nils-herrmann/clean_scirepeval huggingface dataset:
Local descriptors matching for 3D face recognition

An original solution to 3D face recognition, which supports face matching also in the case of probes with varying expressions and missing parts is proposed in this work. Distinguishing traits of the face are captured by first extracting 3D keypoints of the face scan, then measuring how the face surface changes in the neighborhood of the keypoints using a local descriptor. To this end, an adaptation of the meshDOG detector to the case of 3D faces is proposed, together with a multi-ring geometric histogram descriptor. Face similarity is then evaluated by comparing local keypoint descriptors across inlier pairs of matching keypoints between probe and gallery scans. Experiments have been performed on the Bosphorus database, showing competitive results with respect to existing solutions for 3D face biometrics.
Sensations and psychological effects in color image database

The development of a system supporting querying of image databases by color content tackles a major design choice about properties of colors which are referenced within user queries. On the one hand, low-level properties directly reflect numerical features and concepts tied to the machine representation of color information. On the other hand, high-level properties address concepts such as the perceptual quality of colors and the sensations that they convey. Color-induced sensations include warmth, accordance or contrast, harmony, excitement, depression, anguish etc. In particular, paintings are an example where the message is contained more in the high-level color qualities and spatial arrangements than in the physical properties of colors. Starting from this observation, Johannes Itten (1961) introduced a formalism to analyze the use of color in art and the effects that this induces on the user's psyche. In this paper, we present a system which translates the Itten theory into a formal language that allows us to express the semantics associated with the combination of chromatic properties of color images.
Reconstructing High-Resolution Face Models From Kinect Depth Sequences

Performing face recognition across 3D scans with different resolution is now attracting an increasing interest thanks to the introduction of a new generation of depth cameras, capable of acquiring color/depth images over time. In fact, these devices acquire and provide depth data with much lower resolution compared with the 3D high-resolution scanners typically used for face recognition applications. If data are acquired without user cooperation, the problem is even more challenging, and the gap of resolution between probe and gallery scans can yield to a severe loss in terms of recognition accuracy. Based on these premises, we propose a method to build a higher resolution 3D face model from 3D data acquired by a low-resolution scanner. This face model is built using data acquired when a person passes in front of the scanner, without assuming any particular cooperation. The 3D data are registered and filtered by combining a model of the expected distribution of the acquisition error with a variant of the lowess method to remove outliers and build the final face model. The proposed approach is evaluated in terms of accuracy of face reconstruction and face recognition.
Indexed retrieval by shape appearance

Modern visual information retrieval systems support retrieval by visual content also by directly addressing image visual features such as color, texture, shape and spatial relationships. Combining useful representations and similarity models with efficient index structures is a problem that has been largely underestimated. This problem is particularly challenging in the case of retrieval by shape similarity. In this paper we discuss retrieval by shape similarity, using local features and metric indexing. Shape is partitioned into tokens following curvature analysis. Each token is modeled by a set of perceptually salient attributes and two distinct distance functions are used to model token similarity and shape similarity. Shape indexing is obtained by arranging tokens into a M-tree index structure. Examples from a prototype system an expounded with considerations about the effectiveness of the approach.
3D Face Recognition by Spatial Arrangement of Iso-Geodesic Surfaces

An original framework for description and matching of three dimensional (3D) faces for recognition purposes is proposed in this paper. Face information is captured by extracting iso-geodesic surfaces of 3D face models. A compact representation of the faces is then constructed through a modeling technique capable to express the basic shape of iso-geodesic surfaces and quantitatively measure their spatial relationships in the 3D space. This information is encoded in an attributed relational graph. Experimental results on a 3D face database and baseline comparison show that the proposed solution attains high face recognition accuracy and is reasonably robust to facial expression changes. Experiments also target the identification of portions of the face surface which are most significant for the purpose of discriminating between faces.
Indexed retrieval by shape appearance

Efficient retrieval by content of visual information requires that visual content descriptors and similarity models are combined with efficient index structures. This problem is particularly challenging in the case of retrieval by shape similarity. The paper discusses retrieval by shape similarity, using local features and effective indexing. Shapes are partitioned into tokens following curvature analysis and each token is modelled by a set of perceptually salient attributes. Two distinct distance functions are used to model token similarity and shape similarity. Shape indexing is obtained by arranging tokens into an M-tree index structure. Examples from a prototype system and computational experiences are reported for both retrieval accuracy and indexing efficiency.
4-D Facial Expression Recognition by Learning Geometric Deformations

In this paper, we present an automatic approach for facial expression recognition from 3-D video sequences. In the proposed solution, the 3-D faces are represented by collections of radial curves and a Riemannian shape analysis is applied to effectively quantify the deformations induced by the facial expressions in a given subsequence of 3-D frames. This is obtained from the dense scalar field, which denotes the shooting directions of the geodesic paths constructed between pairs of corresponding radial curves of two faces. As the resulting dense scalar fields show a high dimensionality, Linear Discriminant Analysis (LDA) transformation is applied to the dense feature space. Two methods are then used for classification: 1) 3-D motion extraction with temporal Hidden Markov model (HMM) and 2) mean deformation capturing with random forest. While a dynamic HMM on the features is trained in the first approach, the second one computes mean deformations under a window and applies multiclass random forest. Both of the proposed classification schemes on the scalar fields showed comparable results and outperformed earlier studies on facial expression recognition from 3-D video sequences.
Retrieval by Shape Similarity with Perceptual Distance and Effective Indexing

An important problem in accessing and retrieving visual information is to provide efficient similarity matching in large databases. Though much work is being done on the investigation of suitable perceptual models and the automatic extraction of features, little attention is given to the combination of useful representations and similarity models with efficient index structures. In this paper we propose retrieval by shape similarity using local descriptors and effective indexing. Shapes are partitioned into tokens in correspondence with their protrusions, and each token is modeled according to a set of perceptually salient attributes. Shape indexing is obtained by arranging shape tokens into a suitably modified M-tree index structure. Two distinct distance functions model respectively, token and shape perceptual similarity. Examples from a prototype system and computational experiences are reported for both retrieval accuracy and indexing efficiency. Shape retrieval has been tested under shape scaling, orientation changes, and partial shape occlusions. A comparative analysis of different indexing structures, for shape retrieval is presented.
Merging results of distributed image libraries

Exploitation of information repositories available on the Internet requires users to separately query each repository and manually gather retrieved results. Such a solution could be simplified by using a centralized server that acts as a gateway between the user and repositories: the centralized server forwards the user query to federated repositories and fuses retrieved documents for presentation to the user. To perform these tasks efficiently, the centralized server should perform two main functions: resource selection and data fusion. The former is required to forward the user query only to the repositories that are candidate to contain relevant documents. The latter is used to gather all retrieved documents and conveniently arrange them for presentation to the user. In the case of image repositories, data fusion is particularly challenging owing to the difficulty to normalize document scores returned by different repositories. In this paper a novel solution is presented for fusion of results returned by different image repositories. Experimental results are presented that show the potential of the proposed approach.
Weighting spatial arrangement of colors in content based image retrieval

Content-based access to image databases relies on models which permit one to abstract images in some space of visual features. A large number of such models, addressing shape, texture, color and spatial arrangement features, have been proposed and experimented with. The fusion of multiple features within a single model is now being investigated as a way to increase the effectiveness of retrieval. In particular, the integration of color and spatial models is receiving growing interest. In this paper, we propose a representation which captures the spatial arrangement of the color sets of a picture, without talking into account their distribution in separate non-connected regions. A spatial modeling technique, which we call "weighted walkthroughs", is presented, and its suitability to support this representation is motivated and experimented on.
A look-ahead strategy for graph matching in retrieval by spatial arrangement

Image retrieval by spatial arrangement underlies a matching problem for the interpretation of entities specified in the user query on the entities appearing in the image of the database, and for the joint comparison of their features and spatial relationships. We provide a graph-theoretical formulation and discuss the size of the problem of spatial matching as encountered in the specific application context of retrieval by visual content. We then expound and evaluate an original state-space search algorithm which supports efficient comparison of models through an heuristic bounding approach.
Modeling spatial relationships between color sets

Modeling of image content based on chromatic arrangement includes representation of the spatial relationship between complex sets of pixels. We propose a model of spatial directional relationship between extended sets. This involves the same computational and programming complexity as that of conventional representations based on centroids, but it is able to account for the overall sets of pixels without reducing them to a single representative point or to a bounding rectangle. The gain in effectiveness is evaluated in a user-based comparison with a representation based on mutual centroid orientation.
Analysis and retrieval of 3D facial models using iso-geodesic stripes

In this paper, we describe a framework for analysis, representation and matching of three-dimensional faces. Basic traits of a face are encoded by extracting iso-geodesic stripes from the surface of a face model. A compact representation is then constructed through a modeling technique capable to express the basic shape of iso-geodesic stripes and quantitatively measure their spatial relationships in a three-dimensional space. This information is encoded in an attributed relational graph. In this way, the structural similarity between two face models is evaluated by matching their corresponding graphs. Experimental results on a 3D face database and baseline comparison show that the proposed solution attains high face recognition accuracy and is reasonably robust to facial expression and pose changes. Experiments also target the identification of portions of the face surface which are most significant for the purpose of discriminating between faces.
Modeling Spatial Relationships between 3D Objects

In this paper, we propose an original modeling technique which enables quantitative non-symbolic representation and comparison of the mutual spatial positioning of extended entities in a 3D space. The representation accounts for the overall distribution of relationships among the individual points of two objects. Properties of the model are expounded to develop an efficient computation technique and to motivate and assess a metric of similarity for quantitative comparison of spatial relationships between object pairs. Experiments compare the proposed approach against two different solutions
Retrieval of 3D objects using curvature correlograms

Along with images and videos, 3D models have raised a certain interest for a number of reasons, including advancements in 3D hardware and software technologies, their ever decreasing prices and increasing availability, affordable 3D authoring tools, and the establishment of open standards for 3D data interchange. The resulting proliferation of 3D models demands for tools supporting their effective and efficient management, including archival and retrieval. In order to support effective retrieval by content of 3D objects and enable retrieval by object parts, information about local object structure should be combined with spatial information on object surface. In this paper, as a solution to this requirement, we present a method relying on curvature correlograms to perform description and retrieval by content of 3D objects. Experimental results are presented both to show results of sample queries by content and to compare-in terms of precision/recall figures-the proposed solution to alternative techniques.
Investigating Nuisance Factors in Face Recognition with DCNN Representation

Deep learning based approaches proved to be dramatically effective to address many computer vision applications, including "face recognition in the wild". It has been extensively demonstrated that methods exploiting Deep Convolutional Neural Networks (DCNN) are powerful enough to overcome to a great extent many problems that negatively affected computer vision algorithms based on hand-crafted features. These problems include variations in illumination, pose, expression and occlusion, to mention some. The DCNNs excellent discriminative power comes from the fact that they learn low-and high-level representations directly from the raw image data. Considering this, it can be assumed that the performance of a DCNN are influenced by the characteristics of the raw image data that are fed to the network. In this work, we evaluate the effect of different bounding box dimensions, alignment, positioning and data source on face recognition using DCNNs, and present a thorough evaluation on two well known, public DCNN architectures.
Computing Local Binary Patterns on Discrete Manifolds

In this paper, we present a novel and original framework for computing Local Binary Pattern (LBP)-like patterns on a triangular mesh manifold. This framework, that we called mesh-LBP, can be adapted to all the LBP variants employed in 2D image analysis. As such, it allows extending the related techniques to mesh surfaces. First, we describe the foundations, the construction and the features of the mesh-LBP. In the experiments, we first show evidence of the presence of the "uniformity" aspect in the mesh-LBP patterns. Then, we report about the application of mesh-LBP to the problem of 3D texture-classification in comparison to standard 3D surface descriptors and show the mesh-LBP robustness to mesh irregularities.
Effective 3D based frontalization for unconstrained face recognition

In this paper, we propose a new and effective frontalization algorithm for frontal rendering of unconstrained face images, and experiment it for face recognition. Initially, a 3DMM is fit to the image, and an interpolating function maps each pixel inside the face region on the image to the 3D model's. Thus, we can render a frontal view without introducing artifacts in the final image thanks to the exact correspondence between each pixel and the 3D coordinate of the model. The 3D model is then back projected onto the frontalized image allowing us to localize image patches where to extract the feature descriptors, and thus enhancing the alignment between the same descriptor over different images. Our solution outperforms other frontalization techniques in terms of face verification. Results comparable to state-of-the-art on two challenging benchmark datasets are also reported, supporting our claim of effectiveness of the proposed face image representation.
Robust low complexity feature tracking

In this paper, we present the Kanade-Lucas-Tomasi (KLT) tracking algorithm coupled with a varying integration window, which tracks a small subset of feature points to initialize the approximate motion model between images. For the remaining larger subset of the feature points initial tracking location is predicted by using this motion model, thus improving the tracking result. For an image of size 1000×700, the computational cost is reduced by a factor of 9.5 and tracking 500 features with our method runs in only 64 ms on a commodity 2GHz CPU with 1GB RAM.
Scene-adapted structured light

In order to overcome several limitations of structured light 3D acquisition methods, the colors, intensities, and shapes of the projected patterns are adapted to the scene. Based on a crude estimate of the scene geometry and reflectance characteristics, the local intensity ranges in the projected patterns are adapted, in order to avoid over- and under-exposure in the image. This avoids the infamous specularity problems and generally increases accuracy. The estimated geometry also helps to limit the effect of aliasing caused by the sampling of foreshortened patterns. Furthermore, the approach also accounts for the adverse effects that small motions during scanning would normally have. Moreover, the approach yields a confidence measure at every pixel of the range image. Last but not least, the scanner consists of consumer products only, and therefore is cheap.
Energy-efficient ConvNets through approximate computing

Recently convolutional neural networks (ConvNets) have come up as state-of-the-art classification and detection algorithms, achieving near-human performance in visual detection. However, ConvNet algorithms are typically very computation and memory intensive. In order to be able to embed ConvNet-based classification into wearable platforms and embedded systems such as smartphones or ubiquitous electronics for the internet-of-things, their energy consumption should be reduced drastically. This paper proposes methods based on approximate computing to reduce energy consumption in state-of-the-art ConvNet accelerators. By combining techniques both at the system- and circuit level, we can gain energy in the systems arithmetic: up to 30× without losing classification accuracy and more than 100× at 99% classification accuracy, compared to the commonly used 16-bit fixed point number format.
Hunting Nessie - Real-time abnormality detection from webcams

We present a data-driven, unsupervised method for unusual scene detection from static webcams. Such time-lapse data is usually captured with very low or varying framerate. This precludes the use of tools typically used in surveillance (e.g., object tracking). Hence, our algorithm is based on simple image features. We define usual scenes based on the concept of meaningful nearest neighbours instead of building explicit models. To effectively compare the observations, our algorithm adapts the data representation. Furthermore, we use incremental learning techniques to adapt to changes in the data-stream. Experiments on several months of webcam data show that our approach detects plausible unusual scenes, which have not been observed in the data-stream before.
Dynamic 3D Scene Analysis from a Moving Vehicle

In this paper, we present a system that integrates fully automatic scene geometry estimation, 2D object detection, 3D localization, trajectory estimation, and tracking for dynamic scene interpretation from a moving vehicle. Our sole input are two video streams from a calibrated stereo rig on top of a car. From these streams, we estimate structure-from-motion (SfM) and scene geometry in real-time. In parallel, we perform multi-view/multi-category object recognition to detect cars and pedestrians in both camera images. Using the SfM self-localization, 2D object detections are converted to 3D observations, which are accumulated in a world coordinate frame. A subsequent tracking module analyzes the resulting 3D observations to find physically plausible spacetime trajectories. Finally, a global optimization criterion takes object-object interactions into account to arrive at accurate 3D localization and trajectory estimates for both cars and pedestrians. We demonstrate the performance of our integrated system on challenging real-world data showing car passages through crowded city areas.
What's going on? Discovering spatio-temporal dependencies in dynamic scenes

We present two novel methods to automatically learn spatio-temporal dependencies of moving agents in complex dynamic scenes. They allow to discover temporal rules, such as the right of way between different lanes or typical traffic light sequences. To extract them, sequences of activities need to be learned. While the first method extracts rules based on a learned topic model, the second model called DDP-HMM jointly learns co-occurring activities and their time dependencies. To this end we employ Dependent Dirichlet Processes to learn an arbitrary number of infinite Hidden Markov Models. In contrast to previous work, we build on state-of-the-art topic models that allow to automatically infer all parameters such as the optimal number of HMMs necessary to explain the rules governing a scene. The models are trained offline by Gibbs Sampling using unlabeled training data.
One-shot active 3D shape acquisition

An active 3D acquisition system is presented that is based on the projection a simple pattern of squares. Through the observation of the projected pattern, and assuming pseudo-orthographic projection, the 3D shape of the scene can be retrieved up to a single parameter, that can be determined through a particularly simple calibration. An algorithm has been developed to automatically detect the lines and crossings of the projected pattern in the image. Experiments on a variety of scenes show that the algorithm is robust and provides accurate 3D reconstructions. Its one-shot operation enables the system to retrieve the shape of moving objects.
A distributed camera system for multi-resolution surveillance

We describe an architecture for a multi-camera, multi-resolution surveillance system. The aim is to support a set of distributed static and pan-tilt-zoom (PTZ) cameras and visual tracking algorithms, together with a central supervisor unit. Each camera (and possibly pan-tilt device) has a dedicated process and processor. Asynchronous interprocess communications and archiving of data are achieved in a simple and effective way via a central repository, implemented using an SQL database.