nils-herrmann/clean_scirepeval huggingface dataset:
Retrieval by shape using multidimensional indexing structures

In modern visual information retrieval systems, visual content is directly addressed by features such as color, texture, shape and spatial relationship. While a large amount of work is being done around the perceptual soundness of models and around the automatic extraction of features, only a limited amount of attention is being placed on the combination of useful representations and similarity models with efficient index structures for shape. In this paper, we discuss retrieval by shape similarity using local features and metric indexing. Shape is partitioned into tokens in correspondence with its protrusions, following curvature analysis. Each token is modeled by a set of perceptually salient attributes, and two distinct distance functions are used to model token similarity and shape similarity. Shape indexing is obtained by arranging shape tokens into a M-tree index structure, suitably modified. Examples from a prototype system are expounded with considerations about the effectiveness of the approach and a comparative performance analysis.
Boosting 3D LBP-Based Face Recognition by Fusing Shape and Texture Descriptors on the Mesh

In this paper, we present a novel approach for fusing shape and texture local binary patterns (LBPs) on a mesh for 3D face recognition. Using a recently proposed framework, we compute LBP directly on the face mesh surface, then we construct a grid of the regions on the facial surface that can accommodate global and partial descriptions. Compared with its depth-image counterpart, our approach is distinguished by the following features: 1) inherits the intrinsic advantages of mesh surface (e.g., preservation of the full geometry); 2) does not require normalization; and 3) can accommodate partial matching. In addition, it allows early level fusion of texture and shape modalities. Through experiments conducted on the BU-3DFE and Bosphorus databases, we assess different variants of our approach with regard to facial expressions and missing data, also in comparison to the state-of-the-art solutions.
Face recognition by SVMS classification of 2D and 3D Radial Geodesics

An original approach to represent 2D and 3D faces using radial geodesic distances (RGDs) is proposed in this work. In 3D, the RGD of a generic point of the face surface is computed as the length of the geodesic connecting the point with a reference point along a radial direction. In 2D, the RGD of a pixel with respect to a reference pixel accounts for the difference of gray level intensities of the two pixels and the Euclidean distance between them. Support Vector Machines (SVMs) are used to perform face recognition using 2D- and 3D-RGDs. Due to the high dimensionality of face representations based on RGDs, embedding into lower-dimensional spaces is applied before SVMs classification. Experimental results are reported for 3D-3D and 2D-3D face recognition using the proposed approach.
Object-Based Image Retrieval Using Active Nets

In this work, extraction of relevant objects from images and their matching for retrieval is proposed. Objects are represented by using a two dimensional deformable structure referred to as active net, capable to adapt to relevant image regions according to chromatic and edge information. In particular, this representation allows a joint description of color, shape and structural information of extracted objects. A similarity measure between active nets is also defined and validated in a set of retrieval experiments on the ETH-80 objects database
Managing the complexity of match in retrieval by spatial arrangement

Image retrieval by spatial arrangement underlies a matching algorithm, which "interprets" entities specified in the user query on the entities appearing in the image of the database, and which jointly compares their features and their spatial relationships. In this paper we provide a graph-theoretical formulation and we discuss the size for the problem of spatial matching as encountered in the specific application context of retrieval by visual content. We then expound and evaluate three state-space search algorithms which support efficient comparison of models through a heuristic bounding approach.
A Set of Selected SIFT Features for 3D Facial Expression Recognition

In this paper, the problem of person-independent facial expression recognition is addressed on 3D shapes. To this end, an original approach is proposed that computes SIFT descriptors on a set of facial landmarks of depth images, and then selects the subset of most relevant features. Using SVM classification of the selected features, an average recognition rate of 77.5% on the BU-3DFE database has been obtained. Comparative evaluation on a common experimental setup, shows that our solution is able to obtain state of the art results.
The Mesh-LBP: A Framework for Extracting Local Binary Patterns From Discrete Manifolds

In this paper, we present a novel and original framework, which we dubbed mesh-local binary pattern (LBP), for computing local binary-like-patterns on a triangular-mesh manifold. This framework can be adapted to all the LBP variants employed in 2D image analysis. As such, it allows extending the related techniques to mesh surfaces. After describing the foundations, the construction and the main features of the mesh-LBP, we derive its possible variants and show how they can extend most of the 2D-LBP variants to the mesh manifold. In the experiments, we give evidence of the presence of the uniformity aspect in the mesh-LBP, similar to the one noticed in the 2D-LBP. We also report repeatability experiments that confirm, in particular, the rotation-invariance of mesh-LBP descriptors. Furthermore, we analyze the potential of mesh-LBP for the task of 3D texture classification of triangular-mesh surfaces collected from public data sets. Comparison with state-of-the-art surface descriptors, as well as with 2D-LBP counterparts applied on depth images, also evidences the effectiveness of the proposed framework. Finally, we illustrate the robustness of the mesh-LBP with respect to the class of mesh irregularity typical to 3D surface-digitizer scans.
Weighted walkthroughs between extended entities for retrieval by spatial arrangement

In the access to image databases, queries based on the appearing visual features of searched data reduce the gap between the user and the engineering representation. To support this access modality, image content can be modeled in terms of different types of features such as shape, texture, color, and spatial arrangement. An original framework is presented which supports quantitative nonsymbolic representation and comparison of the mutual positioning of extended nonrectangular spatial entities. Properties of the model are expounded to develop an efficient computation technique and to motivate and assess a metric of similarity for quantitative comparison of spatial relationships. Representation and comparison of binary relationships between entities is then embedded into a graph-theoretical framework supporting representation and comparison of the spatial arrangements of a picture. Two prototype applications are described.
Multiresolution spatial partitioning for shape representation

An original solution for shape representation is proposed which relies on a spatial partitioning approach. The representation selects a discrete set of reference points with respect to which a relationship matrix is computed, accounting for the spatial distribution of shape pixels. This is accomplished at different levels of resolution by a tree based representation. Depending on the number of points, coarse to fine region and boundary shape information are captured. Properties of the representation are discussed and assessed through an experimental evaluation on a set of sample shapes.
A Dictionary Learning-Based 3D Morphable Shape Model

Face analysis from 2D images and videos is a central task in many multimedia applications. Methods developed to this end perform either face recognition or facial expression recognition, and in both cases results are negatively influenced by variations in pose, illumination, and resolution of the face. Such variations have a lower impact on 3D face data, which has given the way to the idea of using a 3D morphable model as an intermediate tool to enhance face analysis on 2D data. In this paper, we propose a new approach for constructing a 3D morphable shape model (called DL-3DMM) and show our solution can reach the accuracy of deformation required in applications where fine details of the face are concerned. For constructing the model, we start from a set of 3D face scans with large variability in terms of ethnicity and expressions. Across these training scans, we compute a point-to-point dense alignment, which is accurate also in the presence of topological variations of the face. The DL-3DMM is constructed by learning a dictionary of basis components on the aligned scans. The model is then fitted to 2D target faces using an efficient regularized ridge-regression guided by 2D/3D facial landmark correspondences in order to generate pose-normalized face images. Comparison between the DL-3DMM and the standard PCA-based 3DMM demonstrates that in general a lower reconstruction error can be obtained with our solution. Application to action unit detection and emotion recognition from 2D images and videos shows competitive results with state of the art methods on two benchmark datasets.
The Mesh-LBP: Computing Local Binary Patterns on Discrete Manifolds

In this paper, we present a novel and original framework for computing Local Binary Pattern (LBP)-like patterns on a triangular mesh manifold. This framework, dubbed mesh-LBP can be adapted to all the LBP variants employed in 2D image analysis. As such, it allows extending the related techniques to mesh surfaces. First, we describe the foundations, the construction and the features of the mesh-LBP. In the experiments, we first show evidence of the presence of the ``uniformity'' aspect in the mesh-LBP patterns. Then, we show the mesh-LBP repeatability across different instances of same objects, %and the potentials for analyzing a variety of triangular mesh surfaces, reporting also the application of mesh-LBP to the problem of 3D texture-classification in comparison to standard 3D surface descriptors.
Description and retrieval of 3D cellular structures

Recent advances in management of multimedia digital libraries enable effective retrieval of information in the form of audio, image and video. Many archives of 3D objects already exist and are expected to grow both in relevance and size. However, retrieval of information in the form of 3D objects has received limited attention. We address the problem of effective description and retrieval of 3D data representing intracellular structures. These structures are represented as image stacks, where an image stack is constituted by a set of 2D images representing sections of a cellular body at different heights. In the proposed approach 2D visual feature descriptors and hidden Markov models are combined to obtain a representation model which is able to distinguish such intracellular structures as Golgi, nucleus, endoplasmic reticulum and lysosomes. Preliminary results are presented to show the effectiveness of the proposed representation model.
SHREC’08 entry: 3D face recognition using integral shape information

In this work, we shortly describe an original 3D face recognition approach and its performance as resulted from the 3D Shape Retrieval Contest of 3D Face Scans organized by SHREC 2008 with the support of the Network of Excellence AIM copySHAPE. In particular, the evaluation shows that the proposed approach attains the highest performance on the SHREC 2008 data set.
3D Mesh Partitioning for Retrieval by Parts Applications

A solution for part segmentation of 3D objects is proposed in this paper. The approach is targeted to identify salient visual parts of a mesh by determining its main protrusions and discarding, at the same time, parts originated by un-relevant local properties. This is obtained by first breaking the 3D mesh into seed regions according to the sum of geodesic distances between vertices, then by using topological and curvature information to refine the number of regions and their boundaries. In so doing, effective segmentation is regarded as a prerequisite to enable retrieval of 3D objects based on similarity of parts. Experimental results show the applicability of the proposed solution to complex shapes and its effectiveness in the identification of object parts
Spontaneous Expression Detection from 3D Dynamic Sequences by Analyzing Trajectories on Grassmann Manifolds

In this paper, we propose a framework for online spontaneous emotion detection, such as happiness or physical pain, from depth videos. Our approach consists on mapping the video streams onto a Grassmann manifold (i.e., space of <inline-formula><tex-math notation="LaTeX">$k$</tex-math><alternatives> <inline-graphic xlink:href="benamor-ieq1-2623718.gif"/></alternatives></inline-formula>-dimensional linear subspaces) to form time-parameterized trajectories. To this end, depth videos are decomposed into short-time subsequences, each approximated by a <inline-formula><tex-math notation="LaTeX">$k$</tex-math><alternatives> <inline-graphic xlink:href="benamor-ieq2-2623718.gif"/></alternatives></inline-formula>-dimensional linear subspace, which is in turn a point on the Grassmann manifold. Then, the temporal evolution of subspaces gives rise to a precise mathematical representation of trajectories on the underlying manifold. In the final step, extracted spatio-temporal features based on computing the velocity vectors along the trajectories, termed Geometric Motion History (GMH), are fed to an early event detector based on Structured Output SVM, which enables online emotion detection from partially-observed data. Experimental results obtained on the publicly available Cam3D Kinect and BP4D-spontaneous databases validate the proposed solution. The first database has served to exemplify the proposed framework using depth sequences of the upper part of the body collected using depth-consumer cameras, while the second database allowed the application of the same framework to physical pain detection from high-resolution and long 3D-face sequences.
Local 3D Shape Analysis for Facial Expression Recognition

We investigate the problem of facial expression recognition using 3D face data. Our approach is based on local shape analysis of several relevant regions of a given face scan. These regions or patches from facial surfaces are extracted and represented by sets of closed curves. A Riemannian framework is used to derive the shape analysis of the extracted patches. The applied framework permits to calculate a similarity (or dissimilarity) distances between patches, and to compute the optimal deformation between them. Once calculated, these measures are employed as inputs to a commonly used classification techniques such as AdaBoost and Support Vector Machines (SVM). A quantitative evaluation of our novel approach is conducted on a subset of the publicly available BU-3DFE database.
Representing 3D texture on mesh manifolds for retrieval and recognition applications

In this paper, we present and experiment a novel approach for representing texture of 3D mesh manifolds using local binary patterns (LBP). Using a recently proposed framework [37], we compute LBP directly on the mesh surface, either using geometric or photometric appearance. Compared to its depth-image counterpart, our approach is distinguished by the following features: a) inherits the intrinsic advantages of mesh surface (e.g., preservation of the full geometry); b) does not require normalization; c) can accommodate partial matching. In addition, it allows early-level fusion of the geometry and photometric texture modalities. Through experiments conducted on two application scenarios, namely, 3D texture retrieval and 3D face recognition, we assess the effectiveness of the proposed solution with respect to state of the art approaches.
Shape representation by spatial partitioning for content based retrieval applications

Shape representations for classification or retrieval purposes, have been extensively investigated, but only a few methods have tried to represent shapes as extended entities without reducing them to their boundary profiles or to synthetic geometric descriptors. An original solution for shape representation is proposed; it relies on a modelling technique originally developed to express directional spatial relationships between extended spatial entities. The representation selects a discrete set of points with respect to which a relationship matrix is computed, accounting for the spatial distribution of shape pixels. This is accomplished at different levels of resolution by a tree based structural representation. Properties of the representation and a measure of shape similarity are discussed. The efficiency and effectiveness of the proposed solution have also been assessed in the context of content based retrieval applications, through an experimental evaluation using a shape collection
Boosting 3D LBP-based face recognition by fusing shape and texture descriptors on the mesh

In this paper, we present a novel approach for fusing shape and texture local binary patterns (LBP) for 3D face recognition. Using the framework proposed in , we compute LBP directly on the face mesh surface, then we construct a grid of the regions on the facial surface that can accommodate global and partial descriptions. Compared to its depth-image counterpart, our approach is distinguished by the following features: a) inherits the intrinsic advantages of mesh surface; b) does not require normalization; c) can accommodate partial matching. In addition, it allows early-level fusion of texture and shape modalities. Through experiments conducted on the BU-3DFE and Bosphorus databases, we assess different variants of our approach with regard to facial expressions and missing data.
3D Face Identification Based on Arrangement of Salient Wrinkles

In this paper, we propose an original framework for three dimensional face representation and matching for identification purposes. Basic traits of a face are encoded by extracting curves of salient ridges and ravines from the surface of a dense mesh. A compact graph representation is then extracted from these curves through an original modeling technique capable to quantitatively measure spatial relationships between curves in a three dimensional space. In this way, face recognition is obtained by matching 3D graph representations of faces. Experimental results on a 3D face database show that the proposed solution attains high recognition accuracy and is quite robust to facial expression and pose changes
Dictionary Learning Based 3D Morphable Model Construction for Face Recognition with Varying Expression and Pose

In this paper, we propose a new approach for constructing a 3D morph able model (3DMM) and experiment its application to face recognition. Differently from existing solutions, the proposed 3DMM is constructed from a training set that includes a large spectrum of variability in terms of ethnicity and facial expressions. By exploiting annotated landmarks available in the training data, we are able of establishing dense correspondence across training scans also in the presence of strong facial expressions. The 3DMM is then constructed by learning a dictionary of basis components, instead of using the traditional approach based on PCA decomposition. Finally, we cast the proposed dictionary learning DL-3DMM to a rigid/non-rigid deformation framework, which includes pose estimation and regularized ridge-regression fitting to 2D images. Comparative results between the DL-3DMM and its PCA counterpart are reported, together with face recognition results for images with large pose and expression variations.
Recognizing Actions from Depth Cameras as Weakly Aligned Multi-part Bag-of-Poses

Recently released depth cameras provide effective estimation of 3D positions of skeletal joints in temporal sequences of depth maps. In this work, we propose an efficient yet effective method to recognize human actions based on the positions of joints. First, the body skeleton is decomposed in a set of kinematic chains, and the position of each joint is expressed in a locally defined reference system which makes the coordinates invariant to body translations and rotations. A multi-part bag-of-poses approach is then defined, which permits the separate alignment of body parts through a nearest-neighbor classification. Experiments conducted on the Florence 3D Action dataset and the MSR Daily Activity dataset show promising results.
Combined shape analysis of human poses and motion units for action segmentation and recognition

Recognizing human actions or analyzing human behaviors from 3D videos is an important problem currently investigated in many research domains. The high complexity of human motions and the variability of gesture combinations make this task challenging. Local (over time) analysis of a sequence is often necessary in order to have a more accurate and thorough understanding of what the human is doing. In this paper, we propose a method based on the combination of pose-based and segment-based approaches in order to segment an action sequence into motion units (MUs). We jointly analyze the shape of the human pose and the shape of its motion using a shape analysis framework that represents and compares shapes in a Riemannian manifold. On one hand, this allows us to detect periodic MUs and thus perform action segmentation. On another hand, we can remove repetitions of gestures in order to handle with failure cases for the task of action recognition. Experiments are performed on three representative datasets for the task of action segmentation and action recognition. Competitive results with state-of-the-art methods are obtained in both the tasks.
Face Recognition by Super-Resolved 3D Models From Consumer Depth Cameras

Face recognition based on the analysis of 3D scans has been an active research subject over the last few years. However, the impact of the resolution of 3D scans on the recognition process has not been addressed explicitly, yet being an element of primal importance to enable the use of the new generation of consumer depth cameras for biometric purposes. In fact, these devices perform depth/color acquisition over time at standard frame-rate, but with a low resolution compared to the 3D scanners typically used for acquiring 3D faces in recognition applications. Motivated by these considerations, in this paper, we define a super-resolution approach for 3D faces by which a sequence of low-resolution 3D face scans is processed to extract a higher resolution 3D face model. The proposed solution relies on the scaled iterative closest point procedure to align the low-resolution scans with each other, and estimates the value of the high-resolution 3D model through a 2D box-spline functions approximation. To evaluate the approach, we built-and made it publicly available-the Florence Superface dataset that collects high-resolution and low-resolution data for about 50 different persons. Qualitative and quantitative results are reported to demonstrate the accuracy of the proposed solution, also in comparison with alternative techniques.