nils-herrmann/clean_scirepeval huggingface dataset:
Data-driven animation of hand-object interactions

Animating hand-object interactions is a frequent task in applications such as the production of 3d movies. Unfortunately this task is difficult due to the hand's many degrees of freedom and the constraints on the hand motion imposed by the geometry of the object. However, the causality between the object state and the hand's pose can be exploited in order to simplify the animation process. In this paper, we present a method that takes an animation of an object as input and automatically generates the corresponding hand motion. This approach is based on the simple observation that objects are easier to animate than hands, since they usually have fewer degrees of freedom. The method is data-driven; sequences of hands manipulating an object are captured semi-automatically with a structured-light setup. The training data is then combined with a new animation of the object in order to generate a plausible animation featuring the hand-object interaction.
Ground Plane Estimation Using a Hidden Markov Model

We focus on the problem of estimating the ground plane orientation and location in monocular video sequences from a moving observer. Our only assumptions are that the 3D ego motion t and the ground plane normal n are orthogonal, and that n and t are smooth over time. We formulate the problem as a state-continuous Hidden Markov Model (HMM) where the hidden state contains t and n and may be estimated by sampling and decomposing homographies. We show that using blocked Gibbs sampling, we can infer the hidden state with high robustness towards outliers, drifting trajectories, rolling shutter and an imprecise intrinsic calibration. Since our approach does not need any initial orientation prior, it works for arbitrary camera orientations in which the ground is visible.
Material Classification under Natural Illumination Using Reflectance Maps

Research on visual material recognition has traditionally been based on texture analysis. Whereas older work has focused on uncluttered scenes, more recent contributions allowed for material recognition 'in the wild'. Quite some objects have untextured surfaces, however. Especially man-made examples are legion. The most obvious cue to use in such cases would be reflection information. Yet, methods to that effect are lacking. It is clear that more than an estimate of a scalar albedo is needed, and a more complete reflectance model has to be derived. Rather than using an extensive lab setup, we propose a system that only requires the 3D shape of objects and a regular, commercial camera to capture their appearance in a single image, to perform material classification under unknown illumination. To this end, we rely on a Gaussian Process Latent Variable Model (GPLVM) with a discriminative prior to learn a low-dimensional manifold suitable for material classification of reflectance maps, i.e. from a 2D image of a singlematerial sphere under natural illumination. We evaluated our method based on experiments generated from synthetic and real-life data. Although recognizing materials without texture (or object recognition) is not a trivial problem, our method achieves about 75% recognition accuracy, about 27% higher than human performance.
Simultaneous Segmentation and 3D Reconstruction of Monocular Image Sequences

When trying to extract 3D scene information and camera motion from an image sequence alone, it is often necessary to cope with independently moving objects. Recent research has unveiled some of the mathematical foundations of the problem, but a general and practical algorithm, which can handle long, realistic sequences, is still missing. In this paper, we identify the necessary parts of such an algorithm, highlight both unexplored theoretical issues and practical challenges, and propose solutions. Theoretical issues include proper handling of different situations, in which the number of independent motions changes: objects can enter the scene, objects previously moving together can split and follow independent trajectories, or independently moving objects can merge into one common motion. We derive model scoring criteria to handle these changes in the number of segments. A further theoretical issue is the resolution of the relative scale ambiguity between such changes. Practical issues include robust 3D reconstruction of freely moving foreground objects, which often have few and short feature tracks. The proposed framework simultaneously tracks features, groups them into rigidly moving segments, and reconstructs all segments in 3D. Such an online approach, as opposed to batch processing techniques, which first track features, and then perform segmentation and reconstruction, is vital in order to handle small foreground objects.
Deep visual words: Improved fisher vector for image classification

Image classification has been revolutionized by deep convolutiosnal neural networks. Using previous state-of-the-art classification methods like Fisher vector encoding in combination with deep CNNs has been shown to be promising. Motivated by the recent work on dense CNN features to extract Fisher encoding(FV-CNN), we present a scheme to discover better visual words with CNNs, to obtain improved Fisher vector features. Our method (Deep Visual Words-DVW) learns semantic visual clusters per each category, by iteratively learning and refining groups of visual patches. DVW represents an efficient feature space embedding to capture the discriminative potential between meaningful visual clusters. We evaluate our approach on popular datasets in object, scene and action classification and outperformed the state-of-the-art: scene classification MIT indoor, object categorization PASCAL VOC 2007 and Stanford40 human actions.
Server-side object recognition and client-side object tracking for mobile augmented reality

In this paper we present a system for mobile augmented reality (AR) based on visual recognition. We split the tasks of recognizing an object and tracking it on the user's screen into a server-side and a client-side task, respectively. The capabilities of this hybrid client-server approach are demonstrated with a prototype application on the Android platform, which is able to augment both stationary (landmarks) and non stationary (media covers) objects. The database on the server side consists of hundreds of thousands of landmarks, which is crawled using a state of the art mining method for community photo collections. In addition to the landmark images, we also integrate a database of media covers with millions of items. Retrieval from these databases is done using vocabularies of local visual features. In order to fulfill the real-time constraints for AR applications, we introduce a method to speed-up geometric verification of feature matches. The client-side tracking of recognized objects builds on a multi-modal combination of visual features and sensor measurements. Here, we also introduce a motion estimation method, which is more efficient and precise than similar approaches. To the best of our knowledge this is the first system, which demonstrates a complete pipeline for augmented reality on mobile devices with visual object recognition scaled to millions of objects combined with real-time object tracking.
Actionness Estimation Using Hybrid Fully Convolutional Networks

Actionness was introduced to quantify the likelihood of containing a generic action instance at a specific location. Accurate and efficient estimation of actionness is important in video analysis and may benefit other relevant tasks such as action recognition and action detection. This paper presents a new deep architecture for actionness estimation, called hybrid fully convolutional network (HFCN), which is composed of appearance FCN (A-FCN) and motion FCN (M-FCN). These two FCNs leverage the strong capacity of deep models to estimate actionness maps from the perspectives of static appearance and dynamic motion, respectively. In addition, the fully convolutional nature of H-FCN allows it to efficiently process videos with arbitrary sizes. Experiments are conducted on the challenging datasets of Stanford40, UCF Sports, and JHMDB to verify the effectiveness of H-FCN on actionness estimation, which demonstrate that our method achieves superior performance to previous ones. Moreover, we apply the estimated actionness maps on action proposal generation and action detection. Our actionness maps advance the current state-of-the-art performance of these tasks substantially.
Finding nemo: Deformable object class modelling using curve matching

An image search for “clownfish” yields many photos of clownfish, each of a different individual of a different 3D shape in a different pose. Yet, to the human observer, this set of images contains enough information to infer the underlying 3D deformable object class. Our goal is to recover such a deformable object class model directly from unordered images. For classes where feature-point correspondences can be found, this is a straightforward extension of non-rigid factorization, yielding a set of 3D basis shapes to explain the 2D data. However, when each image is of a different object instance, surface texture is generally unique to each individual, and does not give rise to usable image point correspondences. We overcome this sparsity using curve correspondences (crease-edge silhouettes or class-specific internal texture edges). Even rigid contour reconstruction is difficult due to the lack of reliable correspondences. We incorporate correspondence variation into the optimization, thereby extending contour-based reconstruction techniques to deformable object modelling. The notion of correspondence is extended to include mappings between 2D image curves and corresponding parts of the desired 3D object surface. Combined with class-specific priors, our method enables effective de-formable class reconstruction from unordered images, despite significant occlusion and the scarcity of shared 2D image features.
Automatic Tool Landmark Detection for Stereo Vision in Robot-Assisted Retinal Surgery

Computer vision and robotics are being increasingly applied in medical interventions. Especially in interventions where extreme precision is required, they could make a difference. One such application is robot-assisted retinal microsurgery. In recent works, such interventions are conducted under a stereo-microscope, and with a robot-controlled surgical tool. The complementarity of computer vision and robotics has, however, not yet been fully exploited. In order to improve the robot control, we are interested in three-dimensional (3-D) reconstruction of the anatomy and in automatic tool localization using a stereo microscope. In this letter, we solve this problem for the first time using a single pipeline, starting from uncalibrated cameras to reach metric 3-D reconstruction and registration, in retinal microsurgery. The key ingredients of our method are 1) surgical tool landmark detection, and 2) 3-D reconstruction with the stereo microscope, using the detected landmarks. To address the former, we propose a novel deep learning method that detects and recognizes keypoints in high-definition images at higher than real-time speed. We use the detected two-dimensional keypoints along with their corresponding 3-D coordinates obtained from the robot sensors to calibrate the stereo microscope using an affine projection model. We design an online 3-D reconstruction pipeline that makes use of smoothness constraints and performs robot-to-camera registration. The entire pipeline is extensively validated on open-sky porcine eye sequences. Quantitative and qualitative results are presented for all steps.
Recognition and semi-differential invariants

Semidifferential invariants, combining coordinates in different points together with their derivatives, are used for the description of planar contours. Their use can be seen as a tradeoff between two extreme strategies currently used in shape recognition: (invariant) feature extraction methods, involving high-order derivatives, and invariant coordinate descriptions, leading to the correspondence problem of reference points. The method for the derivation of such invariants, based on Lie group theory and applicable to a wide spectrum of transformation groups, is described. As an example, invariant curve parameterizations are developed for affine and projective transformations. The usefulness of the approach is illustrated with two examples: (1) recognition of a test set of 12 planar objects viewed under conditions allowing affine approximations, and (2) the detection of symmetry in perspective projections of curves.<<ETX>>
DLDR: Deep Linear Discriminative Retrieval for Cultural Event Classification from a Single Image

In this paper we tackle the classification of cultural events from a single image with a deep learning based method. We use convolutional neural networks (CNNs) with VGG-16 architecture , pretrained on ImageNet or the Places205 dataset for image classification, and fine-tuned on cultural events data. CNN features are robustly extracted at 4 different layers in each image. At each layer Linear Discriminant Analysis (LDA) is employed for discriminative dimensionality reduction. An image is represented by the concatenated LDA-projected features from all layers or by the concatenation of CNN pooled features at each layer. The classification is then performed through the Iterative Nearest Neighbors-based Classifier (INNC) [20]. Classification scores are obtained for different image representation setups at train and test. The average of the scores is the output of our deep linear discriminative retrieval (DLDR) system. With 0.80 mean average precision (mAP) DLDR is a top entry for the ChaLearn LAP 2015 cultural event recognition challenge.
Accurate and efficient stereo matching with robust piecewise voting

In this paper, we propose an efficient local stereo algorithm for accurate disparity estimation. First, we attain initial disparity estimates by iterating a cross-based cost aggregation process. Then, we propose a robust voting scheme to refine the initial estimates based on a piecewise smoothness prior, improving the quality in occluded regions and low-textured regions effectively. The refinement is guided by the segmentation result of input images. Unreliable initial estimates, which are detected using an efficient left-right consistency check, are rejected to increase the reliability of the voting results. Evaluated with the Middlebury stereo benchmark, our method is among the top performing local methods in accuracy. Compared to other local methods with similar accuracy, our method is faster by a factor of about two orders.
3D Reconstruction from Multiple Images: Part 1 - Principles

This issue discusses methods to extract three-dimensional (3D) models from plain images. In particular, the 3D information is obtained from images for which the camera parameters are unknown. The principles underlying such uncalibrated structure-from-motion methods are outlined. First, a short review of 3D acquisition technologies puts such methods in a wider context and highlights their important advantages. Then, the actual theory behind this line of research is given. The authors have tried to keep the text maximally self-contained, therefore also avoiding to rely on an extensive knowledge of the projective concepts that usually appear in texts about self-calibration 3D methods. Rather, mathematical explanations that are more amenable to intuition are given. The explanation of the theory includes the stratification of reconstructions obtained from image pairs as well as metric reconstruction on the basis of more than two images combined with some additional knowledge about the cameras used. Readers who want to obtain more practical information about how to implement such uncalibrated structure-from-motion pipelines may be interested in two more Foundations and Trends issues written by the same authors. Together with this issue they can be read as a single tutorial on the subject.
Adaptive real-time 3D acquisition and contour tracking within a multiple structured light system

Active 3D acquisition setups currently capture 3D data from a single viewpoint. We propose a system that supports the effective use of multiple structured light modules placed around the object of interest. We show how oppositely positioned modules could work together and how we can avoid those parts of the background that unnecessarily take up computation time. Key to the setup is having the structured light patterns masked outside the outlines of the object to be captured. This is done very fast, so that the system can be used for motion capture. Only off-the-shelf hardware is required.
Multi-view traffic sign detection, recognition, and 3D localisation

Several applications require information about street furniture. Part of the task is to survey all traffic signs. This has to be done for millions of km of road, and the exercise needs to be repeated every so often. A van with 8 roof-mounted cameras drove through the streets and took images every meter. The paper proposes a pipeline for the efficient detection and recognition of traffic signs. The task is challenging, as illumination conditions change regularly, occlusions are frequent, 3D positions and orientations vary substantially, and the actual signs are far less similar among equal types than one might expect. We combine 2D and 3D techniques to improve results beyond the state-of-the-art, which is still very much preoccupied with single view analysis.
Bayesian Pixel Classification for Human Tracking

We present a monocular object tracker, able to detect and track multiple objects in non-controlled environments. Bayesian per-pixel classification is used to build a tracking framework that segments an image into foreground and background objects, based on observations of object appearances and motions. Gaussian mixtures are used to build the color appearance models. The system adapts to changing lighting conditions, handles occlusions, and works in real-time.
DEX: Deep EXpectation of Apparent Age from a Single Image

In this paper we tackle the estimation of apparent age in still face images with deep learning. Our convolutional neural networks (CNNs) use the VGG-16 architecture  and are pretrained on ImageNet for image classification. In addition, due to the limited number of apparent age annotated images, we explore the benefit of finetuning over crawled Internet face images with available age. We crawled 0.5 million images of celebrities from IMDB and Wikipedia that we make public. This is the largest public dataset for age prediction to date. We pose the age regression problem as a deep classification problem followed by a softmax expected value refinement and show improvements over direct regression training of CNNs. Our proposed method, Deep EXpectation (DEX) of apparent age, first detects the face in the test image and then extracts the CNN predictions from an ensemble of 20 networks on the cropped face. The CNNs of DEX were finetuned on the crawled images and then on the provided images with apparent age annotations. DEX does not use explicit facial landmarks. Our DEX is the winner (1st place) of the ChaLearn LAP 2015 challenge on apparent age estimation with 115 registered teams, significantly outperforming the human reference.
Real-time sign language letter and word recognition from depth data

In this work, we present a system for recognizing letters and finger-spelled words of the American sign language (ASL) in real-time. To this end, the system segments the hand and estimates the hand orientation from captured depth data. The letter classification is based on average neighborhood margin maximization and relies on the segmented depth data of the hands. For word recognition, the letter confidences are aggregated. Furthermore, the word recognition is used to improve the letter recognition by updating the training examples of the letter classifiers on-line.
Separating rigid motion from linear local deformation models

In this paper we deal with the problem of recovering deformable 3D surfaces from image observations, a topic which has received a lot of attention in recent years. As the problem is inherently under-constrained, additional information has to be used to regularize the solution. Linear local deformation models have been presented as a solution that fits into convex programming schemes. In contrast to more complex statistical models, they also offer very good generalizability. However, in existing work, they are used to model not only the local non-rigid deformations, but also the global and local rigid transformations. We show that by not estimating the rigid transformations separately, a systematic reconstruction error that depends on the transformation is introduced. We then propose separating the rigid and non-rigid parts and demonstrate how to fit the resulting problem into the existing SOCP scheme. We finally compare our method to the baseline approach and show that our method outperforms it.
Transferring activities: Updating human behavior analysis

One of the great open challenges in visual recognition is the ability to cope with unexpected stimuli. In this work, we present a technique to interpret detected anomalies and update the existing knowledge of normal situations. The addressed context is the analysis of human behavior in indoor surveillance scenarios, where new activities might need to be learned, once the system is already in operation. Our approach is based on human tracking with multiple activity trackers. The main contribution is to integrate a learning stage, where labeled and unlabeled information is collected and analyzed. To this end we develop a new multi-class version of transfer learning which requires minimal human interaction but still provides semantic labels of the new classes. The activity model is then updated with the new activities. Experiments show promising results.
3D reconstruction of freely moving persons for re-identification with a depth sensor

In this work, we describe a novel method for creating 3D models of persons freely moving in front of a consumer depth sensor and we show how they can be used for long-term person re-identification. For overcoming the problem of the different poses a person can assume, we exploit the information provided by skeletal tracking algorithms for warping every point cloud frame to a standard pose in real time. Then, the warped point clouds are merged together to compose the model. Re-identification is performed by matching body shapes in terms of whole point clouds warped to a standard pose with the described method. We compare this technique with a classification method based on a descriptor of skeleton features and with a mixed approach which exploits both skeleton and shape features. We report experiments on two datasets we acquired for RGB-D re-identification which use different skeletal tracking algorithms and which are made publicly available to foster research in this new research branch.