nils-herrmann/clean_scirepeval huggingface dataset:
Video-based tracking of user's motion for augmented desk interface

This paper presents an overview of our project on an augmented desk interface system called the EnhancedDesk. The EnhancedDesk is equipped with an infrared camera for tracking a users' hand motions, a color video camera for recognizing objects, and projectors for displaying various kinds of digital information onto the desk. The key technical innovations of the EnhancedDesk include: fast and accurate tracking of multiple fingers for direct manipulation of both real and projected objects with bare hands, interactive object registration and recognition with hand gestures, vision-based tracking of a user's gaze, and addition of interactive functionality overlaid on the desk. This paper presents a brief description of the components, as well as some prototypes of applications of our augmented desk interface system.
From RGB to Spectrum for Natural Scenes via Manifold-Based Mapping

Spectral analysis of natural scenes can provide much more detailed information about the scene than an ordinary RGB camera. The richer information provided by hyperspectral images has been beneficial to numerous applications, such as understanding natural environmental changes and classifying plants and soils in agriculture based on their spectral properties. In this paper, we present an efficient manifold learning based method for accurately reconstructing a hyperspectral image from a single RGB image captured by a commercial camera with known spectral response. By applying a nonlinear dimensionality reduction technique to a large set of natural spectra, we show that the spectra of natural scenes lie on an intrinsically low dimensional manifold. This allows us to map an RGB vector to its corresponding hyperspectral vector accurately via our proposed novel manifold-based reconstruction pipeline. Experiments using both synthesized RGB images using hyperspectral datasets and real world data demonstrate our method outperforms the state-of-the-art.
Learning-by-Synthesis for Appearance-Based 3D Gaze Estimation

Inferring human gaze from low-resolution eye images is still a challenging task despite its practical importance in many application scenarios. This paper presents a learning-by-synthesis approach to accurate image-based gaze estimation that is person- and head pose-independent. Unlike existing appearance-based methods that assume person-specific training data, we use a large amount of cross-subject training data to train a 3D gaze estimator. We collect the largest and fully calibrated multi-view gaze dataset and perform a 3D reconstruction in order to generate dense training data of eye images. By using the synthesized dataset to learn a random regression forest, we show that our method outperforms existing methods that use low-resolution eye images.
Photometric stereo with auto-radiometric calibration

We propose a novel method for estimating surface normals and a radiometric response function of a camera at the same time. Photometric stereo assumes that a camera is radiometrically calibrated in advance, so that image irradiance values can be determined from observed pixel values. Our proposed method avoids such often cumbersome radiometric calibration of a camera by the simultaneous estimation of surface normals and a radiometric response function. The key idea of our method is to make use of the consistency; the irradiance values converted from pixel values by using the radiometric response function should be equal to the corresponding irradiance values calculated from the surface normals. We show that the simultaneous estimation results in a linear least-square problem with linear constraints. The experimental results demonstrate that our method can estimate surface normals accurately even when images are captured by using cameras with nonlinear radiometric response functions.
Early facial expression recognition with high-frame rate 3D sensing

This work investigates a new challenging problem: how to exactly recognize facial expression as early as possible, while most works generally focus on improving the recognition rate of facial expression recognition. The features of facial expressions in their early stage are unfortunately very sensitive to noise due to their low intensity. So, we propose a novel wavelet spectral subtraction method to spatio-temporally refine the subtle facial expression features. Moreover, in order to achieve early facial expression recognition, we newly introduce an early AdaBoost algorithm for facial expression recognition problem. Experiments using our database established by using a high-frame rate 3D sensing showed that the proposed method has a promising performance on early facial expression recognition.
Illumination distribution from shadows

The image irradiance of a three-dimensional object is known to be the function of three components: the distribution of light sources, the shape, and reflectance of a real object surface. In the past, recovering the shape and reflectance of an object surface from the recorded image brightness has been intensively investigated. On the other hand, there has been little progress in recovering illumination from the knowledge of the shape and reflectance of a real object. In this paper, we propose a new method for estimating the illumination distribution of a real scene from image brightness observed on a real object surface in that scene. More specifically, we recover the illumination distribution of the scene from a radiance distribution inside shadows cast by an object of known shape onto another object surface of known shape and reflectance. By using the occlusion information of the incoming light, we are able to reliably estimate the illumination distribution of a real scene, even in a complex illumination environment.
The study of spectroscopic properties of Nd: PTR glass

In order to scale up the optical energy density of laser media, the gain control technology in each small domain of laser media is required. Considering recent spreading of laser resonator control that uses volume Bragg diffraction grating (VBG) , it is convenient for VBG to have an ability of laser action. VBG is made of photo-thermo-refractive (PTR) glass that is one of photosensitive glasses, and co-doping of PTR glass with the second rare earth component does not eliminate its photosensitivity for making Bragg gratings . If we choose this second rare-earth appropriately, doped luminous ions provide additional function to VBG. In this work we evaluated the spectroscopic properties of Nd3+-doped photo-thermo-refractive glass (Nd:PTR). Absorption coefficient, Stimulated emission cross section, fluorescence lifetime, and radiative quantum efficiency of Nd:PTR are discussed, which aims at the investigation of the laser performance.
Motion Control of Self-Moving Trays for Human Supporting Production Cell "Attentive Workbench"

We propose “Attentive Workbench (AWB),” a new cell production system in which an intelligent system supports human workers. Using cameras, projectors, planar motor driven self-moving trays and other devices, the system supports workers from both physical and information aspects, recognizing worker’s condition and intention. This paper outlines AWB and deals with physical assembly support using self-moving parts trays. Two different schemes, centralized and decentralized, for controlling multiple parts trays are evaluated through experiments and simulations. The centralized control scheme is found to have the higher performance than the decentralized control scheme.
Recovering shape and reflectance properties from a sequence of range and color images

For synthesizing realistic images of a real three dimensional object, reflectance properties of the object surface, as well as the object shape, need to be measured. This paper describes one approach to create a three dimensional object model with physically correct reflectance properties by observing a real object. The approach consists of three steps. First, a sequence of range images and color images is measured by rotating a real object on a rotary table with fixed viewing and illumination directions. Then, the object shape is obtained as a collection of triangular patches by merging multiple range images. Secondly, by using the recovered object shape, color pixel intensities of the color image sequence are separated into the diffuse reflection component and the specular reflection component. Finally the separated reflection components are used to estimate parameters of the Lambertian reflection model and a simplified Torrance-Sparrow reflection model. We have successfully tested our approach by using images of a real object. Synthesized images of the object under arbitrary illumination conditions are shown in this paper.
Exploiting Spectral-Spatial Correlation for Coded Hyperspectral Image Restoration

Conventional scanning and multiplexing techniques for hyperspectral imaging suffer from limited temporal and/or spatial resolution. To resolve this issue, coding techniques are becoming increasingly popular in developing snapshot systems for high-resolution hyperspectral imaging. For such systems, it is a critical task to accurately restore the 3D hyperspectral image from its corresponding coded 2D image. In this paper, we propose an effective method for coded hyperspectral image restoration, which exploits extensive structure sparsity in the hyperspectral image. Specifically, we simultaneously explore spectral and spatial correlation via low-rank regularizations, and formulate the restoration problem into a variational optimization model, which can be solved via an iterative numerical algorithm. Experimental results using both synthetic data and real images show that the proposed method can significantly outperform the state-of-the-art methods on several popular coding-based hyperspectral imaging systems.
Real-time input of 3D pose and gestures of a user's hand and its applications for HCI

Introduces a method for tracking a user's hand in 3D and recognizing the hand's gesture in real time without the use of any invasive devices attached to the hand. Our method uses multiple cameras for determining the position and orientation of a user's hand moving freely in a 3D space. In addition, the method identifies pre-determined gestures in a fast and robust manner by using a neural network which has been properly trained beforehand. This paper also describes results of user study of our proposed method and several types of applications, including 3D object handling for a desktop system and a 3D walkthrough for a large immersive display system.
Reflectance analysis under solar illumination

The paper describes reflectance analysis of real images taken in an outdoor environment. In the past, the fundamental difficulties involved in analyzing outdoor images have prevented the use of traditional techniques for reflectance analysis such as shape from shading and photometric stereo. Those difficulties include: 1. multiple reflection components due to two light sources of different spectral and spatial distributions, namely the sunlight and the skylight, 2. ambiguity, in surface normal determination caused by the sun's restricted motion on the ecliptic, and 3. non-uniform surface albedo of object surfaces. In order to overcome those difficulties, we developed a reflectance model under solar illumination. Based on this model, we proposed solutions for each of these problems. We have successfully tested our algorithms using real color images taken both in a laboratory setup simulating sunlight, and in an outdoor environment under solar illumination.
Shape Reconstruction Based on Similarity in Radiance Changes under Varying Illumination

This paper presents a technique for determining an object's shape based on the similarity of radiance changes observed at points on its surface under varying illumination. To examine the similarity, we use an observation vector that represents a sequence of pixel intensities of a point on the surface under different lighting conditions. Assuming convex objects under distant illumination and orthographic projection, we show that the similarity between two observation vectors is closely related to the similarity between the surface normals of the corresponding points. This enables us to estimate the object's surface normals solely from the similarity of radiance changes under unknown distant lighting by using dimensionality reduction. Unlike most previous shape reconstruction methods, our technique neither assumes particular reflection models nor requires reference materials. This makes our method applicable to a wide variety of objects made of different materials.
Detecting Forgery From Static-Scene Video Based on Inconsistency in Noise Level Functions

Recently developed video editing techniques have enabled us to create realistic synthesized videos. Therefore, using video data as evidence in places such as courts of law requires a method to detect forged videos. In this study, we developed an approach to detect suspicious regions in a video of a static scene on the basis of the noise characteristics. The image signal contains irradiance-dependent noise the variance of which is described by a noise level function (NLF) as a function of irradiance. We introduce a probabilistic model providing the inference of an NLF that controls the characteristics of the noise at each pixel. Forged pixels in the regions clipped from another video camera can be differentiated by using maximum a posteriori estimation for the noise model when the NLFs of the regions are inconsistent with the rest of the video. We demonstrate the effectiveness of our proposed method by adapting it to videos recorded indoors and outdoors. The proposed method enables us to highly accurately evaluate the per-pixel authenticity of the given video, which achieves denser estimation than prior work based on block-level validation. In addition, the proposed method can be applied to various kinds of videos such as those contaminated by large noise and recorded with any scan formats, which limits the applicability of the existing methods.
Illumination and reflectance spectra separation of a hyperspectral image meets low-rank matrix factorization

This paper addresses the illumination and reflectance spectra separation (IRSS) problem of a hyperspectral image captured under general spectral illumination. The huge amount of pixels in a hypersepctral image poses tremendous challenges on computational efficiency, yet in turn offers greater color variety that might be utilized to improve separation accuracy and relax the restrictive subspace illumination assumption in existing works. We show that this IRSS problem can be modeled into a low-rank matrix factorization problem, and prove that the separation is unique up to an unknown scale under the standard low-dimensionality assumption of reflectance. We also develop a scalable algorithm for this separation task that works in the presence of model error and image noise. Experiments on both synthetic data and real images have demonstrated that our separation results are sufficiently accurate, and can benefit some important applications, such as spectra relighting and illumination swapping.
Measurement of surface orientations of transparent objects using polarization in highlight

This paper proposes a method for obtaining surface orientations of transparent objects using polarization in highlight. Since the highlight, the specular component of reflection light from objects, is observed only near the specular direction, it appears merely on limited parts on an object surface. In order to obtain orientations of a whole object surface, we employ a spherical extended light source. This paper reports its experimental apparatus, a shape recovery algorithm, and its performance evaluation.
Reflectance and Fluorescent Spectra Recovery Based on Fluorescent Chromaticity Invariance under Varying Illumination

In recent years, fluorescence analysis of scenes has received attention. Fluorescence can provide additional information about scenes, and has been used in applications such as camera spectral sensitivity estimation, 3D reconstruction, and color relighting. In particular, hyperspectral images of reflective-fluorescent scenes provide a rich amount of data. However, due to the complex nature of fluorescence, hyperspectral imaging methods rely on specialized equipment such as hyperspectral cameras and specialized illuminants. In this paper, we propose a more practical approach to hyperspectral imaging of reflective-fluorescent scenes using only a conventional RGB camera and varied colored illuminants. The key idea of our approach is to exploit a unique property of fluorescence: the chromaticity of fluorescence emissions are invariant under different illuminants. This allows us to robustly estimate spectral reflectance and fluorescence emission chromaticity. We then show that given the spectral reflectance and fluorescent chromaticity, the fluorescence absorption and emission spectra can also be estimated. We demonstrate in results that all scene spectra can be accurately estimated from RGB images. Finally, we show that our method can be used to accurately relight scenes under novel lighting.
Robust Content-Dependent Photometric Projector Compensation

We wish to compensate for irregularities in the output of digital projectors that occur when they are used in non-ideal situations, such as those with varying surface reflectance and ambient light. We transform the image to be displayed into a compensation image that will produce the desired appearance. In contrast to previous methods, the transformation is based on both a radiometric model of the system and the content of the image. We present a five-stage framework for performing content-dependent photometric compensation, and the details of a specific implementation. The original image is converted to a perceptually-uniform space, the desired chrominance is fitted to the gamut of the projector, a luminance range is calculated within which the fitted chrominance values can be produced, the original luminance is fitted to that range, and finally the fitted values are converted to a compensation image. Our method balances strict compensation against dynamic range in the final output, so we can produce a good result even when removal of all visible spatial variation is not possible.
Recognizing Micro-Actions and Reactions from Paired Egocentric Videos

We aim to understand the dynamics of social interactions between two people by recognizing their actions and reactions using a head-mounted camera. Our work will impact several first-person vision tasks that need the detailed understanding of social interactions, such as automatic video summarization of group events and assistive systems. To recognize micro-level actions and reactions, such as slight shifts in attention, subtle nodding, or small hand actions, where only subtle body motion is apparent, we propose to use paired egocentric videos recorded by two interacting people. We show that the first-person and second-person points-of-view features of two people, enabled by paired egocentric videos, are complementary and essential for reliably recognizing micro-actions and reactions. We also build a new dataset of dyadic (two-persons) interactions that comprises more than 1000 pairs of egocentric videos to enable systematic evaluations on the task of micro-action and reaction recognition.
Consensus surfaces for modeling 3D objects from multiple range images

In this paper, we present a robust method for creating a triangulated surface mesh from multiple range images. Our method merges a set of range images into a volumetric implicit-surface representation which is converted to a surface mesh using a variant of the marching-cubes algorithm. Unlike previous techniques based on implicit-surface representations, our method estimates the signed distance to the object surface by finding a consensus of locally coherent observations of the surface. We call this method the consensus-surface algorithm. This algorithm effectively eliminates many of the troublesome effects of noise and extraneous surface observations without sacrificing the accuracy of the resulting surface. We utilize octrees to represent volumetric implicit surfaces-effectively reducing the computation and memory requirements of the volumetric representation without sacrificing accuracy of the resulting surface. We present results which demonstrate that our consensus-surface algorithm can construct accurate geometric models from rather noisy input range data.
Attached shadow coding: Estimating surface normals from shadows under unknown reflectance and lighting conditions

We present a novel technique, termed attached shadow coding, for estimating surface normals from shadows when the reflectance and lighting conditions are unknown. Our key idea is encoding surface points via attached shadows observed under different light source directions and then estimating surface normals on the basis of the similarity of the attached shadow codes. Because shadows do not rely on reflectance properties, our method is applicable to surfaces with various complex reflectances such as anisotropic and composite materials. Moreover, our method is robust against noise because it takes advantage of the combination of weak constraints imposed by a number of light sources. We theoretically show that the distance between the codes at two surface points is equal to the angle between the corresponding surface normals under the assumption of uniform lighting and a convex object. Our method embeds high-dimensional codes into a 3D surface normal space so that the inter-code distances are preserved. Furthermore, we extend the method in order to alleviate the effects of nonuniform lighting and cast shadows. Experimental results demonstrate the effectiveness of our method.
Eigen-texture method: Appearance compression based on 3D model

Image-based and model-based methods are two representative rendering methods for generating virtual images of objects from their real images. Extensive research on these two methods has been made in CV and CG communities. However, both methods still have several drawbacks when it comes to applying them to the mixed reality where we integrate such virtual images with real background images. To overcome these difficulties, we propose a new method which we refer to as the Eigen-Texture method. The proposed method samples appearances of a real object under various illumination and viewing conditions, and compresses them in the 2D coordinate system defined on the 3D model surface. The 3D model is generated from a sequence of range images. The Eigen-Texture method is practical because it does not require any detailed reflectance analysis of the object surface, and has great advantages due to the accurate 3D geometric models. This paper describes the method, and reports on its implementation.
Robust localization for 3D object recognition using local EGI and 3D template matching with M-estimators

A teleoperated system in a robot greatly reduces the demands on the human operator, although some human intervention is still required to perform such tasks as insulator recognition, positional adjustments of the robot, and guidance toward electric lines and insulators. In order to automate some of the robot's capabilities, we have developed a 3D object-localization method for the robot's positional adjustment. The method is designed to be insensitive to noise, outliers and occlusions while, at the same time, it has optimal run-time efficiency. The main contribution of our algorithm is the use of an objective function which is specified to reduce the effect of noise and outliers in the range image and a method for minimizing this function. The objective function is efficiently minimized by dynamically recomputing correspondences as the pose improves. Our algorithm is general enough to be applied not only to our dual-armed mobile robots but also to other teleoperation robots. This algorithm should greatly reduce the burden of operators when applied. This paper first describes our algorithm, and then presents a performance evaluation.