nils-herrmann/clean_scirepeval huggingface dataset:
Multi-modal Sequential Monte Carlo for On-Line Hierarchical Graph Structure Estimation in Model-based Scene Interpretation

We present a computationally efficient, on-line graph structure estimation method for model-based scene interpretation. Different scenes have different hierarchical graphical models composed of place, objects, and parts. Generally, it is very difficult and time-consuming to estimate dynamic graph structures. The key idea is to represent hypothesized graph structures as multi-modal particles instead of joint particle representation. Such Monte Carlo representation makes the one-line hierarchical graph structure estimation feasible. The proposed method is supported by the neurobiological inference model. Large-scale experimental results in an indoor (12 places, 112 3D objects) validate the feasibility of the proposed inference method
Calibration and 3D structure recovery under varying cameras using known angles

We present an algorithm for the calibration of a camera and the recovery of 3D scene structure up to a scale from image sequences using known angles between lines in the scene. The proposed method computes the intrinsic parameters of the camera using the invariance of angles under the similarity transformation. Specifically, we recover the matrix that is the homography between the projective structure and the Euclidean structure using angles. Since this matrix is a unique one in the given set of image sequences, we can easily deal with the problem of varying intrinsic parameters of the camera. Experimental results on the synthetic and real images demonstrate the feasibility of the proposed algorithm.
RGBD sensor and mirrors: A practical setup for 3D reconstruction of dynamic objects

In order to get full 3D model from consumer depth cameras, previous approaches used set of synchronized depth cameras or moved the depth sensor around the target object. This on-going work introduces a cost effective and convenient framework for filming dynamic 3D object. This set is comprised of a consumer depth camera, or RGBD sensor and a pair of mirrors. Because of the images reflected from two mirrors in the system, it can capture a depth-map of the target object in an extended view. The practicality and effectiveness of this framework is demonstrated.
Efficient Data-Driven MCMC sampling for vision-based 6D SLAM

In this paper, we propose a Markov Chain Monte Carlo (MCMC) sampling method with the data-driven proposal distribution for six-degree-of-freedom (6-DoF) SLAM. Recently, visual odometry priors have been widely used as the process model in the SLAM formulation to improve the SLAM performance. However, modeling the uncertainties of incremental motions estimated by visual odometry is especially difficult under challenging conditions, such as erratic motion. For a particle-based model representation, it can represent the uncertainty of the camera motion well under erratic motion compared to the constant velocity model or a Gaussian noise model, but the manner of representing the proposal distribution and sampling the particles is extremely important, as we can maintain only a limited number of particles in the high-dimensional state space. Hence, we propose an effective sampling approach by exploiting MCMC sampling and the data-driven proposal distribution to propagate the particles. We demonstrate the performance of the proposed approach for 6-DoF SLAM using both synthetic and real datasets and compare the performance with those of other sampling methods.
Thermal Image Enhancement using Convolutional Neural Network

With the advent of commodity autonomous mobiles, it is becoming increasingly prevalent to recognize under extreme conditions such as night, erratic illumination conditions. This need has caused the approaches using multi-modal sensors, which could be complementary to each other. The choice for the thermal camera provides a rich source of temperature information, less affected by changing illumination or background clutters. However, existing thermal cameras have a relatively smaller resolution than RGB cameras that has trouble for fully utilizing the information in recognition tasks. To mitigate this, we aim to enhance the low-resolution thermal image according to the extensive analysis of existing approaches. To this end, we introduce Thermal Image Enhancement using Convolutional Neural Network (CNN), called in TEN, which directly learns an end-to-end mapping a single low resolution image to the desired high resolution image. In addition, we examine various image domains to find the best representative of the thermal enhancement. Overall, we propose the first thermal image enhancement method based on CNN guided on RGB data. We provide extensive experiments designed to evaluate the quality of image and the performance of several object recognition tasks such as pedestrian detection, visual odometry, and image registration.
A novel 2.5D pattern for extrinsic calibration of tof and camera fusion system

Recently, many researchers have made efforts for accurate calibration of a Time-of-Flight camera to fully utilize its provided depth values. Yet most previous works focus mainly on intrinsic calibration by modeling its systematic errors and noises while extrinsic calibration is also an important factor when constructing sensor fusion system. In this paper, we present a calibration process that can correctly transfer the depth measurements onto the color image. We use 2.5D pattern so that sufficient reprojection error can be considered for both color and ToF cameras. The issues on obtaining the correct correspondences for this pattern are discussed. In the optimization stage, the depth constraint is also employed to ensure the depth measurements to lie on the pattern plane. The strengths of the proposed method over previous approaches are evaluated in several robotic applications which require precise ToF and camera calibration.
Balanced optical flow refinement by bidirectional constraint

We present an efficient optical flow refinement approach based on a bidirectional flow consistency. Our method is an add-on component that improves the existing optical flow estimation to be balanced between forward and backward flows. Most of the state-of-the-art optical flow methods only consider unidirectional motion vectors from a source image to a target image, which can make the estimated flow inconsistent with its backward estimation. The inconsistency can be reduced by considering the bidirectional motion when the optical flow is estimated, but it would be very hard for most of the typical optical flow methods and impossible for some of them. To solve this problem, we propose a sampling-based optimization method for efficiently refining the optical flows with a bidirectional constraint. By evaluating on Middle-bury benchmark and public large displacement datasets, we validate the effectiveness of our method quantitatively and qualitatively and for accuracy.
Object detection using hierarchical graph-based segmentation

Object detection in real images or videos is challenging because the shapes and sizes of objects vary significantly according to their poses, camera viewing direction, and partial occlusion. Previous detection methods employ sliding-window-based schemes that scan windows across an image, requiring many differently shaped windows to capture shape and size variation. In order to solve this problem, we propose an object detection method using hierarchical graph-based segmentation: color-consistent parts are obtained by part-level segmentation and category-consistent regions are found using object-level segmentation. Thus we can avoid scanning a lot of windows across whole images by using part-level segmentation and robustly detect the objects of various shapes and sizes by using object-level segmentation. In addition, we evaluate detection performance using various classifiers with our detection approach.
Robust Multiview Photometric Stereo Using Planar Mesh Parameterization

We propose a robust uncalibrated multiview photometric stereo method for high quality 3D shape reconstruction. In our method, a coarse initial 3D mesh obtained using a multiview stereo method is projected onto a 2D planar domain using a planar mesh parameterization technique. We describe methods for surface normal estimation that work in the parameterized 2D space that jointly incorporates all geometric and photometric cues from multiple viewpoints. Using an estimated surface normal map, a refined 3D mesh is then recovered by computing an optimal displacement map in the same 2D planar domain. Our method avoids the need of merging view-dependent surface normal maps that is often required in conventional methods. We conduct evaluation on various real-world objects containing surfaces with specular reflections, multiple albedos, and complex topologies in both controlled and uncontrolled settings and demonstrate that accurate 3D meshes with fine geometric details can be recovered by our method.
A simple and real-time moving object detection invariant to cast shadow

We present a moving objection detection method robust to shadow effects and illumination changes on a static camera. This study addresses the effect of the declining sun at evening, which produces elongated shadow of objects. The illumination condition of the evening time is more rapidly changed. These two factors can affect the quality of the moving object detection. We deal with these artifacts by fusing the several illumination or shadow invariant features. The preliminary results of this study are validated qualitatively on two real datasets. Also, the proposed algorithm achieves the real-time performance.
RANUS: RGB and NIR Urban Scene Dataset for Deep Scene Parsing

In this letter, we present a data-driven method for scene parsing of road scenes to utilize single-channel near-infrared (NIR) images. To overcome the lack of data problem in non-RGB spectrum, we define a new color space and decompose the task of deep scene parsing into two subtasks with two separate CNN architectures for chromaticity channels and semantic masks. For chromaticity estimation, we build a spatially-aligned RGB-NIR image database (40k urban scenes) to infer color information from RGB-NIR spectrum learning process and leverage existing scene parsing networks trained over already available RGB masks. From our database, we sample key frames and manually annotate them (4k ground truth masks) to finetune the network into the proposed color space. Hence, the key contribution of this work is to replace multispectral scene parsing methods with a simple yet effective approach using single NIR images. The benefits of using our algorithm and dataset are confirmed in the qualitative and quantitative experiments.
Two-phase approach for multi-view object extraction

In this paper, we propose an automatic method to extract a foreground object captured from multiple viewpoints. We consider the foreground object is within the visual hull of camera field of views. By exploring the multi-view geometric relationship and color measurements of the input images, we can estimate the foreground segmentations as well as their fractional boundaries. To facilitate efficient computation and high quality mattes, we adopt a two-phase approach. The first phase of our algorithm provides quick and rough binary segmentations of the foreground object using graph-cut; the second phase refines the segmentation boundaries using matting. Our result is the high quality alpha mattes of the foreground object consistently across all different viewpoints. We demonstrate the effectiveness of our method using challenging examples.
3D Display Calibration by Visual Pattern Analysis

Nearly all 3D displays need calibration for correct rendering. More often than not, the optical elements in a 3D display are misaligned from the designed parameter setting. As a result, 3D magic does not perform well as intended. The observed images tend to get distorted. In this paper, we propose a novel display calibration method to fix the situation. In our method, a pattern image is displayed on the panel and a camera takes its pictures twice at different positions. Then, based on a quantitative model, we extract all display parameters (i.e., pitch, slanted angle, gap or thickness, and offset) from the observed patterns in the captured images. For high accuracy and robustness, our method analyzes the patterns mostly in the frequency domain. We conduct two types of experiments for validation; one with optical simulation for quantitative results and the other with real-life displays for qualitative assessment. Experimental results demonstrate that our method is quite accurate, about a half order of magnitude higher than prior work; is efficient, spending less than 2s for computation; and is robust to noise, working well in the SNR regime as low as 6dB.
Behavior-based mobile robot using active sensor fusion

The authors present a navigation system using multiple sensors for unknown and dynamic indoor environments. To achieve robustness and flexibility in the mobile robot, a behavior-based system architecture is developed consisting of multilayered behaviors using multiple sensors that were ultrasonic sensors and a video camera. Basic behaviors required for navigation, such as avoiding obstacles, moving toward free space, and following targets are redundantly developed as agents and combined in a behavior-based system architecture. The capabilities of the system were demonstrated in unstructured real office environments using an indoor mobile robot.<<ETX>>
A Kalman filter based visual tracking algorithm for an object moving in 3D

Robust and effective real-time visual tracking is realized by combining the first order differential invariants with stochastic filtering. The Kalman filter as an optimal stochastic filter is used to estimate the motion parameters, namely the plant state vector of the moving object with the unknown dynamics in successive image frames. Using the fact that the relative motion between the moving object and the moving observer causes the deformation, we compute the first differential invariants of the image velocity field. The surface orientation and the depth estimate between the observer and the object are computed based on these first order differential invariants. We demonstrate the robustness and feasibility of the proposed tracking algorithm through real experiments in which an X-Y Cartesian robot tracks a toy vehicle moving along 3D rails.
Motion deblurring using coded exposure for a wheeled mobile robot

We present a motion deblurring framework for a wheeled mobile robot. Motion blur is an inevitable problem in a mobile robot, especially side-view cameras severely suffer from motion blur when a mobile robot moves forward. To handle motion blur in a robot, we develop a fast motion deblurring framework using the concept of coded exposure. We estimate a blur kernel by a simple template matching between adjacent frames with a motion prior and a blind deconvolution algorithm with a Gaussian prior is exploited for fast deblurring. Our system is implemented using an off-the-shelf machine vision camera and enables us to achieve high-quality deblurring results with little computation time. We demonstrate the effectiveness of our system to handle motion blur and validate it is useful for many robot applications such as text recognition and visual structure from motion.
Vision-based autonomous navigation based on motion estimation

This paper addresses a navigation method which copes with dynamic environments, e.g. objects moving in the environment and environment changes. Because many vision-based navigation methods mainly focus on finding consistent corresponding parts with one of the database images, dynamic environments can cause the failure of autonomous navigation due to visual occlusion. To solve these problems, we propose a motion-based navigation method in contrast with appearance-based approaches. We also solve the kidnapping problem, which is defined as the problem of re-localizing a mobile robot after it is undergone an unknown motion or visual occlusion. This algorithm is based on motion estimation by a camera to plan the next movement and an efficient pose recovery scheme after kidnapping problems. Various experimental results demonstrate the capability of vision-based autonomous navigation against dynamic environments.
High-Quality Depth Map Upsampling and Completion for RGB-D Cameras

This paper describes an application framework to perform high-quality upsampling and completion on noisy depth maps. Our framework targets a complementary system setup, which consists of a depth camera coupled with an RGB camera. Inspired by a recent work that uses a nonlocal structure regularization, we regularize depth maps in order to maintain fine details and structures. We extend this regularization by combining the additional high-resolution RGB input when upsampling a low-resolution depth map together with a weighting scheme that favors structure details. Our technique is also able to repair large holes in a depth map with consideration of structures and discontinuities utilizing edge information from the RGB input. Quantitative and qualitative results show that our method outperforms existing approaches for depth map upsampling and completion. We describe the complete process for this system, including device calibration, scene warping for input alignment, and even how our framework can be extended for video depth-map completion with the consideration of temporal coherence.
Visual Categorization Robust to Large Intra-Class Variations using Entropy-guided Codebook

Categorizing visual elements is fundamentally important for autonomous mobile robots to get intelligence such as new object acquisition and topological place classification. The main problem of visual categorization is how to reduce the large intra-class variations, especially surface markings of man-made objects. In this paper, we present a robust method by introducing intermediate blurring and entropy-guided codebook selection in a bag-of-words framework. Intermediate blurring can filter out the high frequency of surface markings and provide dominant shape information. Entropy of a hypothesized codebook can provide the necessary measure for the semantic parts among training exemplars. From the first step, a generative optimal codebook for each category is learned using the MDL (minimum description length) principle guided by entropy information. From the second step, a final set of codebook is learned using the discriminative method guided by the inter-category entropy of the codebook. We select the necessary parameters through various evaluations and validate the effect of the surface marking reduction method using a Caltech-101 DB, which has large intra-class variations. Finally, we briefly introduce the impact of the method to the object categorization and segmentation problem
Intra-class key feature weighting method for vocabulary tree based image retrieval

With the existing feature weighting methods of image retrieval field, it was impossible to use the fact that images have different key features depending on their classes because the same weight is applied to every image class. We propose a method of indexing features of each class in order of importance and giving them relevant weights, which can be applied to image retrieval. We designed a simple weight mapping function in order to enhance the distinctiveness between the image classes and also proposed a method to re-rank sub-class image set to apply different weight vectors to image retrieval framework. We demonstrated the proposed method on the existing image retrieval framework to compare and verify the performance. Proposed method was evaluated with UKBench Dataset and the result showed a noticeable improvement.
Scalable temporal interest points for abstraction and classification of video events

The image sequence of a static scene includes similar or redundant information over time. Hence, motion-discontinuous instants can efficiently characterize a video shot or event. However, such instants (key frames) are differently identified according to the change of velocity and acceleration of motion, and such scales of change might be different on each sequence of the same event. In this paper, we present a scalable video abstraction in which the key frames are obtained by the maximum curvature of camera motion at each temporal scale. The scalability means dealing with the velocity and acceleration change of motion. In the temporal neighborhood determined by the scale, the scene features (motion, color, and edge) can be used to index and classify the video events. Therefore, those key frames provide temporal interest points (TIPs) for the abstraction and classification of video events.
Calibrating a Non-isotropic Near Point Light Source Using a Plane

We show that a non-isotropic near point light source rigidly attached to a camera can be calibrated using multiple images of a weakly textured planar scene. We prove that if the radiant intensity distribution (RID) of a light source is radially symmetric with respect to its dominant direction, then the shading observed on a Lambertian scene plane is bilaterally symmetric with respect to a 2D line on the plane. The symmetry axis detected in an image provides a linear constraint for estimating the dominant light axis. The light position and RID parameters can then be estimated using a linear method. Specular highlights if available can also be used for light position estimation. We also extend our method to handle non-Lambertian reflectances which we model using a biquadratic BRDF. We have evaluated our method on synthetic data quantitavely. Our experiments on real scenes show that our method works well in practice and enables light calibration without the need of a specialized hardware.
Self-calibration using the linear projective reconstruction

Self-calibration algorithms that use only the information in the image have been actively researched. However, most algorithms require bundle adjustment in the projective reconstruction or in the nonlinear minimization. We propose a practical self-calibration algorithm that only requires a linear projective reconstruction. We overcome the sensitivity of the algorithm due to image noises by adding another constraint on the principal point. Also, we propose a variant of linear auto-calibration algorithm which uses the similar assumption of the work of Pollefeys et al. (1998), based on the property of the absolute quadric. Experimental results using real and synthetic images demonstrate the feasibility of the proposed algorithm.
Haze removal using visible and infrared image fusion

In order to remove haze from images, we propose an improved method for fusing a visible and a near-infrared (NIR) image of the same scene. MR images are inherently haze-free. The NIR information allows dehazing without building and analyzing a model of haze image. We suggest a multi-resolution fusion using edge-preserving filter, bilateral filter, to reduce artifacts and computational time compared to the existing methods. Experimental results validate our approach.
Personalized Cinemagraphs Using Semantic Understanding and Collaborative Learning

Cinemagraphs are a compelling way to convey dynamic aspects of a scene. In these media, dynamic and still elements are juxtaposed to create an artistic and narrative experience. Creating a high-quality, aesthetically pleasing cinemagraph requires isolating objects in a semantically meaningful way and then selecting good start times and looping periods for those objects to minimize visual artifacts (such a tearing). To achieve this, we present a new technique that uses object recognition and semantic segmentation as part of an optimization method to automatically create cinemagraphs from videos that are both visually appealing and semantically meaningful. Given a scene with multiple objects, there are many cinemagraphs one could create. Our method evaluates these multiple candidates and presents the best one, as determined by a model trained to predict human preferences in a collaborative way. We demonstrate the effectiveness of our approach with multiple results and a user study.