nils-herrmann/clean_scirepeval huggingface dataset:
Real-time markerless tracking for augmented reality: the virtual visual servoing framework

Tracking is a very important research subject in a real-time augmented reality context. The main requirements for trackers are high accuracy and little latency at a reasonable cost. In order to address these issues, a real-time, robust, and efficient 3D model-based tracking algorithm is proposed for a "video see through" monocular vision system. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose. In this paper, nonlinear pose estimation is formulated by means of a virtual visual servoing approach. In this context, the derivation of point-to-curves interaction matrices are given for different 3D geometrical primitives including straight lines, circles, cylinders, and spheres. A local moving edges tracker is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating an M-estimator into the visual control law via an iteratively reweighted least squares implementation. This approach is then extended to address the 3D model-free augmented reality problem. The method presented in this paper has been validated on several complex image sequences including outdoor environments. Results show the method to be robust to occlusion, changes in illumination, and mistracking.
ViSP for visual servoing: a generic software platform with a wide class of robot control skills

ViSP (Visual Servoing Platform), a fully functional modular architecture that allows fast development of visual servoing applications, is described. The platform takes the form of a library which can be divided in three main modules: control processes, canonical vision-based tasks that contain the most classical linkages, and real-time tracking. ViSP software environment features independence with respect to the hardware, simplicity, extendibility, and portability. ViSP also features a large library of elementary tasks with various visual features that can be combined together, an image processing library that allows the tracking of visual cues at video rate, a simulator, an interface with various classical framegrabbers, a virtual 6-DOF robot that allows the simulation of visual servoing experiments, etc. The platform is implemented in C++ under Linux.
Dynamic sensor planning in visual servoing

We present an approach to dynamic sensor planning problems in visual servoing. Specifically, one of the main problems an image-based visual servoing is to plan the camera trajectory in order to avoid undesired configurations (e.g., features out of view, collision with obstacles, etc.). Our approach uses the robot redundancy and employs a control scheme based on the task function approach. It combines the regulation of the selected vision-based task with the minimization of a secondary cost function, which reflects given constraints on the manipulator trajectory. We describe how this methodology is applied to common problems in robotic vision: occlusion avoidance, field of view constraint and obstacle avoidance. We demonstrate the validity of this approach with various experiments.
VISP: a software environment for eye-in-hand visual servoing

We describe a modular software that allows fast development of eye-in-hand image-based visual servoing applications (VISP states for "Visual Servoing Platform"). Visual servoing consists of specifying a task as the regulation in the image of a set of visual features. Various issues have thus to considered in the design of such application: among these issues we find the control of camera motions and the tracking of visual features. Our environment features a wide class of control skills as well as a library of real-time tracking processes. Some applications that used this modular architecture on a six DOF cartesian robot are finally presented.
Tracking complex targets for space rendezvous and debris removal applications

In the context of autonomous rendezvous and space debris removal, visual model-based tracking can be particularly suited. Some classical methods achieve the tracking by relying on the alignment of projected lines of the 3D model with edges detected in the image. However, processing complete 3D models of complex objects, of any shape, presents several limitations, and is not always suitable for real-time applications. This paper proposes an approach to avoid these shortcomings. It takes advantage of GPU acceleration and 3D rendering. From the rendered model, visible edges are extracted, from both depth and texture discontinuities. Correspondences with image edges are found thanks to a 1D search along the edge normals. Our approach addresses the pose estimation task as the full scale nonlinear minimization of a distance to a line. A multiple hypothesis solution is also proposed, improving tracking robustness. Our method has been evaluated on both synthetic images (provided with ground truth) and real images.
Camera localization using mutual information-based multiplane tracking

This paper deals with dense visual tracking robust towards scene perturbations using 3D information to provide a space-time coherency. The proposed method is based on an piecewise-planar scenes visual tracking algorithm which aims to minimize an error between an observed image and a reference template by estimating the parameters of a rigid 3D transformation taking into acount the relative positions of the planes in the scene. The major drawback of this approch stems from the registration function used to perform the minimization (the sum of squared differences) as it is very poorly robust towards scene variations. In this paper, the tracking process is adapted to take into account two more complex registration functions. First, the sum of conditional variance. Since it is invariant to global illumination variations, the proposed algorithm is robust with relation to those conditions whilst keeping a low computation complexity. Then, the mutual information is considered. In that case the complexity is greater but so is the robustness towards non global illumination variations, specularities or occlusions. The proposed approaches, after being described, are tested on different scenes under varying illumination conditions to assess their respective efficiency.
Through-the-eyes control of a virtual humanoid

We present an animation technique to control a humanoid in a virtual environment. The automatic generation of humanoid motion is difficult and needs to be simplified. The solution proposed in this paper consists of controlling humanoid motion through the image it "perceives": through-the-eyes control. The considered approach is based on the visual servoing concept. It allows the automatic generation of (virtual) camera motion by simply specifying the task in the image space. This approach is suited to highly reactive contexts (video games, virtual reality). We also discuss the integration of such techniques in a more complex behavioral system.
Real-time 3D model-based tracking: combining edge and texture information

This paper proposes a real-time, robust and efficient 3D model-based tracking algorithm. A nonlinear minimization approach is used to register 2D and 3D cues for monocular 3D tracking. The integration of texture information in a more classical nonlinear edge-based pose computation highly increases the reliability of more conventional edge-based 3D tracker. Robustness is enforced by integrating a M-estimator into the minimization process via an iteratively re-weighted least squares implementation. The method presented in this paper has been validated on several video sequences as well as in visual servoing experiments considering various objects. Results show the method to be robust to large motions and textured environments
Texture-less planar object detection and pose estimation using Depth-Assisted Rectification of Contours

This paper presents a method named Depth-Assisted Rectification of Contours (DARC) for detection and pose estimation of texture-less planar objects using RGB-D cameras. It consists in matching contours extracted from the current image to previously acquired template contours. In order to achieve invariance to rotation, scale and perspective distortions, a rectified representation of the contours is obtained using the available depth information. DARC requires only a single RGB-D image of the planar objects in order to estimate their pose, opposed to some existing approaches that need to capture a number of views of the target object. It also does not require to generate warped versions of the templates, which is commonly needed by existing object detection techniques. It is shown that the DARC method runs in real-time and its detection and pose estimation quality are suitable for augmented reality applications.
One Click Focus with Eye-in-hand/Eye-to-hand Cooperation

A critical assumption of many multi-view control systems is the initial visibility of the regions of interest from all the views. An initialization step is proposed for a hybrid eye-in-hand/eye-to-hand grasping system to fulfil this requirement. In this paper, the object of interest is assumed to be within the eye-to-hand field of view, whereas it may not be within the eye-in-hand one. The object model is unknown and no database is used. The object lies in a complex scene with a cluttered background. A method to automatically focus on the object of interest is presented, tested and validated on a multi view robotic system.
A new application for saliency maps: synthetic vision of autonomous actors

We present in this paper a new and original application for saliency maps, intending to simulate the visual perception of a synthetic actor. Within computer graphics field, simulating virtual humans has become a challenging task. Animating such an autonomous actor within a virtual environment requires most of the time in modeling of a perception-decision-action cycle. To model a part of the perception process, we have designed a new model of saliency map, based on geometric and depth information, allowing our synthetic humanoid to perceive its environment in a biologically plausible way.
Pose Estimation for Augmented Reality: A Hands-On Survey

Augmented reality (AR) allows to seamlessly insert virtual objects in an image sequence. In order to accomplish this goal, it is important that synthetic elements are rendered and aligned in the scene in an accurate and visually acceptable way. The solution of this problem can be related to a pose estimation or, equivalently, a camera localization process. This paper aims at presenting a brief but almost self-contented introduction to the most important approaches dedicated to vision-based camera localization along with a survey of several extension proposed in the recent years. For most of the presented approaches, we also provide links to code of short examples. This should allow readers to easily bridge the gap between theoretical aspects and practical implementations.
Active sensor placement for complete scene reconstruction and exploration

This paper deals with the 3D structure estimation and exploration of a scene using active vision. We have used the structure from controlled motion approach to obtain a precise and robust estimation of the 3D structure of geometrical primitives. Since it involves gazing successively on the considered primitives, we have developed perceptual strategies able to perform a succession of robust estimations without any assumption on the number and on the localization of the different objects. An exploration process centered on current visual features and on the structure of the previously studied primitives is presented. This leads to a gaze planning strategy that mainly uses a representation of known and unknown areas as a basis for selecting viewpoints. The proposed strategy ensures the completeness of the reconstruction.
Motion-based obstacle detection and tracking for car driving assistance

The paper is concerned with the detection and tracking of obstacles from a camera mounted on a vehicle with a view to driver assistance. To achieve this goal, we have designed a technique entirely based on image motion analysis. We perform the robust estimation of the dominant image motion assumed to be due to the camera motion. Then by considering the outliers to the estimated dominant motion, we can straightforwardly detect obstacles in order to assist car driving. We have added to the detection step a tracking module that also relies on a motion consistency criterion. Time-to-collision is then computed for each validated obstacle. We have thus developed an application-oriented solution which has proven accurate, reliable and efficient as demonstrated by experiments on numerous real situations.
Reflectance and Illumination Estimation for Realistic Augmentations of Real Scenes

The acquisition of surface material properties and lighting conditions is a fundamental step for photo-realistic Augmented Reality (AR). In this paper, we present a new method for the estimation of diffuse and specular reflectance properties of indoor real static scenes. Using an RGB-D sensor, we further estimate the 3D position of light sources responsible for specular phenomena and propose a novel photometry-based classification for all the 3D points. Our algorithm allows convincing AR results such as realistic virtual shadows as well as proper illumination and specularity occlusions.
A Dense and Direct Approach to Visual Servoing Using Depth Maps

This paper presents a novel 3-D servoing approach using dense depth maps to perform robotic tasks. With respect to position-based approaches, our method does not require the estimation of the 3-D pose (direct), nor the extraction and matching of 3-D features (dense), and only requires dense depth maps provided by 3-D sensors. Our approach has been validated in various servoing experiments using the depth information from a low-cost Red Green Blue-Depth (RGB-D) sensor. Positioning tasks are properly achieved despite noisy measurements, even when partial occlusions or scene modifications occur. We also show that, in cases where a reference depth map cannot be easily available, synthetic ones generated with a rendering engine still lead to satisfactory positioning performances. Application of the approach to the navigation of a mobile robot is also demonstrated.
Improvements in robust 2D visual servoing

A fundamental step towards broadening the use of real world image-based visual servoing is to deal with the important issues of reliability and robustness. In order to address this issue, a closed loop control law is proposed that simultaneously accomplishes a visual servoing task and is robust to a general class of image processing errors. This is achieved with the application of widely accepted statistical techniques of robust M-estimation. Furthermore, improvement have been added in the weight computation process: memory, initialization. Indeed, when the error between current visual features and desired ones are large, which occurs when large robot displacement are required, M-estimator may not detect outliers. To address this point, the method we propose to initialize the confidence in each feature is based on the LMedS estimators. Experimental results are presented which demonstrate visual servoing tasks which resist severe outlier contamination.
Visual servoing through mirror reflection

Apart the use of catadioptric cameras, only few visual servoing works exploit the use of mirror. Such a configuration is however interesting since it allows to overpass the limited camera field of view. Based on the known projection equations involved in such a system, this paper introduces the theoretical background that allows the use of planar mirror for visual servoing in different configurations. Limitations intrinsic to such systems, such as the number of d.o.f actually controllable, is then discussed. Experiments using a mirror mounted on the end-effector of a 6 d.o.f robot validate the proposed approach.
A new redundancy-based iterative scheme for avoiding joint limits. Application to visual servoing

We propose in this paper new redundancy-based solutions to avoid robot joint limits of a manipulator. We use a control scheme based on the task function approach. We first recall the classical gradient projection approach and we then present a far more efficient method that relies on the iterative computation of motion that does not affect the task achievement and ensures the avoidance problem. We apply this new method in a visual servoing application and we demonstrate on various real experiments the validity of the approach.
Robust real-time visual tracking using a 2D-3D model-based approach

We present an original method for tracking, in an image sequence, complex objects which can be approximately modeled by a polyhedral shape. The approach relies on the estimation of the 2D object image motion along with the computation of the 3D object pose. The proposed method fulfills real-time constraints along with reliability and robustness requirements. Real tracking experiments and results concerning a visual servoing positioning task are presented.
Improving mutual information-based visual servoing

In a previous paper , we proposed a new way to achieve visual servoing. Rather than minimizing the error between the position of two set of geometric features, we proposed to maximize the mutual information shared by the current and desired images. This leads to a new information theoretic approach to visual servoing. Mutual information is a well known alignment function. Thanks to its robustness toward illumination variations, occlusions and multi modality, it has been widely used in medical applications for alignment as well as in general tracking problems. Despite those previous works, no highlight has been given on the problem of Hessian computation that yields, in the case of common approximations, to divergence of the optimization process. In this paper we focus on the need of computing the second order derivative of the mutual information in visual servoing. Experiments on a 6 dof robot demonstrates the significance of this work on visual servoing tasks.
Direct 3D servoing using dense depth maps

This paper proposes a novel 3D servoing approach using dense depth maps to achieve robotic tasks. With respect to position-based approaches, our method does not require the estimation of the 3D pose (direct), nor the extraction and matching of 3D features (dense) and only requires dense depth maps provided by 3D sensors. Our approach has been validated in servoing experiments using the depth information from a low cost RGB-D sensor. Positioning tasks are properly achieved despite the noisy measurements, even when partial occlusions or scene modifications occur.
Visual perception based on salient features

We present in this paper an original model to simulate visual perception based on the detection of salient features. Salient features correspond to the maximum of conspicuity in a static image or in a sequence of images. We intend to use this information to provide visually interesting targets that can be used in multiple contexts. The extraction process of such an information is performed through multiple steps that correspond to different features locally encoding the conspicuity of a spatial location. We mainly use for those features spatial orientation and motion information, but other types of feature could be used as well. Two kinds of application are presented in this paper: video surveillance application and simulation of the visual perception of a synthetic actor.
Visual servoing from lines using a planar catadioptric system

In this paper, we propose a complete scheme to control a mirror, using a visual servoing scheme, using lines as a set of visual features. Considering the equations of the projection of the reflection of a lines on a mirror, this paper introduces the theoretical background that allows to control the mirror using visual information. Experiments using a mirror mounted on the end-effector of a 6 d.o.f robot validate the proposed approach.
Direct visual servoing based on multiple intensity histograms

Classically Visual servoing considered the regulation in the image of a set of visual features (usually geometric features). Recently direct visual servoing scheme, such as photometric visual servoing, have been introduced in order to consider every pixel of the image as a primary source of information and thus avoid the extraction and the tracking of such geometric features. Previous works proposed methods to use directly the image intensities in the definition of the control law, by using for example mutual information. In this paper, we propose a method to extend these works by using a global descriptor, namely intensity histograms, on the whole or multiple sub-sets of the images in order to achieve control of a 6 degrees of freedom (DoF) robot. The results are then demonstrated through experimental validations.
Mutual Information-Based Visual Servoing

In this paper, we propose a new information theoretic approach to achieve visual servoing directly utilizing the information (as defined by Shannon) contained in the images. A metric derived from information theory, i.e., mutual information, is considered. Mutual information is widely used in multimodal image registration since it is insensitive to changes in the lighting condition and to a wide class of nonlinear image transformations. In this paper, mutual information is used as a new visual feature for visual servoing, which allows us to build a new control law that can control the six degrees of freedom (DOF) of a robot. Among various advantages, this approach requires no matching or tracking step, is robust to large illumination variations, and allows the consideration of different image modalities within the same task. Experiments on a real robot demonstrate the efficiency of the proposed visual-servoing approach.
Visual servoing using the sum of conditional variance

In this paper we propose a new way to achieve direct visual servoing. The novelty is the use of the sum of conditional variance to realize the optimization process of a positioning task. This measure, which has previously been used successfully in the case of visual tracking, has been shown to be invariant to non-linear illumination variations and inexpensive to compute. Compared to other direct approaches of visual servoing, it is a good compromise between techniques using the illumination of pixels which are computationally inexpensive but non robust to illumination variations and other approaches using the mutual information which are more complicated to compute but offer more robustness towards the variations of the scene. This method results in a direct visual servoing task easy and fast to compute and robust towards non-linear illumination variations. This paper describes a visual servoing task based on the sum of conditional variance performed using a Levenberg-Marquardt optimization process. The results are then demonstrated through experimental validations and compared to both photometric-based and entropy-based techniques.
Combining complementary edge, keypoint and color features in model-based tracking for highly dynamic scenes

This paper focuses on the issue of estimating the complete 3D pose of the camera with respect to a complex object, in a potentially highly dynamic scene, through model-based tracking. We propose to robustly combine complementary geometrical edge and point features with color based features in the minimization process. A Kalman filtering and pose prediction process is also suggested to handle potential large interframe motions. In order to deal with complex 3D models, our method takes advantage of hardware acceleration. Promising results, outperforming classical state-of-art approaches, have been obtained on various real and synthetic image sequences, with a focus on space robotics applications.