nils-herrmann/clean_scirepeval huggingface dataset:
3D models from extended uncalibrated video sequences: addressing key-frame selection and projective drift

In this paper, we present an approach that is able to reconstruct 3D models from extended video sequences captured with an uncalibrated hand-held camera. We focus on two specific issues: (1) key-frame selection; and (2) projective drift. Given a long video sequence it is often not practical to work with all video frames. In addition, to allow for effective outlier rejection and motion estimation it is necessary to have a sufficient baseline between frames. For this purpose, we propose a key-frame selection procedure based on a robust model selection criterion. Our approach guarantees that the camera motion can be estimated reliably by analyzing the feature correspondences between three consecutive views. Another problem for long uncalibrated video sequences is projective drift. Error accumulation leads to a non-projective distortion of the model. This causes the projective basis at the beginning and the end of the sequence to become inconsistent and leads to the failure of self-calibration. We propose a self-calibration approach that is insensitive to this global projective drift. After self-calibration triplets of key-frames are aligned using absolute orientation and hierarchically merged into a complete metric reconstruction. Next, we compute a detailed 3D surface model using stereo matching. The 3D model is textured using some of the frames.
Non-parametric Structure-Based Calibration of Radially Symmetric Cameras

We propose a novel two-step method for estimating the intrinsic and extrinsic calibration of any radially symmetric camera, including non-central systems. The first step consists of estimating the camera pose, given a Structure from Motion (SfM) model, up to the translation along the optical axis. As a second step, we obtain the calibration by finding the translation of the camera center using an ordering constraint. The method makes use of the 1D radial camera model, which allows us to effectively handle any radially symmetric camera, including non-central ones. Using this ordering constraint, we show that the we are able to calibrate several different (central and non-central) Wide Field of View (WFOV) cameras, including fisheye, hyper-catadioptric and spherical catadioptric cameras, as well as pinhole cameras, using a single image or jointly solving for several views.
Towards Urban 3D Reconstruction from Video

The paper introduces a data collection system and a processing pipeline for automatic geo-registered 3D reconstruction of urban scenes from video. The system collects multiple video streams, as well as GPS and INS measurements in order to place the reconstructed models in geo- registered coordinates. Besides high quality in terms of both geometry and appearance, we aim at real-time performance. Even though our processing pipeline is currently far from being real-time, we select techniques and we design processing modules that can achieve fast performance on multiple CPUs and GPUs aiming at real-time performance in the near future. We present the main considerations in designing the system and the steps of the processing pipeline. We show results on real video sequences captured by our system.
Flexible 3D acquisition with a monocular camera

One of the key problems for robots and autonomous vehicles is the acquisition of 3D information about their environment. In this paper a flexible technique for 3D acquisition is proposed. This technique only requires an uncalibrated monocular camera. No prior knowledge about the scene or about the camera is necessary to build metric 3D models of the environment. In addition zoom and focus can be used freely. The feasibility of the approach has been tested on both real and synthetic data and is illustrated here on real image sequences.
USAC: A Universal Framework for Random Sample Consensus

A computational problem that arises frequently in computer vision is that of estimating the parameters of a model from data that have been contaminated by noise and outliers. More generally, any practical system that seeks to estimate quantities from noisy data measurements must have at its core some means of dealing with data contamination. The random sample consensus (RANSAC) algorithm is one of the most popular tools for robust estimation. Recent years have seen an explosion of activity in this area, leading to the development of a number of techniques that improve upon the efficiency and robustness of the basic RANSAC algorithm. In this paper, we present a comprehensive overview of recent research in RANSAC-based robust estimation by analyzing and comparing various approaches that have been explored over the years. We provide a common context for this analysis by introducing a new framework for robust estimation, which we call Universal RANSAC (USAC). USAC extends the simple hypothesize-and-verify structure of standard RANSAC to incorporate a number of important practical and computational considerations. In addition, we provide a general-purpose C++ software library that implements the USAC framework by leveraging state-of-the-art algorithms for the various modules. This implementation thus addresses many of the limitations of standard RANSAC within a single unified package. We benchmark the performance of the algorithm on a large collection of estimation problems. The implementation we provide can be used by researchers either as a stand-alone tool for robust estimation or as a benchmark for evaluating new techniques.
3D Modeling on the Go: Interactive 3D Reconstruction of Large-Scale Scenes on Mobile Devices

This paper presents a system for 3D reconstruction of large-scale outdoor scenes based on monocular motion stereo. Ours is the first such system to run at interactive frame rates on a mobile device (Google Project Tango Tablet), thus allowing a user to reconstruct scenes "on the go" by simply walking around them. We utilize the device's GPU to compute depth maps using plane sweep stereo. We then fuse the depth maps into a global model of the environment represented as a truncated signed distance function in a spatially hashed voxel grid. We observe that in contrast to reconstructing objects in a small volume of interest, or using the near outlier-free data provided by depth sensors, one can rely less on free-space measurements for suppressing outliers in unbounded large-scale scenes. Consequently, we propose a set of simple filtering operations to remove unreliable depth estimates and experimentally demonstrate the benefit of strongly filtering depth maps. We extensively evaluate the system with real as well as synthetic datasets.
Fast 3D Reconstruction of Faces with Glasses

We present a method for the fast 3D face reconstruction of people wearing glasses. Our method explicitly and robustly models the case in which a face to be reconstructed is partially occluded by glasses. We propose a simple and generic model for glasses that copes with a wide variety of different shapes, colors and styles, without the need for any database or learning. Our algorithm is simple, fast and requires only small amounts of both memory and runtime resources, allowing for a fast interactive 3D reconstruction on commodity mobile phones. The thorough evaluation of our approach on synthetic and real data demonstrates superior reconstruction results due to the explicit modeling of glasses.
A benchmarking tool for MAV visual pose estimation

The large collections of datasets for researchers working on the Simultaneous Localization and Mapping problem are mostly collected from sensors such as wheel encoders and laser range finders mounted on ground robots. The recent growing interest in doing visual pose estimation with cameras mounted on micro-aerial vehicles however has made these datasets less useful. In this paper, we describe our work in creating new datasets collected from a sensor suite mounted on a quadrotor platform. Our sensor suite includes a forward looking camera, a downward looking camera, an inertial measurement unit and a Vicon system for groundtruth. We propose the use our datasets as benchmarking tools for future works on visual pose estimation for micro-aerial vehicles. We also show examples of how the datasets could be used for benchmarking visual pose estimation algorithms.
3D modeling for communications

The media and communications providers share an increasing interest in 3D models of people, objects, and scenes. The paper focuses on features that 3D acquisition systems ought to have in order to optimally serve these markets, where emphasis is on realistic visualisation. It is argued that 3D acquisition techniques developed for traditional applications such as visual inspection aren't necessarily the best option. Techniques should be developed that are dedicated to visualisation-specific requirements. This is exemplified with two systems that have been developed recently. One takes uncalibrated video data as input from which it generates a 3D model. A second system projects a grid of lines and gets dense 3D from a single image. This system needs some calibration, but the corresponding procedure is extremely simple. It can also be used to capture detailed, 3D scene dynamics.
A model change detection approach to dynamic scene modeling

In this work we propose a dynamic scene model to provide information about the presence of salient motion in the scene, and that could be used for focusing the attention of a pan/tilt/zoom camera, or for background modeling purposes. Rather than proposing a set of saliency detectors, we define what we mean by salient motion, and propose a precise model for it. Detecting salient motion becomes equivalent to detecting a model change. We derive optimal online procedures to solve this problem, which enable a very fast implementation. Promising results show that our model can effectively detect salient motion even in severely cluttered scenes, and while a camera is panning and tilting.
Camera network calibration from dynamic silhouettes

In this paper we present an automatic method for calibrating a network of cameras from only silhouettes. This is particularly useful for shape-from-silhouette or visual-hull systems, as no additional data is needed for calibration. The key novel contribution of this work is an algorithm to robustly compute the epipolar geometry from dynamic silhouettes. We use the fundamental matrices computed by this method to determine the projective reconstruction of the complete camera configuration. This is refined into a metric reconstruction using self-calibration. We validate our approach by calibrating a four camera visual-hull system from archive data where the dynamic object is a moving person. Once the calibration parameters have been computed, we use a visual-hull algorithm to reconstruct the dynamic object from its silhouettes.
Discovering and exploiting 3D symmetries in structure from motion

Many architectural scenes contain symmetric or repeated structures, which can generate erroneous image correspondences during structure from motion (Sfm) computation. Prior work has shown that the detection and removal of these incorrect matches is crucial for accurate and robust recovery of scene structure. In this paper, we point out that these incorrect matches, in fact, provide strong cues to the existence of symmetries and structural regularities in the unknown 3D structure. We make two key contributions. First, we propose a method to recover various symmetry relations in the structure using geometric and appearance cues. A set of structural constraints derived from the symmetries are imposed within a new constrained bundle adjustment formulation, where symmetry priors are also incorporated. Second, we show that the recovered symmetries enable us to choose a natural coordinate system for the 3D structure where gauge freedom in rotation is held fixed. Furthermore, based on the symmetries, 3D structure completion is also performed. Our approach significantly reduces drift through ”structural” loop closures and improves the accuracy of reconstructions in urban scenes.
Articulated and Restricted Motion Subspaces and Their Signatures

Articulated objects represent an important class of objects in our everyday environment. Automatic detection of the type of articulated or otherwise restricted motion and extraction of the corresponding motion parameters are therefore of high value, \eg in order to augment an otherwise static 3D reconstruction with dynamic semantics, such as rotation axes and allowable translation directions for certain rigid parts or objects. Hence, in this paper, a novel theory to analyse relative transformations between two motion-restricted parts will be presented. The analysis is based on linear subspaces spanned by relative transformations. Moreover, a signature for relative transformations will be introduced which uniquely specifies the type of restricted motion encoded in these relative transformations. This theoretic framework enables the derivation of novel algebraic constraints, such as low-rank constraints for subsequent rotations around two fixed axes for example. Lastly, given the type of restricted motion as predicted by the signature, the paper shows how to extract all the motion parameters with matrix manipulations from linear algebra. Our theory is verified on several real data sets, such as a rotating blackboard or a wheel rolling on the floor amongst others.
Real-time velocity estimation based on optical flow and disparity matching

A high update rate of metric velocity values is crucial for a robust operation of navigation control loops of mobile robots such as micro aerial vehicles (MAVs). An efficient way for obtaining metric velocity of robots without external reference are image-based optical flow measurements, scaled with the distance between camera and the observed scene. However, since optical flow and stereo vision are computationally intensive tasks, metric optical flow calculations on embedded systems are typically only possible at limited frame rate. In this work, we therefore present an FPGA-based platform with the capability of calculating real-time metric optical flow at 127 frames per second and 376×240 resolution. Radial undistortion, image rectification, disparity estimation and optical flow calculation tasks are performed on a single FPGA without the need for external memory. The platform is perfectly suited for mobile robots or MAVs due to its low weight and low power consumption.
Consensus Maximization with Linear Matrix Inequality Constraints

Consensus maximization has proven to be a useful tool for robust estimation. While randomized methods like RANSAC are fast, they do not guarantee global optimality and fail to manage large amounts of outliers. On the other hand, global methods are commonly slow because they do not exploit the structure of the problem at hand. In this paper, we show that the solution space can be reduced by introducing Linear Matrix Inequality (LMI) constraints. This leads to significant speed ups of the optimization time even for large amounts of outliers, while maintaining global optimality. We study several cases in which the objective variables have a special structure, such as rotation, scaled-rotation, and essential matrices, which are posed as LMI constraints. This is very useful in several standard computer vision problems, such as estimating Similarity Transformations, Absolute Poses, and Relative Poses, for which we obtain compelling results on both synthetic and real datasets. With up to 90 percent outlier rate, where RANSAC often fails, our constrained approach is consistently faster than the non-constrained one - while finding the same global solution.
Toroidal Constraints for Two-Point Localization Under High Outlier Ratios

Localizing a query image against a 3D model at large scale is a hard problem, since 2D-3D matches become more and more ambiguous as the model size increases. This creates a need for pose estimation strategies that can handle very low inlier ratios. In this paper, we draw new insights on the geometric information available from the 2D-3D matching process. As modern descriptors are not invariant against large variations in viewpoint, we are able to find the rays in space used to triangulate a given point that are closest to a query descriptor. It is well known that two correspondences constrain the camera to lie on the surface of a torus. Adding the knowledge of direction of triangulation, we are able to approximate the position of the camera from two matches alone. We derive a geometric solver that can compute this position in under 1 microsecond. Using this solver, we propose a simple yet powerful outlier filter which scales quadratically in the number of matches. We validate the accuracy of our solver and demonstrate the usefulness of our method in real world settings.
Multi-view geometry of 1D radial cameras and its application to omnidirectional camera calibration

We study the multi-view geometry of 1D radial cameras. A broad a class of both central and non-central cameras, such as fish-eye and catadioptric cameras, can be reduced to 1D radial cameras under the assumption of known center of radial distortion. For cameras in general configuration, we introduce a quadrifocal tensor that can be computed linearly from 15 or more features seen in four views. From this tensor a metric reconstruction of the 1D cameras as well as the observed features can be obtained. In a second phase this reconstruction can then be used as a calibration object to estimate a non-parametric non-central model for the cameras. We study some degenerate cases, including pure rotation. In the case of a purely rotating camera we obtain a trifocal tensor that can be estimated linearly from 7 points in three views. This allows us to obtain a metric reconstruction of the plane at infinity. Next, we use the plane at infinity as a calibration device to non-parametrically estimate the radial distortion. We demonstrate the results of our approach on real and synthetic images
Illumination Insensitive Model-Based 3D Object Tracking and Texture Refinement

A common approach to model-based tracking is to use a model of the object to predict what will be observed, and then to compare that with real observations. For methods that make use of the object's photometric properties (appearance) in their measurements, illumination inconsistencies between the modeled and actual scene can cause tracking problems. In this paper we address one case: model-based tracking of Lambertian objects under directional light sources. We present an iterative optimization method that uses a Kalman filter to simultaneously refine estimates of the object motion, the illumination, and the model texture. We model the illumination variance between the real and predicted observation using the intensity ratios of corresponding surface points, which we then use to make model-based image predictions consistent with the real lighting. To demonstrate the effectiveness of our method we present experimental results using both synthetic (controlled) and real image sequences.
Multi-View Stereo via Graph Cuts on the Dual of an Adaptive Tetrahedral Mesh

We formulate multi-view 3D shape reconstruction as the computation of a minimum cut on the dual graph of a semi- regular, multi-resolution, tetrahedral mesh. Our method does not assume that the surface lies within a finite band around the visual hull or any other base surface. Instead, it uses photo-consistency to guide the adaptive subdivision of a coarse mesh of the bounding volume. This generates a multi-resolution volumetric mesh that is densely tesselated in the parts likely to contain the unknown surface. The graph-cut on the dual graph of this tetrahedral mesh produces a minimum cut corresponding to a triangulated surface that minimizes a global surface cost functional. Our method makes no assumptions about topology and can recover deep concavities when enough cameras observe them. Our formulation also allows silhouette constraints to be enforced during the graph-cut step to counter its inherent bias for producing minimal surfaces. Local shape refinement via surface deformation is used to recover details in the reconstructed surface. Reconstructions of the Multi- View Stereo Evaluation benchmark datasets and other real datasets show the effectiveness of our method.
Stereo depth map fusion for robot navigation

We present a method to reconstruct indoor environments from stereo image pairs, suitable for the navigation of robots. To enable a robot to navigate solely using visual cues it receives from a stereo camera, the depth information needs to be extracted from the image pairs and combined into a common representation. The initially determined raw depthmaps are fused into a two level heightmap representation which contains a floor and a ceiling height level. To reduce the noise in the height maps we employ a total variation regularized energy functional. With this 2.5D representation of the scene the computational complexity of the energy optimization is reduced by one dimension in contrast to other fusion techniques that work on the full 3D space such as volumetric fusion. While we show only results for indoor environments the approach can be extended to generate heightmaps for outdoor environments.
A Factorization-Based Approach to Articulated Motion Recovery

This paper addresses the subspace properties and the recovery of articulated motion. We point out that the global motion subspace of an articulated object is a combination of a number of intersecting rigid motion subspaces of the parts. Depending on whether a link of two parts is a joint or an axis, the global motion subspace loses one or two in rank for each link. The rank loss results from the intersection between the rigid motion subspaces of linked parts. Furthermore, the intersection is, in fact, the motion subspace of the link. From these observations, we describe the rank constraint of the global motion subspace of an articulated object; we give an algorithm to recover the image motion of a link, either a joint or an axis; and we propose a novel but simple approach, which is based on subspace clustering, to recover articulated shape and motion from a single-view image sequence.
Auto-calibration of multi-projector display walls

By treating projectors as pin-hole cameras, we show it is possible to calibrate the projectors of a casually-aligned, multi-projector display wall using the principles of planar auto-calibration. We also use a pose estimation technique for planar scenes to reconstruct the relative pose of a calibration camera, the projectors and the plane they project on. Together with assumptions about the pose of the camera, we use the reconstruction to automatically compute the projector-display homographies needed to render properly scaled and oriented imagery on the display wall. The main contribution of this paper is thus to provide a fully automated approach to calibrate a multi-projector display wall without the need for fiducials or interaction.