nils-herrmann/clean_scirepeval huggingface dataset:
Automatic 3D reconstruction of manifold meshes via delaunay triangulation and mesh sweeping

In this paper we propose a new approach to incrementally initialize a manifold surface for automatic 3D reconstruction from images. More precisely we focus on the automatic initialization of a 3D mesh as close as possible to the final solution; indeed many approaches require a good initial solution for further refinement via multi-view stereo techniques. Our novel algorithm automatically estimates an initial manifold mesh for surface evolving multi-view stereo algorithms, where the manifold property needs to be enforced. It bootstraps from 3D points extracted via Structure from Motion, then iterates between a state-of-the-art manifold reconstruction step and a novel mesh sweeping algorithm that looks for new 3D points in the neighborhood of the reconstructed manifold to be added in the manifold reconstruction. The experimental results show quantitatively that the mesh sweeping improves the resolution and the accuracy of the manifold reconstruction, allowing a better convergence of state-of-the-art surface evolution multi-view stereo algorithms.
A factorization-based approach to articulated motion recovery

This paper addresses the subspace properties and the recovery of articulated motion. We point out that the global motion subspace of an articulated object is a combination of a number of intersecting rigid motion subspaces of the parts. Depending on whether a link of two parts is a joint or an axis, the global motion subspace loses one or two in rank for each link. The rank loss results from the intersection between the rigid motion subspaces of linked parts. Furthermore, the intersection is, in fact, the motion subspace of the link. From these observations, we describe the rank constraint of the global motion subspace of an articulated object; we give an algorithm to recover the image motion of a link, either a joint or an axis; and we propose a novel but simple approach, which is based on subspace clustering, to recover articulated shape and motion from a single-view image sequence.
Temporally Consistent Reconstruction from Multiple Video Streams Using Enhanced Belief Propagation

We present an approach for 3D reconstruction from multiple video streams taken by static, synchronized and calibrated cameras that is capable of enforcing temporal consistency on the reconstruction of successive frames. Our goal is to improve the quality of the reconstruction by finding corresponding pixels in subsequent frames of the same camera using optical flow, and also to at least maintain the quality of the single time-frame reconstruction when these correspondences are wrong or cannot be found. This allows us to process scenes with fast motion, occlusions and self- occlusions where optical flow fails for large numbers of pixels. To this end, we modify the belief propagation algorithm to operate on a 3D graph that includes both spatial and temporal neighbors and to be able to discard messages from outlying neighbors. We also propose methods for introducing a bias and for suppressing noise typically observed in uniform regions. The bias encapsulates information about the background and aids in achieving a temporally consistent reconstruction and in the mitigation of occlusion related errors. We present results on publicly available real video sequences. We also present quantitative comparisons with results obtained by other researchers.
Real-World Normal Map Capture for Nearly Flat Reflective Surfaces

Although specular objects have gained interest in recent years, virtually no approaches exist for marker less reconstruction of reflective scenes in the wild. In this work, we present a practical approach to capturing normal maps in real-world scenes using video only. We focus on nearly planar surfaces such as windows, facades from glass or metal, or frames, screens and other indoor objects and show how normal maps of these can be obtained without the use of an artificial calibration object. Rather, we track the reflections of real-world straight lines, while moving with a hand-held or vehicle-mounted camera in front of the object. In contrast to error-prone local edge tracking, we obtain the reflections by a robust, global segmentation technique of an ortho-rectified 3D video cube that also naturally allows efficient user interaction. Then, at each point of the reflective surface, the resulting 2D-curve to 3D-line correspondence provides a novel quadratic constraint on the local surface normal. This allows to globally solve for the shape by integrability and smoothness constraints and easily supports the usage of multiple lines. We demonstrate the technique on several objects and facades.
Multi-view reconstruction using photo-consistency and exact silhouette constraints: a maximum-flow formulation

This paper describes a novel approach for reconstructing a closed continuous surface of an object from multiple calibrated color images and silhouettes. Any accurate reconstruction must satisfy (1) photo-consistency and (2) silhouette consistency constraints. Most existing techniques treat these cues identically in optimization frameworks where silhouette constraints are traded off against photo-consistency and smoothness priors. Our approach strictly enforces silhouette constraints, while optimizing photo-consistency and smoothness in a global graph-cut framework. We transform the reconstruction problem into computing max-flow/min-cut in a geometric graph, where any cut corresponds to a surface satisfying exact silhouette constraints (its silhouettes should exactly coincide with those of the visual hull); a minimum cut is the most photo-consistent surface amongst them. Our graph-cut formulation is based on the rim mesh, (the combinatorial arrangement of rims or contour generators from many views) which can be computed directly from the silhouettes. Unlike other methods, our approach enforces silhouette constraints without introducing a bias near the visual hull boundary and also recovers the rim curves. Results are presented for synthetic and real datasets.
Real-time photo-realistic 3D mapping for micro aerial vehicles

We present a real-time photo-realistic 3D mapping framework for micro aerial vehicles (MAVs). RGBD images are generated from either stereo or structured light cameras, and fed into the processing pipeline. A visual odometry algorithm runs on-board the MAV. We improve the computational performance of the visual odometry by using the IMU readings to establish a 1-point RANSAC instead of using the standard 3-point RANSAC to estimate the relative motion between consecutive frames. We use local bundle adjustment to refine the pose estimates. At the same time, the MAV builds a 3D occupancy grid from range data, and transmits this grid together with images and pose estimates over a wireless network to a ground station. We propose a view-dependent projective texture mapping method that is used by the ground station to incrementally build a 3D textured occupancy grid over time. This map is both geometrically accurate and photo-realistic; the map provides real-time visual updates on the ground to a remote operator, and is used for path planning as well.
Efficient visual exploration and coverage with a micro aerial vehicle in unknown environments

In this paper, we propose a novel and computationally efficient algorithm for simultaneous exploration and coverage with a vision-guided micro aerial vehicle (MAV) in unknown environments. This algorithm continually plans a path that allows the MAV to fulfil two objectives at the same time while avoiding obstacles: observe as much unexplored space as possible, and observe as much of the surface of the environment as possible given viewing angle and distance constraints. The former and latter objectives are known as the exploration and coverage problems respectively. Our algorithm is particularly useful for automated 3D reconstruction at the street level and in indoor environments where obstacles are omnipresent. By solving the exploration problem, we maximize the size of the reconstructed model. By solving the coverage problem, we maximize the completeness of the model. Our algorithm leverages the state lattice concept such that the planned path adheres to specified motion constraints. Furthermore, our algorithm is computationally efficient and able to run on-board the MAV in real-time. We assume that the MAV is equipped with a forward-looking depth-sensing camera in the form of either a stereo camera or RGB-D camera. We use simulation experiments to validate our algorithm. In addition, we show that our algorithm achieves a significantly higher level of coverage as compared to an exploration-only approach while still allowing the MAV to fully explore the environment.
Indoor Scan2BIM: Building information models of house interiors

We present a system to generate building information models (BIMs) of house interiors from 3D scans. The strength of our approach is its simplicity and low runtime which allows for mobile processing applications. We consider scans of single floor, Manhattan-like indoor scenes for which our method creates metric room layouts by detecting walls and performing a subsequent reasoning about their neighborhood relations. The output of our method is a 3D BIM with hierarchical semantic annotations for individual rooms being refined by walls, ceilings, floors and doors. A variety of experiments demonstrate the effectiveness of our approach. Our reconstruction results compare well to other state-of-art methods in both reconstruction quality as well as runtime.
3D model search and pose estimation from single images using VIP features

This paper describes a method to efficiently search for 3D models in a city-scale database and to compute the camera poses from single query images. The proposed method matches SIFT features (from a single image) to viewpoint invariant patches (VIP) from a 3D model by warping the SIFT features approximately into the orthographic frame of the VIP features. This significantly increases the number of feature correspondences which results in a reliable and robust pose estimation. We also present a 3D model search tool that uses a visual word based search scheme to efficiently retrieve 3D models from large databases using individual query images. Together the 3D model search and the pose estimation represent a highly scalable and efficient city-scale localization system. The performance of the 3D model search and pose estimation is demonstrated on urban image data.
An overview of recent progress in volumetric semantic 3D reconstruction

This paper gives an overview of a recently proposed method of solving dense 3D reconstruction and semantic segmentation from multiple input images in a joint fashion, i.e. as semantic 3D reconstruction. The formulation is cast as a volumetric fusion of depth maps and pixel-wise semantic classification scores. By posing the two problems as a joint optimization problem, both of the tasks can benefit from the other task's information. This leads to formulations which can reconstruct hidden unobserved surfaces. We give an overview of several papers which describe different ways of modeling the data term from the input data and also works which introduce object shape priors to the formulation. We present the basic convex multi-label formulation on which the method builds and also discuss the relation to other reconstruction algorithms which extract semantically annotated 3D models from images.
From Point Clouds to Mesh Using Regression

Surface reconstruction from a point cloud is a standard subproblem in many algorithms for dense 3D reconstruction from RGB images or depth maps. Methods, performing only local operations in the vicinity of individual points, are very fast, but reconstructed models typically contain lots of holes. On the other hand, regularized volumetric approaches, formulated as a global optimization, are typically too slow for real-time interactive applications. We propose to use a regression forest based method, which predicts the projection of a grid point to the surface, depending on the spatial configuration of point density in the grid point neighborhood. We designed a suitable feature vector and efficient oct-tree based GPU evaluation, capable of predicting surface of high resolution 3D models in milliseconds. Our method learns and predicts surfaces from an observed point cloud sparser than the evaluation grid, and therefore effectively acts as a regularizer.
Robust Radiometric Calibration and Vignetting Correction

In many computer vision systems, it is assumed that the image brightness of a point directly reflects the scene radiance of the point. However, the assumption does not hold in most cases due to nonlinear camera response function, exposure changes, and vignetting. The effects of these factors are most visible in image mosaics and textures of 3D models where colors look inconsistent and notable boundaries exist. In this paper, we propose a full radiometric calibration algorithm that includes robust estimation of the radiometric response function, exposures, and vignetting. By decoupling the effect of vignetting from the response function estimation, we approach each process in a manner that is robust to noise and outliers. We verify our algorithm with both synthetic and real data, which shows significant improvement compared to existing methods. We apply our estimation results to radiometrically align images for seamless mosaics and 3D model textures. We also use our method to create high dynamic range (HDR) mosaics that are more representative of the scene than normal mosaics.
Automatic Kinematic Chain Building from Feature Trajectories of Articulated Objects

We investigate the problem of learning the structure of an articulated object, i.e. its kinematic chain, from feature trajectories under affine projections. We demonstrate this possibility by proposing an algorithm which first segments the trajectories by local sampling and spectral clustering, then builds the kinematic chain as a minimum spanning tree of a graph constructed from the segmented motion subspaces. We test our method in challenging data sets and demonstrate the ability to automatically build the kinematic chain of an articulated object from feature trajectories. The algorithm also works when there are multiple articulated objects in the scene. Furthermore, we take into account non-rigid articulated parts that exist in human motions. We believe this advance will have impact on articulated object tracking and dynamical structure from motion.
Do It Yourself Hyperspectral Imaging with Everyday Digital Cameras

Capturing hyperspectral images requires expensive and specialized hardware that is not readily accessible to most users. Digital cameras, on the other hand, are significantly cheaper in comparison and can be easily purchased and used. In this paper, we present a framework for reconstructing hyperspectral images by using multiple consumer-level digital cameras. Our approach works by exploiting the different spectral sensitivities of different camera sensors. In particular, due to the differences in spectral sensitivities of the cameras, different cameras yield different RGB measurements for the same spectral signal. We introduce an algorithm that is able to combine and convert these different RGB measurements into a single hyperspectral image for both indoor and outdoor scenes. This camera-based approach allows hyperspectral imaging at a fraction of the cost of most existing hyperspectral hardware. We validate the accuracy of our reconstruction against ground truth hyperspectral images (using both synthetic and real cases) and show its usage on relighting applications.
Design and implementation of an unmanned tail-sitter

We present the design and implementation of a small Vertical-Take-Off-and-Landing (VTOL) aircraft. The vehicle requires minimal additional components to achieve the hover capability and is thus very efficient in forward flight. We improve over the state of the art by using a single controller in all flight modes without using blending between hover condition and fixed wing controllers or gain scheduling. We present a compact airflow estimation model for VTOL airframes which rely on the slipstream across control surfaces for hover attitude control. Furthermore we show attitude and position control results in simulation. Finally we show outdoor flight experiments validating our simulation results.
PIXHAWK: A system for autonomous flight using onboard computer vision

We provide a novel hardware and software system for micro air vehicles (MAV) that allows high-speed, low-latency onboard image processing. It uses up to four cameras in parallel on a miniature rotary wing platform. The MAV navigates based on onboard processed computer vision in GPS-denied in- and outdoor environments. It can process in parallel images and inertial measurement information from multiple cameras for multiple purposes (localization, pattern recognition, obstacle avoidance) by distributing the images on a central, low-latency image hub. Furthermore the system can utilize low-bandwith radio links for communication and is designed and optimized to scale to swarm use. Experimental results show successful flight with a range of onboard computer vision algorithms, including localization, obstacle avoidance and pattern recognition.
SGM-Nets: Semi-Global Matching with Neural Networks

This paper deals with deep neural networks for predicting accurate dense disparity map with Semi-global matching (SGM). SGM is a widely used regularization method for real scenes because of its high accuracy and fast computation speed. Even though SGM can obtain accurate results, tuning of SGMs penalty-parameters, which control a smoothness and discontinuity of a disparity map, is uneasy and empirical methods have been proposed. We propose a learning based penalties estimation method, which we call SGM-Nets that consist of Convolutional Neural Networks. A small image patch and its position are input into SGMNets to predict the penalties for the 3D object structures. In order to train the networks, we introduce a novel loss function which is able to use sparsely annotated disparity maps such as captured by a LiDAR sensor in real environments. Moreover, we propose a novel SGM parameterization, which deploys different penalties depending on either positive or negative disparity changes in order to represent the object structures more discriminatively. Our SGM-Nets outperformed state of the art accuracy on KITTI benchmark datasets.
Absolute scale in structure from motion from a single vehicle mounted camera by exploiting nonholonomic constraints

In structure-from-motion with a single camera it is well known that the scene can be only recovered up to a scale. In order to compute the absolute scale, one needs to know the baseline of the camera motion or the dimension of at least one element in the scene. In this paper, we show that there exists a class of structure-from-motion problems where it is possible to compute the absolute scale completely automatically without using this knowledge, that is, when the camera is mounted on wheeled vehicles (e.g. cars, bikes, or mobile robots). The construction of these vehicles puts interesting constraints on the camera motion, which are known as “nonholonomic constraints”. The interesting case is when the camera has an offset to the vehicle's center of motion. We show that by just knowing this offset, the absolute scale can be computed with a good accuracy when the vehicle turns. We give a mathematical derivation and provide experimental results on both simulated and real data over a large image dataset collected during a 3 Km path. To our knowledge this is the first time nonholonomic constraints of wheeled vehicles are used to estimate the absolute scale. We believe that the proposed method can be useful in those research areas involving visual odometry and mapping with vehicle mounted cameras.
Hyperpoints and Fine Vocabularies for Large-Scale Location Recognition

Structure-based localization is the task of finding the absolute pose of a given query image w.r.t. a pre-computed 3D model. While this is almost trivial at small scale, special care must be taken as the size of the 3D model grows, because straight-forward descriptor matching becomes ineffective due to the large memory footprint of the model, as well as the strictness of the ratio test in 3D. Recently, several authors have tried to overcome these problems, either by a smart compression of the 3D model or by clever sampling strategies for geometric verification. Here we explore an orthogonal strategy, which uses all the 3D points and standard sampling, but performs feature matching implicitly, by quantization into a fine vocabulary. We show that although this matching is ambiguous and gives rise to 3D hyperpoints when matching each 2D query feature in isolation, a simple voting strategy, which enforces the fact that the selected 3D points shall be co-visible, can reliably find a locally unique 2D-3D point assignment. Experiments on two large-scale datasets demonstrate that our method achieves state-of-the-art performance, while the memory footprint is greatly reduced, since only visual word labels but no 3D point descriptors need to be stored.
Stereo Reconstruction of Building Interiors with a Vertical Structure Prior

Image-based computation of a 3D map for an indoor environment is a very challenging task, but also a useful step for vision-based navigation and path planning for autonomous systems, and for efficient visualization of interior spaces. Since computational stereo is a highly ill-posed problem for the typically weakly textured, specular, and even sometimes transparent indoor environments, one has to incorporate very strong prior assumptions on the observed geometry. A natural assumption for building interiors is that open space is bounded (i) by parallel ground and ceiling planes, and (ii) by vertical (not necessarily orthogonal) wall elements. We employ this assumption as a strong prior in dense depth estimation from stereo images. The additional assumption of smooth vertical elements allows our approach to fill in plausible extensions of e.g. walls in case of (non-vertical) occlusions. It is also possible to explicitly detect non-vertical regions in the images, and to revert to more general stereo methods only in those areas. We demonstrate our method on several challenging stereo images of office environments.
Reactive avoidance using embedded stereo vision for MAV flight

High speed, low latency obstacle avoidance is essential for enabling Micro Aerial Vehicles (MAVs) to function in cluttered and dynamic environments. While other systems exist that do high-level mapping and 3D path planning for obstacle avoidance, most of these systems require high-powered CPUs on-board or off-board control from a ground station. We present a novel entirely on-board approach, leveraging a light-weight low power stereo vision system on FPGA. Our approach runs at a frame rate of 60 frames a second on VGA-sized images and minimizes latency between image acquisition and performing reactive maneuvers, allowing MAVs to fly more safely and robustly in complex environments. We also suggest our system as a light-weight safety layer for systems undertaking more complex tasks, like mapping the environment. Finally, we show our algorithm implemented on a lightweight, very computationally constrained platform, and demonstrate obstacle avoidance in a variety of environments.
Leveraging Topographic Maps for Image to Terrain Alignment

With the wide-spread availability of photographic and cartographic data, it becomes desirable to be able to geo-localize any picture in the world. Existing approaches have so far shown impressive results, but they are still lacking in either precision or applicability. In the present work, we explore as an additional cue, semantic image labeling coupled with topographic maps. As an intermediate step towards the ultimate goal of universal geo-localiztion, we show that these cues are suitable for estimating the viewing direction of a terrestrial image, given the image's location.