nils-herrmann/clean_scirepeval huggingface dataset:
Pulling Things out of Perspective

The limitations of current state-of-the-art methods for single-view depth estimation and semantic segmentations are closely tied to the property of perspective geometry, that the perceived size of the objects scales inversely with the distance. In this paper, we show that we can use this property to reduce the learning of a pixel-wise depth classifier to a much simpler classifier predicting only the likelihood of a pixel being at an arbitrarily fixed canonical depth. The likelihoods for any other depths can be obtained by applying the same classifier after appropriate image manipulations. Such transformation of the problem to the canonical depth removes the training data bias towards certain depths and the effect of perspective. The approach can be straight-forwardly generalized to multiple semantic classes, improving both depth estimation and semantic segmentation performance by directly targeting the weaknesses of independent approaches. Conditioning the semantic label on the depth provides a way to align the data to their physical scale, allowing to learn a more discriminative classifier. Conditioning depth on the semantic class helps the classifier to distinguish between ambiguities of the otherwise ill-posed problem. We tested our algorithm on the KITTI road scene dataset and NYU2 indoor dataset and obtained obtained results that significantly outperform current state-of-the-art in both single-view depth and semantic segmentation domain.
Calibration of Pan-Tilt-Zoom (PTZ) Cameras and Omni-Directional Cameras

In the first part we discuss the problem of recovering the calibration of a network of pan-tilt-zoom cameras. The intrinsic parameters of each camera over its full range of zoom settings are estimated through a two step procedure. We first determine the intrinsic parameters at the camera's lowest zoom setting very accurately by capturing an extended panorama. The camera intrinsics are then determined at discrete steps in a monotonically increasing zoom sequence that spans the full zoom range of the cameras. Both steps are fully automatic and do not assume any knowledge of the scene structure. We validate our approach by calibrating two different types of pan tilt zoom cameras placed in an outdoor environment. We also show the high-resolution panoramic mosaics built from the images captured during this process. The second section deals with the calibration of omnidirectional cameras. A broad class of both central and non-central cameras, such as fish-eye and catadioptric cameras, can be reduced to 1D radial cameras under the assumption of known center of radial distortion. We study the multi-view geometry of 1D radial cameras. For cameras in general configuration, we introduce a quadrifocal tensor. From this tensor a metric reconstruction of the 1D cameras as well as the observed features can be obtained. In a second phase this reconstruction can then be used as a calibration object to estimate a non-parametric non-central model for the cameras. We study some degenerate cases, including pure rotation. In the case of a purely rotating camera we obtain a trifocal tensor. This allows us to obtain a metric reconstruction of the plane at infinity. Next, we use the plane at infinity as a calibration device to non-parametrically estimate the radial distortion.
Geometric Change Detection in Urban Environments Using Images

We propose a method to detect changes in the geometry of a city using panoramic images captured by a car driving around the city. The proposed method can be used to significantly optimize the process of updating the 3D model of an urban environment that is changing overtime, by restricting this process to only those areas where changes are detected. With this application in mind, we designed our algorithm to specifically detect only structural changes in the environment, ignoring any changes in its appearance, and ignoring also all the changes which are not relevant for update purposes such as cars, people etc. The approach also accounts for the challenges involved in a large scale application of change detection, such as inaccuracies in the input geometry, errors in the geo-location data of the images as well as the limited amount of information due to sparse imagery. We evaluated our approach on a small scale setup using high resolution, densely captured images and a large scale setup covering an entire city using instead the more realistic scenario of low resolution, sparsely captured images. A quantitative evaluation was also conducted for the large scale setup consisting of 14,000 images.
Repetition-based dense single-view reconstruction

This paper presents a novel approach for dense reconstruction from a single-view of a repetitive scene structure. Given an image and its detected repetition regions, we model the shape recovery as the dense pixel correspondences within a single image. The correspondences are represented by an interval map that tells the distance of each pixel to its matched pixels within the single image. In order to obtain dense repetitive structures, we develop a new repetition constraint that penalizes the inconsistency between the repetition intervals of the dynamically corresponding pixel pairs. We deploy a graph-cut to balance between the high-level constraint of geometric repetition and the low-level constraints of photometric consistency and spatial smoothness. We demonstrate the accurate reconstruction of dense 3D repetitive structures through a variety of experiments, which prove the robustness of our approach to outliers such as structure variations, illumination changes, and occlusions.
Visual-hull reconstruction from uncalibrated and unsynchronized video streams

We present an approach for automatic reconstruction of a dynamic event using multiple video cameras recording from different viewpoints. Those cameras do not need to be calibrated or even synchronized. Our approach recovers all the necessary information by analyzing the motion of the silhouettes in the multiple video streams. The first step consists of computing the calibration and synchronization for pairs of cameras. We compute the temporal offset and epipolar geometry using an efficient RANSAC-based algorithm to search for the epipoles as well as for robustness. In the next stage the calibration and synchronization for the complete camera network is recovered and then refined through maximum likelihood estimation. Finally, a visual-hull algorithm is used to the recover the dynamic shape of the observed object. For unsynchronized video streams silhouettes are interpolated to deal with subframe temporal offsets. We demonstrate the validity of our approach by obtaining the calibration, synchronization and 3D reconstruction of a moving person from a set of 4 minute videos recorded from 4 widely separated video cameras.
Two Cameras and a Screen: How to Calibrate Mobile Devices?

We propose a new approach to estimate the geometric extrinsic calibration of all the elements of a smart phone or tablet (such as the screen, the front and the back cameras) by using a planar mirror. By moving a smart phone in front of a single static planar mirror, it is possible to establish correspondences between the images and a pattern displayed on the screen, and therefore estimate the geometric relationship between the non-overlapping cameras with respect to the screen location. The newly proposed setup (static mirror, moving smart phone) enables to both improve the state-of-the-art by working in the minimal case of two images, and improve the accuracy when more images are available. We analyze the minimal case for different calibration scenarios and evaluate the proposed approach on several data. We also show an application of this geometric calibration for specular surface reconstruction, by observing the reflection of a known pattern displayed on the screen.
City-Scale Change Detection in Cadastral 3D Models Using Images

In this paper, we propose a method to detect changes in the geometry of a city using panoramic images captured by a car driving around the city. We designed our approach to account for all the challenges involved in a large scale application of change detection, such as, inaccuracies in the input geometry, errors in the geo-location data of the images, as well as, the limited amount of information due to sparse imagery. We evaluated our approach on an area of 6 square kilometers inside a city, using 3420 images downloaded from Google Street View. These images besides being publicly available, are also a good example of panoramic images captured with a driving vehicle, and hence demonstrating all the possible challenges resulting from such an acquisition. We also quantitatively compared the performance of our approach with respect to a ground truth, as well as to prior work. This evaluation shows that our approach outperforms the current state of the art.
MAV visual SLAM with plane constraint

Bundle adjustment (BA) which produces highly accurate results for visual Simultaneous Localization and Mapping (SLAM) could not be used for Micro-Aerial Vehicles (MAVs) with limited processing power because of its O(N3) complexity. We observed that a consistent ground plane often exists for MAVs flying in both the indoor and outdoor urban environments. Therefore, in this paper, we propose a visual SLAM algorithm that make use of the plane constraint to reduce the complexity of BA. The reduction of complexity is achieved by refining only the current camera pose and most recent map points with BA that minimizes the reprojection errors and perpendicular distances between the most recent map points and the best fit plane with all the pre-existing map points. As a result, our algorithm is approximately constant time since the number of current camera pose and most recent map points remain approximately constant. In addition, the minimization of the perpendicular distances between the plane and map points would enforce consistency between the reconstructed map points and the actual ground plane.
Motion Estimation for Self-Driving Cars with a Generalized Camera

In this paper, we present a visual ego-motion estimation algorithm for a self-driving car equipped with a close-to-market multi-camera system. By modeling the multi-camera system as a generalized camera and applying the non-holonomic motion constraint of a car, we show that this leads to a novel 2-point minimal solution for the generalized essential matrix where the full relative motion including metric scale can be obtained. We provide the analytical solutions for the general case with at least one inter-camera correspondence and a special case with only intra-camera correspondences. We show that up to a maximum of 6 solutions exist for both cases. We identify the existence of degeneracy when the car undergoes straight motion in the special case with only intra-camera correspondences where the scale becomes unobservable and provide a practical alternative solution. Our formulation can be efficiently implemented within RANSAC for robust estimation. We verify the validity of our assumptions on the motion model by comparing our results on a large real-world dataset collected by a car equipped with 4 cameras with minimal overlapping field-of-views against the GPS/INS ground truth.
Vision-Controlled Micro Flying Robots: From System Design to Autonomous Navigation and Mapping in GPS-Denied Environments

Autonomous microhelicopters will soon play a major role in tasks like search and rescue, environment monitoring, security surveillance, and inspection. If they are further realized in small scale, they can also be used in narrow outdoor and indoor environments and represent only a limited risk for people. However, for such operations, navigating based only on global positioning system (GPS) information is not sufficient. Fully autonomous operation in cities or other dense environments requires microhelicopters to fly at low altitudes, where GPS signals are often shadowed, or indoors and to actively explore unknown environments while avoiding collisions and creating maps. This involves a number of challenges on all levels of helicopter design, perception, actuation, control, and navigation, which still have to be solved. The Swarm of Micro Flying Robots (SFLY) project was a European Union-funded project with the goal of creating a swarm of vision-controlled microaerial vehicles (MAVs) capable of autonomous navigation, three-dimensional (3-D) mapping, and optimal surveillance coverage in GPS-denied environments. The SFLY MAVs do not rely on remote control, radio beacons, or motion-capture systems but can fly all by themselves using only a single onboard camera and an inertial measurement unit (IMU). This article describes the technical challenges that have been faced and the results achieved from hardware design and embedded programming to vision-based navigation and mapping, with an overview of how all the modules work and how they have been integrated into the final system. Code, data sets, and videos are publicly available to the robotics community. Experimental results demonstrating three MAVs navigating autonomously in an unknown GPS-denied environment and performing 3-D mapping and optimal surveillance coverage are presented.
Automated valet parking and charging for e-mobility

Automated valet parking services provide great potential to increase the attractiveness of electric vehicles by mitigating their two main current deficiencies: reduced driving ranges and prolonged refueling times. The European research project V-Charge aims at providing this service on designated parking lots using close-to-market sensors only. For this purpose the project developed a prototype capable of performing fully automated navigation in mixed traffic on designated parking lots and GPS-denied parking garages with cameras and ultrasonic sensors only. This paper summarizes the work of the project, comprising advances in network communication and parking space scheduling, multi-camera calibration, semantic mapping concepts, visual localization and motion planning. The project pushed visual localization, environment perception and automated parking to centimetre precision. The developed infrastructure-based camera calibration and semi-supervised semantic mapping concepts greatly reduce maintenance efforts. Results are presented from extensive month-long field tests.
PTZ camera network calibration from moving people in sports broadcasts

In sports broadcasts, networks consisting of pan-tilt-zoom (PTZ) cameras usually exhibit very wide baselines, making standard matching techniques for camera calibration very hard to apply. If, additionally, there is a lack of texture, finding corresponding image regions becomes almost impossible. However, such networks are often set up to observe dynamic scenes on a ground plane. Corresponding image trajectories produced by moving objects need to fulfill specific geometric constraints, which can be leveraged for camera calibration. We present a method which combines image trajectory matching with the self-calibration of rotating and zooming cameras, effectively reducing the remaining degrees of freedom in the matching stage to a 2D similarity transformation. Additionally, lines on the ground plane are used to improve the calibration. In the end, all extrinsic and intrinsic camera parameters are refined in a final bundle adjustment. The proposed algorithm was evaluated both qualitatively and quantitatively on four different soccer sequences.
PX4: A node-based multithreaded open source robotics framework for deeply embedded platforms

We present a novel, deeply embedded robotics middleware and programming environment. It uses a multithreaded, publish-subscribe design pattern and provides a Unix-like software interface for micro controller applications. We improve over the state of the art in deeply embedded open source systems by providing a modular and standards-oriented platform. Our system architecture is centered around a publish-subscribe object request broker on top of a POSIX application programming interface. This allows to reuse common Unix knowledge and experience, including a bash-like shell. We demonstrate with a vertical takeoff and landing (VTOL) use case that the system modularity is well suited for novel and experimental vehicle platforms. We also show how the system architecture allows a direct interface to ROS and to run individual processes either as native ROS nodes on Linux or nodes on the micro controller, maximizing interoperability. Our microcontroller-based execution environment has substantially lower latency and better hardware connectivity than a typical Robotics Linux system and is therefore well suited for fast, high rate control tasks.
Obstacle detection for self-driving cars using only monocular cameras and wheel odometry

Mapping the environment is crucial to enable path planning and obstacle avoidance for self-driving vehicles and other robots. In this paper, we concentrate on ground-based vehicles and present an approach which extracts static obstacles from depth maps computed out of multiple consecutive images. In contrast to existing approaches, our system does not require accurate visual inertial odometry estimation but solely relies on the readily available wheel odometry. To handle the resulting higher pose uncertainty, our system fuses obstacle detections over time and between cameras to estimate the free and occupied space around the vehicle. Using monocular fisheye cameras, we are able to cover a wider field of view and detect obstacles closer to the car, which are often not within the standard field of view of a classical binocular stereo camera setup. Our quantitative analysis shows that our system is accurate enough for navigation purposes of self-driving cars and runs in real-time.
A simple and efficient rectification method for general motion

In this paper a new rectification method is proposed. The method is both simple and efficient and can deal with all possible camera motions. A minimal image size without any pixel loss is guaranteed. The only required information is the oriented fundamental matrix. The whole rectification process is carried out directly in the images. The idea consists of using a polar parametrization of the image around the epipole. The transfer between the images is obtained through the fundamental matrix. The proposed method has important advantages compared to the traditional rectification schemes. In some cases these approaches yield very large images or can not rectify at all. Even the recently proposed cylindrical rectification method can encounter problems in some cases. These problems are mainly due to the fact that the matching ambiguity is not reduced to half epipolar lines. Although this last method is more complex than the one proposed in this paper the resulting images are in general larger. The performance of the new approach is illustrated with some results on real image pairs.
Visual Hull Construction in the Presence of Partial Occlusion

In this paper, we propose a visual hull algorithm, which guarantees a correct construction even in the presence of partial occlusion, while "correct" here means that the real shape is located inside the visual hull. The algorithm is based on a new idea of the "extended silhouette", which requires the silhouette from background subtraction and the "occlusion mask" of the same view. In order to prepare the occlusion mask, we also propose a novel concept of "effective boundary" of moving foreground objects in a video obtained from a static camera. The accumulation of the effective boundary through time automatically gives robust occluder boundaries. We theoretically prove that our algorithm deterministically computes the tightest, correct visual hull in the presence of occlusion. Both synthetic and real examples are given as a demonstration of the correctness of the algorithm. Finally we analyze that this new algorithm is still within the time complexity of the traditional method.
Real-time stereo matching failure prediction and resolution using orthogonal stereo setups

Estimating the depth from two images with a baseline has a well-known regular problem: When a line is parallel to the epipolar geometry it is not possible to estimate the depth from pixels on this line. Moreover, the classic measure for the certainty of the depth estimate fails as well: The matching score between the template and any pixel on the epipolar line is perfect. This results for common scenes in incorrect matches with very high confidence, some even resistant to left-right image checks. It is straightforward to try to address this by adding a second stereo head in a perpendicular direction. However, it is nontrivial to identify the failure and fuse the two depth maps in a real-time system. A simple weighted average will alleviate the problem but still result in a very large error in the depth map. Our contributions are: 1) We derive a model to predict the failure of stereo by leveraging the matching scores and 2) we propose a combined cost function to fuse two depth maps from orthogonal stereo heads using the failure prediction, matching score and consistency. We show the resulting system in real-time operation on a low-latency system in indoor, urban and natural environments.
A new minimal solution to the relative pose of a calibrated stereo camera with small field of view overlap

In this paper we present a new minimal solver for the relative pose of a calibrated stereo camera (i.e. a pair of rigidly mounted cameras). Our method is based on the fact that a feature visible in all four images (two image pairs acquired at two points in time) constrains the relative pose of the second stereo camera to lie on a sphere around this feature, which has a known, triangulated position in the first stereo camera coordinate frame. This constraint leaves three degrees of freedom; two for the location of the second camera on the sphere, and the third for the rotation in the respective tangent plane. We use three 2D correspondences, in particular two correspondences from the left (or right) camera and one correspondence from the other camera, to solve for these three remaining degrees of freedom. This approach is amenable to stereo cameras having a small overlap in their views. We present an efficient solution for this novel relative pose problem, describe the incorporation of our proposed solver into the RANSAC framework, evaluate its performance given noise and outliers, and demonstrate its use in a real-time structure from motion system.
Do ambiguous reconstructions always give ambiguous images?

In many cases self-calibration is not able to yield a unique solution for the 3D reconstruction of a scene. This is due to the occurrence of critical motion sequences. If this is the case, an ambiguity is left on the reconstruction. In this paper we derive under which conditions correct novel views can be generated from ambiguous reconstructions. The problem is first approached from a theoretical point of view: It is proven that novel views are correct as long as the inclusion of the next view the sequence yields the same ambiguity on the reconstruction. The problem is therefore much related to the problem of critical motion sequences since the virtual camera can be arbitrarily moved within the smallest critical motion set that the recovered camera motion without distortions becoming visible. Based on these results a practical measure for the expected ambiguity on a novel view generated from the recovered structure and motion is derived. As an application a viewer was built that indicates if a specific novel view can be trusted or not by altering the background color.