Query: Camera network calibration

title: A Model for Auto Calibration of Stereo Camera and Lidar Based on Neural Networks
abstract: The methodologies of cameras and Lidar fusion is widely researched over recent years, and the fusion of both sensors can provide us essential information from surrounding environment. However, it is not a straightforward task to utilize two sensor data due to the calibration challenges that align Lidar point cloud with the camera image pixels. One could calibrate both camera and Lidar before using, but in reality, errors are cumulated along with the time because of vibration or distortion of the platforms that sensors settled on. In this paper, we implemented a methodology to auto calibrate stereo cameras with a Lidar sensor using a deep neural network. The neural network takes two depth maps generated from stereo cameras and Lidar sensor as the input. Next, two separate convolutional neural network will extract features from the inputs. The features will be fused and fed to a global regression net, which will infer the calibration errors between two depth maps. Our evaluation showed that auto calibration network managed to fix the errors with different error ranges and a iterative calibration method is used to fix the cumulated error gradualy.

title: Neural Geometric Parser for Single Image Camera Calibration
abstract: We propose a neural geometric parser learning single image camera calibration for man made scenes. Unlike previous neural approaches that rely only on semantic cues obtained from neural networks, our approach considers both semantic and geometric cues, resulting in significant accuracy improvement. The proposed framework consists of two networks. Using line segments of an image as geometric cues, the first network estimates the zenith vanishing point and generates several candidates consisting of the camera rotation and focal length. The second network evaluates each candidate based on the given image and the geometric cues, where prior knowledge of man made scenes is used for the evaluation. With the supervision of datasets consisting of the horizontal line and focal length of the images, our networks can be trained to estimate the same camera parameters. Based on the Manhattan world assumption, we can further estimate the camera rotation and focal length in a weakly supervised manner. The experimental results reveal that the performance of our neural approach is significantly higher than that of existing state of the art camera calibration techniques for single images of indoor and outdoor scenes.

title: PST900: RGB Thermal Calibration, Dataset and Segmentation Network
abstract: In this work we propose long wave infrared (LWIR) imagery as a viable supporting modality for semantic segmentation using learning based techniques. We first address the problem of RGB thermal camera calibration by proposing a passive calibration target and procedure that is both portable and easy to use. Second, we present PST900, a dataset of 894 synchronized and calibrated RGB and Thermal image pairs with per pixel human annotations across four distinct classes from the DARPA Subterranean Challenge. Lastly, we propose a CNN architecture for fast semantic segmentation that combines both RGB and Thermal imagery in a way that leverages RGB imagery independently. We compare our method against the state of the art and show that our method outperforms them in our dataset.

title: Camera to Robot Pose Estimation from a Single Image
abstract: We present an approach for estimating the pose of an external camera with respect to a robot using a single RGB image of the robot. The image is processed by a deep neural network to detect 2D projections of keypoints (such as joints) associated with the robot. The network is trained entirely on simulated data using domain randomization to bridge the reality gap. Perspective n point (PnP) is then used to recover the camera extrinsics, assuming that the camera intrinsics and joint configuration of the robot manipulator are known. Unlike classic hand eye calibration systems, our method does not require an off line calibration step. Rather, it is capable of computing the camera extrinsics from a single frame, thus opening the possibility of on line calibration. We show experimental results for three different robots and camera sensors, demonstrating that our approach is able to achieve accuracy with a single frame that is comparable to that of classic off line hand eye calibration using multiple frames. With additional frames from a static pose, accuracy improves even further. Code, datasets, and pretrained models for three widely used robot manipulators are made

title: Literature Survey on Multi Camera System and Its Application
abstract: A multi camera system combines features from different cameras to exploit a scene of an event to increase the output image quality. The combination of two or more cameras requires prior settings in terms of calibration and architecture. Therefore, this paper surveys the available literature in terms of multi camera systems' physical arrangements, calibrations, algorithms, and their advantages and disadvantages. We also survey the recent developments and advancements in four areas of multi camera system applications, which are surveillance, sports, education, and mobile phones. In the surveillance system, the combination of multiple heterogeneous cameras and the discovery of Pan Tilt Zoom (PTZ) and smart cameras have brought tremendous achievements in the area of multi camera control and coordination. Different approaches have been proposed to facilitate effective collaboration and monitoring among the camera network. Furthermore, the application of multi cameras in sports has made the games more interesting in the aspect of analyses and transparency. The application of the multi camera system in education has taken education beyond the four walls of the class. The method of teaching, student attendance enrollment, determination of students' attention, teacher and student assessment can now be determined with ease, and all forms of proxy and manipulation in education can be reduced by using a multi camera system. Besides, the number of cameras featuring on smartphones is gaining noticeable recognition. However, most of these cameras serve different purposes, from zooming, telephoto, and wider Field of View (FOV) Therefore, future smartphones should be expecting more cameras or the development would be in a different direction.

title: DeepPTZ: Deep Self Calibration for PTZ Cameras
abstract: Rotating and zooming cameras, also called PTZ (Pan Tilt Zoom) cameras, are widely used in modern surveillance systems. While their zooming ability allows acquiring detailed images of the scene, it also makes their calibration more challenging since any zooming action results in a modification of their intrinsic parameters. Therefore, such camera calibration has to be computed online; this process is called self calibration. In this paper, given an image pair captured by a PTZ camera, we propose a deep learning based approach to automatically estimate the focal length and distortion parameters of both images as well as the rotation angles between them. The proposed approach relies on a dual Siamese structure, imposing bidirectional constraints. The proposed network is trained on a large scale dataset automatically generated from a set of panoramas. Empirically, we demonstrate that our proposed approach achieves competitive performance with respect to both deep learning based and traditional state of the art methods. Our code and model will be publicly available at https:/github.com/ChaoningZhang/DeepPTZ.

title: Learning Camera Miscalibration Detection
abstract: Self diagnosis and self repair are some of the key challenges in deploying robotic platforms for long term real world applications. One of the issues that can occur to a robot is miscalibration of its sensors due to aging, environmental transients, or external disturbances. Precise calibration lies at the core of a variety of applications, due to the need to accurately perceive the world. However, while a lot of work has focused on calibrating the sensors, not much has been done towards identifying when a sensor needs to be recalibrated. This paper focuses on a data driven approach to learn the detection of miscalibration in vision sensors, specifically RGB cameras. Our contributions include a proposed miscalibration metric for RGB cameras and a novel semi synthetic dataset generation pipeline based on this metric. Additionally, by training a deep convolutional neural network, we demonstrate the effectiveness of our pipeline to identify whether a recalibration of the camera's intrinsic parameters is required or not. The code is available at http:/github.com/ethz asl/camera_miscalib_detection.

title: Indirect Object to Robot Pose Estimation from an External Monocular RGB Camera
abstract: We present a robotic grasping system that uses a single external monocular RGB camera as input. The object to robot pose is computed indirectly by combining the output of two neural networks: one that estimates the object to camera pose, and another that estimates the robot to camera pose. Both networks are trained entirely on synthetic data, relying on domain randomization to bridge the sim to real gap. Because the latter network performs online camera calibration, the camera can be moved freely during execution without affecting the quality of the grasp. Experimental results analyze the effect of camera placement, image resolution, and pose refinement in the context of grasping several household objects. We also present results on a new set of 28 textured household toy grocery objects, which have been selected to be accessible to other researchers. To aid reproducibility of the research, we offer 3D scanned textured models, along with pre trained weights for pose

title: Optimized calibration method for ultra field dual bands cameras based on thermal radiation checkerboard
abstract: Camera calibration is a vital and important work for machine vision. In the paper, an optimized calibration method for ultra field dual bands cameras is proposed, which is based on the designed thermal radiation checkerboard. The optimizations are achieved in subpixel location, center location and mapping relationship, respectively. Specifically, the subpixel corner is located by performing the designed variable quadrilateral template on edge gradients and calculating their minimum orthogonal value. Then, the center location is accomplished through Gaussian fitting, in which the bottom point is taken as the potential center in iterations. Moreover, the RBF network is employed to replace the polynomial fitting and fit the mapping relationship between the camera coordinates and sensor coordinates. Thus, the accuracy of the mapping is increased, which contributes to the optimization of the intrinsic and extrinsic parameters. The experiments results show that the average root mean square error of the reprojection is reduced by nearly 50% The calibrated parameters are accurate and have physical significance in the proposed calibration scheme.

title: A 3D Deep Learning based Augmented Reality Calibration Method for Robotic Environments using Depth Sensor Data
abstract: Augmented Reality and mobile robots are gaining increased attention within industries due to the high potential to make processes cost and time efficient. To facilitate augmented reality, a calibration between the Augmented Reality device and the environment is necessary. This is a challenge when dealing with mobile robots due to the mobility of all entities making the environment dynamic. On this account, we propose a novel approach to calibrate Augmented Reality devices using 3D depth sensor data. We use the depth camera of a Head Mounted Augmented Reality Device, the Microsoft Hololens, for deep learning based calibration. Therefore, we modified a neural network based on the recently published VoteNet architecture which works directly on raw point cloud input observed by the Hololens. We achieve satisfying results and eliminate external tools like markers, thus enabling a more intuitive and flexible work flow for Augmented Reality integration. The results are adaptable to work with all depth cameras and are promising for further research. Furthermore, we introduce an open source 3D point cloud labeling tool, which is to our knowledge the first open source tool for labeling raw point cloud data.