# Stereo Vision Based Sensory Substitution for the Visually Impaired
## 3 Stereo Based Reconstruction and Segmentation in Outdoor Environments
### 3D Reconstruction  
The system relies on passive stereo vision, so, in order to perform the incremental reconstruction we need to employ three operations for each frame: (1) Estimation of depth data from stereo; (2) estimation of camera motion with respect to the previous frame; (3) fusion of current frame data into the reconstruction. In the following we detail these parts of the reconstruction process. **(1) Estimation of depth:** In order to work properly, the stereo vision system requires a set of intrinsic and extrinsic parameters which can be used to obtain a correct rectification of the acquired left and right image. These parameters are computed in an off-line calibration process, while rectification, which ensures distortion removal and stereo alignment, is an on-line process. Once these two processes are completed, the most important task is solving the correspondence problem between the left and right image. We use the stereo correspondence method by [PERSON] [33] to compute a dense disparity map. We employ a parallelization of the original algorithm in order to produce a dense disparity map, without the need for a global optimization, and still stay within the real-time operation constraints. The depth values, \(z_{ij}\), are computed based on the disparity values using the equation \(z_{ij}=bf/d_{ij}\), where \(d_{ij}\) is the disparity of pixel \((i,j)\) in the image, \(b\) is the stereo baseline and \(f\) is the focal length of the camera. **(2) Camera motion estimation:** In order to perform the registration of the previous measurements with the camera coordinate system of the current frame, we need to estimate the camera pose in the current frame. This is a complex task generated by a combination of the motion of the head mounted stereo camera and body motion. The recovery of this 6-DOF motion needs to comply with the high update rate required by the application. The performance evaluation of any camera motion estimation technique needs to include: high accuracy in relation with ground truth, real-time behavior, robustness to motion-blur induced by body motion. In the end we selected the camera motion estimation technique  
Figure 3: Comparison between our reconstruction and single frame based reconstruction. (**Top-left**) RGB image from the left camera. (**Top-right**) Disparity map computed with the approach of [33]. (**Bottom-left**) 3D reconstruction constrained within 5 m in front of the camera, based solely on the disparity map of the current frame. Errors in the estimation of the stereo correspondence lead to the presence of false structures in the reconstruction. (**Bottom-right**) Result of our incremental reconstruction showing the accumulated stable points (confidence above a predefined threshold). proposed by [PERSON] in the _libviso2_ framework [34]. This technique is frequently used in automotive and robotics applications. For applications with head-mounted cameras, the internal parameters required adaptation. For parameter tuning, a 3D virtual environment was designed. This allowed the generation of benchmark testing stereo sequences for human assistive devices in different outdoor illumination conditions and recreate some special situations of real life environments. The tests revealed valuable data to be processed for computing the efficiency of the algorithms while acknowledging the worst case scenarios. In addition, tests revealed the highest attitude errors to be equal to \(\pm\)10 degrees for orientation and \(\pm\)0.1 m for position. These errors establish the confidence interval that maintain the system reliable, as showed in Section 4. Based on the collected data, the internal parameters of the libviso framework were tuned for the best performance of our stereo camera's motion estimation. In the current design the IMU is used to improve ground detection. Future refinements will include the visual-inertial fusion for motion estimation.