# Stereo Vision Based Sensory Substitution for the Visually Impaired
## 1 Introduction  
Scene understanding is an important topic in a large variety of computer vision based applications, from autonomous driving to assistive systems. Humans generally rely on their visual sense to perform this task. Mobile robots, autonomous cars and visually impaired persons require complex systems and processes based on artificial vision to infer information about the surrounding environment in order to accomplish their tasks or goals that involve mobility and perception. This paper discusses the design and development of a computer vision based system aimed at assisting the visually impaired persons in accomplishing scene understanding and safe navigation in outdoor environments. The solution presented in this paper is integrated in a sensory substitution device (SSD), Sound of Vision (SoV) [1], a wearable assistive equipment providing acoustic and tactile descriptions of both indoor and outdoor environments. In this context, the actual scene understanding is performed by the user, based on the audio and haptic feedback provided by the SoV system regarding the objects in the scene. Investigating the state of the art on development of assistive systems for visually impaired, we find that many works have been described in the literature [2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20]. Nonetheless, from the perspective of the end-user there are still some important steps to be considered before noticeable communities of visuallyimpaired users adopt this technology. The reasons for not having such consumer grade systems largely available for the end-user are mostly related to the incomplete fulfillment of key requirements, out of which some are also technically conflicting: Wearability, real time operation, pervasiveness, functioning in any kind of environments, both indoor and outdoor usability (e.g., offering rich and natural feedback, efficient training instruments, customization possibilities), cost. The Sound of Vision project aimed to tackle the sensory substitution for the visually impaired in a comprehensive way. That is, it considers all the above challenging aspects and delivers a user friendly, wearable SSD, that works in both indoor and outdoor environments, invariant to the illumination conditions. Real time operation and wearability are conflicting requirements, especially as such systems need to employ complex 3D processing algorithms for environment elements of interest identification. Wearability induces hard constraints on the form factor of the acquisition/processing/rendering devices that can be integrated in the SSD. It also has a significant impact on aspects such as power consumption, heat dissipation. A minority of the reported assistive systems consider the pervasiveness aspect [18; 19; 21]. Most work is either in indoor or outdoor environments. This disadvantage mainly comes from the properties of the integrated 3D sensors. The infrared-based sensors, e.g., Kinect, are not meant to handle scenes with bright illumination from the sun. On the other hand, stereo sensors give unreliable depth maps in indoor environments with poor artificial lighting or uniformly colored/textured surfaces. Moreover, the computation pipelines are designed based on the exploitation of reliable depth input coming from high quality sensors, Kinect-like cameras [4; 15; 22; 23] or the Bumblebee stereo camera [3; 5; 6; 14]. With such devices, the wearability constraint is far from being fulfilled. The systems reported as being reliable in both indoor and outdoor environments deal with different limitations. CASBlip proposed by [2] has an operational range up to 5 m and uses reflected laser light beams for static object detection. The system deals with some problems when used in open environments (outdoor) due to external noises as the sound feedback is not perceived as clearly as it is in indoor environments (e.g., laboratory). [7]'s system relies on a pair of glasses with an embedded stereo camera mounted for depth sensing.