# Stereo Vision Based Sensory Substitution for the Visually Impaired
## 2 The Sound of Vision System  
The SoV system aims to improve both perception and mobility in unknown environments for users with visual impairment. Moreover, it provides feedback to the user in both indoor and outdoor environments and irrespective of the illumination conditions. The extended categories of information that can be provided by the proposed system and its user-centric hw & sw design surpass the limitations discussed in the previous section. The SOV device aims at improving the lifestyle of visually impaired persons in categories 3 to 5 according to the World Health Organization ([[http://apps.who.int/classifications/icd10/browse/2015/en](http://apps.who.int/classifications/icd10/browse/2015/en)]([http://apps.who.int/classifications/icd10/browse/2015/en](http://apps.who.int/classifications/icd10/browse/2015/en))). In these categories are included visually impaired persons who require assistance in their daily lives by using a white cane, a guide dog or relying on a guiding person. Sound of Vision was designed, developed and tested with the strong involvement of end-users, orientation and mobility instructors and specialists in psycho-physics and behavioral science. This multi- and inter-disciplinary approach produced an original hardware and software system, accompanied by a set of innovative and efficient training instruments [27]. The SoV device is foreseen to be used together with the white cane. Still, evidence from the performed usability experiments suggests that, after some amount of training, the users feel confident enough to use it without the cane. Thus, the SoV device provides the user with both redundant and complementary information to the white cane. The system architecture is based on four repetitive steps: (i) 3D data acquisition from the environment; (ii) 3D environment reconstruction and segmentation; (iii) haptic and audio modeling of the processed 3D scene; (iv) feedback the haptic and audio information to the user. A general overview of the entire 3D processing component of the Sound of Vision system was presented in [28], focusing on the philosophy behind the design that meets the requirements stated in the previous section. In order to work in both indoor and outdoor environments, and invariant to the illumination conditions, the 3D acquisition system in the SoV device integrates two different types of depth sensors: an IR-based depth sensor (Structure Sensor PS1080 from Occipital, Inc., San Francisco, CA, USA) and a stereo vision camera (LI-OV580 from Leopard Imaging Inc., Fremont, CA, USA). An IMU device (LPMS-CURS2 from Lp-Research Inc., Tokyo, Japan) is also used for recovering the orientation of the head/cameras, as the acquisition devices are rigidly mounted on a headgear (Figure 1). companies, please confirm  
The system can be set to work on four 3D acquisition modes: (1) Stereo camera input; (2) structure sensor input; (3) stereo-structure dual input and (4) recording playback. The 3D processing module makes use of different combinations of sensor data to maximize the system usability in different  
Figure 1: The Sound of Vision headgear: (**left**) 3D acquisition devices integrated in the SoV system—stereo camera (LI-OV580 from Leopard Imaging), IR depth sensor (Structure Sensor PS1080 from Occipital), IMU device (LPMS-CURS2 from Lp-Research) and a prototype implementation of the SoV headgear (**right**). situations and still provide environmental information in conditions atypical to standalone sensors [29]. The fusion of data in the dual mode allows the processing of both color and depth information to detect various structures such as doors, signs, text in indoor environments. Irrespective of the input, the core approach for scene segmentation relies on efficient detection of the ground (and ceiling, in indoor environments) surfaces, and partitioning of the remaining 3D information in obstacles. Still, different 3D processing approaches are employed to deal with the specific 3D input, structure and composition of indoor and outdoor environments, respectively.