Audio navigation assistance

Embodiments that relate to a head-mounted computing device for providing navigation assistance via audio output are disclosed. For example, in one disclosed embodiment depth image and visible image data are used to generate a three-dimensional mesh of at least a portion of an environment. Using the mesh, at least one feature in the environment is detected. When operating in a familiar navigation mode and based on detecting the feature, a first audio navigation cue is outputted to the user. When operating in an unfamiliar navigation mode and based on detecting the feature, a second audio navigation cue is outputted to the user, where the second audio navigation cue differs from the first audio navigation cue.

BACKGROUND

Persons with physiological visual impairments may utilize one or more mobility aids to navigate their surroundings. Such mobility aids may include, for example, canes, assistance animals, vision enhancement devices, and the like. Similarly, sighted persons encountering situational visual impairment, such as a dark or smoke-filled room, may also benefit from mobility aids that provide navigation assistance for their current environment.

Some mobility aids provide tactile feedback that indicates aspects of the user's surroundings, such as a cane contacting a curb on a street. However, using such aids may occupy a significant amount of the user's attention along with a hand of the user. Additionally, such aids generally provide the same amount and type of feedback of the user's immediate surroundings regardless of the context of those surroundings. As such, these aids are generally incapable of modifying or enhancing the utility of their feedback based on the context of the user's current environment.

SUMMARY

Various embodiments are disclosed herein that relate to providing navigation assistance in an environment via audio output. For example, one disclosed embodiment provides, in a head-mounted computing device, a method comprising receiving depth image data and visible image data from the environment. Using the depth image data and visible image data, a three-dimensional mesh of at least a portion of the environment is generated. Using the three-dimensional mesh, at least one feature in the environment is detected. When the head-mounted computing device is operating in a familiar navigation mode, and based on detecting the feature, a first audio navigation cue of the environment is outputted to the user via one or more transducers. When the head-mounted computing device is operating in the unfamiliar navigation mode, and based on detecting the feature, a second audio navigation cue of the environment is outputted to the user via the one or more transducers, where the second audio navigation cue differs from the first audio navigation cue.

DETAILED DESCRIPTION

FIG. 1shows a schematic view of one embodiment of head-mounted computing device10for providing navigation assistance to a user14. The head-mounted computing device10includes a navigation module16that may be stored in mass storage18of the head-mounted computing device10. The navigation module16may be loaded into memory20and executed by a processor22of the head-mounted computing device10to perform one or more of the methods and processes described in more detail below.

The navigation module16includes a plurality of navigation modes24comprising a familiar navigation mode26and an unfamiliar navigation mode28. Advantageously and described in more detail below, one or more of the navigation modes24may be selectively engaged to provide navigation assistance that is tailored to a user's particular needs in a context-appropriate manner. For example, the familiar navigation mode26may be engaged, programmatically or via user input30, when the user14is navigating an environment that is familiar to the user. In the familiar navigation mode26, navigation assistance may be tailored to account for the user's familiarity with her surroundings.

In a similar manner, the unfamiliar navigation mode28may be engaged, programmatically or via user input30, when the user14is navigating an environment that is unfamiliar to the user. In the unfamiliar navigation mode28, navigation assistance may be tailored to account for the user's unfamiliarity with her surroundings. It will be appreciated that the navigation module16may also include one or more additional navigation modes24that relate to environments or surroundings having particular characteristics or aspects such as, for example, an indoor navigation mode, outdoor navigation mode, urban navigation mode, rural navigation mode, etc. In some examples, two or more navigation modes may be engaged simultaneously, such as for example a familiar navigation mode and an indoor navigation mode.

The head-mounted computing device10may include various sensors and related systems that receive physical environment data from a physical environment32. With reference now also toFIG. 2, one example of a head-mounted computing device200in the form of a pair of wearable glasses is provided. In some examples, the head-mounted computing device200may comprise a transparent, semi-transparent or non-transparent display204that is supported in front of a viewer's eye or eyes. In other examples, the head-mounted computing device200may take a variety of other forms that may or may not comprise a display that is supported in front of a user's eye or eyes.

For example, in some embodiments the head-mounted computing device10may comprise a pair of eyeglass frames that do not include a display. Accordingly, many other types and configurations of head-mounted computing devices10having various form factors may also be used and are within the scope of the present disclosure. It will also be appreciated that the head-mounted computing device10shown inFIG. 1may take the form of the head-mounted computing device200shown inFIG. 2, as described in more detail below, or any other suitable head-mounted computing device.

With reference toFIGS. 1 and 2, in this example the head-mounted computing device10includes a depth sensor system34that includes one or more depth cameras208that generate depth image data38. Depth sensor system34may also detect movements within its field of view, such as gesture-based inputs or other movements performed by user14or by a person or physical object within the user's field of view. In one example, each depth camera208may include left and right cameras of a stereoscopic vision system. Time-resolved images from one or more of these depth cameras may be registered to each other and/or to images from another optical sensor such as a visible spectrum camera, and may be combined to yield depth-resolved video.

In other examples, a structured light depth camera may be configured to project a structured infrared illumination, and to image the illumination reflected from a scene onto which the illumination is projected. A depth map of the scene may be constructed based on spacings between adjacent features in the various regions of an imaged scene. In still other examples, a depth camera may take the form of a time-of-flight depth camera configured to project a pulsed infrared illumination onto a scene and detect the illumination reflected from the scene. It will be appreciated that any other suitable depth camera may be used within the scope of the present disclosure.

The head-mounted computing device10may also include a visible light sensor system42that utilizes at least one outward facing sensor212, such as an RGB camera or other optical sensor. The visible light sensor system42may generate visible image data46that is provided to the navigation module16. Outward facing sensor212may capture two-dimensional image information from physical environment32and features50within the environment. As described in more detail below, features50may include physical objects52and hazards54in the environment32. In some examples, a physical object52may also comprise a hazard54. Outward facing sensor212may capture images of the physical environment32in which user14is situated.

The head-mounted computing device10may also include a 3D modeling program56that uses depth image data38and visible image data46to generate a three-dimensional mesh58that models at least a portion of the physical environment32surrounding the user14. In some examples, the 3D modeling program56may utilize polygonal modeling techniques in which vertices in 3D space are connected by line segments to form a textured, 3D polygonal mesh. It will also be appreciated that other techniques for 3D modeling may be utilized including, but not limited to, curve modeling and digital sculpting techniques.

The head-mounted computing device10may also include a transducer system60comprising one or more actuators that convert an electrical signal from the navigation module16into another form of energy. As described in more detail below, the navigation module16may utilize the transducer system60to output navigation cues62of the environment32to the user14. With reference again toFIG. 2and in some examples, the transducer system60may include one or more speakers216for generating audio navigation cues64. In other examples, the one or more speakers216may take the form of headphones or in-ear ear buds worn by the user14.

In some examples, the navigation module16and/or transducer system60may process the audio navigation cues64to enable the user14to perceive that the cues originate at a particular location in 3D space of the physical environment32. For example, one or more crosstalk cancellation mechanisms may be utilized and configured such that a first audio signal (e.g., left channel) is delivered to a first ear (e.g., left ear) and a second audio signal (e.g., right channel) is delivered to a second ear (e.g., right ear) while substantially attenuating the delivery of the first audio signal to the second ear and delivery of the second audio signal to the first ear.

In other examples, the provision of three-dimensional audio may be based on a head-related transfer function “HRTF” and/or head-related impulse response “HRIR” to create the illusion that sound is originating from a particular location in 3D acoustic space. The HRTF describes how a given sound wave input is filtered by the diffraction and reflection properties of the head and pinna before the sound reaches the eardrum and inner ear. In other words, an HRTF may be defined based on the difference between a sound in free air and the sound as it arrives at the eardrum.

In other examples the transducer system60may include one or more tactile transducers220for generating haptic navigation cues to the user14, such as vibrations.

The head-mounted computing device10may also include a microphone system66and one or more microphones224for receiving audio input from the physical environment32. In some examples, a microphone array that includes a plurality of microphones224positioned at various locations on the head-mounted computing device200may be provided. The microphones224may comprise omnidirectional microphones and/or unidirectional microphones that are configured to receive speech and other audio inputs from the physical environment32.

In some examples, the microphone system66and/or navigation module16may utilize one or more acoustic source localization techniques to locate audio sources in the 3D space of the physical environment32, including sources located behind the user14. For example, the microphone system66and/or navigation module16may apply one or more beamforming techniques to at least a portion of the audio inputs from the microphone array. For example, a single, directionally-adaptive sound signal may be determined in any suitable manner. The directionally-adaptive sound signal may be determined based on a time-invariant beamforming technique, adaptive beamforming technique, or a combination of time-invariant and adaptive beamforming techniques. The resulting combined signal may have a narrow directivity pattern, which may be steered in a direction of an audio source. It will also be appreciated that any suitable acoustic source localization technique may be used to identify the location of an audio source.

As noted above, in some examples the head mounted computing device200may include a display system70and display204that enables visual navigation cues to be delivered to the eyes of a user14. In one example, the display204may comprise a transparent display configured to visually augment an appearance of the physical environment32to user14viewing the environment through the transparent display. For example, the appearance of the physical environment32may be augmented by graphical content (e.g., one or more pixels each having a respective color and brightness) that is presented via the transparent display to create a mixed reality environment.

In this example the transparent display204may also be configured to enable a user to view a physical, real-world object52in the physical environment32through one or more partially transparent pixels that are displaying a virtual object representation. In one example, the transparent display204may include image-producing elements located within lenses210(such as, for example, a see-through Organic Light-Emitting Diode (OLED) display). As another example, the transparent display204may include a light modulator on an edge of the lenses210. In this example the lenses210may serve as a light guide for delivering light from the light modulator to the eyes of a user. Such a light guide may enable a user to perceive a 2D image or a 3D holographic image located within the physical environment32that the user is viewing, while also allowing the user to view physical objects52in the physical environment.

With reference again to the head-mounted computing device10ofFIG. 1, it will be appreciated that the processor22may comprise a logic subsystem and the memory20may comprise a storage subsystem, as discussed in more detail below with respect toFIG. 9, that are in communication with the various sensors and corresponding systems described above. In one example, the storage subsystem may include instructions that are executable by the logic subsystem to receive signal inputs from the sensors and forward such inputs (in unprocessed or processed form) to the navigation module16.

Additionally, the example illustrated inFIG. 1shows the mass storage18, memory20and processor22integrated into the head-mounted computing device10. It will be appreciated that in other examples one or more of the mass storage18, memory20and processor22may be located in one or more other computing devices to which the head-mounted computing device10is communicatively coupled. For example, the head-mounted computing device10may be operatively connected with another computing device using a wired connection, or may employ a wireless connection via WiFi, Bluetooth, or any other suitable wireless communication protocol. Additional details regarding the components and computing aspects of the head-mounted computing device10and other computing device(s) are described in more detail below with reference toFIG. 9.

It will be appreciated that the head-mounted computing device10and related sensors and other components described above and illustrated inFIGS. 1 and 2are provided by way of example. These examples are not intended to be limiting in any manner, as any other suitable sensors, components, and/or combination of sensors and components may be utilized. Therefore it is to be understood that the head-mounted computing device10may include additional and/or alternative sensors, cameras, microphones, input devices, output devices, etc. without departing from the scope of this disclosure. Further, the physical configuration of the head-mounted computing device10and its various sensors and subcomponents may take a variety of different forms without departing from the scope of this disclosure.

With reference now toFIGS. 3-7, descriptions of example use cases and embodiments of the head-mounted computing device10will now be provided. InFIG. 3a user14wearing the head-mounted computing device200ofFIG. 2may be walking in his home300. The user14may have a vision impairment and may use the head-mounted computing device200for navigation assistance. In one example, the user14may initialize the head-mounted computing device200by walking from room to room in his home300and, for each room, providing user input30of a room identifier that identifies the room.

For example, while the user14walks around the kitchen304, the user may say, “This is my kitchen”. The navigation module16may receive this user input30, process it using a voice recognition program and tag the kitchen304with a “Kitchen” identifier. The navigation module16may use the visible image data46and depth image data38of the kitchen304to generate a 3D mesh of the kitchen and its various features50, such as appliances, furniture, walls, contours, surfaces, etc. The navigation module16may also generate a corresponding map80of the home300that includes the kitchen304tagged with the “Kitchen” identifier. In a similar manner, the navigation module16may use visible image data46and depth image data38of the living room308to generate a 3D mesh of this room and its various features50. The navigation module may tag the living room with a “Living Room” identifier based on user input30from the user14, and may add the living room308tagged with the “Living Room” identifier to the map80of the home300. As the user14walks from room to room and around the home300, the navigation module16may also recognize and store common pathways the user takes when navigating the house.

Subsequently, when the user14enters one of these rooms, the navigation module16may identify the room via the map80and detect the room as tagged with the corresponding room identifier. Accordingly and based at least on detecting the room as tagged with the corresponding room identifier, the navigation module16may engage the familiar navigation mode26. In other examples, a room or other portion of a physical environment32may not be associated with an identifier, but may have been previously visited by the user14. In these examples, the navigation module16may recognize the room or other portion of the environment32from the user's one or more previous visits by utilizing, for example, machine learning techniques. If the user has visited the room or other portion of the environment32at least a predetermined number of times, such as 1, 3, 5 or any other suitable number, then the navigation module16may engage the familiar navigation mode26. In still other examples, the user14may provide user input30to the navigation module16that engages the familiar navigation mode26.

With continued reference toFIG. 3and as described in more detail below, while in the familiar navigation mode26the navigation module16may provide navigation cues62to the user14that are different from the navigation cues provided when the navigation module16is operating in the unfamiliar navigation mode28. In one example, the user14may be walking in the general direction of the ottoman312located at a usual location314. In the familiar navigation mode26, the navigation module16may not provide a navigation cue62corresponding to the ottoman312, under the assumption that the user14remembers the usual location314of the ottoman312from previous visits to the living room308. In another example, the navigation module16may output an audio navigation cue64that comprises a gentle reminder of the ottoman312, such as vocalizing “Ottomon” in a soft voice via speaker216.

In another example, the user14may be a first-time guest in the home300. Accordingly the navigation module16may engage the unfamiliar navigation mode28. In this mode, as the user14walks in the general direction of the ottoman312, the navigation module16may provide an audio navigation cue64to the user that alerts the user to the location of the ottoman. For example, the navigation module16may output a brief, distinctive sound (or earcon), illustrated as familiar earcon320, using one or more acoustic source localization techniques that cause the user14to perceive the earcon as originating at the usual, 3D location314corresponding to the ottoman312. In another example, the navigation module16may vocalize the word “Ottoman” to the user14from the usual location314using an acoustic source localization technique

In another example and while in the familiar navigation mode26, the navigation module16may recognize that the ottoman312is in an unusual location316that is different from its usual location314. Further and also based on the user's direction of movement, the navigation module16may determine that in unusual location316the ottoman312is in the direct path of the user14. The navigation module16may determine that in this situation the ottoman312poses a tripping hazard to the user14.

Accordingly, the navigation module16may classify the ottoman312as a hazard54and may output an audio navigation cue64in the form of an alert earcon322utilizing an acoustic source localization technique that warns the user of the unusual location316of the ottoman312. In this example, the ottoman312may be classified as both an object52and a hazard54by the navigation module16. Additionally and to convey a sense of urgency to the user14, the alert earcon322may be outputted with a higher volume as compared to the familiar earcon320, as depicted inFIG. 3by the alert earcon322having a larger size than the familiar earcon320. In other examples, the alert earcon322may comprise a different distinctive sound or sounds as compared to the familiar earcon320.

In some examples, the navigational module16may output audio navigation cues64for a feature50in the environment32at increasing volumes as the proximity of the feature to the user14decreases. For example and with reference now toFIG. 4, the user14may be in the lobby404of a hotel400for the first time. Accordingly, the navigation module16may be operating in the unfamiliar navigation mode28. The user14may be at an initial position410that is a first distance away from the sculpture414, and may be walking towards the sculpture in the center of the lobby404. When the user14is at the initial position410, the navigation module16may output a sculpture earcon418at a first volume that causes the user14to perceive the sculpture earcon as originating at the 3D location420corresponding to the sculpture. As the user14continues walking toward the sculpture414and reaches a subsequent position422, the navigation module16may output the sculpture earcon at a second volume greater than the first volume, as indicated by the larger sculpture earcon418′, to alert the user that he is closer to the sculpture414.

In some examples in the unfamiliar navigation mode28, the navigation module16may output an audio navigation cue64corresponding to each feature50in the physical environment32in which the user14is located. For example, inFIG. 4the navigation module16may output an audio navigation cue64corresponding to each feature that is within the field of view of the user14. When the user14is at the initial position410, the navigation module16may output an audio navigation cue64corresponding to the sculpture414, desk430, couch434, person438and wall442.

In another example, the navigation module16may use one or more of the depth image data38and visible image data46to recognize the face of the person438sitting on the couch434. The navigation module may then associate the person's face with an identity of the person, using for example a facial recognition database stored in a remote server. The navigation module may then inform the user14of the identity of the person438. Advantageously, the user14may thereby anticipate a person who is in the vicinity of the user.

Further and in some examples, an identity earcon may be assigned to the identity of the person438, with the identity earcon being provided to the navigation module16via user input30or being programmatically generated by the navigation module. In these examples, when the navigation module16associates the face with the identity of the person438, the navigation module16may output the identity earcon to the user to inform the user of the identity of the person.

With reference now toFIG. 5, in another example an audio navigation cue64may be customized to identify a slope in the physical environment32. In one example the user14may be approaching an inclined slope502. The navigation module16may recognize the inclined slope502and may correspondingly output to the user14an audio navigation cue64comprising a plurality of ascending pitched sounds506that indicate an inclined slope.

In another example and with reference now toFIG. 6, the user14may be approaching a declined slope602. The navigation module16may recognize the declined slope602and may correspondingly output to the user14an audio navigation cue64comprising a plurality of descending pitched sounds606that indicate a declined slope.

With reference again toFIG. 5, in some examples the navigation module16may recognize the inclined slope502as one or more steps510, such as in a flight of stairs. In response, the navigation module16may correspondingly output to the user14an audio navigation cue64comprising a plurality of discrete ascending pitched sounds506, as indicated by the individual eighth notes514. In this manner, the navigation module16may communicate to the user14that ascending steps are ahead. In a similar manner, where the navigation module16recognizes a declined slope as one or more steps, the navigation module16may correspondingly output to the user14an audio navigation cue64comprising a plurality of discrete descending pitched sounds. In this manner, the navigation module16may communicate to the user14that descending steps are ahead.

In some examples, the navigation module may identify the number of steps in an inclined or declined slope, and may communicate the number of steps to the user14via a number of discrete pitched sounds that correspond to the number of steps. For example and as shown inFIG. 5, the navigation module16may recognize three steps510in the inclined slope502. Accordingly, the navigation module16may output to the user14three discrete pitched sounds in the form of three eighth notes514. Advantageously, in this manner the user14may be conveniently informed that three ascending steps are ahead.

With reference again toFIG. 6, in some examples the navigation module16may recognize the declined slope602as comprising a continuous slope in the form of a declined grade, such as an downwardly sloped sidewalk, street, hallway, etc., that may be substantially planar as shown inFIG. 6, or may be undulating or somewhat rolling. In response, the navigation module16may correspondingly output to the user14an audio navigation cue64comprising a plurality of continuous descending pitched sounds606, as indicated by the slur610connecting the four eighth notes inFIG. 6.

In some examples, the audio navigation cue64outputted to the user14to identify a slope may be different in the familiar navigation mode26as compared to the unfamiliar navigation mode28. For example and with reference again toFIG. 5, the discrete ascending pitched sounds506indicating steps510may be outputted at a first, lower volume when the navigation module16is operating in the familiar navigation mode26(such as, for example, when the user14is in his home). However when the navigation module16is operating in the unfamiliar navigation mode28(such as, for example, when the user14is in a building for the first time), the discrete ascending pitched sounds506indicating steps510may be outputted at a second, higher volume to provide greater notice to the user of the stairs.

In some examples and as noted above, the navigation module16may be configured to output a haptic navigation cue to the user14via one or more tactile transducers220. For example and with reference also toFIG. 4, the head-mounted computing device200worn by user14may include a tactile transducer220on the right temple arm240of the device, where the tactile transducer is configured to generate vibrations that are felt by the user14. When the user14is at initial position410, and to alert the user of the desk430to the immediate right of the user, the navigation module16may cause the tactile transducer220on the right temple arm240of the head-mounted computing device200to vibrate in a predetermined pattern that indicates an object is nearby and to the right of the user14.

With reference now toFIG. 7, in some examples the navigation module16may also be configured to display a visual navigation cue to the user14via a display system70of a head-mounted computing device10. In one example, the user14may have a sight impairment comprising significant myopia that limits the user's ability to see distant objects. The user14may be walking down a city street, with the view of the street through a transparent display204of the head-mounted computing device200shown inFIG. 7. A van704may be driving toward the user14.

The navigation module16may use depth image data38to determine three-dimensional locations of the van704as it moves toward the user14, and may determine that the van is on a path that may hit the user14. Accordingly, the navigation module16may display multiple, flashing warning icons708on the transparent display204to alert the user to the approaching hazard. In some examples, the navigation module16may also output an audio navigation cue64alerting the user to the approaching hazard. For example, the audio navigation cue64may comprise one or more vocalized warnings and related instructions712, such as “Warning! Car approaching rapidly! Move to your right!”

As noted above, in some examples the head-mounted computing device10may include a display system70. Where the head-mounted computing device10includes a display system70, in some examples the head-mounted computing device10may output audio navigation cues64as described above while not displaying images or other visual content to the user via the display system70.

FIGS. 8A and 8Billustrate a flow chart of a method800for providing navigation assistance to a user in an environment via audio output according to an embodiment of the present disclosure. The following description of method800is provided with reference to the software and hardware components of the head-mounted computing device10described above and shown inFIGS. 1-7. It will be appreciated that method800may also be performed in other contexts using other suitable hardware and software components.

With reference toFIG. 8A, at804the method800may include receiving depth image data from the environment. At808the method800may include receiving visible image data from the environment. At812the method800may include, using the depth image data and visible image data, generating a three-dimensional mesh of at least a portion of the environment. At816the method800may include, using the three-dimensional mesh, detecting at least one feature in the environment. At820the method800may include, when the head-mounted computing device is operating in a familiar navigation mode and based on detecting the feature, outputting a first audio navigation cue of the environment to the user via the one or more transducers.

At824the method800may include, when the head-mounted computing device is operating in an unfamiliar navigation mode and based on detecting the feature, outputting a second audio navigation cue of the environment to the user via the one or more transducers, wherein the second audio navigation cue differs from the first audio navigation cue. At828the method800may include outputting one or more of the first audio navigation cue and the second audio navigation cue to the user in a manner that causes the user to perceive the cue as originating at a three-dimensional location of the feature. At832the feature may comprise a slope, and the first audio navigation cue and the second audio navigation cue comprise a plurality of ascending pitched sounds that indicate an inclined slope or a plurality of descending pitched sounds that indicate a declined slope.

At836the first audio navigation cue and the second audio navigation cue may comprise a plurality of continuous pitched sounds that indicate a continuous slope or a plurality of discrete pitched sounds that indicate a plurality of steps. With reference now toFIG. 8B, at840the method800may include outputting the first audio navigation cue and the second audio navigation cue to the user at a volume that increases as a proximity of the detected object to the user decreases. At844the method800may include recognizing a face in the environment using one or more of the depth image data and the visible image data. At848the method800may include associating the face with an identity of a person. At852the method800may include informing the user of the identity of the person.

At856the method800may include assigning an identity earcon to the identity of the person. At860the method800may include outputting the identity earcon to the user via the one or more transducers. At864the method800may include outputting a haptic navigation cue to the user via the one or more transducers. At868the method800may include, based on detecting the feature, displaying a visual navigation cue to the user via a display system of the head-mounted computing device, wherein the visual navigation cue is based on at least a three-dimensional location of feature in the environment.

It will be appreciated that method800is provided by way of example and is not meant to be limiting. Therefore, it is to be understood that method800may include additional and/or alternative steps than those illustrated inFIGS. 8A and 8B. Further, it is to be understood that method800may be performed in any suitable order. Further still, it is to be understood that one or more steps may be omitted from method800without departing from the scope of this disclosure.

FIG. 9schematically shows a nonlimiting embodiment of a computing system900that may perform one or more of the above described methods and processes. Head-mounted computing device10may take the form of or include one or more aspects of computing system900. Computing system900is shown in simplified form. It is to be understood that virtually any computer architecture may be used without departing from the scope of this disclosure. In different embodiments, computing system900may take the form of a mainframe computer, server computer, desktop computer, laptop computer, tablet computer, home entertainment computer, network computing device, mobile computing device, mobile communication device, gaming device, etc.

As shown inFIG. 9, computing system900includes a logic subsystem904, storage subsystem908, and sensor subsystem912. Computing system900may optionally include a display subsystem916, communication subsystem920, input subsystem922and/or other subsystems and components not shown inFIG. 9. Computing system900may also include computer readable media, with the computer readable media including computer readable storage media and computer readable communication media. Computing system900may also optionally include other user input devices such as keyboards, mice, game controllers, and/or touch screens, for example. Further, in some embodiments the methods and processes described herein may be implemented as a computer application, computer service, computer API, computer library, and/or other computer program product in a computing system that includes one or more computers.

Logic subsystem904may include one or more physical devices configured to execute one or more instructions. For example, the logic subsystem904may be configured to execute one or more instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more devices, or otherwise arrive at a desired result.

The logic subsystem904may include one or more processors that are configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic subsystem may be single core or multicore, and the programs executed thereon may be configured for parallel or distributed processing. The logic subsystem may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and/or configured for coordinated processing. One or more aspects of the logic subsystem may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

Storage subsystem908may include one or more physical, persistent devices configured to hold data and/or instructions executable by the logic subsystem904to implement the herein described methods and processes. When such methods and processes are implemented, the state of storage subsystem908may be transformed (e.g., to hold different data).

In some embodiments, aspects of logic subsystem904and storage subsystem908may be integrated into one or more common devices through which the functionally described herein may be enacted, at least in part. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC) systems, and complex programmable logic devices (CPLDs), for example.

FIG. 9also shows an aspect of the storage subsystem908in the form of removable computer readable storage media924, which may be used to store data and/or instructions executable to implement the methods and processes described herein. Removable computer-readable storage media924may take the form of CDs, DVDs, HD-DVDs, Blu-Ray Discs, EEPROMs, and/or floppy disks, among others.

It is to be appreciated that storage subsystem908includes one or more physical, persistent devices. In contrast, in some embodiments aspects of the instructions described herein may be propagated in a transitory fashion by a pure signal (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for at least a finite duration. Furthermore, data and/or other forms of information pertaining to the present disclosure may be propagated by a pure signal via computer-readable communication media.

Sensor subsystem912may include one or more sensors configured to sense different physical phenomenon (e.g., visible light, infrared light, sound, acceleration, orientation, position, etc.) as described above. Sensor subsystem912may be configured to provide sensor data to logic subsystem904, for example. As described above, such data may include image information, ambient lighting information, depth information, audio information, position information, motion information, user location information, and/or any other suitable sensor data that may be used to perform the methods and processes described above.

When included, display subsystem916may be used to present a visual representation of data held by storage subsystem908. As the above described methods and processes change the data held by the storage subsystem908, and thus transform the state of the storage subsystem, the state of the display subsystem916may likewise be transformed to visually represent changes in the underlying data. The display subsystem916may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic subsystem904and/or storage subsystem908in a shared enclosure, or such display devices may be peripheral display devices. The display subsystem916may include, for example, the optional display system70and display program72of the head-mounted computing device10.

When included, communication subsystem920may be configured to communicatively couple computing system900with one or more networks and/or one or more other computing devices. Communication subsystem920may include wired and/or wireless communication devices compatible with one or more different communication protocols. As nonlimiting examples, the communication subsystem920may be configured for communication via a wireless telephone network, a wireless local area network, a wired local area network, a wireless wide area network, a wired wide area network, etc. In some embodiments, the communication subsystem may allow computing system900to send and/or receive messages to and/or from other devices via a network such as the Internet.

The terms “module” and “program” may be used to describe an aspect of the head-mounted computing device10that is implemented to perform one or more particular functions. In some cases, such a module or program may be instantiated via logic subsystem904executing instructions held by storage subsystem908. It is to be understood that different modules and programs may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module or program may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module” and “program” are meant to encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.