Audible user interface system

Described are techniques for enabling pointer devices to be used by users with vision impairment to interact with rendered content. Sounds may be output to a user to indicate the presence of one or more features within rendered content. Characteristics of the sounds may be used to indicate the position of the features within a user interface, the distance between the features and a pointer associated with a pointer device, and the direction of the features relative to the pointer. Sounds may also be used to indicate the type or size of rendered content. Responsive to the sounds, a user with impaired vision may manipulate the pointer device to reposition the pointer. Additional or modified sounds may be provided to the user based on the subsequent position of the pointer.

BACKGROUND

Users of a computing device having impaired vision are often unable to effectively utilize a pointer device or other types of input devices normally used to maneuver a cursor within a display area of a computing device.

DETAILED DESCRIPTION

A visually impaired user of a computing device may use a screen reader to identify and interpret displayed text and generate speech corresponding to the text. Screen reading software may also output audible speech that describes graphical elements or communicates other information that may be encoded within rendered content. Rendered content may include content provided by a web server and rendered by a rendering engine for presentation to a user. Screen reading software may include Job Access With Speech (JAWS) created by Freedom Scientific, Inc. of St. Petersburg, Fla., USA; NonVisual Desktop Access (NVDA) created by NV Access of Ferny Grove, Queensland, Australia; Microsoft Narrator created by Microsoft Corp. of Redmond, Wash., USA, and so forth. Through use of keyboard presses or other inputs, the user may navigate between sections of a webpage or other content, listen to speech corresponding to the content that is currently displayed, and then interact with that content or navigate to other content. Interaction with rendered content in this manner may be less efficient than conventional methods due to the fact that a user must typically navigate through individual sections of the content, one at a time, listening to a sufficient quantity of speech for each section to determine whether to navigate to the subsequent section. Pointer devices, such as mouse devices or other types of input devices, used to move a pointer, such as a visible cursor, within a display area may permit more direct and efficient interaction with and navigation of content. For example, a mouse device may be used to directly select a displayed feature or section of the content. However, because pointer devices rely on movement of a visible cursor, they are not readily accessible to users with vision impairment.

Described in this disclosure are techniques for enabling users with impaired vision to effectively use input devices configured to move a pointer within a display area by providing audible feedback to the user(s). The audible feedback may include one or more sounds that indicate features of rendered content, such as a type or category associated with the content, a length of the content, or the presence of certain features within the content, such as particular text, buttons, menus, search bars, and so forth. Sounds may also be provided based on the position of a pointer relative to one or more features within the content, the beginning or end of the content, or the edges of the display area. In some implementations, the audible feedback may be used in conjunction with a screen reader to augment the accessibility provided by the screen reader. For example, audible feedback may be used to enable a user to maneuver a pointer into association with a feature of a rendered website, responsive to which the screen reader may output speech indicative of the feature associated with the pointer. As another example, audible feedback may be used to enable a user to determine a number of pages remaining in a chapter of an electronic book, while the screen reader outputs speech corresponding to the text of the book. The volume and frequency of the audible feedback may be configured to minimize interference with the speech output by the screen reader.

Audible feedback may include any manner of sound, such as a tone having a selected frequency, a musical note, a recorded sound (e.g., a WAV file, a MPEG file, and so forth), a series of tones or sounds, and so forth. For example, when a webpage having a particular link or button is rendered, a user may be provided with a first tone, followed by a second tone after passage of an interval of time. The length of the interval of time may correspond to the distance between a pointer associated with a mouse device and the particular link or button. In other implementations, other characteristics of a sound may be altered to indicate the distance between a pointer and a link or button, a direction of the link or button relative to the pointer, and so forth. For example, the frequency, amplitude, or duration of a sound may be modified based on the distance between a feature and a pointer or the direction of the feature relative to the pointer. Different tones or other types of sounds may be used to indicate the other features on the webpage. Tones, notes, and other sounds may also be used to indicate a type or category associated with the webpage, the length of the webpage, and so forth. One or more of the emitted sounds may be provided with a volume less than that of speech produced by accompanying screen reader software to avoid interfering with the ability of a user to hear the speech indicative of the rendered content.

In some implementations, the sounds that are output and the characteristics of the sounds may be determined based on one or more environmental factors associated with a user device. For example, sensors associated with a user device may determine ambient noise, temperature, motion, and so forth. Properties of emitted sounds, such as the frequency, wavelength, amplitude, direction, and so forth, may be modified based on determined environmental factors. Continuing the example, sounds emitted by a user device may be provided with a volume greater than that of ambient noise or with a frequency that differs from that of the ambient noise to facilitate distinguishing the emitted sounds. As another example, sensors associated with a user device may detect speech, such as the user's voice. Sounds emitted by the user device may be provided with a lower volume responsive to determination of the user's speech. As yet another example, responsive to determination of the presence of a service animal, such as a dog, the frequency of sounds emitted by the user device may be selected to minimize upset to the animal. In some implementations, the emitted sounds may be selected to alert the animal to take an action subsequent to an event, such as the completion of an article by a user. The presence of a service animal may be determined based on user input, detection of a radio frequency identification (RFID) tag or similar object worn by the animal, determination of the presence of the animal using cameras associated with the user device, and so forth.

In some implementations, the features of rendered content responsive to which sounds are output and the particular sounds that are emitted by a speaker may be customized by a user. For example, a user may provide user input to a user device to assign a series of tones having frequencies ranging from 10 hertz (Hz) to 360 Hz to one or more types of webpages. When a particular type of webpage is rendered, a tone having the selected frequency corresponding to that webpage type may be emitted. As another example, a user may assign a tone having a frequency ranging from 600 Hz to 1200 Hz to indicate the length of a webpage. When the webpage is rendered, an interval of time corresponding to the webpage length may be determined. A first tone may be emitted, followed by a second tone after the passage of the interval of time. In other implementations, particular frequencies may be used to indicate page length using a single tone, such as a 1,200 Hz tone to indicate a short webpage and a 600 Hz tone to indicate a long webpage. Other tones may be assigned to indicate the position of a pointer relative to the left and right edges of a display area. As another example, a user may assign a 2,000 Hz tone to a particular link. The selected tone may be emitted periodically, at a rate based on the proximity of the pointer to the link. As the pointer is moved closer to the link, the rate at which the tone is repeated may increase. When the pointer reaches a position from which the link may be selected, the tone may change (e.g., in rate, frequency, amplitude, and so forth) to indicate to the user that the link may be accessed. As yet another example, if a webpage or other content is not properly rendered, a notification sound indicative of the failure to render the content may be provided to the user. In some implementations, when subsequent content is not properly rendered, the content previously presented to the user may be retained for interaction by the user. A user may also customize sounds based on environmental factors. For example, a user may select particular frequencies, volumes, or types of sounds to be emitted based on ambient noise, motion, temperature, and so forth.

In some implementations, audio data may be output to a speaker or another type of output device automatically, such as at the time when content is rendered, or periodically at selected intervals. In other implementations, audio data may be output responsive to input received by a user. For example, a user may provide input using a mouse device or another input device requesting audio feedback regarding the position of a pointer relative to one or more features. Audio feedback may be output to the user responsive to that input. As another example, a user may move a pointer device a distance that exceeds a threshold distance. Responsive to this movement, audio feedback may be output to the user based on the new position of the pointer relative to one or more features.

To facilitate a user's access to one or more features of rendered content, in some implementations, a pointer associated with an input device may be automatically positioned within a threshold distance of the feature at the time that the content is rendered. For example, when rendering a webpage for presentation to a user, the pointer associated with a mouse device may be automatically positioned over a button that will link to a subsequent page of the article when the current page of the article is rendered.

Implementations usable within the scope of the present disclosure may enable user with impaired vision to make use of input devices associated with movable pointers, such as mouse devices, trackballs, touchpads, touch sensors, joysticks, puff tubes (e.g., sip-and-puff (SNP)), head trackers, and so forth. By providing audio feedback to a user, the user may be informed of the presence or position of any number of distinct features within rendered content. The characteristics of the audio feedback may be modified as the user moves the pointer, which may guide the user toward certain desired features.

FIG. 1depicts a system100for providing audio output to a user102based on features determined from content rendered to a user device104. Use of audio output in lieu of or in addition to visible content may enable a user102with vision impairment to discern features of the content and interact with the user device104responsive to the audio output. WhileFIG. 1depicts the user device104as a laptop computer, the user device104may include any manner of computing device such as a mobile device, smartphone, set-top box, tablet computer, personal computer, wearable computer, and so forth. In one implementation, the user device104may include a portable device for placement on a floor, shelf, or other surface in a room. Cameras or other sensors associated with the portable device may detect movement or the position of a user's fingers or other body parts for use as input associated with a virtual user interface. One or more speakers associated with the portable device may be used to provide output to the user102in the form of sound. For example, sounds may be output responsive to movement of the user's fingers (e.g., through the air), and the user102may further move fingers or other body parts responsive to the sounds, effectively creating an audio-based user interface.

The user102may provide input to the user device104using one or more input devices106. The input devices106may include one or more mouse devices, trackballs, touchpads, touch sensors, joysticks, SNPs, head trackers, microphones, keyboards, buttons, cameras, and so forth. For example,FIG. 1depicts a first input device106(1) as a mouse device, a second input device106(2) as a touchpad, and a third input device106(3) as a keyboard. The user device104may also include one or more output devices108. For example, the user device104may include one or more speakers that may be used to emit sounds110perceivable by the user102. The speaker(s) may include monaural speakers or multi-channel (e.g., stereo, surround-sound, and so forth) speakers. In some implementations, an array of speakers may be used to control the directionality of the emitted sounds110. For example, beamforming or other techniques may be used to provide for directionality or gain in particular directions. In other implementations, speakers used to emit sounds110may be associated with multiple user devices104. For example, speakers associated with a portable user device104, such as a smartphone, and one or more wearable devices, such as a watch or headphones, may be used to provide the sounds110to the user102. The particular speaker(s) used to emit a sound110may be used to provide the sound110with a directional component or to indicate specific features of accessed content. The user device104may also include a display having a display area112in which rendered content may be visualized. However, in some implementations, content may be rendered by the user device104without necessarily displaying the content using a display. For example, a user device104lacking a display, such as a telephone, wearable device, or computer lacking a monitor or screen, may be used by a user102with vision impairment or by other users102, who may interact with the content based on the sounds110emitted by the speaker(s). In some implementations, the output devices108may include one or more haptic devices. For example, one or more of the input devices106or other portions of the user device104may be provided with a haptic device that may provide a tactile signal to a user102touching one or more of the input devices106or other portions of the user device104. The tactile signal is configured to provide a sensory input to the user102that may be perceived as a touch, texture, and so forth. For example, the haptic device may comprise a mass and actuator configured to generate a vibrotactile sensation to the user102. In another example, the haptic device may include an electrical signal generator to electrically stimulate nerves in the skin of the user102. As another example, the user102may touch one or more wearable haptic devices such as gloves, footpads, and so forth, associated with the user device104, which may be used to provide tactile signals to the user102. Haptic devices may be used in conjunction with speakers. For example, a user102may experience listener fatigue when presented with a large number of audible sounds110for a prolonged period of time. As another example, the quantity of information that may be presented at a single time using sound may be limited, such as due to limitations of a user device104or of the user102. Use of haptic feedback in addition to audio feedback may enable the user102to reduce the quantity of audible sounds110used to navigate rendered content while enhancing the ability of the user102to interact with the content. For example, haptic feedback may be used to indicate particular features or characteristics of the content, such as a specific location or direction corresponding to a feature. Audio feedback may be used to indicate other features or characteristics, such as a distance corresponding to the feature. As another example, audio feedback may be used to indicate a first characteristic of a webpage feature, such as the distance between that feature and a pointer. Continuing the example, haptic feedback may be used to indicate the direction of the feature relative to the pointer, such as by providing a tactile sensation to one or more wearable devices contacting a user's extremities.

A rendering engine114, stored in a data store116associated with the user device104, may output rendered content118for presentation by the user device104. For example, rendered content118may be provided to memory associated with the user device104for access by one or more processors. The data store116may include one or more computer-readable storage media (CRSM) in communication with the user device104. For example, the data store116may include one or more internal, external, removable, or portable CRSM associated with the user device104. In other implementations, the data store116may be associated with a server or one or more other computing devices, which may communicate with the user device104via one or more networks. For example, a web server120may provide source content122to the user device104. The web server120may include one or more computing devices configured to process requests from user devices104to access websites, games, applications, and other types of data and to provide source content122associated with the requested data to a user device104for rendering. The source content122may comprise Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), JavaScript, Java code, image files, and so forth. The rendering engine114may process the source content122to produce rendered content118. For example, the rendering engine114may produce rendered content118that comprises a Document Object Model (DOM) that has been generated from the source content122, vector graphics, bitmapped images, and so forth. The rendered content118may be presented to a user102using one or more of visible indicia within the display area112, audible indicia using speakers, or tactile indicia using haptic devices. In other implementations, rendering engine114may be associated with the web server120or another computing device. The web server120or one or more other computing devices associated with the rendering engine114may produce the rendered content118and provide the rendered content118to the user device104.

The rendered content118may be used to present a user interface to one or more users102in the display area112of the user device104. As described previously, however, in some implementations, use of the display area112may be omitted, or a user device104lacking a display may be used to render content for access by one or more users102. The rendered content118may then function as a virtual user interface. The user interface may include a pointer124associated with one or more of the input devices106. For example, input provided to the user device104using a mouse device or touchpad may move the pointer124within the display area112or select one or more features126depicted within the display area112. Continuing the example, the depicted user interface includes a first feature126(1), which may include a button for purchasing a product shown or described in the display area112. A second feature126(2) may include a text box that may be used to input search queries to locate additional products. Other features126may include links, buttons, menus, scrollbars, graphics, text, the edges of the display area112, the boundaries of the rendered content118, and so forth.

A feature determination module128in the data store116may be used to determine the position of one or more features126or the pointer124within the user interface. For example, the feature determination module128may process the DOM tree provided by the rendering engine114to determine the placement of features126within the display area112. In other implementations, the feature determination module128may access one or more of the rendered content118or the user device104to determine the position of the one or more features126or the pointer124. Determining the position of the one or more features126or the pointer124may include determining a relative position the one or more features126or the pointer124in a two-dimensional space or three-dimensional space represented by the user interface. The determined positions of one or more features126or of the pointer124may be communicated to the user102via audio output, such as tones, musical notes, recorded sounds, melodies, and so forth. For example, the feature determination module128may determine that the pointer124is located a first distance D1from the first feature126(1). The feature determination module128may also determine that the pointer124is located a second distance D2from the second feature126(2). In some implementations, relative distances D of features126from the pointer124may be determined by measuring the distances D relative to a particular point associated with the pointer124, such as the tip of an arrow. The determined distances D may be used by an audio generation module130in the data store116to determine audio output to present to users102.

The audio generation module130may access audio data132for output to one or more speakers, based on the presence or position of the features126determined by the feature determination module128or the relative distances D of the features126from the pointer124. The audio data132may include data corresponding to any manner of sound110, such as tones, musical notes, waveforms, a series of tones, and so forth. The audio data132accessed by the audio generation module130may correspond to the particular features126determined by the feature determination module128, the distances D between the pointer124and the features126, or other characteristics of the rendered content118. For example, the audio generation module130may access first audio data132(1) corresponding to a first sound110(1) to indicate the presence of the first feature126(1). The audio generation module130may access second audio data132(2) corresponding to a second sound110(2) to indicate the presence of the second feature126(2). In some implementations, the audio generation module130may modify one or more characteristics of the first audio data132(1) or the second audio data132(2) based on the distances D between the pointer124and one or more features126. For example, a sound110corresponding to the second feature126(2), which is farther from the pointer124than the first feature126(1), may be modified by providing the sound110with a lower volume, a lower frequency, a longer duration, and so forth, than the sound110corresponding to the first feature126(1). In some implementations, the sound110may include a repeating tone or other type of sound110, and the rate of repetition of the sound110may correspond to the distance D between the pointer124and a feature126. In other implementations, two sounds110corresponding to a feature126may be emitted, spaced apart by an interval of time that corresponds to the distance D between the pointer124and the feature126.

In some implementations, the audio generation module130may determine an interval of time that corresponds to the first distance D1. Continuing the example, the interval of time may be determined using a formula or algorithm, such as an interval of 20 milliseconds (ms) per 10 pixels of distance between the pointer124and the first feature126(1). In other implementations, the interval of time may be determined from a table stored in the data store116in which intervals of time are associated with distances D. The audio generation module130may provide first audio data132(1) to an output device108to produce a first sound110(1) that corresponds to the first feature126(1). For example, the audio generation module130may provide a tone having a particular frequency. The audio generation module130may then determine passage of the interval of time. After passage of the interval of time, the audio generation module130may provide second audio data132(2) to the output device108to produce a second sound110(2). The two sounds110may be identical sounds110or different sounds110. The audio generation module130may similarly determine an interval of time corresponding to the second distance D2. The audio generation module130may provide first audio data132(1) that corresponds to the second feature126(2) to produce a first sound110(1) and then provide second audio data132(2) to produce a second sound110(2) subsequent to the passage of the interval of time. The intervals of time that pass between successive emissions of paired sounds110may inform the user102of the distances D between each feature126and the pointer124. In some implementations, the audio data132may be modified with one or more audio characteristics indicative of the direction that a feature126is located relative to the pointer124. For example, speakers that are part of a surround-sound system may emit sounds110using channels that provide the sounds110with a particular directionality. Continuing the example, responsive to determination of a feature126located on the right portion of a user interface, surround-sound channels that produce a sound110appearing to emanate from the right side of the user102may be used to emit the sound110.

In some implementations, a user102may select one or more of the features126responsive to which particular sounds110are produced. A user102may also select particular characteristics of the features126to be indicated by modifying characteristics of the sounds110. For example, a user interface module134may receive user input that may be stored as configuration data136. Continuing the example, the user102may select one or more types of content that may be of interest to the user102. The user102may also select one or more sounds110that may indicate the selected types of content. When rendered content118having a selected content type is rendered by the rendering engine114, the audio generation module130may determine the audio data132corresponding to the sound110indicative of that content type and provide that audio data132to an output device108.

As another example, the user102, by providing input processed by the user interface module134, may select one or more features126that may be of interest, such as a text box for inputting search queries, a button for purchasing products, one or more particular strings of text, one or more graphics, and so forth. The user102may also select one or more sounds110for indicating selected features126. In some implementations, the user102may further select characteristics of the features126, such as the position of the features126within a user interface (e.g., the section of a webpage where a feature126may be located) or the distance or direction of a feature126relative to a pointer124. The user102may also select characteristics of the sounds110that may be used to indicate characteristics of the features126. The user interface module134may receive and process each selection provided by a user102and generate configuration data136based on the user input. For example, a user102may select a repeating tone to indicate the distance D between a pointer124and a feature126, in which the rate of repetition of the tone changes based on the distance D. As another example, a user102may select tones having different frequencies or amplitudes based on the zone of a user interface or display area112within which a feature126is located or the direction in which a feature126is located relative to the pointer124. Continuing the example, the user102may input characteristics of tones, such as frequencies, durations, or amplitudes, or the user102may select one or more tones from a list or menu. As yet another example, a user102may select characteristics of a sound110that may vary based on distance D, such as the amplitude of a sound110. Sounds110that correspond to features126that are a greater distance D from a pointer124may be emitted at a lesser amplitude than sounds110corresponding to features126that are closer to the pointer124. The user102may provide input to be processed by the user interface module134using one or more input devices106. For example, the user interface module134may maintain data regarding the position of the pointer124within the location of the user interface, such as a set of coordinates corresponding to the position.

FIG. 2depicts a diagram200illustrating an example of rendered content118and a user interface202usable within the scope of the present disclosure. The display area112may present at least a portion of the rendered content118. However, in some implementations, the display area112may remain unused, or a user device104lacking a display may be used. The display area112is bounded by an upper edge204, a lower edge206, a left edge208, and a right edge210. The display area112may not necessarily contain the entirety of the rendered content118presented in the user interface202. In some implementations, audio output may be provided to a user102with regard to portions of the user interface202not presented in the display area112, and a user102may interact with portions of the user interface202not presented in the display area112. For example, a user102may manipulate a pointer124associated with an input device106to indicate portions of the user interface202outside of the display area112.

One or more input devices106, such as a mouse device, touchpad, trackball, joystick, touch sensor, SNP, head tracker, and so forth, may be used to move a pointer124within the display area112. In some implementations, the pointer124may include a visible cursor depicted within the display area112. In other implementations, the pointer124may not be visible. For example, the pointer124may simply include a point or group of points within a user interface202that is currently indicated or selected by an input device106. Continuing the example, a head tracker or a camera may be used to determine one or more points within a display area112indicated by a user102without necessarily generating a visible cursor. As another example, a user102may contact or provide a near-touch to a touchscreen to indicate one or more points displayed to the user102without generating a visible cursor. In implementations where the display area112is not used or where no display device is present, visible cursors would not be generated by any input device106.

As described previously, in some implementations, the rendered content118may exceed the boundaries of the display area112. For example,FIG. 2depicts a portion of the rendered content118extending below the lower edge206of the display area112. In other implementations, the rendered content118may extend beyond other edges of the display area112. The portion of the content contained within the display area112may be changed by manipulating a scroll bar212or by providing other types of interaction using one or more input devices106. For example, certain keystrokes of a keyboard, button movements of a mouse device, or gestures using a touchpad or touch sensor may be used to scroll through rendered content118. WhileFIG. 2depicts a vertically-oriented scroll bar212, in other implementations, a horizontally-oriented scroll bar212may be used to manipulate content in a horizontal direction. The rendered content118may include a content top214and a content bottom216, which may define the boundaries of the content. In some implementations, the upper edge204and the lower edge206of the display area112may define the content top214and the content bottom216, respectively. In other implementations, one or more of the content top214or the content bottom216may be positioned external to the display area112. The content top214and content bottom216may be used to determine a content length L, indicated by the arrow at the left side ofFIG. 2, associated with the rendered content118. The content length L may include a physical length between the content top214and the content bottom216, such as a number of pixels or a linear measurement. In other implementations, the content length L may include a number of lines, a number of words, a number of characters, or a quantity of data (e.g., a number of bytes).

The first feature126(1) and the second feature126(2) are shown within the portion of the content presented in the display area112. A third feature126(3), shown as a section containing product reviews, may be included in a portion of the user interface202not contained in the display area112. A fourth feature126(4), shown as a hidden or suppressed feature, such as content that only becomes visible (e.g., mouse-over content) when a pointer124is placed within a threshold distance of the fourth feature126(4), is also shown within the display area112. As another example, hidden or suppressed features126may include buttons or links that are not visibly presented in the display area112but may be accessed when selected using an input device106, such as by use of a button on a mouse device or keyboard. As yet another example, hidden or suppressed features126may include content accessible by screen reading software. Positioning of a pointer124within a threshold distance of such a feature126may cause screen reading software to output simulated speech intended for receipt by a user102having impaired vision, while the feature remains invisible in the display area112. As described previously, with regard toFIG. 1, the feature determination module128may determine the position of the pointer124and of one or more features126within the user interface202. In some implementations, the feature determination module128may determine the position of one or more features126relative to the pointer124. For example, the pointer124may be positioned a first distance D1from the first feature126(1) and a second distance D2from the second feature126(2). In some implementations, the feature determination module128may be configured to determine the distance between the pointer124and a feature126within the user interface202independent of whether that feature126is currently presented in the display area112. For example, the feature determination module128may determine that the pointer124is positioned a third distance D3from the third feature126(3), which is included in a portion of the user interface202not contained in the display area112. In some implementations, the feature determination module128may be configured to determine whether the user interface202contains any hidden features126, such as the fourth feature126(4). Hidden features126may include mouse-over content, pop-up windows, or other types of features126that may not be presented until the user102provides a particular input. For example, a particular tone may be emitted to indicate to the user102that one or more hidden features126are present in the user interface202. The feature determination module128may be configured to determine the distance D4between the pointer124and a location of the hidden fourth feature126(4) independent of whether the fourth feature126(4) is currently presented in the user interface202.

In some implementations, the feature determination module128may be configured to determine the distance D between the pointer124and the left edge208of the display area112and between the pointer124and the right edge210of the display area112. For example, the pointer124is depicted a fifth distance D5from the left edge208and a sixth distance D6from the right edge210of the display area112. In other implementations, distances D between the pointer124and the upper edge204and between the pointer124and the lower edge206may be determined. The feature determination module128may also determine distances D between the pointer124and the content top214and between the pointer124and the content bottom216. For example, the pointer124may be positioned a seventh distance D7from the content top214and an eighth distance D8from the content bottom216.

When the rendered content118is produced, the audio generation module130may determine audio data132corresponding to one or more of the content type associated with the rendered content118, the content length L, the presence of one or more features126, the position of one or more features126, the position of the pointer124, the position of the pointer124relative to one or more of the features126, the position of the pointer124relative to the content top214or content bottom216, or the position of the pointer124relative to one or more edges of the display area112. As described previously, a user102may provide user input selecting the particular elements of the rendered content118responsive to which sounds110may be emitted. A user102may also select particular sounds110to correspond to each element of the rendered content118. A user102may further select the manner in which a sound110may be modified based on one or more of the size, distance D, length L, and so forth, associated with the rendered content118or with one or more features126. The user input may be stored as configuration data136. In some implementations, the configuration data136may include standardized or preselected audio data132corresponding to one or more preselected elements, independent of user input.

For example, when the rendering engine114produces the rendered content118, the feature determination module128may determine a content type associated with the rendered content118. The audio generation module130may access audio data132corresponding to a sound110indicative of that content type, which may be determined from the configuration data136. The sound110may be output using a speaker or another output device108.

The feature determination module128may also determine the content length L associated with the rendered content118. The audio generation module130may access audio data132corresponding to a sound110indicative of a content length L. The audio generation module130may modify the audio data132using one or more audio characteristics, based on the content length L. For example, the audio generation module130may modify the amplitude, frequency, or duration of the sound110based on a value associated with content length L (e.g., a linear measurement or a quantity of data). Continuing the example, a tone having a high volume, a low frequency, or a long duration may indicate a large content length L. A tone having a low volume, a high frequency, or short duration may indicate a small content length L. In one implementation, the audio generation module130may cause emission of two sounds110, separated by an interval of time. The length of the interval of time may correspond to the content length L. The interval of time may be determined using a table or another type of data structure in which intervals of time are stored in association with content lengths L. In other implementations, the interval of time may be determined using a formula or algorithm, such as an interval of 10 ms per line of content length L.

The feature determination module128may determine the presence of one or more features126within the rendered content118. The audio generation module130may access audio data132corresponding to sounds110indicative of one or more of the features126. The presentation of the sounds110using a speaker or other output device108may inform the user102of the presence of the one or more features126within the user interface202. In some implementations, the feature determination module128may determine the position of one or more of the features126. The presented sounds110may be modified by the audio generation module130based on the determined position(s) of the features126. For example, one or more zones of the user interface202may be associated with a particular sound110or with a particular audio characteristic. Continuing the example, the presence of a feature126within the lower right quarter of a user interface202may be indicated by repeating a tone three times, by providing the tone with a longer or shorter duration, by increasing or decreasing the volume of the tone, and so forth. The feature determination module128may also determine the position of one or more features126relative to the pointer124, relative to one or more edges of the display area112, relative to the content top214or content bottom216, or relative to other features126. The audio data132selected for output and one or more audio characteristics used to modify the audio data132may be determined based on the position of one or more features126relative to other elements.

For example, the feature determination module128may determine a position of the pointer124relative to one or more features126. Based on one or more of the distance D between the pointer124and a feature126or the direction of the feature126relative to the pointer124, the sound110associated with that feature126may be modified. Continuing the example, the configuration data136may associate a tone having a frequency of 2000 Hz with the first feature126(1). When the rendering engine114produces the rendered content118, the feature determination module128may determine the position of the first feature126(1), the position of the pointer124, and the distance D1between the pointer124and the first feature126(1). The audio generation module130may determine audio data132corresponding to the 2,000 Hz tone and an audio characteristic, such as the amplitude or duration of the tone, corresponding to the distance D1. The audio data132and the audio characteristic may be determined from the configuration data136. For example, the configuration data136may indicate that the distance D1between the pointer124and the first feature126(1) may be indicated by modifying the rate of repetition of the 2,000 Hz tone. When the rendered content118is produced, the 2,000 Hz tone may be output at a first rate. As the user102moves the pointer124within the user interface202, the rate at which the tone is repeated may increase as the pointer124is moved closer to the first feature126(1). The rate at which the tone is repeated may decrease as the pointer124is moved away from the first feature126(1). In other implementations, the frequency, volume, or duration of the tone may be modified as the distance D1between the pointer124and the first feature126(1) changes.

As another example, the feature determination module128may determine the position of the third feature126(3) and the distance D3between the third feature126(3) and the pointer124. While the third feature126(3) is not currently within the display area112, the feature determination module128may determine the presence of the third feature126(3) within the user interface202. Sounds110corresponding to one or more of the presence, position, distance D3from the pointer124, or direction from the pointer124of the third feature126(3) may be presented to the user102to indicate the presence, position, distance D3, or direction of the third feature126(3).

In some implementations, one or more sounds110indicative of the position of the pointer124within the user interface202may be output. For example, a sound110corresponding to the position of the pointer124may be emitted based on the section of the display area112where the pointer124is positioned. In other implementations, sounds110may be presented based on the distance D5between the pointer124and the left edge208of the display area112or the distance D6between the pointer124and the right edge210of the display area112. As another example, sounds110may be presented based on the distance D7between the pointer124and the content top214or the distance D8between the pointer124and the content bottom216. Continuing the example, a sound110indicative of the distance D8between the pointer124and the content bottom216may indicate a portion of the rendered content118that has not yet been navigated by the user102.

FIG. 3is a block diagram300illustrating examples of rendered content118and configuration data136.

Rendered content118may include any type of data configured to be rendered by the user device104, a server, or another computing device in communication with the user device104. For example, rendered content118may include data for presenting a webpage, an electronic book or other publication, or other content for presentation to a user102. While a portion of the rendered content118may be displayed using a display device, in some implementations, use of a display device may be omitted, and a user102may navigate the rendered content118based on the sounds110presented to the user102.

The rendered content118may include one or more features126. Features126may include buttons, links, menus, text boxes, and other elements with which a user102may interact using an input device106. For example, a feature126may include a button for purchasing a product, a link to a subsequent webpage, a drop down menu, a text box for entering a search query, hidden content, and so forth. Hidden content may include a button, link, and so forth, that is not visibly presented but may be accessible when clicked or otherwise selected. For example, hidden content may become visible when the pointer124is within proximity of a hidden feature126, such as when the position of the pointer124is within the boundary of the hidden content. Features126may also include particular text, graphics, and so forth. For example, a feature126may include a section of a webpage containing user-submitted product reviews or a section of an electronic book indicating the beginning of a chapter. The presence of one or more features126and the position of the features126within the rendered content118may be determined by the feature determination module128. In some implementations, the rendered content118may include indications of one or more features126, which may be determined by the feature determination module128. For example, a DOM tree or other metadata associated with the rendered content118may indicate the presence of a particular button, link, search bar, and so forth. In other implementations, the feature determination module128may perform object recognition or text recognition to determine features126within the rendered content118.

The rendered content118may also include one or more content types302corresponding to the content. A content type302may include a general type or category associated with the rendered content118, such as an electronic book, a webpage, and so forth. A content type302may include a subcategory associated with the rendered content118, such as a webpage describing a product, depicting the contents of a user's electronic shopping cart, for purchasing a product, containing user reviews of a product, containing user account information, and so forth. A content type302may also include an indication regarding whether the rendered content118contains secured information (e.g., content designated for special handling). For example, secured information may include financial information, medical information, or other types of information intended to remain confidential. In some implementations, the content type302may include a determination regarding whether the rendered content118is accessed via a secure (e.g., Hypertext Transfer Protocol Secure (HTTPS)) connection. A user102may select one or more sounds110to be associated with one or more content types302. When rendered content118having a particular content type302is produced, the sound110corresponding to that content type302may be emitted by a speaker or other output device108. The content type302associated with rendered content118may be determined by a flag, metadata, DOM tree, or another portion of the rendered content118indicative of the nature of the rendered content118. In other implementations, the feature determination module128may perform object recognition or text recognition to determine a content type302associated with rendered content118. For example, correspondence between stored data indicative of a content type302and one or more text strings, one or more graphics, or a particular layout of graphical objects from the rendered content118may indicate the content type302associated with the rendered content118.

The rendered content118may also include a content size304. In some implementations, the content size304may include a linear measurement. For example, the content size304may include a content length L associated with the rendered content118, such as the distance D between a content top214and a content bottom216of a webpage, as shown inFIG. 2. Content size304may also include a width of rendered content118, such as the distance D between the left and right boundaries of the rendered content118. In other implementations, content size304may include a non-linear measurement. For example, non-linear measurements may include the number of pages in an electronic book or the number of pages within a chapter or section of an electronic book, a number of lines or characters in a webpage, a number of bytes of rendered data, and so forth. The content size304of the rendered content118may be determined based on a DOM tree, metadata, or other data associated with the rendered content118. In other implementations, the feature determination module128may perform object recognition or text recognition to determine a content size304associated with the rendered content118. For example, a rendered webpage may include a scroll bar212indicative of the content length L of the webpage. As another example, an electronic book may include an indication of a number of pages within the electronic book or within a chapter of the electronic book, a percentage of completed or remaining pages within the book or chapter, and so forth.

A user102may select one or more sounds110to be associated with one or more content sizes304. For example, an interval of time corresponding to a length of a webpage or the size of an electronic book may be determined. A first sound110(1) may be emitted prior to passage of the interval of time. After passage of the interval of time, a second sound110(2) may be emitted. The first sound110(1) and the second sound110(2) may be identical sounds110or different sounds110. As another example, a sound110may be emitted having a duration, amplitude, or frequency corresponding to the content size304associated with rendered content118. Continuing the example, a tone may be provided with a duration corresponding to the number of pages or lines of text remaining in a current chapter of an electronic book, such as a duration of 10 ms per page. Sounds110may be provided to the user102indicative of a portion of a content size304that has already been navigated by a user102, a portion of the content size304that has not yet been navigated by a user102, a portion of rendered content118located above or below a pointer124, a total length or size associated with rendered content118, and so forth. For example, the duration, amplitude, frequency, rate of repetition, and so forth, may indicate the number of pages within a chapter of an electronic book subsequent to a current position within the electronic book associated with a pointer124(e.g., the current page rendered for presentation by a user device104).

The rendered content118may further include one or more content boundaries306. Content boundaries306may include the beginning and ending of rendered content118, such as the first and last pages of an electronic book or of a chapter of an electronic book. Content boundaries306may include physical borders that limit scrolling or navigation of rendered content118, such as a content top214, a content bottom216, left and right edges of a user interface202, and so forth. In some implementations, the content boundaries306may be used to determine the content size304associated with rendered content118. For example, the distance D between a content top214and a content bottom216of a webpage may be used to determine a content length L of the webpage. As another example, the first and last pages of an electronic book may be used to determine the content size304associated with the electronic book (e.g., the number of pages or lines between the first and last pages).

The rendered content118may additionally include one or more navigation elements308. Navigation elements308may include portions of rendered content118that may be used to scroll or otherwise navigate the rendered content118, such as a scroll bar212, the pointer, buttons, links, and so forth. For example, rendered content118may include a scroll bar212with which a user102may interact to change the portion of the rendered content118accessible to a pointer124. As another example, the rendered content118may include an electronic book, article, webpage, or similar content that may be navigated by selecting buttons, links, or regions of the rendered content118using an input device106. As yet another example, rendered content118may be scrolled or accessed by moving a pointer124to the boundaries of a display area112, by selecting and “dragging” a portion of the content presented in a user interface202, and so forth.

Other rendered content310may include tags, flags, metadata, or other indicia that may be used to determine one or more features126, a content type302, a content size304, and so forth, associated with rendered content118. For example, other rendered content310may include a layout or arrangement of elements that may be identified by the feature determination module128to determine one or more features126, content types302, content sizes304, and so forth.

Configuration data136may include data associating one or more elements of rendered content118with audio data132corresponding to sounds110that may be emitted by a speaker responsive to determination of the elements of the rendered content118. For example, a user102may select one or more features126, content types302, or other characteristics (e.g., position of features126, relative distances D or directions, and so forth) of rendered content118, such as the content size304. The user102may also select one or more particular sounds110that may be emitted responsive to those characteristics. When rendered content118that includes a particular feature126or other element indicated in the configuration data136is presented to the user102, the corresponding sound110may be emitted to indicate the presence of that feature126or element. Characteristics of the sound110may also indicate characteristics of that feature126or element. In some implementations, a characteristic of one or more elements of the rendered content118, such as a value corresponding to a content size304or a distance D between a feature126and the pointer124may be determined. The audio data132may be modified to produce a modified sound110that indicates the characteristic of the element(s). For example, the amplitude, duration, frequency, or rate of repetition of a tone may be modified based on the distance D between a feature126corresponding to that tone and a pointer124. The particular characteristics responsive to which audio data132may be modified may be selected by a user102providing user input using one or more input devices106. In other implementations, the audio data132associated with one or more features126or other elements of rendered content118, or the modifications to audio data132that may be made responsive to determination of characteristics of the element(s), may be determined independent of user input. For example, the configuration data136may include one or more preselected or default associations between audio data132and one or more features126. As another example, the configuration data136may include a preselected or default setting in which the duration of a sound110indicative of a feature126is one ms per pixel of distance D between the feature126and the pointer124.

The configuration data136may include one or more content types302, one or more content sizes304, or one or more features126, at least a portion of which may be selected by a user102or determined independent of user input. Each content type302, content size304, or feature126stored as configuration data136may include an indication of audio data132associated therewith. The audio data132may be indicative of a sound110corresponding to the associated content type302, content size304, or feature126. For example, a first feature126(1) may be stored in association with audio data132indicative of a first sound110(1), such as a tone having a frequency of 1,200 Hz. A second feature126(2) may be stored in association with audio data132indicative of a second sound110(2), such as a tone having a frequency of 1,500 Hz. In a similar manner, tones that differ in duration, amplitude, phase, or other characteristics may be used to enable a user102to differentiate between the presence of the first feature126(1) and the second feature126(2) within a user interface202. Other types of sounds110, such as a series of tones, a recorded sound110such as a WAV file or MPEG file, and so forth, may be used. The characteristics of other types of sounds110, such as frequency, amplitude, phase, duration, and so forth, may be varied to differentiate between the presence of the first feature126(1) and the second feature126(2) within the user interface202.

The configuration data136may also include one or more size characteristics312. Size characteristics312may include a type associated with a content size304. For example, a size characteristic312may include data indicating that a determined content size304corresponds to a content length L of a webpage, a number of pages within an electronic book or other type of document, and so forth. Size characteristics312may also include a value corresponding to a determined content size304. For example, a size characteristic312may include a number of pixels, lines, characters, pages, or bytes of data associated with rendered content118.

The configuration data136may further include one or more feature characteristics314. Feature characteristics314may include a position of a feature126within a user interface202. For example, a feature126may be located within a particular portion or zone of a webpage. As another example, a feature126may be located within a distance D of an edge of a user interface202. Feature characteristics314may include a distance D of a feature126from the pointer124. The distance D between the pointer124and a feature126may change as a user102moves the pointer124within the user interface202using an input device106. Feature characteristics314may also include a status or type associated with one or more features126. For example, a feature characteristic314may include a specific type of link, a destination of the link, an indication that a link has previously been selected, and so forth. Feature characteristics314may further include a shape, dimensions, a color, or other physical characteristics of a particular feature126.

Each size characteristic312and feature characteristic314may be stored in a data store116in association with one or more audio characteristics316. An audio characteristic316may include a trait of an emitted sound110that may be modified to indicate one or more of the size characteristics312or feature characteristics314. For example, an audio characteristic316may include the frequency of a tone, note, or other type of sound110. Continuing the example, a tone having a low frequency may be emitted to indicate a first feature characteristic314or size characteristic312, such as a feature126that is a large distance D from a pointer124or content having a large content size304. A tone having a high frequency may indicate a feature126that is a short distance D from a pointer124or content having a small content size304. In some implementations, the frequency of a sound110may be modulated as the sound110is emitted. For example, the frequency of a tone may increase as a sound110is produced to indicate a feature126that is located within an upper portion of a user interface202. The frequency of the tone may decrease as the sound110is produced to indicate a feature126located within a lower portion of the user interface202.

As another example, an audio characteristic316may include the amplitude of a tone, note, or other type of sound110. Continuing the example, a tone having a high volume may indicate a feature126that is a short distance D from a pointer124or content having a large content size304. A tone having a low volume may indicate a feature126that is a large distance D from the pointer124or content having a small content size304. An audio characteristic316may also include the duration of a tone or other type of sound110. For example, a tone having a long duration may indicate a feature126that is a large distance D from a pointer124or content having a large content size304. A tone having a short duration may indicate a feature126that is a short distance D from the pointer124or content having a small content size304.

Audio characteristics316may also include the timing at which one or more sounds110are produced. For example, the rate of repetition of sound110indicative of a feature126may increase as the distance D between a pointer124and the feature126decreases, and the rate of repetition may decrease as the distance D increases. As another example, the rate of repetition of a sound110indicative of a portion of a content length L below a pointer124may increase as the pointer124moves toward the content bottom216and decrease as the pointer124moves toward the content top214. In some implementations, a sound110may be provided twice, separated by an interval of time based on a feature characteristic314or size characteristic312. In other implementations, two sounds110may be provided in succession, separated by the interval of time. For example, an interval of time corresponding to the length of a webpage may be determined. A first sound110(1) may be emitted when the webpage is rendered. A second sound110(2) may subsequently be emitted after the passage of the interval of time. Thus, a webpage having a short content length L may be indicated by two sounds110emitted in close proximity to one another. A webpage having a large content length L may be indicated by two sounds110spaced apart by a significant length of time. As another example, an interval of time corresponding to the distance D between the pointer124and a feature126may be determined. A first sound110(1) may be emitted when a webpage is rendered. A second sound110(2) may be subsequently emitted after passage of the interval of time. Thus, two sounds110emitted in close proximity to one another may indicate a feature126that is a short distance D from the pointer124. Two sounds110spaced apart by a significant length of time may indicate a feature126that is a significant distance D from the pointer124.

Any combination of audio characteristics316may be associated with one or more size characteristics312or feature characteristics314. For example, one or more of the frequency, duration, amplitude, or rate of repetition of a tone may be modified by the audio generation module130based on the distance D associated with a feature126or the length of rendered content118. The particular characteristics of audio data132that are modified and the size characteristics312or feature characteristics314responsive to which the audio data132is modified may be determined based on user input. In other implementations, particular audio characteristics316may be associated with particular size characteristics312or feature characteristics314independent of user input.

Other configuration data318may include other characteristics of rendered content118, such as characteristics of a content type302, content boundaries306, navigation elements308, and so forth, stored in association with audio characteristics316. Other configuration data318may also include threshold values relating to one or more size characteristics312or feature characteristics314. For example, audio data132generated by the audio generation module130may be modified (e.g., provided with one or more different audio characteristics316) if the distance D between a feature126and the pointer124changes by more than a threshold quantity.

FIG. 4illustrates a method400for determining a distance D between a feature126and a pointer124and presenting sounds110to a user102based on the distance D. At402, rendered content118may be produced to prepare a user interface202for presentation to a user102. For example, a rendering engine114associated with the user device104or another computing device in communication with the user device104may receive or access source content122, such as HTML, CSS, JavaScript, Java code, image files, and so forth, corresponding to a user interface202. The rendering engine114may render the rendered content118, which may induce a DOM, vector graphics, bitmapped images, and so forth, for presentation using the user device104. In some implementations, the rendered content118may be displayed on a display device. However, in other implementations, the rendered content118may be rendered without necessarily displaying the rendered content118. The user interface202may include one or more features126and a pointer124associated with one or more input devices106. The pointer124may include a visible cursor. However, in some implementations, the pointer124may include one or more positions within the user interface202indicated by or associated with an input device106independent of the generation of any visible representation of the pointer124.

At404, a distance D between a feature126of the rendered content118and the pointer124may be determined. For example, a feature determination module128associated with the user device104may determine the position of one or more features126. The feature determination module128may determine the position of the pointer124from the user interface module134. Based on the positions of the pointer124and a particular feature126of interest, the distance D between the pointer124and the particular feature126may be determined. The distance D may include a linear measurement, such as a unit of length (e.g., millimeters (mm)) or a number of pixels between the pointer124and the feature126. In other implementations, the distance D may include a qualitative measurement, such as a number of lines or characters, a percentage of the total distance across the user interface202, or a quantity of data (e.g., number of bytes), indicative of the space between the pointer124and the feature126.

At406, an interval of time corresponding to the distance D may be determined. For example, an audio generation module130associated with the user device104may access a table or other arrangement of data in which distances D are associated with intervals of time. In other implementations, the audio determination module130may determine an interval of time using an algorithm based on the distance D. Continuing the example, an algorithm may indicate an interval of time equal to 20 ms per 10 pixels of distance D between the pointer124and the feature126.

At408, a first sound110(1) corresponding to the feature126may be emitted. For example, the audio generation module130may access configuration data136to determine audio data132associated with a particular feature126. The configuration data136may also include data indicative of audio characteristics316that may be used to modify the audio data132based on one or more feature characteristics314, such as the distance D. Continuing the example, the audio data132may correspond to a tone having a frequency associated with the feature126. The audio characteristic316modified based on the distance D may include the interval of time between repetitions of one or more sounds110. Upon emission of the first sound110(1) the audio generation module130may begin determining passage of the interval of time. For example, the first sound110(1) may be emitted at a first time T(1).

At410, passage of the interval of time may be determined. For example, the audio generation module130may determine that a quantity of time corresponding to the distance D has passed subsequent to emission of the first sound110(1). One or more clocks may be associated with the user device104to determine information indicative of date, time, ticks, and so forth. Data from the clocks may be used to trigger subsequent actions.

At412, a second sound110(2) corresponding to the feature126may be emitted after passage of the interval of time. For example, the audio generation module130may determine a second time T(2) after the passage of the interval of time, after which the second sound110(2) may be emitted. In some implementations, the second sound110(2) may be identical to the first sound110(1). In other implementations, the second sound110(2) may differ from the first sound110(1). For example, the second sound110(2) may include a different frequency, amplitude, duration, and so forth. As another example, the second sound110(2) may include a different recorded sound110(e.g., a different WAV of MPEG file). The length of the interval of time may indicate the distance D between the feature126and the pointer124, analogous to a sonar ping and echo. For example, the second sound110(2) may follow closely after the first sound110(1) when the feature126and the pointer124are within close proximity of one another. The interval of time may have a greater length when the feature126and the pointer124are spaced a significant distance D from one another.

FIG. 5illustrates a method500for determining movement of a pointer124and emitting one or more sounds110responsive to the movement. At502, rendered content118may be rendered to prepare a user interface202for presentation to a user102. For example, a rendering engine114associated with a user device104may receive source content122and render the rendered content118. As described previously, the rendered content118may be displayed using a display device. However, in other implementations, the user interface202may not necessarily be displayed, and a user102may interact with the user interface202based on output from one or more output devices108, such as audio devices or haptic devices.

At504, a first distance D1between a feature126of the rendered content118and a pointer124may be determined. As described previously, a feature determination module128associated with the user device104may determine the position of one or more features126, and the user interface module134may determine the position of the pointer124. Based on the positions of the pointer124and a particular feature126of interest, a first distance D1between the pointer124and the particular feature126may be determined. In other implementations, the feature determination module128may be configured to determine a direction of the feature126relative to the pointer124.

At506, a first sound110(1) corresponding to the first distance D1may be emitted. For example, an audio generation module130may determine audio data132that corresponds to the feature126and one or more audio characteristics316that correspond to the first distance D1. As described previously with regard toFIG. 3, configuration data136may include audio data132stored in association with data indicative of one or more features126. The audio data132may correspond to one or more sounds110associated with one or more particular features126. The audio data132may be modified using one or more audio characteristics316. The audio characteristics316may be determined based on one or more feature characteristics314. For example, the distance D1between the feature126and the pointer124may constitute a feature characteristic314. Based on the value of the distance D1, one or more audio characteristics316of the audio data132that corresponds to the feature126may be used to modify the audio data132. Other feature characteristics314may include the dimensions (e.g., width or height) of a feature126, the position of the feature126within a user interface202, the direction of the feature126relative to the pointer124, and so forth. The first sound110(1) may therefore include one or more of a frequency, duration, amplitude, phase, directionality, or rate of repetition corresponding to the determined audio characteristic(s)316. Continuing the example, the first sound110(1) may have a low amplitude when the distance D1between the feature126and the pointer124is large.

At508, the pointer124may be moved responsive to user input to a second distance D2from the feature126. For example, a user102may provide input to the user device104using a mouse device, touchpad, touch sensor, joystick, SNP, head tracker, or one or more other types of input devices106. The input may be processed by the user interface module134and may be operative to move the pointer124within the user interface202. For example, responsive to the first sound110(1), the user102may move the pointer124relative to the feature126to change the distance D between the pointer124and feature126. Audio output responsive to the changed distance D may be used to determine the proximity of the feature126to the pointer124, the direction of the feature126relative to the pointer124, and so forth.

At510, the movement of the pointer124may be determined to exceed a threshold quantity of movement. As described previously with regard toFIG. 3, the configuration data136may include one or more threshold distances DT. In some implementations, movement of a pointer124a distance D less than the threshold distance DT may be disregarded. Movement of the pointer124a distance greater than the threshold distance DT may result in a determination of movement responsive to which the user device104may perform one or more actions. For example, the user interface module134, the feature determination module128, or another module associated with the user device104may determine the difference between the first distance D1and the second distance D2, which may yield the distance D moved by the pointer124. If the distance D moved by the pointer124is determined to be greater than or equal to the threshold distance DT, the user device104may perform one or more subsequent actions, as described below.

At512, responsive to the determination of movement of the pointer124, a second sound110(2) corresponding to the second distance D2may be emitted. The second sound110(2) may include one or more audio characteristics316that differ from those corresponding to the first sound110(1). For example, the second sound110(2) may be provided with an amplitude greater than that of the first sound110(1) to indicate that the second distance D2is shorter than the first distance D1. As another example, the second sound110(2) may be provided with a duration greater than that of the first sound110(1). As yet another example, the second sound110(2) may be provided with a higher frequency than that of the first sound110(1).

FIG. 6illustrates a method600for determining a direction of a feature126relative to a pointer124and presenting sounds110indicative of the determined direction. At602, rendered content118may be produced to prepare a user interface202for presentation to a user102. For example, the rendered content118may be presented to the user102via a user device104, which may include one or more output devices108, such as speakers. Continuing the example, the user device104may include at least a first output device108(1) and a second output device108(2), which may include multiple speakers of a surround-sound audio system. In other implementations, the output device(s)108may include a monaural speaker, one or more haptic devices, and so forth. The user interface202may include a pointer124associated with one or more input devices106and one or more features126, such as a text box configured for receiving search queries.

At604, a first direction606(1) of a feature126of the rendered content118may be determined relative to the pointer124. For example, a feature determination module128associated with the user device104may determine a position of the feature126, and the user interface module134may determine the position of the pointer124. Based on the positions of the pointer124and the feature126, the direction606(1) in which the feature126is located relative to the pointer124may be determined.FIG. 6depicts the feature126positioned above and toward the right of the pointer124.

At608, a first sound110(1) having one or more audio characteristics316corresponding to the first direction606(1) may be emitted. For example, an audio characteristic316may include the directionality of the sound110(1). Responsive to the determination that the feature126is located toward the right of the pointer124, the second output device108(2), which may be located to the right of the user102, may be used to provide the first sound110(1) that appears to emanate from the same direction in which the feature126is located relative to the pointer124. Other audio characteristics316of the first sound110(1), such as the frequency, amplitude, rate of repetition, and so forth, may also be modified to indicate the first direction606(1) or to indicate the distance D between the pointer124and the feature126. In other implementations, such as in the absence of speakers capable of altering the directionality of the first sound110(1), one or more of the frequency, amplitude, rate of repetition, and so forth, of the first sound110(1) may be used to indicate the first direction606(1). For example, a tone that modulates from a low frequency to a high frequency may be used to indicate a feature126that is positioned to the right of the pointer124. The rate or amount of the modulation may indicate the distance D between the pointer124and the feature126. As another example, haptic feedback may be used to indicate the first direction606(1). For example, a user102may contact multiple haptic devices, each positioned in contact with a particular extremity of the user102. Haptic devices located on one or more of a user's102right arm or right leg may be used to indicate features126located toward the right, while haptic devices located on one or more of a user's left arm or left leg may be used to indicate features located toward the left. Similarly, haptic devices located on one or more of a user's102legs may be used to indicate features located below the pointer124, while haptic devices located on one or more of a user's102arms may be used to indicate features above the pointer124.

At610, the pointer124may be moved responsive to user input. For example, a user102may use an input device106to attempt to move the pointer124toward or away from the feature126, responsive to the emitted first sound110(1).FIG. 6depicts the pointer124positioned toward the right and above the previous position of the pointer124.

At612, a second direction606(2) of the feature126relative to the pointer124may be determined. For example, based on the new position of the pointer124, the feature determination module128may determine that the feature126is located upward and toward the left of the pointer124.

At614, a second sound110(2) corresponding to the second direction606(2) may be emitted. For example, responsive to the determination that the feature126is located toward the left of the pointer124, the first output device108(1), which may be located to the left of the user102, may be used to provide a second sound110(2) that appears to emanate from the same direction in which the feature126is located relative to the pointer124. As described previously, other audio characteristics316, or haptic feedback, may also be used to indicate the second direction606(2) in place of or in addition to the directionality of the second sound110(2).

FIG. 7illustrates a method700for determining a position of a pointer124within a user interface202and presenting sounds110responsive to the position. At702, rendered content118may be produced to prepare a user interface202for presentation to a user102. As described previously, a rendering engine114associated with a user device104may receive or access source content122and render the rendered content118for presentation. The rendered content118may be presented with or without use of a display device. The user interface202may include a pointer124associated with one or more input devices106. The pointer124may be positioned at a point along the X-axis and Y-axis of the user interface202.

At704, a first position706(1) of the pointer124may be determined along a first axis (e.g., the X-axis) of the user interface202. For example, a feature determination module128associated with the user device104may determine a position of the pointer124between the left and right edges of the user interface202. The feature determination module128may determine the first position706(1) of the pointer124from the user interface module134. In other implementations, the feature determination module128may determine the first position706(1) of the pointer124based on data associated with the rendered content118, data determined from one or more input devices106, or data determined from the user device104. In other implementations, the feature determination module128may determine the first position706(1) of the pointer124using object recognition. For example, correspondence between particular text or a particular graphic within the rendered content118and stored data indicative of particular features124may indicate the presence of the features124within the rendered content118.

At708, a second position706(2) of the pointer124may be determined along a second axis (e.g., the Y-axis) of the user interface202. For example, the feature determination module128may determine the second position706(2) of the pointer124within the content length L of the rendered content118. Continuing the example, the feature determination module128may determine the second position706(2) of the pointer124relative to the content top214or the content bottom216. In other implementations, the feature determination module128may determine the second position706(2) of the pointer124from the user input module134using object recognition.

At710, a first sound110(1) associated with the first position706(1) and a second sound110(2) associated with the second position706(2) may be determined. For example, an audio generation module130associated with the user device104may access configuration data136. The configuration data136may include audio data132stored in association with one or more features126or feature characteristics314. For example, the configuration data136may include audio data132corresponding to the position of the pointer124within particular zones or regions of the user interface202, the position of the pointer124relative to one or more edges of the user interface202, the position of the pointer124along one or more axes of the user interface202, the position of the pointer124relative to one or more features126within the user interface202, and so forth. Continuing the example, the configuration data136may include audio data corresponding to tones having a frequency ranging from 2,000 Hz to 4,000 Hz associated with a first position706(1) of the pointer124along the X-axis of the user interface202. The configuration data136may include audio data132corresponding to tones having a frequency ranging from 10 Hz to 500 Hz associated with a second position706(2) of the pointer124along the Y axis of the user interface202. Based on the particular position of the pointer124along both axes, the first sound110(1) corresponding to the first position706(1) may include a 2,725 Hz tone. The second sound110(2) corresponding to the second position706(2) may include a 410 Hz tone.

At712, the first sound110(1) and the second sound110(2) may be emitted. In some implementations, audio data132corresponding to the sounds110may be generated and provided to an output device108, such as one or more speakers associated with the user device104. In other implementations, the audio data132may be determined and generated by one or more computing devices remote from the user device104, and the audio data132may be provided to the speakers by the associated computing devices to cause output of the sounds110.

FIG. 8is a flow diagram800illustrating a method for determining elements of rendered content118, such as features124, a content size304, a content type302, and so forth, and outputting sounds110responsive to the determined elements.

Block802accesses configuration data136that includes audio data132associated with one or more of a content type302, a content size304, or one or more features126. In some implementations, configuration data136may be received via user input. For example, a user102may indicate particular features126or content types302responsive to which the user102desires to receive audio feedback. In other implementations, configuration data136may include audio data132stored in association with one or more elements of the rendered content118independent of user input. For example, a particular tone or series of tones may be used to indicate a content size304in the absence of user input. The configuration data136may be used to determine particular elements of rendered content118responsive to which audio feedback may be provided. The configuration data136may also be used to determine the particular sounds110that are provided responsive to determination of the particular elements of the rendered content118. The configuration data136may further be used to determine audio characteristics316that may be used to modify audio data132responsive to determination of particular size characteristics312or feature characteristics314. For example, based on the distance D between a feature126and a pointer124or other feature characteristics314, such as the size or position of a feature126, one or more of the frequency, amplitude, phase, or rate of repetition of a sound110may be modified by the audio generation module130.

Block804presents rendered content118to a user device104. As described previously, source content122may be received from a web server120or another source. A rendering engine114associated with the user device104may output rendered content118. In some implementations, the rendered content118may be presented on a display device associated with the user device104. In other implementations, use of a display device may be omitted, and the rendered content118may be rendered without being visibly presented to a user102. For example, in the absence of a display device, the rendered content118may constitute a virtual user interface202.

Block806determines one or more of a content type302, a content size304, or one or more features126associated with the rendered content118. A feature determination module128may determine content types302, content sizes304, and features126based on a DOM tree or other metadata associated with rendered content118. In other implementations, the feature determination module128may perform object recognition or text recognition on rendered content118to determine the presence of features126and the presence of one or more indicia, such as the visible appearance of a scroll bar212or an arrangement of one or more features124, that may indicate the content type302or content size304associated with the rendered content118.

Block808accesses audio data132corresponding to the determined content type(s)302, content size(s)304, or feature(s)126. In some implementations, data corresponding to haptic output may also be accessed from the data store116. For example, correspondence between at least a portion of the content types302, content sizes304, or features126of the configuration data136and the elements identified within the rendered content118by the feature determination module128may be determined by the audio generation module130. Content types302, content sizes304, and features126of the configuration data136may have audio data132or data corresponding to haptic output associated therewith. The audio data132associated with the portions of the configuration data136that correspond to the rendered content118may be indicative of sounds110to be emitted responsive to particular content types302, content sizes304, or features126. In some implementations, the audio data132may be modified using one or more audio characteristics316. The audio characteristics316may be determined based on one or more size characteristics312or feature characteristics314determined by one or more of the feature determination module128or the audio generation module130.

Block810provides the audio data132to an output device108to produce one or more sounds110. The sounds110, including any audio characteristics316that may be used to modify the audio data132, may be used to indicate particular content types302, particular content sizes304, the presence of particular features126, or feature characteristics314of the particular features126. Responsive to the sounds110, a user102may provide user input, such as to move a pointer124within the user interface202to interact with one or more of the features126for which a sound110was produced.

FIG. 9is a block diagram900of a computing device902configured to support operation of the system100. The computing device902may include one or more user devices104, one or more servers, such as web servers120, or one or more other computing devices902in communication with the user device(s)104or server(s). The computing device902may also include one or more CRSM accessible to any of the computing devices902. In some implementations, the computing device902may include one or more input devices106. For example, one or more modules, configuration data136, audio data132, and so forth, may be stored in association with a mouse device or another type of input device106.

One or more power supplies904may be configured to provide electrical power suitable for operating the components of the computing device902. In some implementations, the power supply904may include a rechargeable battery, fuel cell, photovoltaic cell, power conditioning circuitry, and so forth.

The computing device902may include one or more hardware processor(s)906(processors) configured to execute one or more stored instructions. The processor(s)906may include one or more cores. One or more clocks908may provide information indicative of date, time, ticks, and so forth. For example, the processor(s)906may use data from the clock908to generate a timestamp, trigger a preprogrammed action, and so forth. As another example, a clock908may be used to determine the rate of repetition associated with one or more sounds110or the passage of an interval of time between subsequent emissions of sound110.

The computing device902may include one or more communication interfaces910, such as input/output (I/O) interfaces912, network interfaces914, and so forth. The communication interfaces910may enable the computing device902, or components of the computing device902, to communicate with other computing devices902or components of the other computing devices902. The I/O interfaces912may include interfaces such as Inter-Integrated Circuit (I2C), Serial Peripheral Interface bus (SPI), Universal Serial Bus (USB) as promulgated by the USB Implementers Forum, RS-232, and so forth.

The I/O interface(s)912may couple to one or more input devices106or one or more output devices108. The input devices106and output devices108may include any manner of device able to be used in association with a user device104, a server, or another computing device902in communication with a user device104or a server. For example, input devices106may include one or more pointer devices916. Pointer devices916may include any device that may be used to produce a pointer124or otherwise indicate a particular point or set of points within a user interface202. In some implementations, a pointer device916may produce a visible cursor within a display area112. In other implementations, a pointer device916may be used to indicate or interact with a particular point or set of points within a user interface202without producing visible indicia. For example, pointer devices916may include mouse devices, trackballs, touchpads, touch sensors, joysticks, SNPs, head trackers, game controllers, and so forth. Other input devices918may include devices for providing input to a computing device902independent of a pointer124. For example, other input devices918may include keyboards, buttons, microphones, scanners, cameras, other types of sensors, and so forth. Output devices108may include devices that may be used to provide visible, audible, or tactile indicia to a user102. For example, output devices108may include one or more speakers920, one or more haptic devices922, or one or more display devices924. Other output devices926may include one or more lights, printers, and so forth. Other output devices926may also include devices for providing other sensory stimuli to a user102, such as air moving devices, temperature control devices, devices for outputting scents, and so forth. For example, a fan may be used as another output device926to blow air onto the user102. In some implementations, input devices106or output devices108may be physically incorporated with the computing device902or may be externally placed.

The network interfaces914may be configured to provide communications between the computing device902and other devices, such as the input devices106, output devices108, routers, access points, and so forth. The network interfaces914may include devices configured to couple to one or more networks including local area networks (LANs), wireless LANs, wide area networks (WANs), wireless WANs, and so forth. For example, the network interfaces914may include devices compatible with Ethernet, Wi-Fi, Bluetooth, ZigBee, Z-Wave, 3G, 4G, LTE, and so forth.

The computing device902may include one or more busses or other internal communications hardware or software that allows for the transfer of data between the various modules and components of the computing device902.

As shown inFIG. 9, the computing device902may include one or more memories928. The memory928may include one or more CRSM. The CRSM may be any one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, a mechanical computer storage medium, and so forth. The memory928may provide storage of computer-readable instructions, data structures, program modules, and other data for the operation of the computing device902. A few example modules are shown stored in the memory928, although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SoC).

The memory928may include one or more operating system (OS) modules930. The OS module930may be configured to manage hardware resource devices such as the I/O interfaces912, the network interfaces914, the input devices106, the output devices108, and to provide various services to applications or modules executing on the processors906. The OS module930may implement a variant of the FreeBSD operating system as promulgated by the FreeBSD Project; UNIX or a UNIX-like operating system; a variation of the Linux operating system as promulgated by Linus Torvalds; the Windows operating system from Microsoft Corporation of Redmond, Wash., USA; or other operating systems.

A data store116and one or more of the following modules may also be stored in the memory928. The modules may be executed as foreground applications, background tasks, daemons, and so forth. The data store116may use a flat file, database, linked list, tree, executable code, script, or other data structure to store information. In some implementations, the data store116or a portion of the data store116may be distributed across one or more other devices including other computing devices902, network attached storage devices, and so forth.

A communication module932may be configured to establish communications with one or more other computing devices902, such as user devices104, servers, such as web servers120, storage media, or other computing devices902in communication therewith. The communications may be authenticated, encrypted, and so forth.

The memory928may store the rendering engine114, which may be used to receive source content122and output rendered content118. The rendering engine114may render the rendered content118for presentation. In some implementations, the rendered content118may be output using one or more output devices108, such as a display device924. In other implementations, content may be rendered to provide a virtual user interface202with which a user102may interact without necessarily providing the rendered content118to an output device108. For example, sounds110may be produced responsive to features126determined from the rendered content118independent of whether the rendered content118is provided to an output device108.

The memory928may also store the feature determination module128. The feature determination module128may determine the presence of one or more features126within rendered content118, the position of one or more features126, and feature characteristics314associated with one or more features126. For example, the feature determination module128may determine the type, size, and position of one or more features126, such as buttons, links, menus, text boxes, text, graphic elements, and so forth. The feature determination module128may also determine the position of one or more edges of a display area112or of the rendered content118. The feature determination module128may additionally determine the position706of the pointer124from the user interface module134. The feature determination module128may further determine distances D between features126, pointers124, edges, and so forth, and the directions606in which elements are located relative to other elements. In some implementations, the feature determination module128may be configured to determine features126and feature characteristics314from a DOM tree or other data associated with rendered content118. In other implementations, the feature determination module128may be configured to perform text, image, or object recognition or processing to determine the presence, position, or characteristics associated with one or more features126. For example, image processing may be performed at least in part by using one or more tools available in the OpenCV library as developed by Intel Corporation of Santa Clara, Calif., USA; Willow Garage of Menlo Park, Calif., USA; and Itseez of Nizhny Novgorod, Russia, with information available at www.opencv.org.

In some implementations, one or more of the rendering engine114or the feature determination module128may be configured to determine whether at least a portion of the rendered content118includes secure content. For example, the rendering engine114may determine whether rendered content118is accessed via a secure connection. The rendering engine114may also determine whether the content type302includes content designated for special handling (such as financial information, medical information, or other types of confidential information). The feature determination module128may determine the presence of content designated for special handling or the secure connection, such as by accessing the DOM tree corresponding to the rendered content118or performing text or object recognition on the rendered content118. Responsive to this determination, the audio generation module130may cause emission of an audible tone or another type of sound may be emitted to indicate the presence of secure content within the user interface202.

The memory928may additionally store the audio generation module130. The audio generation module130may access configuration data136to determine audio data132associated with the features126determined by the feature determination module128. The audio generation module130may further determine audio characteristics316that may be used to modify the audio data132based on feature characteristics314determined by the feature determination module128. The audio generation module130may access audio data132for output, based on the features126determined by the feature determination module128. Based on determined audio characteristics316, the audio generation module130may modify the audio data132. In some implementations, the audio generation module130may provide the audio data132, modified using determined audio characteristics316, to one or more speakers920. In other implementations, the audio data132may be prepared for presentation at a later time. For example, a webpage may be rendered, one or more features126within the webpage may be determined, and corresponding audio data132may be generated, independent of user input. The audio data132may subsequently be output at a later time when a user102accesses the webpage.

The memory928may further store the user interface module134. The user interface module134may receive user input that may be stored as configuration data136. A user102may select one or more types of content that may be of interest to the user102, and one or more sounds110that may indicate selected types of content. A user102may select one or more ranges of content sizes304that may be of interest and one or more sounds110to indicate selected content sizes304. A user102may also select one or more features126that may be of interest and one or more sounds110for use indicating selected features126. In some implementations, the user102may further select characteristics of the features126, such as the position of the features126within a user interface202(e.g., the section of a webpage where a feature126may be located), the distance or direction of a feature126relative to a pointer124, the size, shape, or color of a feature126, and so forth. The user102may also select audio characteristics316that may be used to indicate the feature characteristics314. The user interface module134may receive and process each selection provided by a user102and generate configuration data136based on the user input. The user interface module134may also receive user input provided by the pointer devices916and maintain the position of a pointer124, such as by storing coordinates indicative of the position of the pointer124within a user interface202.

Other modules934may also be present in the memory928. For example, audio processing modules may be used to receive recorded sounds110provided by a user102that may be subsequently used as audio data132for output. Compression and encryption modules may be used to encrypt, decrypt, compress, and decompress audio data132and rendered content118provided between computing devices902and modules.

Other modules934may also include text to speech modules or screen reading software, which may be used to output simulated speech using the one or more speakers920concurrently with the determination of features126within the rendered content118and the output of audio data132. In some implementations, the audio data132may be provided in a manner that does not interfere with the simulated speech. For example, the sounds110may be provided at a volume less than that of the simulated speech. As another example, the sounds110may be suppressed or delayed responsive to a determination that the screen reading software is currently generating a large quantity of simulated speech or speech communicating critical information.

Other data936may include user preferences, such as configurations and settings associated with user devices104, libraries and other resources for performing data processing, registration keys and other data that may be used to verify data provided between computing devices902, and so forth.

In different implementations, different computing devices902may have different capabilities or capacities. For example, a web server120may have significantly more processor906capability and memory928capacity compared to the processor906capability and memory928capacity of a user device104.