Patent Description:
For example, a typical microphone used to capture speech audio for a computing application is built-in to a user device, such as a smartphone, tablet or notebook computer. These microphones capture low-quality audio which exhibits, for example, low signal-to-noise ratios and low sampling rates. Even off-board, consumer-grade microphones provide poor quality audio when used in a typical audio-unfriendly physical environment.

Moreover, a user typically lacks the knowledge and/or the time to control audio processing so as to improve the intelligibility of the recorded audio during playback. Such control would involve the tuning of several individual audio processing parameters over several iterative steps. Such steps cannot be easily hardcoded due to differences in voice pitch, frequencies, etc..

Systems are desired to provide improved speech intelligibility via intuitive and efficient user control over speech audio processing.

<CIT> describes an interactive, head-mounted eyepiece with an electrically adjustable liquid lens that adjusts a focus of the displayed content for the user. The electrically adjustable lens may be of the type wherein the lens is moved back and forth with a small worm drive. Alternatively, the electrically adjustable lens may be of the type that uses two liquids, a conducting liquid and a non-conducting liquid. The liquid lens may instead be a tunable liquid crystal cell, in which a non-uniform magnetic field is used to shape the liquid crystal and adjust a focus for the user.

<CIT> describes that systems and methods disclosed herein provide for low cost hearing assistance to improve intelligible hearing for those with normal hearing and to greatly improve hearing intelligibility for those with hearing problems. One goal of the systems and methods disclosed herein is to make hearing assistance algorithms easily accessible and available by implementing such algorithms using non-dedicated hardware platforms such as non-dedicated mobile computing devices, e.g., smartphones, PDA's and the like. In exemplary embodiments, the systems and method of the present disclosure integrate hearing assistance algorithms with multi-media algorithms in an API stack (similar to the implementation of audio effects such as stereo widening and psychoacoustic bass enhancement) thereby addressing processing delay concerns.

<CIT> describes techniques and devices which include a memory configured to store audio data within a first audio zone, or a second audio zone in a layered soundfield. The memory is coupled to one or more processors and the memory is configured to store the audio data in the first audio zone and the second audio data in the layered soundfield. The one or more processors are configured to receive an interaction command to control the audio data in the first audio zone and the second audio zone in the layered soundfield, and generate one or more indicators that the interaction command was received to control the audio data, in the first audio zone or the second audio zone of the layered soundfield.

The following description is provided to enable any person in the art to make and use the described embodiments. Various modifications, however, will remain apparent to those in the art.

Embodiments described herein provide a technical solution to the technical problem of addressing poor-quality speech audio playback in a computing environment. According to some embodiments, multiple audio processing parameters are abstracted into a single, multidimensional user control. In a mixed-reality environment, such a control may support simple interaction, reduce complexity and efficiently increase intelligibility of recorded speech content.

As an initial introduction to some embodiments, <FIG> illustrates a mixed-reality environment according to some embodiments. Some embodiments may be used in conjunction with mixed-, augmented-, and/or virtual-reality systems, as well as in conventional computer display systems.

According to the example, a user is looking at a mixed-reality display while physically located within in environment <NUM>. Every object shown in <FIG> is also located in environment <NUM> (i.e., the user sees the "real" object), except for user <NUM>. The image of user <NUM> may be acquired by a camera of a remote system and provided to the mixed-reality display via a communication application (e.g., a videoconferencing application). As is known in the art, the mixed-reality display operates to insert an image of user <NUM> into the scene viewed by the current user.

Environment <NUM> includes Loudness control <NUM> and EQ control <NUM>. Controls <NUM> and <NUM> may be displayed as holographic images or in any other image format. Each of controls <NUM> and <NUM> is associated with a respective slider <NUM> and <NUM>. According to some embodiments, the current user manipulates a slider <NUM> or <NUM> using an input device (e.g., a hand, a mixed-reality controller) to set a value corresponding to the associated control <NUM> or <NUM>.

Controls <NUM> and <NUM> may be manipulated to affect the processing of speech audio being played back to the user within environment <NUM>. For example, speech audio signals of the current user or of user <NUM> may be recorded and played back in environment <NUM> such that it may be perceived by the current user. Moving either of slider <NUM> or <NUM> changes values of two or more audio processing parameters used to process the audio signals prior to playback thereof.

According to one example, Loudness control <NUM> is bipolar with selectable values ranging from -<NUM> to <NUM> in increments of one. Embodiments are not limited to this range or granularity of values. According to some embodiments, selection of values less than <NUM> (e.g., via slider <NUM>) will increase the compression (if any) applied to the speech audio signals. Compression reduces the dynamic range of the signals, causing quiet sounds to become louder, and loud sounds to become quieter. Selection of values greater than <NUM> will cause expansion of the signal (e.g., to compensate for strong compression from recording hardware). Expansion increases the dynamic range of the signal, causing quiet sounds to become quieter, and louder sounds to become louder.

Also according to an example, EQ control <NUM> is unipolar with selectable values ranging from <NUM> to <NUM> in increments of one. Embodiments are also not limited to this range or granularity of values. Increasing the value of EQ control <NUM> increases the frequency (i.e., the bandwidth) of an equalization filter applied to the input speech audio signal. In some embodiments, such control facilitates the application of an equalization filter corresponding to the speaker's voice. Increasing the value of EQ control <NUM> may also change the profile (shape) of the equalization filter. For example, as the bandwidth moves up in frequency range, the size of the equalization filter changes due to the logarithmic relationship between frequency and perception.

Embodiments are not limited to two multimodal controls, nor to a slider input metaphor. One or more multimodal controls may be displayed, and each control may comprise any suitable type of control (control knob, selectable buttons, etc.) that are or become known.

<FIG> is a block diagram of system <NUM> to process speech audio signals based on multimodal audio control values according to some embodiments. Generally, processing system <NUM> receives input speech audio signal <NUM> and generates output speech audio signal <NUM> based on control values received from a user via controls <NUM> and <NUM>. Processing system <NUM> may comprise any one or more suitable computing devices, including but not limited a desktop or laptop computer, a computer server, and a mixed-reality headset.

As shown, the single value selected using Loudness control <NUM> may determine several audio processing parameter values represented within Loudness processing component <NUM>. For example, based on a value selected by slider <NUM> of Loudness control <NUM>, processing component <NUM> may determine one or more parameter values defining a compression function, one or more parameter values defining a limiting function, one or more parameter values defining an attack function, and one or more parameter values defining a gain function. Each determination may be based on a calculation associating loudness values with one or more parameter values of each processing function.

The single value selected using EQ control <NUM> may also determine several audio processing parameter values, represented within EQ processing component <NUM>. Based on a value selected by slider <NUM> of EQ control <NUM>, processing component <NUM> may determine an equalization filter bandwidth range and an equalization filter profile. According to some embodiments, ranges of values of EQ control <NUM> correspond to respective pre-defined bandwidth ranges and filter profiles. For example, values between <NUM> and <NUM> may correspond to a first bandwidth range and a first filter profile, values between <NUM> and <NUM> may correspond to a second bandwidth range and a second filter profile, and values between <NUM> and <NUM> may correspond to a third bandwidth range and a third filter profile. In some embodiments, component <NUM> calculates the equalization filter bandwidth range and the equalization filter profile based on the selected value of EQ control <NUM>.

As illustrated by the dashed lines of <FIG>, in some embodiments component <NUM> may determine the equalization processing parameter values based at least in part on the selected value of loudness control <NUM>. Similarly, component <NUM> may determine the loudness-related processing parameter values based at least in part on the selected value of EQ control <NUM>.

<FIG> is a flow diagram of process <NUM> according to some embodiments. Process <NUM> and the other processes described herein may be performed using any suitable combination of hardware and software. Software program code embodying these processes may be stored by any non-transitory tangible medium, including a fixed disk, a volatile or non-volatile random access memory, a DVD, a Flash drive, or a magnetic tape, and executed by any number of processing units, including but not limited to processors, processor cores, and processor threads. Embodiments are not limited to the examples described below.

A mixed-reality environment is presented to a user at S310. The mixed-reality environment may be presented via execution of an application such as a videoconferencing, gaming, or other application providing audio recording and playback. The mixed-reality environment may include zero or more real objects and zero or more computer-generated images in the user's field of vision. The mixed-reality environment may be presented by a mixed-reality headset, a flat-panel display, or any suitable one or more systems.

At S320, a loudness control and an equalization control are presented to the user in the mixed-reality environment. As described with respect to <FIG>, S320 may comprise presentation of Loudness control <NUM> and EQ control <NUM> as holographic images or in any other image format.

The controls may be presented in response to a command input by the user via an input device. The command may comprise a command to open a Settings or Audio menu associated with the application. In some embodiments, the command is a hand gesture and the input device is a motion sensor.

A Loudness value is received from the user via the Loudness control at S330, and an EQ value is received from the user at S340. Continuing with the <FIG> example, the user may manipulate slider <NUM> or <NUM> using an input device (e.g., a hand, a mixed-reality controller) to set a value corresponding to the associated control <NUM> or <NUM>.

The user may input the Loudness and EQ values in response to hearing speech audio signals played back in the mixed-reality environment. For example, the user may speak and hear his own voice played back through loudspeakers, or may hear another user's (e.g., user <NUM>'s) speech audio signals being played back. These audio signals are processed based on an initial set of audio-processing parameters, an initial equalization filter bandwidth, and an initial equalization filter profile.

The user may then manipulate one or both of controls <NUM> and <NUM> in an attempt to increase the intelligibility of the played back speech audio signals. If the user chooses to manipulate only one control, only one of a Loudness value and an EQ are received at S330 and S340.

Next, at S350, values are determined for a plurality of loudness-related audio processing parameters based on the received Loudness value. As described above, the determination at S350 may also be based on the received EQ value. Examples of Loudness-related audio processing parameters for which values may be determined at S350 include but are not limited to Compression, Limiting, Attack Value and Gain. The value of a parameter may be determined at S350 based on a function associated with the parameter which takes the user-provided Loudness value as input, based on a look-up table for which the user-provided Loudness value is an index, or by any other suitable mechanism.

An equalization bandwidth and an equalization profile are determined at S360 based on the received equalization value. In some embodiments, certain ranges of equalization values correspond to respective pre-defined bandwidths and equalization profiles. Accordingly, a pre-defined bandwidth and equalization profile may be determined at S360 by identifying a range in which the received equalization value falls. In some embodiments, the equalization filter bandwidth range and the equalization filter profile are calculated based on functions which take the received equalization value as input. Such functions may also depend on the received Loudness value.

Audio processing is then applied to received speech audio signals at S370. The audio processing utilizes the values determined for various audio processing parameters at S350, and the equalization bandwidth and equalization profile determined at S360. It should be noted that if no Loudness value or equalization value is received from a user at S330 or S340, the initial speech audio signal processing and equalization scheme continues to be used.

The processed speech audio signal is played back in the mixed-reality environment at S380. Flow then returns to S330, at which point the user may again choose to manipulate the Loudness control and/or the EQ control to increase the intelligibility of the played back signal. If so, flow continues therefrom as described above.

As mentioned above, an initial audio processing scheme is applied prior to user selection of Loudness or EQ values. <FIG> illustrates system <NUM> including components for determining the initial audio processing scheme. Specifically, frequency and loudness detection component <NUM> may determine a frequency and loudness based on input signal <NUM> and pass these values to component <NUM> for determination of initial values of Compression, Limiting, Attack and Gain audio processing parameters, and to component <NUM> for determination of an initial equalization frequency bandwidth and equalization profile.

For example, component <NUM> may determine a signal-to-noise ratio (and/or decibel level) level associated with input signal <NUM> and component <NUM> changes compression, limiting and scaling values based on the detected ratio and/or level. Component <NUM> may also estimate a fundamental frequency of signal <NUM> used by component <NUM> to determine an initial equalization frequency bandwidth and equalization profile.

<FIG> is a view of head-mounted audio/video device <NUM> which may support multimodal audio controls in a mixed-reality environment according to some embodiments. Embodiments are not limited to device <NUM>.

Device <NUM> includes a speaker system for presenting spatialized sound and a display for presenting images to a wearer thereof. The images may completely occupy the wearer's field of view, or may be presented within the wearer's field of view such that the wearer may still view other objects in her vicinity. The images may be holographic.

Device <NUM> may also include sensors (e.g., cameras and accelerometers) for determining the position and motion of device <NUM> in three-dimensional space with six degrees of freedom. Data received from the sensors may assist in determining the size, position, orientation and visibility of images displayed to a wearer.

According to some embodiments, device <NUM> executes process <NUM>. <FIG> is an internal block diagram of some of the components of device <NUM> according to some embodiments. Each component may be implemented using any combination of hardware and software.

Device <NUM> includes a wireless networking component to receive and transmit application/environment data. The data may be received via execution of a communication application on device <NUM> and/or on a computing system to which device <NUM> is wirelessly coupled. The data may include remotely-recorded speech audio signals but embodiments are not limited thereto.

The sensors of device <NUM> may detect room acoustics and the position of objects within the room, as well as the position of device <NUM> within the room. The audio processing component of device <NUM> may utilize this information to process the speech audio signals generated according to some embodiments. The thus-processed audio signals are then provided to the spatial loudspeaker system of device <NUM> for playback and perception by the wearer.

As shown in <FIG>, device <NUM> may also include a graphics processor to assist in presenting images on its display. Such images may comprise mixed-reality images of multimodal audio controls as depicted in <FIG>.

<FIG> illustrates virtual machine-based system <NUM> according to some embodiments. System <NUM> may be cloud-implemented and may include any number of virtual machines, virtual servers and cloud storage instances. System <NUM> may execute an application providing mixed-reality experience and audio processing according to some embodiments.

Device <NUM> may communicate with the application executed by system <NUM> to provide recorded speech audio signals thereto. System <NUM> may receive the speech audio signals, process the signals, and provide the processed speech signals to device <NUM>.

As described above, device <NUM> may play back the signals and present one or more multimodal controls to a user in a mixed-reality environment. The user may manipulate one or more of the controls to transmit one or more control values to system <NUM>. For each of the one or more control values, system <NUM> determines two or more audio processing parameter values associated with speech intelligibility, processes speech audio signals received from device <NUM> based thereon, and returns the processed signals to device <NUM>. Device <NUM> may further process the received speech signals prior to playback, for example based on context information local to device <NUM>.

Each functional component described herein may be implemented at least in part in computer hardware, in program code and/or in one or more computing systems executing such program code as is known in the art. Such a computing system may include one or more processing units which execute processor-executable program code stored in a memory system.

The foregoing diagrams represent logical architectures for describing processes according to some embodiments, and actual implementations may include more or different components arranged in other manners. Other topologies may be used in conjunction with other embodiments. Moreover, each component or device described herein may be implemented by any number of devices in communication via any number of other public and/or private networks. Two or more of such computing devices may be located remote from one another and may communicate with one another via any known manner of network(s) and/or a dedicated connection. Each component or device may comprise any number of hardware and/or software elements suitable to provide the functions described herein as well as any other functions. For example, any computing device used in an implementation of a system according to some embodiments may include a processor to execute program code such that the computing device operates as described herein.

Claim 1:
A computing system (<NUM>) located in a physical environment (<NUM>), the computing system (<NUM>) comprising:
a mixed-reality display;
a loudspeaker system; and
one or more processing units to execute processor-executable program code to cause the computing system to:
present (S310) a mixed-reality environment on the mixed-reality display, wherein the mixed-reality environment includes real objects of the physical environment and an image of a remote user (<NUM>) acquired by a camera of a remote system;
present (S320) a first audio control and a second audio control within the mixed-reality environment;
receive (S330) a user manipulation of the first audio control within the mixed-reality environment to select a first value, the selected first value being a value of a first audio parameter associated with speech intelligibility, wherein the first audio parameter is loudness;
receive (S340) a user manipulation of a second audio control within the mixed-reality environment to select a second value, the selected second value being a value of a second audio parameter associated with speech intelligibility, wherein the second audio parameter is equalization;
determine (S350) a first two or more audio processing parameter values associated with speech intelligibility based on the selected first value, wherein the first two or more audio processing parameter values comprise a compression value and a limiting value;
determine (S360) a second two or more audio processing parameter values associated with speech intelligibility based on the selected second value and on the
selected first value, wherein the second two or more audio processing parameter values comprise an equalization filter bandwidth and an equalization filter profile;
apply (S370) audio processing to first speech audio signals based on the determined first two or more audio processing parameter values and the determined second two or more audio processing parameter values to generate second speech audio signals, wherein the first speech audio signals are recorded speech audio signals of the remote user (<NUM>); and
transmit (S380) the second speech audio signals to the loudspeaker system for playback.