Patent Description:
Virtual personal assistants (VPAs) are devices that respond to user queries, which may be in the form of spoken queries, by searching for a response to the query of the user in a database, for example, the internet and providing the response to the user, often in the form of an audible response such as synthesized speech. VPAs may also respond to user commands to play audio from a specified audio source, for example, an internet radio station, or to control a smart device, for example, to turn on or off a light or change a setting of another smart device that the VPA has access to, for example, via Wi-Fi signals either directly or through an internet router of the user. Queries or commands are typically provided to a VPA by a user after the user says a wake up word or phrase, for example, "Alexa" that indicates to the VPA that the user is addressing the VPA. VPAs are becoming more prevalent with various companies providing competing devices, for example, the Echo™ VPA from Amazon, Google Home™ VPA from Google, and various devices incorporating the Siri™ application from Apple. Smart speaker systems may include functionality to both stream music or other audio content and to function as VPAs.

<CIT> discloses prior art method and apparatus for signal processing based upon characteristics of music.

The present invention relates to a method and a streaming audio player according to the independent claims. Advantageous embodiments are recited in dependent claims.

Aspects and implementations disclosed herein are not limited to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. Aspects and implementations disclosed herein are capable of being practiced or of being carried out in various ways.

Aspects and implementations disclosed herein may be applicable to a wide variety of audio players, for example, streaming audio players or smart speaker systems that may incorporate virtual personal assistant (VPA) functionality or smart speakers that communicate with a VPA. Aspects and implementations of audio players disclosed herein include functionality that renders the audio players capable of differentiating between different forms of content in audio streams and rendering the audio streams in a manner that varies based on the type of content. For example, when an audio player is providing a response to a user query or command, it may render the response with a first equalization or frequency response. When the audio player is playing music, it may render the music with a second equalization or frequency response. In some implementations, an audio player may be playing entertainment audio and responsive to detecting a wake up word or phrase may lessen the volume of the entertainment audio, await a query or command from a user, and respond to the query or command of the user prior to resuming play of the entertainment audio at the original volume.

<FIG> illustrates an exemplary streaming audio player <NUM> including an enclosure <NUM>. A graphical interface <NUM> (e.g., an OLED display) resides on the enclosure <NUM> which can provide a user with information regarding currently playing ("Now Playing") audio content (e.g., streaming music) or other information regarding the system status. A screen <NUM> conceals one or more electro-acoustic transducers <NUM> (<FIG>). The streaming audio player <NUM> also includes a user input interface <NUM>. As shown in <FIG>, the user input interface <NUM> includes a plurality of preset indicators <NUM>, which are hardware buttons in the illustrated example. The preset indicators <NUM> (numbered <NUM>-<NUM>) provide the user with easy, one press access to entities assigned to those buttons.

As illustrated in <FIG>, the user input interface <NUM> may also include one or more microphones <NUM> to receive voice queries or commands from a user. In some implementations, the one or more electro-acoustic transducers <NUM> (<FIG>) may be utilized both to render audio content and to receive voice queries or commands from a user.

With reference to <FIG>, the streaming audio player <NUM> also includes a network interface <NUM>, a processor <NUM>, audio hardware <NUM>, power supplies <NUM> for powering the various streaming audio player components, and memory <NUM>. Each of the processor <NUM>, the graphical interface <NUM>, the network interface <NUM>, the audio hardware <NUM>, the power supplies <NUM>, and the memory <NUM> are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate. VPA functionality may be included in the processor <NUM> with associated programming residing in, for example, the memory <NUM>.

The network interface <NUM> may provide either or both of a wireless interface <NUM> and a wired interface <NUM>. The wireless interface <NUM> allows the streaming audio player <NUM> to communicate wirelessly with other devices in accordance with a communication protocol such as IEEE <NUM>. The wired interface <NUM> provides network interface functions via a wired (e.g., Ethernet) connection.

Digital audio coming from network packets may be directed from the network media processor <NUM> through a USB bridge <NUM> to the processor <NUM> and run into the decoders, DSP, and eventually be played back (rendered) via the electro-acoustic transducer(s) <NUM>.

The network interface <NUM> can also include a Bluetooth low energy (BTLE) system-on-chip (SoC) <NUM> for Bluetooth low energy applications (e.g., for wireless communication with a Bluetooth enabled controller. A suitable BTLE SoC is the CC2540 available from Texas Instruments, with headquarters in Dallas, Texas.

Streamed data passes from the network interface <NUM> to the processor <NUM>. The processor <NUM> can execute instructions within the streaming audio player (e.g., for performing, among other things, digital signal processing, decoding, and equalization functions), including instructions stored in the memory <NUM>. The processor <NUM> may provide, for example, for coordination of other components of the streaming audio player <NUM>, such as control of user interfaces, or applications run by the streaming audio player <NUM>.

The processor <NUM> provides a processed digital audio signal to the audio hardware <NUM> which includes one or more digital-to-analog (D/A) converters for converting the digital audio signal to an analog audio signal. The audio hardware <NUM> also includes one or more amplifiers which provide amplified analog audio signals to the electroacoustic transducer(s) <NUM> for playback. In addition, the audio hardware <NUM> may include circuitry for processing analog input signals to provide digital audio signals for sharing with other devices.

The memory <NUM> stores information within the streaming audio player <NUM>. In this regard, the memory <NUM> may store account information, such as information regarding the audio station or channel presets.

The memory <NUM> may include, for example, flash memory and/or non-volatile random access memory (NVRAM). In some implementations, instructions (e.g., software) are stored in an information carrier. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory <NUM>, or memory on the processor). The instructions may include instructions for performing decoding (i.e., the software modules include the audio codecs for decoding the digital audio streams), as well as digital signal processing and equalization.

The network interface <NUM> provides for communication between the streaming audio player <NUM> and a controller (for example, a remote control or a smart phone or computer having an appropriate control application installed), a server connected to the Internet or cloud-based server that may contain an account database including information regarding an audio system account of a user, audio sources, and other streaming audio players <NUM> via one or more communications protocols. The network interface <NUM> may also provide for communication between the streaming audio player <NUM> and a cloud-based service, for example, Alexa Voice Service, used to obtain information in response to a query by a user to use to prepare and render an audio response to the query of the user. Communication between the network interface <NUM> and the cloud-based service may be through an internet router. The service will take an uploaded audio (voice) file, recorded by the microphone <NUM>, and will perform automatic speech recognition and natural language understanding on the voice file to provide an appropriate response. The response will be fed back to the streaming audio player <NUM>, for example, as a digital audio file. For example, a user may ask the VPA residing on the streaming audio player <NUM> what the current weather forecast is. The VPA will provide a recorded voice file with that inquiry to the voice service, and will receive therefrom a digital audio file that includes the local weather forecast for playback on the streaming audio player <NUM>.

<FIG> illustrates a user <NUM> proving a spoken query 100A (e.g., triggered by the utterance of a wake word) to a streaming audio player <NUM> including VPA functionality as disclosed herein. The streaming audio player <NUM> recognizes the spoken query 100A and accesses a cloud-based service in the cloud <NUM> via an internet router <NUM> and obtains information necessary to respond to the query 100A. The streaming audio player <NUM> receives the requested information from the cloud-based service in the cloud <NUM> via the internet router <NUM>, performs a text-to-speech transformation of the received information if the received information is not already in audio format, and provides a response 100B to the query 100A via synthesized speech. In some cases, the cloud-based service may provide the requested information in audio formation (e.g., the cloud-based service may perform text-to-speech transformation of search results). If the streaming audio player <NUM> were playing entertainment audio, for example, music when the response 100B to the query 100A was to be rendered, the response 100B to the query 100A may be rendered at an elevated volume relative to the entertainment audio. The entertainment audio may be temporarily reduced in volume or turned off during rendering of the response 100B to the query 100A.

In other implementations, VPA functionality, e.g., sending a request for information to a VPA service provider or other source of information and receiving a response to the request for information from the VPA service provider or other source of information may be performed in a device separate from a device that receives a user query or command or renders the response to the user query or command. For example, in some implementations the streaming audio player <NUM> may lack functionality to send a request for information to a VPA service provider or other source of information and receive a response to the request for information from the VPA service provider or other source of information. The streaming audio player <NUM> may thus communicate with a separate device including VPA functionality to send and receive information from the VPA service provider or other source of information.

As illustrated in <FIG>, a user <NUM> may provide a spoken query 100A to a streaming audio player <NUM>. The user <NUM> may speak a wake word to the streaming audio player <NUM> prior to providing the spoken query 100A so the streaming audio player <NUM> will interpret the spoken query 100A as one to which the user <NUM> desires a response. The streaming audio player <NUM> may relay the spoken query 100A, optionally after recording the spoken query 100A, to a VPA enabled device <NUM> (also referred to herein as simply a "VPA") having the ability to request and receive a response to the user query 100A from a VPA service provider or other source of information as described above, for example, to service provider or other source of information in the cloud <NUM>. The VPA <NUM> may receive a response to the user query from the VPA service provider or other source of information and communicate the response to the streaming audio player <NUM> for rendering. The streaming audio player <NUM> may render the response as an audio response 100B to the user <NUM> after applying appropriate equalization to the response as disclosed herein.

The VPA <NUM> may include a processor, a memory, and a network interface that may be configured or may include functionality similar to the processor <NUM>, memory <NUM>, and network interface <NUM> described above with reference to the streaming audio player <NUM>. The processor of the VPA <NUM> may implement instructions stored in the memory of the VPA <NUM> that provides for the VPA <NUM> to send a request for information to a VPA service provider or other source of information and receive a response to the request for information from the VPA service provider or other source of information as well as to receive queries from the streaming audio player <NUM> and to send responses to queries to the streaming audio player <NUM>.

Communications between the streaming audio player <NUM> and VPA <NUM> may be through a router <NUM> as illustrated in <FIG> or may be in the form of direct communication (wired or wireless) between the streaming audio player <NUM> and VPA <NUM>.

It should be understood that reference to an streaming audio player <NUM> herein includes systems in which a single component receives spoken user queries and provides audio responses to a user as well as requests and receives responses to the queries from an external source, as well as to systems as illustrated in <FIG> in which a first device (e.g., n streaming audio player <NUM>) receives user queries and renders responses to a user and a second device (e.g., a VPA <NUM>) requests and receives responses to the user queries and communicates the responses to the first device for rendering.

In accordance with some aspects and implementations, the memory <NUM> of the streaming audio player <NUM> includes instructions that when executed by the processor causes the processor to label audio streams with labels (also referred to herein as digital tags or simply tags) specific to the type of content included in the stream. For example, the processor may include a first type of digital tag in an audio stream including a VPA response to a user query or command identifying the first audio stream as such and may include a second type of digital tag in a second audio stream including music identifying the second audio stream as entertainment audio. When rendering an audio stream the audio hardware <NUM> of the streaming audio player <NUM> may apply different signal conditioning, for example, different types of equalization to the audio stream based on the type of digital tag included in the audio stream. For example, if the digital tag in an audio stream is a digital tag associated with music, the audio hardware <NUM> of the streaming audio player <NUM> may render the audio stream with a higher amplitude of bass frequencies than if the digital tag was one associated with speech. If the digital tag in an audio stream is a digital tag associated with a response to a user query made to the streaming audio player <NUM> the audio hardware <NUM> of the streaming audio player <NUM> may render the audio stream with a lesser amplitude of bass frequencies than if the digital tag was one associated with music such that the response may be easier for the user to understand.

The processor of the streaming audio player <NUM> may differentiate audio streams into more than just streams including a VPA response to a user query or command and audio streams including entertainment audio. The processor of the streaming audio player <NUM> may differentiate audio streams into further classifications, such as spoken voice, entertainment audio, chimes indicative of, for example, a doorbell ringing or a text message or phone call being received, or different types of music, for example, classical music vs. rock music. The processor may embed digital tags representative of any of these different types of audio in audio streams received at the streaming audio player <NUM> and different predetermined equalization profiles may be applied to each different type of audio based on the particular digital tag embedded in the respective audio streams. The different types of audio may include, for example, voice (e.g., text-to-speech, talk-radio, news broadcast), music, movie, audio-chime, etc. The different types of audio in the audio streams may be identified by the processor of the streaming audio player <NUM> based on one or more of the frequency profiles associated with the different types of audio that the processor of the streaming audio player <NUM> may attempt to match to audio in a particular audio stream, sources of the different types of audio, or other identifying metadata already present in the audio streams including the different types of audio.

As illustrated in <FIG> the streaming audio player <NUM> may include a parser <NUM>, a ring buffer <NUM>, a decoder <NUM>, a sample buffer <NUM>, a synchronization module (SM) <NUM>, an asynchronous sample rate converter (ASRC) <NUM>, and an equalizer <NUM>. These components may be in addition to the components illustrated in <FIG> or may be included in, for example, the processor <NUM>, audio hardware <NUM>, and/or memory <NUM> illustrated in <FIG>. At the beginning of a stream, the data (encoded audio, e.g., entertainment audio or a response to a voice request) starts to flow to the streaming audio player <NUM> where it is parsed by the parser <NUM> to identify frame boundaries. The parser <NUM> strips away any container (e.g., MP3) that encoded audio is packed in. The streaming audio player <NUM> determines the type of the encoded audio, and appends a digital tag associated with the type of the encoded audio to the packet header of the encoded audio. The parsed but still encoded data is stored in the master's ring buffer <NUM>. Next, the encoded data is decoded and a time offset is generated and affixed to the header of the audio frame and the decoded audio frames are stored in the sample buffer <NUM>. The offset is used by the synchronization module <NUM> to determine when the audio samples from the corresponding audio frame are fed into the ASRC <NUM>. The ASRC <NUM> ensures a constant sample-rate for rendering. The output of the ASRC <NUM> is fed into the equalizer <NUM>, which applies the appropriate equalization profile (as indicated by the digital tag) before it is fed to a digital to analog converter of the audio hardware <NUM> and ultimately transduced to acoustic energy by the transducer <NUM>.

In some implementations, multiple streaming audio players <NUM> may be grouped together to provide synchronized, multi-room playback. Generally, in such a group, one of the devices will serve the role of a master, and the remaining devices will operate as slaves. The master device will provide an audio stream, playback timing information, and a master clock time to the slaves. The slaves can then use the playback timing information and master clock time to reproduce the streamed audio in synchrony with the master - and with each other. The master device provides the clock data (i.e., the master device acts as a time server) to the slave devices, which then use that clock data to update their respective clocks to synchronize with that of the master device. The clock data may be provided periodically (e.g., every <NUM> to <NUM> seconds) to keep the slave devices updated and in sync with the master.

The master device also provides a "play at" time to the slave devices. This "play at" time represents the time that the devices are to start playing a first sample in an audio stream. The "play at" time may be communicated in control data that is separate from the audio stream. Every new track or stream will get a new "play at" time.

The slave devices receive the first sample in a stream and begin playback at the designated "play at time. " Since all devices have the same current clock time, they all begin playback at the same time. From there, the devices all provide playback at a constant sample rate, and, consequently, stay in sync.

For multi-room synchronization, the encoded data is immediately pulled out of the master's ring buffer and is provided to the slave playback device(s) (a/k/a slave(s)) ring buffer. From there, the slaves follow the same process as outlined above. Each slave will decode the encoded audio pulled from the master, assign an offset to the frame header, and store the decoded audio frames in their respective sample buffers. The slaves each apply their own offsets to the audio frames, but these offsets will be the same as those applied by the master since each device is receiving the same stream and is using the same decoder software. The slave devices will also use the digital tag that was appended to the audio data to apply the appropriate equalization profile to the audio. In that regard, each device may have a library of equalization profiles stored in memory - a look-up table can be used to associate the digital tag with a corresponding equalization profile. In some examples, a same tag could cause different slave devices to utilize different equalization profiles for audio content, for example, based on previous user input and selection. For example, a particular internet radio station may be rendered with an equalization profile associated with voice content on one slave device, and with an equalization profile associated with music on another slave device.

For example, as illustrated in <FIG>, a master streaming audio player <NUM> may be in communication with one or more slave streaming audio players 10A, 10B, 10C via a router <NUM>. Alternatively, as illustrated in <FIG>, the master streaming audio player <NUM> and slave streaming audio players 10A, 10B, 10C may be in direct communication with one another, for example, utilizing the network interfaces in each of the master and slave devices.

VPA functionally of the master streaming audio player <NUM> and/or slave streaming audio players 10A, 10B, 10C may be triggered by a wake word from a user that is detected by the master streaming audio player <NUM> and/or slave streaming audio players 10A, 10B, 10C, which is then followed by a voice request. The one of the master streaming audio player <NUM> and/or slave streaming audio players 10A, 10B, 10C that detects the wake word and user voice request or query 100A will record the voice request when its microphone <NUM> detects the wake word. If it is the master streaming audio player <NUM> that receives the user voice request or query 100A it may provide a synthesized voice response 100B to the user as described above.

In some instances, one of the slave streaming audio players 10A, 10B, 10C may receive the voice request. Since a user may not know which device in a group is the master streaming audio player <NUM>, or even that there is a master streaming audio player <NUM>, the user may unknowingly direct a voice request to one of the slave streaming audio players 10A, 10B, 10C (e.g., the user may simply direct a voice request to the streaming audio player that is closest to him/her). The recipient slave streaming audio player 10A, 10B, 10C could communicate the voice request to a cloud-based voice service; however, typically, a response from a cloud-based voice service is provided back over a secure socket to the same device that communicated the voice request to the cloud-based voice service - meaning the response from the voice service may be returned to the recipient slave streaming audio player 10A, 10B, 10C, which may be otherwise ill equipped to distribute the audio to the other streaming audio players. To address this, in situations in which a voice request is picked-up by a microphone <NUM> of a slave streaming audio player 10A, 10B, 10C, the slave streaming audio player can forward the corresponding audio file to the master streaming audio player <NUM> for communicating the same to the cloud-based voice service. This ensures that the response will be directed back to the master streaming audio player <NUM>, which can then label and distribute the audio to the slave streaming audio players 10A, 10B, 10C. In some instances, the master streaming audio player <NUM> may record an indication of which of the slave streaming audio players 10A, 10B, 10C forwarded the user request 100A to the master streaming audio player <NUM>, and may forward the response 100B to the same slave streaming audio player that forwarded the user request 100A to the master streaming audio player <NUM>. Alternatively, the response 100B may be sent to each slave streaming audio player 10A, 10B, 10C for rendering. The response 100B may also or alternatively be rendered by the master streaming audio player <NUM>.

One option for distributing a VPA synthesized voice response 100B to a user query 100A would be to mix it, at the master streaming audio player <NUM>, with any entertainment audio stream that may be playing, and then distribute a single, mixed audio stream for playback at the slave streaming audio players 10A, 10B, 10C. This option takes advantage of the fact that the slave streaming audio players 10A, 10B, 10C are already synchronized to the clock time of the master streaming audio player <NUM>. A problem with that option, however, is that the slave streaming audio players 10A, 10B, 10C may not be able to distinguish and separate the voice response 100B from the entertainment audio, and, thus, will not be able to apply different equalization profiles (e.g., different levels of amplification of different frequency bands) to those audio types prior to rendering.

To more easily enable slave streaming audio players 10A, 10B, 10C to distinguish different types of audio streams (e.g., VPA responses 100B v. entertainment audio) the master streaming audio player <NUM> may distribute multiple separate streams of audio to the slave streaming audio players 10A, 10B, 10C, each audio stream having its own playback timing information. The streams may be distributed in parallel. These streams may include one stream for the entertainment audio, and a separate stream for a VPA response 100B to a voice request. The slave streaming audio players 10A, 10B, 10C may already be synchronized to the clock of the master streaming audio player <NUM> for playback of the entertainment audio. However, on the slave side, each audio stream may be processed separately (e.g., each stream may have its own buffer, decoder, asynchronous sample rate converter (ASRC), and equalization profile), allowing for different equalization to be applied to the different streams. The processing of the two streams can be done in parallel. Since the slave streaming audio players 10A, 10B, 10C would generally be unaware of the content type or source of the content in an audio stream, the master streaming audio player <NUM> can label the streams with the corresponding content type to ensure that the appropriate equalization is applied by the slave streaming audio players 10A, 10B, 10C prior to rendering the content. For example, the master streaming audio player <NUM> may include an identification of the audio content type in the header of audio packets provided to the slave streaming audio players 10A, 10B, 10C. The slave streaming audio players 10A, 10B, 10C may be pre-programmed with different equalization profiles (e.g., different amplification factors for different frequencies in an audio stream) to apply to different types of audio content based on the identification of the audio content type provided in the audio streams communicated from the master streaming audio player <NUM>. The slave streaming audio players 10A, 10B, 10C may be pre-programmed with different volumes at which to render different types of audio content based on the identification of the audio content type provided in the audio streams communicated from the master streaming audio player <NUM>. The slave streaming audio players 10A, 10B, 10C may be pre-programmed to alter the volume of a first audio stream being rendered upon receipt of a second type of audio stream so the second type of audio stream may be audible over the first. For example, the slave streaming audio players 10A, 10B, 10C may be pre-programmed to reduce the volume of entertainment audio being rendered when a VPA response 100B is received and while the VPA response 100B is being rendered so that the VPA response 100B may be audible over the entertainment audio.

In other examples, the different equalization profiles and/or volume adjustments may be applied by the master streaming audio player <NUM> to the different types of audio streams prior to sending the different types of audio streams to the slave streaming audio player(s) 10A, 10B, 10C for rendering. For example, the master streaming audio player <NUM> may apply equalization emphasizing lower frequencies in an audio stream identified as including rock music and may apply equalization emphasizing higher frequencies in an audio stream identified as including voice or a VPA response 100B. In such examples, the audio streams received at the slave streaming audio player(s) 10A, 10B, 10C for rendering may already have had appropriate equalization applied thereto by the master streaming audio player <NUM> and the slave streaming audio player(s) 10A, 10B, 10C may not need to check for tags identifying the type of audio in the audio streams or to apply audio type specific equalization the received audio streams.

Implementations are not limited to sending only two types of audio streams (e.g., entertainment audio and VPA responses 100B) from a master streaming audio player <NUM> to slave streaming audio players 10A, 10B, 10C. In some examples a master streaming audio player <NUM> may synchronize and send additional audio streams with different identification labels to slave streaming audio players 10A, 10B, 10C. One example of an additional type of audio stream may be audio chimes, for example, indicators of a doorbell being rung or of an incoming telephone call or text message. The slave streaming audio players 10A, 10B, 10C can follow the same rules as described above with respect to the VPA responses 100B to apply different equalization profiles to audio streams labeled as audio chimes than to audio streams labeled as entertainment audio. In other examples, a prioritization hierarchy may be defined in memories of the slave streaming audio players 10A, 10B, 10C (or the master streaming audio player <NUM>) for the different types of audio. Based on the prioritization hierarchy, an audio stream including a first type of audio, for example, an audio chime or a VPA response 100B may be rendered at a higher volume than a concurrently received audio stream including a second type of audio, for example, music that may be considered less important than the first type of audio.

Having thus described several aspects of at least one implementation, it is to be appreciated various alterations, modifications, and improvements will readily occur to those skilled in the art. The acts of methods disclosed herein may be performed in alternate orders than illustrated, and one or more acts may be omitted, substituted, or added. One or more features of any one example disclosed herein may be combined with or substituted for one or more features of any other example disclosed. Accordingly, the foregoing description and drawings are by way of example only.

Claim 1:
A method comprising:
receiving a user query (100A) spoken by a user at a microphone (<NUM>) of a streaming audio player (<NUM>);
rendering a speech response (100B) to the user query with the streaming audio player with a first equalization profile applied to the response to the user query; and
rendering entertainment audio with the streaming audio player with a second equalization profile different than the first equalization profile applied to the entertainment audio,
wherein the streaming audio player identifies a wake word spoken by the user and preceding the user query so that the streaming audio player interprets the user query as one to which the user desires a response, and
wherein the streaming audio player lowers a volume of the entertainment audio responsive to identifying the wake word.