Patent Description:
Access to electronic media, such as music and video content, has expanded dramatically over time. As a departure from physical media, media content providers stream media to electronic devices across wireless networks, improving the convenience with which users can digest and experience such content.

Some devices are enabled to receive voice commands as a means of controlling media playback. When two or more devices are in proximity with each other, the device receiving the voice commands may also receive audio output, such as a media item, that is playing on the other device. Thus, the audio output from a second device can make it more challenging for a first device to accurately identify and interpret voice commands.

There is a need for systems and methods of generating a cleaned version of ambient sound such that voice commands and/or wake words can be identified at a first device, even while media content is played back at a second device. For example, when the second device is located close to the first device, the media content that is played back at the second device contributes to ambient sound that is picked up by the first device. The first device generates a cleaned version of the ambient sound so that the cleaned version can be analyzed and voice commands received by the first device can be detected and identified. In some embodiments, the first device tracks the media content that is played back at the second device so that the first device is enabled to subtract audio signals produced by the media content playing on the second device from the ambient sound that is detected at the first device. Subtracting audio signals produced by media content playing on a second device from sound detected at a first device improves detection of voice commands by the first device. Improving detection of voice commands reduces incorrect detection of voice commands or missed detection of voice commands, which in turn reduces the amount of processing power required for detecting and interpreting voice commands. By improving the accuracy of detecting voice commands (e.g., as inputs from a user), the performance of the device is improved by allowing faster recognition of the inputs and reducing the processing load of the device when performing recognition.

In accordance with some embodiments, a method is performed at a first electronic device. The first electronic device has one or more processors and memory storing instructions for execution by the one or more processors. The method includes, while a media content item is emitted by a second electronic device that is remote from the first electronic device, receiving data that includes: timing information, offset information that indicates a difference between an initial position of the media content item and a current playback position of the media content item, and an audio stream that corresponds to the media content item. The method also includes detecting ambient sound that includes sound corresponding to the media content item emitted by the second electronic device and generating a cleaned version of the ambient sound, including using the timing information and the offset information to align the audio stream with the ambient sound and performing a subtraction operation to substantially subtract the audio stream from the ambient sound.

In some embodiments, the timing information includes a clock signal. In some embodiments, the timing information includes an indication of latency between the second electronic device and the first electronic device.

In some embodiments, the offset information includes an indication of a time duration between the initial position of the media content item and the current playback position of the media content item.

In some embodiments, the offset information includes an indication of a data amount that corresponds to the difference between the initial position of the media content item and the current playback position of the media content item.

In some embodiments, the timing information is received from the second electronic device.

In some embodiments, the timing information is received from a first server.

In some embodiments, the offset information is received from the second electronic device.

In some embodiments, the offset information is received from a first server.

In some embodiments, the audio stream has a lower data rate than the media content item stream that is provided to the second electronic device.

In some embodiments, the audio stream is received from the second electronic device.

In some embodiments, the audio stream is received from a first server. In some embodiments, the audio stream is received from a second server distinct from a first server.

In some embodiments, the timing information is embedded in the audio stream.

In some embodiments, the method includes analyzing the cleaned version of the ambient sound to determine whether a command is present in the ambient sound.

In some embodiments, the first electronic device is not playing the media content item.

In some embodiments, the first electronic device is playing the media content item.

In accordance with some embodiments, a first electronic device (e.g., a server system, a client device, etc.) includes one or more processors and memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for, while a media content item is emitted by a second electronic device that is remote from the first electronic device, receiving data that includes: timing information, offset information that indicates a difference between an initial position of the media content item and a current playback position of the media content item, and an audio stream that corresponds to the media content item. The one or more programs also include instructions for detecting ambient sound that includes sound corresponding to the media content item emitted by the second electronic device and generating a cleaned version of the ambient sound, including using the timing information and the offset information to align the audio stream with the ambient sound and performing a subtraction operation to substantially subtract the audio stream from the ambient sound.

In accordance with some embodiments, a computer-readable storage medium has stored therein instructions that, when executed by an electronic device, cause the first electronic device to, while a media content item is emitted by a second electronic device that is remote from the first electronic device, receive data that includes: timing information, offset information that indicates a difference between an initial position of the media content item and a current playback position of the media content item, and an audio stream that corresponds to the media content item. The instructions further cause the first electronic device to detect ambient sound that includes sound corresponding to the media content item emitted by the second electronic device and generate a cleaned version of the ambient sound, including using the timing information and the offset information to align the audio stream with the ambient sound and performing a subtraction operation to substantially subtract the audio stream from the ambient sound.

Thus, systems are provided with improved methods for providing information associated with media content items based on a proxy media content item.

The embodiments disclosed herein are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. Like reference numerals refer to corresponding parts throughout the drawings and specification.

Reference will now be made to embodiments, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide an understanding of the various described embodiments.

It will also be understood that, although the terms first, second, etc. are, in some instances, used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first electronic device could be termed a second electronic device, and, similarly, a second electronic device could be termed a first electronic device, without departing from the scope of the various described embodiments. The first electronic device and the second electronic device are both electronic devices, but they are not the same electronic device.

The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "includes," "including," "comprises," and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

<FIG> is a block diagram illustrating a media content delivery system <NUM>, in accordance with some embodiments. The media content delivery system <NUM> includes one or more electronic devices <NUM> (e.g., electronic device <NUM>-<NUM> to electronic device <NUM>-m, where m is an integer greater than one), one or more media content provider servers <NUM>, and/or one or more content delivery networks (CDNs) <NUM>. The one or more media content provider servers <NUM> are associated with (e.g., at least partially compose) a media-providing service. The one or more CDNs <NUM> store and/or provide one or more content items (e.g., to electronic devices <NUM>). In some embodiments, the one or more CDNs <NUM> are associated with the media-providing service. In some embodiments, the CDNs <NUM> are included in the media content provider servers <NUM>. One or more networks <NUM> communicably couple the components of the media content delivery system <NUM>. In some embodiments, the one or more networks <NUM> include public communication networks, private communication networks, or a combination of both public and private communication networks. For example, the one or more networks <NUM> can be any network (or combination of networks) such as the Internet, other wide area networks (WAN), local area networks (LAN), virtual private networks (VPN), metropolitan area networks (MAN), peer-to-peer networks, and/or ad-hoc connections.

In some embodiments, an electronic device <NUM> is associated with one or more users. In some embodiments, an electronic device <NUM> is a personal computer, mobile electronic device, wearable computing device, laptop computer, tablet computer, mobile phone, feature phone, smart phone, digital media player, a speaker, television (TV), digital versatile disk (DVD) player, and/or any other electronic device capable of presenting media content (e.g., controlling playback of media items, such as music tracks, videos, etc.). Electronic devices <NUM> may connect to each other wirelessly and/or through a wired connection (e.g., directly through an interface, such as an HDMI interface). In some embodiments, an electronic device <NUM> is a headless client. In some embodiments, electronic devices <NUM>-<NUM> and <NUM>-m are the same type of device (e.g., electronic device <NUM>-<NUM> and electronic device <NUM>-m are both speakers). Alternatively, electronic device <NUM>-<NUM> and electronic device <NUM>-m include two or more different types of devices. In some embodiments, electronic device <NUM>-<NUM> (e.g., or electronic device <NUM>-<NUM>) includes a plurality (e.g., a group) of electronic devices. For example, electronic device <NUM>-<NUM> comprises two or more speakers (e.g., or a combination of two or more electronic devices, such as a speaker, a TV, and a smart phone). In some embodiments, the two or more speakers are communicatively coupled to each other (e.g., to synchronize playback over the two or more speakers).

In some embodiments, electronic devices <NUM>-<NUM> and <NUM>-m send and receive media-control information through network(s) <NUM>. For example, electronic devices <NUM>-<NUM> and <NUM>-m send media control requests (e.g., requests to play music, movies, videos, or other media items, or playlists thereof) to media content provider server <NUM> through network(s) <NUM>. Additionally, electronic devices <NUM>-<NUM> and <NUM>-m, in some embodiments, also send indications of media content items to media content provider server <NUM> through network(s) <NUM>. In some embodiments, the media content items are uploaded to electronic devices <NUM>-<NUM> and <NUM>-m before the electronic devices forward the media content items to media content provider server <NUM>.

In some embodiments, electronic device <NUM>-<NUM> communicates directly with electronic device <NUM>-m (e.g., as illustrated by the dotted-line arrow), or any other electronic device <NUM>. As illustrated in <FIG>, electronic device <NUM>-<NUM> is able to communicate directly (e.g., through a wired connection and/or through a short-range wireless signal, such as those associated with personal-area-network (e.g., BLUETOOTH / BLE) communication technologies, radio-frequency-based near-field communication technologies, infrared communication technologies, etc.) with electronic device <NUM>-m. In some embodiments, electronic device <NUM>-<NUM> communicates with electronic device <NUM>-m through network(s) <NUM>. In some embodiments, electronic device <NUM>-<NUM> uses the direct connection with electronic device <NUM>-m to stream content (e.g., data for media items) for playback on the electronic device <NUM>-m.

In some embodiments, electronic device <NUM>-<NUM> and/or electronic device <NUM>-m include a media application <NUM> (<FIG>) that allows a respective user of the respective electronic device to upload (e.g., to media content provider server <NUM>), browse, request (e.g., for playback at the electronic device <NUM>), and/or present media content (e.g., control playback of music tracks, videos, etc.). In some embodiments, one or more media content items are stored locally by an electronic device <NUM> (e.g., in memory <NUM> of the electronic device <NUM>, <FIG>). In some embodiments, one or more media content items are received by an electronic device <NUM> in a data stream (e.g., from the CDN <NUM> and/or from the media content provider server <NUM>). In some embodiments, the electronic device(s) <NUM> are capable of receiving media content (e.g., from the CDN <NUM>) and presenting the received media content. For example, electronic device <NUM>-<NUM> may be a component of a network-connected audio/video system (e.g., a home entertainment system, a radio/alarm clock with a digital display, and/or an infotainment system of a vehicle). In some embodiments, the CDN <NUM> sends media content to the electronic device(s) <NUM>. In some embodiments, the CDN <NUM> sends media content to the electronic device(s) <NUM> in response to receiving a request (e.g., from one or more electronic device(s) <NUM> and/or from media content provider server <NUM>).

In some embodiments, the CDN <NUM> stores and provides media content (e.g., media content requested by the media application <NUM> of electronic device <NUM>) to electronic device <NUM> via the network(s) <NUM>. Content (also referred to herein as "media items," "media content items," and "content items") is received, stored, and/or served by the CDN <NUM>. In some embodiments, content includes audio (e.g., music, spoken word, podcasts, etc.), video (e.g., short-form videos, music videos, television shows, movies, clips, previews, etc.), text (e.g., articles, blog posts, emails, etc.), image data (e.g., image files, photographs, drawings, renderings, etc.), games (e.g., <NUM>- or <NUM>-dimensional graphics-based computer games, etc.), or any combination of content types (e.g., web pages that include any combination of the foregoing types of content or other content not explicitly listed). In some embodiments, content includes one or more audio media items (also referred to herein as "audio items," "tracks," and/or "audio tracks").

In some embodiments, media content provider server <NUM> receives media requests (e.g., commands) from electronic devices <NUM>. In some embodiments, media content provider server <NUM> includes a voice API <NUM> (e.g., voice recognition module <NUM>, <FIG>), a connect API <NUM> (e.g., Network Communication Module <NUM>, <FIG>), and/or key service <NUM> (e.g., key database <NUM>, <FIG>). In some embodiments, media content provider server <NUM> validates (e.g., using key service) electronic devices <NUM> by exchanging one or more keys (e.g., tokens) with electronic device(s) <NUM>.

In some embodiments, media content provider server <NUM> and/or CDN <NUM> stores one or more playlists (e.g., information indicating a set of media content items). For example, a playlist is a set of media content items defined by a user and/or defined by an editor associated with a media-providing service. The description of the media content provider server <NUM> as a "server" is intended as a functional description of the devices, systems, processor cores, and/or other components that provide the functionality attributed to the media content provider server <NUM>. It will be understood that the media content provider server <NUM> may be a single server computer, or may be multiple server computers. Moreover, the media content provider server <NUM> may be coupled to CDN <NUM> and/or other servers and/or server systems, or other devices, such as other client devices, databases, content delivery networks (e.g., peer-to-peer networks), network caches, and the like. In some embodiments, the media content provider server <NUM> is implemented by multiple computing devices working together to perform the actions of a server system (e.g., cloud computing).

<FIG> is a block diagram illustrating an electronic device <NUM> (e.g., electronic device <NUM>-<NUM> and/or electronic device <NUM>-m, <FIG>), in accordance with some embodiments. The electronic device <NUM> includes one or more central processing units (CPU(s), i.e., processors or cores) <NUM>, one or more network (or other communications) interfaces <NUM>, memory <NUM>, and one or more communication buses <NUM> for interconnecting these components. The communication buses <NUM> optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.

In some embodiments, the electronic device <NUM> includes a user interface <NUM>, including output device(s) <NUM> and/or input device(s) <NUM>. In some embodiments, the input devices <NUM> include a keyboard, mouse, or track pad. Alternatively, or in addition, in some embodiments, the user interface <NUM> includes a display device that includes a touch-sensitive surface, in which case the display device is a touch-sensitive display. In electronic devices that have a touch-sensitive display, a physical keyboard is optional (e.g., a soft keyboard may be displayed when keyboard entry is needed). In some embodiments, the output devices (e.g., output device(s) <NUM>) include a speaker <NUM> (e.g., speakerphone device) and/or an audio jack <NUM> (or other physical output connection port) for connecting to speakers, earphones, headphones, or other external listening devices. Furthermore, some electronic devices <NUM> use a microphone and voice recognition device to supplement or replace the keyboard. Optionally, the electronic device <NUM> includes an audio input device (e.g., a microphone <NUM>) to capture audio (e.g., speech from a user).

Optionally, the electronic device <NUM> includes a location-detection device <NUM>, such as a global navigation satellite system (GNSS) (e.g., GPS (global positioning system), GLONASS, Galileo, BeiDou) or other geo-location receiver, and/or location-detection software for determining the location of the electronic device <NUM> (e.g., module for finding a position of the electronic device <NUM> using trilateration of measured signal strengths for nearby devices).

In some embodiments, the one or more network interfaces <NUM> include wireless and/or wired interfaces for receiving data from and/or transmitting data to other electronic devices <NUM>, a media content provider server <NUM>, a CDN <NUM>, and/or other devices or systems. In some embodiments, data communications are carried out using any of a variety of custom or standard wireless protocols (e.g., NFC, RFID, IEEE <NUM>. <NUM>, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth, ISA100.11a, WirelessHART, MiWi, etc.). Furthermore, in some embodiments, data communications are carried out using any of a variety of custom or standard wired protocols (e.g., USB, Firewire, Ethernet, etc.). For example, the one or more network interfaces <NUM> include a wireless interface <NUM> for enabling wireless data communications with other electronic devices <NUM>, and/or or other wireless (e.g., Bluetooth-compatible) devices (e.g., for streaming audio data to the electronic device <NUM> of an automobile). Furthermore, in some embodiments, the wireless interface <NUM> (or a different communications interface of the one or more network interfaces <NUM>) enables data communications with other WLAN-compatible devices (e.g., electronic device(s) <NUM>) and/or the media content provider server <NUM> (via the one or more network(s) <NUM>, <FIG>).

In some embodiments, electronic device <NUM> includes one or more sensors including, but not limited to, accelerometers, gyroscopes, compasses, magnetometer, light sensors, near field communication transceivers, barometers, humidity sensors, temperature sensors, proximity sensors, range finders, and/or other sensors/devices for sensing and measuring various environmental conditions.

Memory <NUM> includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices; and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory <NUM> may optionally include one or more storage devices remotely located from the CPU(s) <NUM>. Memory <NUM>, or alternately, the non-volatile memory solid-state storage devices within memory <NUM>, includes a non-transitory computer-readable storage medium. In some embodiments, memory <NUM> or the non-transitory computer-readable storage medium of memory <NUM> stores the following programs, modules, and data structures, or a subset or superset thereof:.

<FIG> is a block diagram illustrating a media content provider server <NUM>, in accordance with some embodiments. The media content provider server <NUM> typically includes one or more central processing units/cores (CPUs) <NUM>, one or more network interfaces <NUM>, memory <NUM>, and one or more communication buses <NUM> for interconnecting these components.

Memory <NUM> includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory <NUM> optionally includes one or more storage devices remotely located from one or more CPUs <NUM>. Memory <NUM>, or, alternatively, the non-volatile solid-state memory device(s) within memory <NUM>, includes a non-transitory computer-readable storage medium. In some embodiments, memory <NUM>, or the non-transitory computer-readable storage medium of memory <NUM>, stores the following programs, modules and data structures, or a subset or superset thereof:.

In some embodiments, the media content provider server <NUM> includes web or Hypertext Transfer Protocol (HTTP) servers, File Transfer Protocol (FTP) servers, as well as web pages and applications implemented using Common Gateway Interface (CGI) script, PHP Hyper-text Preprocessor (PHP), Active Server Pages (ASP), Hyper Text Markup Language (HTML), Extensible Markup Language (XML), Java, JavaScript, Asynchronous JavaScript and XML (AJAX), XHP, Javelin, Wireless Universal Resource File (WURFL), and the like.

Each of the above identified modules stored in memory <NUM> and <NUM> corresponds to a set of instructions for performing a function described herein. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or otherwise re-arranged in various embodiments. In some embodiments, memory <NUM> and <NUM> optionally store a subset or superset of the respective modules and data structures identified above. Furthermore, memory <NUM> and <NUM> optionally store additional modules and data structures not described above. In some embodiments, memory <NUM> stores one or more of the above identified modules described with regard to memory <NUM>. In some embodiments, memory <NUM> stores one or more of the above identified modules described with regard to memory <NUM>.

Although <FIG> illustrates the media content provider server <NUM> in accordance with some embodiments, <FIG> is intended more as a functional description of the various features that may be present in one or more media content provider servers than as a structural schematic of the embodiments described herein. In practice, and as recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. For example, some items shown separately in <FIG> could be implemented on single servers and single items could be implemented by one or more servers. In some embodiments, media content database <NUM> and/or metadata database <NUM> are stored on devices (e.g., CDN <NUM>) that are accessed by media content provider server <NUM>. The actual number of servers used to implement the media content provider server <NUM>, and how features are allocated among them, will vary from one implementation to another and, optionally, depends in part on the amount of data traffic that the server system handles during peak usage periods as well as during average usage periods.

<FIG> is a block diagram illustrating a first system architecture for generating a cleaned version of ambient sound, in accordance with some embodiments. In the illustrative example of <FIG>, information (e.g., clock sync, audio position, and/or audio buffer) is received from electronic device <NUM>-<NUM> and used for subtracting audio corresponding to a media content item playing at electronic device <NUM>-<NUM> from a signal received by a microphone of electronic device <NUM>-<NUM>.

In some embodiments, electronic device <NUM>-<NUM> receives, from electronic device <NUM>-<NUM>, information about a media content item. Electronic device <NUM>-<NUM> uses the information to perform a subtraction on ambient sound picked up by microphone <NUM> such that the audio signals output from electronic device <NUM>-<NUM> are at least partially removed from the ambient sound picked up by microphone <NUM>. After performing the subtraction, the detectability of a wake word and/or voice command uttered by a user of electronic device <NUM>-<NUM> is improved (e.g., detected with improved accuracy).

Electronic device <NUM>-<NUM> receives information associated with audio signals that are produced by electronic device <NUM>-<NUM>, including: clock synchronization information, audio position information (e.g., the current playback position of media content playing back at electronic device <NUM>-<NUM>), and an audio buffer (e.g., including data corresponding to media content that is playing back at electronic device <NUM>-<NUM>). Electronic device <NUM>-<NUM> uses the received clock information and audio position information (current playback position) to calculate an offset between a first time at which media content that is playing back at electronic device <NUM>-<NUM> is output by electronic device <NUM>-<NUM> and a second time at which media content that is playing back at electronic device <NUM>-<NUM> is received by electronic device <NUM>-<NUM>. Electronic device <NUM>-<NUM> uses the calculated offset to perform a subtraction, where the audio signal corresponding to the media content playing at electronic device <NUM>-<NUM> is subtracted from the audio signal received by the microphone of electronic device <NUM>-<NUM> (e.g., the ambient sound).

In some embodiments, electronic device <NUM>-<NUM> communicates (e.g., over network(s) <NUM>) with media content provider server <NUM>. For example, communications of electronic device <NUM>-<NUM> with media content provider server <NUM> use a "Connect API" <NUM> (e.g., Network Communication Module <NUM>) and/or a "Key Service" <NUM> (e.g., Key Database <NUM>). In some embodiments, media content provider server <NUM>, using the Connect API <NUM>, sends information, such as a uniform resource identifier (URI) to electronic device <NUM>-<NUM> to establish a connection between media content provider server <NUM> and electronic device <NUM>-<NUM>. In some embodiments, electronic device <NUM>-<NUM> sends, to media content provider server <NUM>, a request for one or more decryption key(s). In some embodiments, media content provider server <NUM> sends one or more decryption keys (from Key Service <NUM>) to electronic device <NUM>-<NUM> to establish a secure communications channel between the media content provider server and the electronic device. In some embodiments, media content provider server <NUM> uses decryption keys to verify a user account associated with electronic device <NUM>-<NUM> (e.g., to authorize and/or provide access to media content received from CDN <NUM>). In some embodiments, after connecting (e.g., via URI) with media content provider server <NUM>, electronic device <NUM>-<NUM> is granted access to media content stored at CDN <NUM>. For example, electronic device <NUM>-<NUM> is able to request and/or receive media content from CDN <NUM>.

In some embodiments, electronic device <NUM>-<NUM> sends (via network(s) <NUM>), to media content provider server <NUM> and/or CDN <NUM>, a request for media content. In some embodiments, media content provider server <NUM> and/or CDN <NUM> determines whether electronic device <NUM>-<NUM> is authorized to access the media content. For example, electronic device <NUM>-<NUM> is authorized to access the media content after connecting (e.g., exchanging decryption keys) with media content provider server <NUM>.

In response to receiving the request for media content and determining that the electronic device <NUM>-<NUM> is authorized, CDN <NUM> sends media content to the electronic device <NUM>-<NUM> for playback. In some embodiments, media content provider server <NUM> sends an instruction to CDN <NUM> to send the media content to electronic device <NUM>-<NUM>.

In some embodiments, electronic device <NUM>-<NUM> is communicatively coupled (e.g., using network communication module(s) <NUM>, <FIG>) with electronic device <NUM>-<NUM>. In some embodiments, electronic device <NUM>-<NUM> is a speaker (e.g., or other mobile device) that includes a microphone. In some embodiments, electronic device <NUM>-<NUM> is a speaker used to playback the content received from CDN <NUM>.

In some embodiments, electronic device <NUM>-<NUM> and electronic device <NUM>-<NUM> perform a handshake to establish a communications channel between the electronic devices. In some embodiments, electronic device <NUM>-<NUM> and electronic device <NUM>-<NUM> exchange, over the communications channel, clock synchronization information (e.g., to match timing between the electronic devices), audio position information (e.g., representing a current playback position of a media content item), and/or audio buffer information (e.g., storing media content as chunks of data). For example, the clock synchronization information is transmitted to enable electronic device <NUM>-<NUM> to align a current time with electronic device <NUM>-<NUM> and/or establish a delay along a communication channel between electronic device <NUM>-<NUM> and electronic device <NUM>-<NUM>. In some embodiments, electronic device <NUM>-<NUM> uses the clock synchronization information to determine a current real time (e.g., shared by electronic device <NUM>-<NUM> and <NUM>-<NUM>) and uses the audio position information to calculate an offset between the electronic device <NUM>-<NUM> and <NUM>-<NUM>. For example, the offset represents the difference from a first time at which a particular position in a media content item is emitted from a speaker <NUM> of electronic device <NUM>-<NUM> and a second time at which the particular position in the media content item is picked up by a microphone of electronic device <NUM>-<NUM>. Using the calculated offset, the electronic device <NUM>-<NUM> performs a subtraction of the audio signals that are emitted by electronic device <NUM>-<NUM> (e.g., as represented by the data in the received audio buffer) from the ambient sound that electronic device <NUM>-<NUM> receives. In some embodiments, the electronic device <NUM>-<NUM> applies a filter on the subtracted audio signals in order to at least partially filter out the audio signal corresponding to the received media content item (e.g., as represented by the data in the audio buffer) from the signal picked up by the microphone of electronic device <NUM>-<NUM> (e.g., acoustic echo cancellation (AEC) is applied to the signal picked up by the microphone of electronic device <NUM>-<NUM>). For example, electronic device <NUM>-<NUM> uses the calculated offset to align the audio buffer data in time with the audio signals emitted from electronic device <NUM>-<NUM> as included in the ambient sound received at electronic device <NUM>-<NUM>.

In some embodiments, the audio position information comprises a current playback time (in milliseconds (ms)) measured from a beginning of the media content item that is presented at electronic device <NUM>-<NUM>. For example, the audio position information represents how much time has passed since the start of presentation of the media content item. In some embodiments, the audio position information comprises a byte offset. For example, when a same version of media content is sent to both electronic devices, a byte offset representing a change in a size of data of the media content. In some embodiments, the byte offset is measured from a beginning (e.g., a full size) of the media content item.

In some embodiments, the audio buffer includes at least a portion of the media content item that is playing at electronic device <NUM>-<NUM>. In some embodiments, the audio buffer includes an encrypted version of at least a portion of the media content item. In some embodiments, electronic device <NUM>-<NUM> decrypts the received encrypted audio buffer. The electronic device streams the audio buffer (e.g., the decrypted audio buffer) to a digital signal processor (e.g., CPU) of the electronic device <NUM>-<NUM>. The electronic device <NUM>-<NUM> reduces the audio signal corresponding to the received media content item that is included in the signal (e.g., ambient sound) picked up by the microphone of electronic device <NUM>-<NUM>. For example, electronic device uses the data from the audio buffer to apply a filter in order to at least partially filter out the audio signal corresponding to the received media content item (e.g., as represented by the data in the audio buffer) from the signal picked up by the microphone of electronic device <NUM>-<NUM> (e.g., acoustic echo cancellation (AEC) is applied to the signal picked up by the microphone of electronic device <NUM>-<NUM>). In this way, the audio signal corresponding to the received media content item is attenuated, allowing a wake word and/or voice command to be more readily identified in the signal picked up by the microphone of electronic device <NUM>-<NUM>.

For example, electronic device <NUM>-<NUM> uses the calculated offset to filter a signal associated with media content played back by electronic device <NUM>-<NUM> (at least partially) out of the ambient sound. In this context, AEC involves recognizing an emitted signal (e.g., the sound from electronic device <NUM>-<NUM>) that appears with some delay in the received signal (e.g., the ambient sound that is detected by electronic device <NUM>-<NUM>). The recognized emitted signal can be removed (e.g., at least partially) by subtracting it from the received signal. In some embodiments, a digital signal processor (e.g., of electronic device <NUM>-<NUM>) is used to perform the subtraction. This allows electronic device <NUM>-<NUM> to generate a cleaned version of the ambient sound to make it easier to detect a wake word that may be present in the ambient sound. Using the cleaned version of the ambient sound improves the detection of user inputs (e.g., wake words and/or voice commands) by allowing electronic device <NUM>-<NUM> to perform a faster and/or more accurate recognition of the inputs, thus improving the technical performance of electronic device <NUM>-<NUM>.

In some embodiments, electronic device <NUM>-<NUM> calculates a delay (e.g., using delay calculation module <NUM>) of the communications channel over which electronic device <NUM>-<NUM> and electronic device <NUM>-<NUM> communicate. For example, the electronic device <NUM>-<NUM> uses the clock synchronization information received from the electronic device <NUM>-<NUM> to calculate the delay. In some embodiments, the handshake between the electronic devices is used to calculate the delay. In some embodiments, the delay calculation is performed periodically (e.g., to refresh and/or adapt a time delay over time). For example, the delay calculation is performed at intervals of <NUM> millisecond - <NUM> milliseconds, such as intervals of <NUM> milliseconds - <NUM> milliseconds, e.g., every <NUM>. In some embodiments, the delay calculation is performed at fixed intervals. In some embodiments, the delay calculation is performed at a rate that varies with time. For example, a higher frequency of sampling is performed after the initial handshake, and the frequency of the sampling decreases over time. In some embodiments, after the initial handshake, the delay calculation is performed with a first frequency (e.g., every <NUM>) and, after a period of time has passed (e.g., a predetermined period of time, such as e.g., <NUM> minutes after the handshake is performed), the delay calculation is performed with a second frequency that is lower than the first frequency (e.g., the delay calculation is performed every <NUM>). In some embodiments, the delay calculation is performed with a periodicity determined using data obtained about a network connection (e.g., a network connection between electronic device <NUM>-<NUM> and <NUM>-<NUM>) to maintain synchronization between the electronic devices.

In some embodiments, electronic device <NUM>-<NUM> presents (e.g., plays) media content received from CDN <NUM>. In some embodiments, electronic device <NUM>-<NUM> receives (e.g., from electronic device <NUM>-<NUM>) an audio position (e.g., timing information) of the media content that is presented at electronic device <NUM>-<NUM>, but electronic device <NUM>-<NUM> does not present (e.g., playback) the media content. For example, electronic device <NUM>-<NUM> is aware of the media content that is playing at electronic device <NUM>-<NUM> such that electronic device <NUM>-<NUM> performs audio filtering (e.g., subtraction) of the audio from electronic device <NUM>-<NUM> and is better able to detect voice commands received at the electronic device <NUM>-<NUM>.

In some embodiments, electronic device <NUM>-<NUM> also receives and plays media content that is the same as the media content that is presented at electronic device <NUM>-<NUM>. In some embodiments, electronic device <NUM>-<NUM> synchronizes playback with electronic device <NUM>-<NUM> (e.g., such that both electronic devices <NUM>-<NUM> and <NUM>-<NUM> present the media content simultaneously).

In some embodiments, electronic device <NUM>-<NUM> compresses (e.g., using compression module <NUM>) the media content received from CDN <NUM> and sends the compressed version of the media content to electronic device <NUM>-<NUM>. For example, when electronic device <NUM>-<NUM> is not playing back the media content, the bandwidth required to send audio information related to the media content item to the electronic device <NUM>-<NUM> is reduced by sending a compressed version of the audio item. In some embodiments, the compressed version of the media content item comprises a lower bitrate than the uncompressed version of the media content item. Sending the compressed version of the audio item improves the performance of the system <NUM> because less bandwidth is used across the channel between the electronic device <NUM>-<NUM> and electronic device <NUM>-<NUM>. In some embodiments, receiving a compressed version of the audio item improves the performance of electronic device <NUM>-<NUM> by reducing the processing load of the electronic device <NUM>-<NUM>. For example, electronic device <NUM>-<NUM> selectively decodes only a portion (e.g., chunks) of the compressed version of the audio item instead of processing the uncompressed version of the media content item. In some embodiments, the electronic device <NUM>-<NUM> forwards the media content item (e.g., without performing compression) to electronic device <NUM>-<NUM>.

In some embodiments, electronic device <NUM>-<NUM> uses the information about the media content item to subtract the media content item playing at electronic device <NUM>-<NUM> from the signal picked up by microphone <NUM> of electronic device <NUM>-<NUM>, such that detectability of a wake word and/or voice command uttered by a user of electronic device <NUM>-<NUM> is improved (e.g., detected with improved efficiency and/or accuracy). For example, a microphone <NUM> of electronic device <NUM>-<NUM> picks up audio signals produced by electronic device <NUM>-<NUM> (e.g., while a wake word and/or voice command are uttered by a user of electronic device <NUM>-<NUM>). Electronic device <NUM>-<NUM> receives information including clock synchronization information (e.g., calculated time delay), the audio position information (e.g., the current playback position), and the audio buffer (e.g., what media content is playing) associated with audio signals that are produced by electronic device <NUM>-<NUM>. Electronic device <NUM>-<NUM> uses the received information to subtract the media content playing at electronic device <NUM>-<NUM> from the audio signal received by the microphone of electronic device <NUM>-<NUM> (e.g., by performing a subtraction). By subtracting (e.g., and filtering) media content playing at electronic device <NUM>-<NUM> from the audio signal received by the microphone <NUM> of electronic device <NUM>-<NUM>, electronic device <NUM>-<NUM> creates a cleaned version of ambient sound detected by electronic device <NUM>-<NUM> (e.g., the media content playing at electronic device <NUM>-<NUM> is at least partially removed from the sound picked up by the microphone <NUM> of electronic device <NUM>-<NUM>). The cleaned version of the ambient sound is analyzed to determine wake words and/or voice commands. In this way, determination of wake words and/or voice commands received by microphone <NUM> of electronic device <NUM>-<NUM> is improved (e.g., in accuracy).

In some embodiments, in accordance with a determination that a wake word is detected in a signal received by microphone <NUM> of electronic device <NUM>-<NUM>, a determination is made regarding whether a voice command is received following the wake word. For example, the signal received by microphone <NUM> is transmitted by electronic device <NUM>-<NUM> to media content provider server <NUM> and voice API <NUM> of media content provider server <NUM> analyzes the received signal to determine a wake word and/or voice command included in the signal. In some embodiments, electronic device <NUM>-<NUM> analyzes the received signal to determine a wake word. In some embodiments, in accordance with a determination that a wake word is detected in the signal received by microphone <NUM>, electronic device <NUM>-<NUM> proceeds to record and/or transmit to media content provider server <NUM> audio that is received after detection of the predefined command. In some embodiments, the audio that is received by electronic device <NUM>-<NUM> is recorded and stored at electronic device <NUM>-<NUM>. In some embodiments, the audio that is received by electronic device <NUM>-<NUM> is forwarded to voice API <NUM> of media content provider server <NUM>, and voice API <NUM> records and stores the audio. For example, the electronic device <NUM>-<NUM> detects (e.g., receives) a wake word, (e.g., "Hey, Spotify") and, in response to detecting the wake word, the electronic device <NUM>-<NUM> transmits to media content provider server <NUM> audio that is received after detection of "Hey, Spotify," such as a voice command (e.g., "Play," "Pause," "Next track," "Play the song 'Yesterday' in the kitchen," "Play Madonna in the kitchen," etc.).

In some embodiments, electronic device <NUM>-<NUM> locally processes (e.g., using voice recognition module <NUM> and/or a voice API running on electronic device <NUM>-<NUM>) a signal received by microphone <NUM> to determine whether a wake word has been received. In some embodiments, in accordance with a determination that a wake word has been received, electronic device <NUM>-<NUM> records and/or transmits a signal received by microphone <NUM> following the wake word to media content provider server <NUM>. In some embodiments, electronic device <NUM>-<NUM> transmits a signal received by microphone <NUM> to media content provider server <NUM> and wake word module <NUM> of media content provider server <NUM> determines whether a wake word has been received. In some embodiments, in accordance with a determination that a wake word has been received, voice recognition module <NUM> of media content provider server <NUM> determines whether a voice command has been received. For example, the voice API <NUM> (e.g., voice recognition module <NUM>) of media content provider server <NUM> processes (e.g., parses) the signal received by microphone <NUM> to identify a media request corresponding to the voice command. In some embodiments, in accordance with a determination that a media request corresponding to the voice command is identified, the media content provider server <NUM> sends an instruction associated the media request. In some embodiments, the media content provider server <NUM> sends the instruction to electronic device <NUM>-<NUM> for execution. In some embodiments, the media content provider server <NUM> sends the instruction to electronic device <NUM>-<NUM>. In some embodiments, the media content provider server <NUM> sends the instruction to CDN <NUM> (e.g., the media content provider server <NUM> transmits an instruction to CDN <NUM> to transmit a media content item to electronic device <NUM>-<NUM> and/or electronic device <NUM>-<NUM>).

In some embodiments, an instruction generated by media content provider server <NUM> includes a target device (e.g., a target device identified in the voice command). For example, in response to receiving a voice command to "Play 'Yesterday' in the kitchen," the media content provider server generates an instruction to initiate playback of the song "Yesterday" at electronic device <NUM>-<NUM> (located in the kitchen). In some embodiments, the media content provider server sends an instruction to a first electronic device (e.g., electronic device <NUM>-<NUM>) and the first electronic device sends the instruction to a second electronic device (electronic device <NUM>-<NUM>). In some embodiments, the media content provider server <NUM> sends the command to a plurality of electronic devices and/or CDN <NUM>.

<FIG> is a block diagram illustrating a second system architecture for generating a cleaned version of ambient sound, in accordance with some embodiments. In the illustrative example of <FIG>, information (e.g., track playback, clock sync, and/or audio position) is received from media content provider server <NUM> and used for subtracting audio corresponding to a media content item playing at electronic device <NUM>-<NUM> from a signal received by a microphone of electronic device <NUM>-<NUM>. After performing the subtraction, the detectability of a wake word and/or voice command uttered by a user of electronic device <NUM>-<NUM> is improved (e.g., detected with improved efficiency and/or accuracy). For example, using the cleaned (e.g., subtracted) version of the ambient sound improves the detection of user inputs (e.g., wake words and/or voice commands) by allowing electronic device <NUM>-<NUM> to perform a faster and/or more accurate recognition of the inputs, thus improving the technical performance of electronic device <NUM>-<NUM>.

In this architecture, CDN <NUM> sends content to electronic device <NUM>-<NUM> and to electronic device <NUM>-<NUM>. In some embodiments, electronic devices <NUM>-<NUM> and <NUM>-<NUM> receive timing information (e.g., clock sync) from media content provider server <NUM> (e.g., NTP <NUM>). In some embodiments, electronic device <NUM>-<NUM> transmits audio position information to media content provider server <NUM> and electronic device <NUM>-<NUM> receives the audio position information to media content provider server <NUM>. In some embodiments, the timing information and/or audio position information are embedded in the media content (e.g., received by electronic device <NUM>-<NUM> and/or electronic device <NUM>-<NUM> from CDN <NUM>). For example, timing information and/or audio position information are encoded as a watermark within the media content item (e.g., within the audio buffer) (e.g., as audio steganography where the information is concealed within the audio file such that a user does not detect a change in the playback of the audio file). For example, electronic device <NUM>-<NUM> and electronic device <NUM>-<NUM> process the audio signal to extract the information that is encoded in the audio signal. In some embodiments, the timing information and/or audio position are encoded at fixed intervals within the media content item.

In some embodiments, a secure communications channel between media content provider server <NUM> and electronic device <NUM>-<NUM> is established (e.g., as described with reference to <FIG>). For example, electronic device <NUM>-<NUM> communicates (e.g., over network(s) <NUM>) with media content provider server <NUM>, including a "Connect API" (e.g., Network Communication Module <NUM>) and/or "Key Service" <NUM> (e.g., Key Database <NUM>). In some embodiments, media content provider server <NUM>, using the Connect API <NUM>, sends a uniform resource identifier (URI) to electronic device <NUM>-<NUM> to establish a connection between media content provider server <NUM> and electronic device <NUM>-<NUM>. In some embodiments, after connecting (via URI) with media content provider server <NUM>, electronic device <NUM>-<NUM> is granted access to media content stored at CDN <NUM>. For example, electronic device <NUM>-<NUM> is able to request and/or receive media content from CDN <NUM>.

In some embodiments, electronic device <NUM>-<NUM> and/or electronic device <NUM>-<NUM> sends a request for media content to media content provider server <NUM> and/or CDN <NUM>. In response to the request for media content, media content provider server <NUM> and/or CDN <NUM> sends the requested content to electronic device <NUM>-<NUM> and electronic device <NUM>-<NUM> (e.g., media content provider server <NUM> sends an instruction to CDN <NUM> to send the requested content).

In some embodiments, CDN <NUM> sends the same content to electronic device <NUM>-<NUM> and electronic device <NUM>-<NUM>. In some embodiments, CDN <NUM> sends a compressed (e.g., smaller) version of content to electronic device <NUM>-<NUM> and an uncompressed (e.g., larger and/or at least partially uncompressed) version of the content to electronic device <NUM>-<NUM>. In some embodiments, the compressed version of the media content item has a lower bitrate than the uncompressed version of the media content item. For example, electronic device <NUM>-<NUM> receives the uncompressed version of the content such that electronic device <NUM>-<NUM> is enabled to present a high definition (e.g., high bitrate, such as <NUM> kbit/second) version of the media content item. Electronic device <NUM>-<NUM> decodes the compressed version (e.g., with a lower bitrate, such as <NUM> kbit/second) of the content to determine audio signals (e.g., to be subtracted) corresponding to a current playback position of the media content playing at electronic device <NUM>-<NUM>. Transmitting a compressed version of the content to electronic device <NUM>-<NUM> reduces the bandwidth consumed by transmission of the content, while providing sufficient information to allow electronic device <NUM>-<NUM> to determine (e.g., using a clock sync) a current playback position of media content and the audio signals of the media content (e.g., for the current playback position)playing at electronic device <NUM>-<NUM>. By determining a current playback position and the audio signals of media content playing at electronic device <NUM>-<NUM>, electronic device <NUM>-<NUM> is enabled to generate a signal (e.g., corresponding to the decoded compressed version of the audio) for subtracting the media content playing at electronic device <NUM>-<NUM> from a signal detected by microphone <NUM> of electronic device <NUM>-<NUM>.

Electronic device <NUM>-<NUM> receives, via microphone <NUM>, ambient sound. For example, electronic device <NUM>-<NUM> receives audio signals that include voices (e.g., voice command and/or background conversation) and/or media content (e.g., that is playing at electronic device <NUM>-<NUM>). In some embodiments, electronic device <NUM>-<NUM> is configured to track (e.g., using clock sync and audio position information) the media content that is playing at electronic device <NUM>-<NUM>. Electronic device <NUM>-<NUM> uses the timing information (e.g., to calculate an offset and/or latency between the electronic device <NUM>-<NUM> and electronic device <NUM>-<NUM>) and the audio position information related to the media content played back by electronic device <NUM>-<NUM> (e.g., the audio position information received from media content provider server <NUM>) to filter media content audio emitted by electronic device <NUM>-<NUM> from the signal that is received microphone <NUM> of the electronic device <NUM>-<NUM> (e.g., as described with regard to <FIG>). For example, the electronic device <NUM>-<NUM> generates a cleaned version of the ambient audio by at least partially removing audio corresponding to the media content item playing at electronic device <NUM>-<NUM> from the ambient audio such that voices are more detectable in the ambient audio that is analyzed by electronic device <NUM>-<NUM>.

In some embodiments, electronic device <NUM>-<NUM> receives track playback information, indicating what media content is playing on electronic device <NUM>-<NUM>, from media content provider server <NUM> (e.g., a Track Playback API (TPAPI)). In some embodiments, the track playback information includes data that represents audio signals that are emitted by electronic device <NUM>-<NUM>. In some embodiments, electronic device <NUM>-<NUM> transmits, to media content provider server <NUM> (e.g., TPAPI <NUM>) the track playback information that indicates the media content being played at electronic device <NUM>-<NUM>. In some embodiments, the track playback information enables the media content provider server <NUM> to resolve playback of media content items (e.g., tracks) with electronic devices <NUM>. For example, both electronic device <NUM>-<NUM> and electronic device <NUM>-<NUM> agree on which media content item is playing based on the track playback information received from media content provider server <NUM>. Using the clock sync and audio position information received from media content provider server <NUM>, the electronic device <NUM>-<NUM> performs a subtraction of the audio signals that are emitted by electronic device <NUM>-<NUM> (e.g., as represented by the data in the received track playback) from the ambient sound that electronic device <NUM>-<NUM> receives.

In some embodiments, electronic device <NUM>-<NUM> transmits received audio to a voice API <NUM> of media content provider server <NUM> that corresponds to voice recognition module <NUM>, <FIG>). In some embodiments, electronic device <NUM>-<NUM> processes received audio using voice recognition module <NUM> before transmitting the audio to voice API <NUM>. For example, electronic device <NUM>-<NUM> performs a subtraction (e.g., using filtering module <NUM>) to generate a cleaned version of the audio and detects a wake word (e.g., using wake word module <NUM>), and the electronic device <NUM>-<NUM> sends the cleaned version of the audio (e.g., the detected wake word and/or the audio (voice command) received after detection of the wake word) to Voice API <NUM>. In some embodiments, voice API <NUM> is used by media content provider server <NUM> to detect and/or interpret a wake word and/or a voice command (e.g., as described above with reference to <FIG>).

<FIG> are flow diagrams illustrating a method <NUM> for generating a cleaned version of ambient sound, in accordance with some embodiments. Method <NUM> may be performed (<NUM>) at a first electronic device (e.g., electronic device <NUM>-<NUM>), the electronic device having one or more processors and memory storing instructions for execution by the one or more processors. In some embodiments, the method <NUM> is performed by executing instructions stored in the memory (e.g., memory <NUM>, <FIG>) of the electronic device. In some embodiments, the method <NUM> is performed by a combination of the server system (e.g., including media content provider server <NUM> and/or CDN <NUM>) and an electronic device (e.g., electronic device <NUM>-<NUM>). The method <NUM> provides an improvement in the performance of the first electronic device by improving its ability to detect of user inputs (e.g., wake words and/or voice commands). For example, the electronic device performs a faster and/or more accurate recognition of the inputs on a cleaned version of ambient sound because at least a portion of the noise present in the ambient sound has been detected and removed (e.g., subtracted and/or filtered).

Referring now to <FIG>, in performing the method <NUM>, while a media content item is emitted by a second electronic device (e.g., electronic device <NUM>-<NUM>) that is remote from the first electronic device (e.g., electronic device <NUM>-<NUM>) (<NUM>), the first electronic device receives (<NUM>) data that includes timing information (e.g., clock sync, <FIG> and <FIG>), offset information (e.g., audio position, <FIG> and <FIG>) that indicates a difference between an initial position of the media content item and a current playback position of the media content item, and an audio stream (e.g., audio buffer, <FIG> and/or track playback, <FIG>) that corresponds to the media content item. For example, the second electronic device (e.g., electronic device <NUM>-<NUM>) is playing the media content item. In some embodiments, the second electronic device <NUM>-<NUM> (e.g., active device) includes a speaker and the first electronic device <NUM>-<NUM> (e.g., master device) includes a microphone. Providing the first electronic device with timing information, offset information and an audio stream that corresponds to the media content item enables the first electronic device to more accurately determine the audio signals that are emitted from the second electronic device (e.g., corresponding to the media content item). The performance of the first electronic device is improved because the first electronic device more quickly determines a current playback position of the media content item using the received information, thus reducing the processing load of the first electronic device.

In some embodiments, the first electronic device is not playing (<NUM>) the media content item. For example, as described with reference to <FIG> and <FIG>, a first electronic device (e.g., electronic device <NUM>-<NUM>) is not playing the media content item. For example, a second electronic device (e.g., electronic device <NUM>-<NUM>) emits the media content item but the first electronic device does not emit the media content item.

In some embodiments, the first electronic device is playing (<NUM>) the media content item. For example, first electronic device <NUM>-<NUM> plays the media content item while electronic device <NUM>-<NUM> also plays the media content item. In some embodiments, the first electronic device subtracts the locally played media content item from the ambient sound picked up by the microphone <NUM> of electronic device <NUM>-<NUM>. For example, media content emitted by electronic device <NUM>-<NUM> is filtered out of the audio signal picked up by the microphone <NUM> of electronic device <NUM>-<NUM> and media content emitted by electronic device <NUM>-<NUM> is filtered out of the audio signal picked up by the microphone <NUM> of electronic device <NUM>-<NUM> to generate the cleaned version of the ambient sound.

In some embodiments, the timing information includes (<NUM>) a clock signal. For example, in some embodiments, the clock signal (e.g., clock synchronization information, also referred to herein as "clock sync") is received from an NTP server (e.g., NTP <NUM>, <FIG>). In some embodiments, the timing information is sent from NTP server (e.g., NTP <NUM>) to the first electronic device (e.g., electronic device <NUM>-<NUM>) and the second electronic device (e.g., electronic device <NUM>-<NUM>). In some embodiments, the timing information is used to calculate an offset between a first time at which media content that is playing back at electronic device <NUM>-<NUM> is output by electronic device <NUM>-<NUM> and a second time at which media content that is playing back at electronic device <NUM>-<NUM> is received (e.g., as described with regard to <FIG>).

In some embodiments, the timing information includes (<NUM>) an indication of latency between the second electronic device and the first electronic device. For example, the indication of latency includes a round trip time between the second electronic device (e.g., electronic device <NUM>-<NUM>) and the first electronic device (e.g., electronic device <NUM>-<NUM>). For example, the timing information is established during a handshaking process between the first electronic device and the second electronic device. In some embodiments, the handshaking process is performed initially, periodically, and/or in response to a request (from either of the electronic devices <NUM> and/or media content provider server <NUM>).

In some embodiments, the timing information is received (<NUM>) from the second electronic device (e.g., electronic device <NUM>-<NUM>). For example, during the handshaking process between the first electronic device and the second electronic device, the second electronic device sends, to the first electronic device, the timing information (e.g., as illustrated in <FIG>). For example, the timing information (e.g., clock sync, <FIG>) includes a timestamp that indicates a time (e.g., in a universal time code (UTC)) that is currently stored at electronic device <NUM>-<NUM>.

In some embodiments, the timing information is received (<NUM>) from a first server (e.g., an network time protocol (NTP) server). For example, the clock synchronization signal is received from media content provider server <NUM> (e.g., as illustrated in <FIG>).

In some embodiments, the offset information includes (<NUM>) an indication of a time duration between the initial position of the media content item and the current playback position of the media content item. For example, electronic device <NUM>-<NUM> transmits to electronic device <NUM>-<NUM> an indication of a time duration between the start of a media content item playing at electronic device <NUM>-<NUM> and a current playback position of the media content item playing at electronic device <NUM>-<NUM>. For example, as described with regard to <FIG>, an audio position is sent from electronic device <NUM>-<NUM> to electronic device <NUM>-<NUM>.

In some embodiments, the offset information includes (<NUM>) an indication of a data amount that corresponds to the difference between the initial position of the media content item and the current playback position of the media content item (e.g., a byte offset as described with reference to <FIG>).

In some embodiments, the offset information is received (<NUM>) from the second electronic device (e.g., electronic device <NUM>-<NUM>). In some embodiments, the offset information and the timing information are received from the second electronic device. For example, as illustrated in <FIG>, the offset information (e.g., audio position) and timing information (e.g., clock synchronization) are sent from the second electronic device <NUM>-<NUM> to the first electronic device <NUM>-<NUM>.

In some embodiments, the offset information is received (<NUM>) from a first server (e.g., media content provider server <NUM>). For example, the timing information and the offset information are received from the first server. For example, as illustrated in <FIG>, the timing information (e.g., clock synchronization) and offset information (e.g., audio position) are received, by the first electronic device <NUM>-<NUM> and by the second electronic device <NUM>-<NUM>, from media content provider server <NUM>.

In some embodiments, the audio stream has (<NUM>) a lower data rate than the media content item stream that is provided (e.g., by the first server or the second server) to the second electronic device. This improves the performance of the system <NUM> by decreasing the bandwidth used to provide the audio stream. For example, the media content (e.g., audio buffer) received by the first electronic device <NUM>-<NUM> is a compressed version (e.g., a lower bitrate version) of the media content received by the second electronic device <NUM>-<NUM> from CDN <NUM>. For example, as described with reference to <FIG>, electronic device <NUM>-<NUM> compresses the media content received from CDN <NUM> and sends, to electronic device <NUM>-<NUM>, the compressed version of the media content (e.g., in an audio buffer). In some embodiments, the processing load of electronic device <NUM>-<NUM> is reduced by only processing (e.g., decoding) a portion of the received compressed version of the media content. For example, the first electronic device determines which portion of the compressed version to decode based on the received timing information and offset information.

In some embodiments, the audio stream is received (<NUM>) from the second electronic device. For example, as shown in <FIG>, the second electronic device <NUM>-<NUM> sends an audio buffer to the first electronic device <NUM>-<NUM>.

In some embodiments, the audio stream is received (<NUM>) from a first server (or CDN) (e.g., as shown in <FIG>). For example, in some embodiments, the audio stream is received from media content provider server <NUM>. In some embodiments, the first server (e.g., media content provider server <NUM>) includes CDN <NUM> and the audio stream is received from the media content provider server <NUM>.

In some embodiments, the audio stream is received (<NUM>) from a second server distinct from a first server. For example, as shown in <FIG>, the audio stream (e.g., content) is received by the electronic devices <NUM> from CDN <NUM>. In some embodiments, CDN <NUM> is distinct from the first server (e.g., media content provider server <NUM>). As explained above, in some embodiments, media content provider server <NUM> sends an instruction to CDN <NUM> that causes CDN <NUM> to send the audio stream to the electronic devices <NUM>.

In some embodiments, the timing information is embedded (<NUM>) in the audio stream. For example, timing and/or offset indication is encoded as a watermark that is superimposed over the audio stream such that it is acoustically masked by the audio stream.

The first electronic device detects (<NUM>) ambient sound that includes sound corresponding to the media content item emitted by the second electronic device. For example, the ambient sound received by a microphone of the first electronic device (e.g., microphone <NUM> of electronic device <NUM>-<NUM>) includes background noise (e.g., voices, external sources of sound) and sound output by the second electronic device (e.g., electronic device <NUM>-<NUM>) playing the media content item.

The first electronic device generates (<NUM>) a cleaned version of the ambient sound, including: using the timing information and the offset information to align (<NUM>) the audio stream with the ambient sound and performing (<NUM>) a subtraction operation to substantially subtract the audio stream from the ambient sound. For example, as explained with reference to <FIG> and <FIG>, electronic device <NUM>-<NUM> uses the timing information to determine, for a respective time period, which portion of the audio stream is output by speaker <NUM> electronic device <NUM>-<NUM> in order to identify corresponding signals in the ambient sound picked up by the microphone <NUM> of electronic device <NUM>-<NUM>. Generating the cleaned version of the ambient sound improves the detection of user inputs (e.g., wake words and/or voice commands) at the electronic device <NUM>-<NUM>. Electronic device <NUM>-<NUM> is enabled to perform a faster and/or more accurate recognition of the inputs.

In some embodiments, the electronic device <NUM>-<NUM> uses the offset information (e.g., determined from clock sync and/or audio position) to determine a difference in time between when the electronic device <NUM>-<NUM> outputs the audio stream and when the electronic device <NUM>-<NUM> receives the audio stream. The electronic device <NUM>-<NUM> uses the timing information and offset information to time-align the audio signal received by electronic device <NUM>-<NUM> with the audio emitted by the speaker of electronic device <NUM>-<NUM> (e.g., based on the offset). Using the offset information improves the cleaned version of the ambient sound by more accurately aligning the audio signal with the audio emitted by the speaker. The electronic device is enabled to more accurately subtract the audio emitted by the speaker from the audio signal and generate a cleaner version of the ambient sound. A cleaner version of the ambient sound further improves the detection of user inputs (e.g., wake words).

In some embodiments, the first electronic device uses acoustic echo cancellation (AEC) to generate the cleaned version of the ambient sound.

For example, the electronic device <NUM>-<NUM> receives, during a first time period, ambient sound. The ambient sound includes the audio signal output by electronic device <NUM>-<NUM>. Based on the determined time-alignment (e.g., for the first time period), the first electronic device determines which portion of the audio stream (e.g., that is output by the second electronic device) is received within the ambient sound during the first time period. The electronic device <NUM>-<NUM> subtracts (e.g., using a digital signal processor), from the ambient sound, the determined portion of the audio stream for the first time period to generate a clean version of the ambient sound. After subtracting the determined portion of the audio stream for the first time period, the audio stream output by the second electronic device is at least partially removed from the ambient sound to generate the cleaned version of the ambient sound.

In some embodiments, the electronic device analyzes (<NUM>) the cleaned version of the ambient sound to determine whether a command is present in them ambient sound. For example, electronic device <NUM>-<NUM>, after generating the cleaned version of the ambient sound, analyzes the cleaned version of the ambient sound to detect a voice command, such as a wake word. As explained above, the processing power of the electronic device is improved by allowing the electronic device to more quickly and more accurately recognize (e.g., analyze) voice commands.

In some embodiments, the electronic device sends, to a first server (e.g., media content provider server <NUM>) an indication that the command is present in the ambient sound. For example, the electronic device <NUM>-<NUM> forwards the cleaned version of the ambient sound to media content provider server <NUM> (e.g., voice API <NUM>). Media content provider server <NUM> is better enabled to parse the cleaned version of the ambient sound, identify the command, and cause one or more electronic devices to execute the identified command. For example, as described with regard to <FIG> and <FIG>, Voice API <NUM> receives a cleaned version of the ambient sound (e.g., including a wake word and/or voice command) from electronic device <NUM>-<NUM>, and the media content provider server <NUM> processes the voice command to cause media content provider server <NUM>, electronic device <NUM>-<NUM>, electronic device <NUM>-<NUM>, and/or CDN <NUM> to execute the command.

Although <FIG> illustrate a number of logical stages in a particular order, stages which are not order dependent may be reordered and other stages may be combined or broken out. Some reordering or other groupings not specifically mentioned will be apparent to those of ordinary skill in the art, so the ordering and groupings presented herein are not exhaustive. Moreover, it should be recognized that the stages could be implemented in hardware, firmware, software, or any combination thereof.

Claim 1:
A method (<NUM>) for generating a cleaned version of ambient sound performed by a first electronic device (<NUM>-<NUM>, <NUM>, <NUM>), the method comprising:
while a media content item is emitted (<NUM>) by a second electronic device (<NUM>-<NUM>) that is remote from the first electronic device:
receiving (<NUM>) data that includes:
timing information,
offset information that indicates a difference between an initial position of the media content item and a current playback position of the media content item, and
an audio stream that corresponds to the media content item, wherein the audio stream is a compressed version of the media content item that is emitted by the second electronic device and has (<NUM>) a lower data rate than the media content item that is emitted by the second electronic device;
detecting (<NUM>) ambient sound that includes sound corresponding to the media content item emitted by the second electronic device; and
generating (<NUM>) a cleaned version of the ambient sound, including:
using (<NUM>) the timing information and the offset information to align the audio stream with the ambient sound; and
performing (<NUM>) a subtraction operation to substantially subtract the audio stream from the ambient sound.