Patent Description:
Methods, systems and media for controlling audio output are provided.

Devices that interact with users primarily through audio commands given by the user, and audio feedback provided to the user through a built-in speaker have recently become more popular. With such devices, users can ask questions, dictate messages, listen to written messages that are read out loud by the device, play music, and perform other tasks through audio interaction without having to interact with a graphical user interface. Some users may wish to connect such devices to external speakers, but this may limit the usefulness of the device if the external speakers are off, or otherwise not playing the audio output by the device.

From <CIT> a solution is known which relates to distributing licensed content across multiple devices.

Accordingly, new methods, systems and media for controlling audio output are provided.

The proposed solution relates to a method of claim <NUM>, a computing device of claim <NUM> and a non-transitory computer-readable medium of claim <NUM>.

In accordance with some implementations of the disclosed subject matter, mechanisms (which can include methods, systems and media) for controlling audio output are provided.

In some implementations, the mechanisms described herein can control the output of audio from a computing device that can interact with a user through an audio user interface to one or more external speakers. In some implementations, a user can request that the computing device present audio content using any suitable technique or combination of techniques. For example, the user can make the request through a voice command by speaking one or more words, through an application on another device such as a smartphone or tablet computer, through a graphical user interface of the computing device, and/or through any other suitable user interface. In some implementations the audio content can be any suitable audio content and/or can be provided from any suitable source. For example, the audio content can be content that is stored as one or more audio signals such as music or an audio book. As another example, the audio content can be content that is converted to audio from text such as an article, a message (e.g., an email, a text message, etc.), a recipe, a web page, etc. In such examples, the audio content can be available from any suitable source such as internal memory of the computing device, internal memory of another device local device (e.g., a smartphone, a tablet computer, a personal computer, network attached storage, etc.) from which the computing device can access the content, remote storage (e.g., a server) that is associated with a user of the computing device, a service that provides access to audio content (e.g., a streaming audio service, a streaming video service, etc.), a web server that provides access to web pages, and/or any other suitable source. In a more particular example, a user can speak one or more words requesting that the computing device play music from a particular artist from a particular streaming music service which the user may access through a user account.

In some implementations, the computing device can receive the audio content to be presented and determine whether to present the audio content through an internal speaker or to output the audio content for presentation by an external speaker. For example, a user can connect an audio output of the computing device to an external sound system that includes external speakers that may provide a subjectively more enjoyable audio experience for the user. In a more particular example, the user can connect a cable having a <NUM> millimeter (mm) plug at one end to a <NUM> millimeter output jack of the computing device, and can connect the other end of the cable to a jack of an audio video receiver that is connected to one or more external speakers.

In some implementations, the computing device can detect that the audio output is connected to a corresponding connector and attempt to determine whether audio that is provided to the connector is presented by external speakers. For example, the computing device can detect that the <NUM> jack is connected to something that has one or more properties that a <NUM> plug has. In some implementations, when the connector is detected, the computing device can provide the audio content to the audio output, and record a signal produced by a microphone of the computing device. In some implementations, the computing device can compare the recorded signal to the output signal to determine whether the signal that is being output is being played by any nearby (e.g., close enough to be recorded at sufficient volume) external speakers. For example, the computing device can record its surroundings with a microphone and compare the recorded signal to the music being output to the audio output to determine whether that same music has been recorded by the microphone.

In some implementations, if the audio that is recorded does not match the audio that is being provided to the audio output, the computing device can present the audio content using the internal speaker. For example, if the audio video receiver is powered down, the computing device can determine that the music being provided to the <NUM> jack is not present in the recorded audio signal, and can begin playing the music over the internal speaker.

Turning to <FIG>, an example <NUM> of a system for controlling audio output is shown in accordance with some implementations of the disclosed subject matter. In some implementations, system <NUM> can include a computing device <NUM>, which can include an output connector <NUM>. Output port <NUM> can be any suitable output connector for communicating audio information with an external sound system <NUM>. For example, output port <NUM> can be a <NUM> jack that accepts a <NUM> connector, a fiber optic audio port (e.g., that accepts a TOSLINK connector), an RCA jack, a High-Definition Multimedia Interface (HDMI) port, a Universal Serial Bus (USB) port (e.g., a USB <NUM>. x port, a USB <NUM>. x port, etc.), a THUNDERBOLT port, an Ethernet port (whether wired or wireless), any other suitable port and/or any suitable combination of ports.

In some implementations, external sound system <NUM> can be any suitable device that can receive an audio signal and output the audio signal (potentially through one or more intermediate devices) to one or more speakers. For example, external sound system <NUM> can be an audio video (AV) receiver, a speaker, an amplifier, an audio switch, an HDMI switch, any other suitable external sound system and/or any suitable combination thereof. More particularly, external sound system <NUM> can include, speakers, hardware and/or software for rendering media content such as analog and/or digital audio and/or video data, a power supply, etc. External sound system <NUM> can include various input ports for receiving audio and/or video data from various sources. Such input ports can include one or more HDMI ports, one or more component video ports, one or more composite video ports, one or more USB ports, one or more S-Video ports, one or more TOSLINK ports, one or more coaxial ports, one or more Ethernet ports (whether wired or wireless), etc. Additionally or alternatively, in some implementations, external sound system <NUM> can be configured to receive audio over any suitable wireless connection, such as over a connection to a local area network (e.g., via a connection that complies with one of the IEEE <NUM>. 11x family of protocols which are sometimes referred to as Wi-Fi connections), or a wireless connection between external sound system <NUM> and computing device <NUM>, such as an ad hoc wireless network connection, a short-range wireless communication protocol (e.g., Bluetooth, Bluetooth Low Energy, Wireless USB, etc.), etc..

In some implementations, computing device <NUM> can be connected to external sound system <NUM> by a connection <NUM>. Computing device <NUM> and external sound system <NUM> can be connected using any suitable technique(s). For example, connection <NUM> can include a suitable cable, such as a stereo cable, an RCA cable, an HDMI cable, that can be connected between output port <NUM> and a port of external audio system <NUM>. As another example, connection <NUM> can include a wireless connection, such as a connection to a local area network (e.g., via a Wi-Fi connection), or a wireless connection established between computing device <NUM> and external sound system <NUM>, such as an ad hoc wireless network connection, a short-range wireless communication protocol (e.g., Bluetooth, Bluetooth Low Energy, Wireless USB, etc.), etc. Additionally or alternatively, media streaming device <NUM> can be included as hardware and/or software of external sound system <NUM>.

In some implementations, computing device <NUM> can be coupled to a user device <NUM> using a communications link <NUM>. User device <NUM> can be any suitable device, such as a smartphone, a tablet computer, a wearable computer, a laptop computer, a personal computer, an electronic reader, a digital media receiver, a smart television, a game console, any other suitable computing device, or any suitable combination thereof. In some implementations, communications link <NUM> can include any suitable communications link, for example, as described below in connection with <FIG>. Additionally, communications link <NUM> can include a network, such as a local area network (LAN) having wired and/or wireless connections. Additionally or alternatively, communications link <NUM> can include a wireless connection between user device <NUM> and computing device <NUM>, such as an ad hoc wireless network connection, a short-range wireless communication protocol (e.g., Bluetooth, Bluetooth Low Energy, Wireless USB, etc.), etc. In some implementations, communications between user device <NUM> and computing device <NUM> over communications link <NUM> can be encrypted using any suitable encryption technique(s). For example, messages sent from user device <NUM> to computing device <NUM> can be encrypted using any suitable encryption scheme, for example, using a public key-private key pair for encryption and decryption of messages. As another example, communications over communications link <NUM> can be encrypted using Transport Layer Security (TLS).

In some implementations, user device <NUM> can include a display <NUM> for presenting a user interface to a user. In some examples described herein, user device <NUM> can be a mobile computing device such as a smartphone or a tablet computer, and display <NUM> can include a touchscreen for receiving input and displaying a user interface and/or media content to a user.

In some implementations, computing device <NUM> and/or user device <NUM> can communicate with a content discovery server <NUM> over a communications link <NUM>. Computing device <NUM> and/or user device <NUM> can communicate with content discovery server <NUM> to discover and/or browse content made available by a content delivery server <NUM>. For example, a user can use user device <NUM> to find media content to be presented by interacting with content discovery server <NUM>. In some implementations, such media content can be presented using display <NUM>, one or more speakers, an audio output (e.g., a <NUM> jack), any other suitable presentation device, or any suitable combination thereof. In some implementations, user device <NUM> can present a media content item that was selected using content discovery server <NUM>, by requesting the media content item from content delivery server <NUM> and receiving the media content item over a communications link <NUM>. In some implementations, each of content delivery server <NUM> and/or content discovery server <NUM> can be implemented on any suitable number of servers. Additionally or alternatively, in some implementations, the functions performed by content delivery server <NUM> and content discovery server <NUM> can be performed by the same device (e.g., by a common server computer). In some implementations, multiple content delivery servers can provide content to user device <NUM> in response to a request for content. For example, in response to a request for a particular media content item from user device <NUM>, content delivery server <NUM> can cause an advertisement to be provided to user device <NUM> prior to, during and/or after the requested content item is presented. In such an example, the server that provides the advertisement can be separate from the server that provides the requested media content item.

In some implementations, communications links <NUM> and/or <NUM> can include a local area network (e.g., a home network) and/or a non-local network (e.g., the Internet). For example, computing device <NUM> and/or user device <NUM> can be connected to a common LAN such as a home network, and can connect to content discovery server <NUM> and/or content delivery server <NUM> through the Internet. As another example, content delivery server <NUM> and/or content discovery server <NUM> can be connected to a LAN that is common to computing device <NUM> and/or user device <NUM>. In a more particular example, content delivery server <NUM> and/or content discovery server <NUM> can be located remotely from computing device <NUM> and/or user device <NUM>, and communications links <NUM> and/or <NUM> can include a non-local network such as the Internet for accessing content from content delivery server <NUM> and/or discovering content using content discovery server <NUM>. In another more particular example, content delivery server <NUM> and/or content discovery server <NUM> can be located locally to computing device <NUM> and/or user device <NUM>, and communications links <NUM> and/or <NUM> can include a local network, such as a home network, to which content delivery server <NUM> and/or content discovery server <NUM> are connected. In such an example, a non-local network of communications links <NUM> and/or <NUM> (if it exists) may not be used when accessing content from content delivery server <NUM> and/or discovering content using content discovery server <NUM>.

In some implementations, a user of user device <NUM> can cause media content made available by content delivery server <NUM> to be transmitted to computing device <NUM>, which can, in turn, present the media content using a speaker integrated in computing device <NUM> and/or output the media content to external sound system <NUM>.

In some implementations, a user of computing device <NUM> can interact with computing device <NUM> to request media content. For example, computing device <NUM> can receive record an audio signal and determine that a user said one or more words, can interpret the words as an instruction to request media content, and can take one or more actions based on the instruction.

In some implementations, when a media content item is to be presented from content delivery server <NUM> using computing device <NUM>, the media content item can be transmitted to computing device <NUM> over a communications link <NUM>. Communications link <NUM> can include a local area network (e.g., a home network) and/or a non-local network (e.g., the Internet). For example, computing device <NUM> and user device <NUM> can both be connected to a common LAN such as a home network, and can connect to content delivery server <NUM> through the Internet. As another example, content delivery server <NUM> can be connected to a LAN that is common to user device <NUM> and/or computing device <NUM>. In a more particular example, content delivery server <NUM> can be located remotely from computing device <NUM> and/or user device <NUM>, and communications link <NUM> can include a non-local network such as the Internet for accessing content from content delivery server <NUM>. In another more particular example, content delivery server <NUM> can be located locally to computing device <NUM> and/or user device <NUM>, and communications link <NUM> can include a local network, such as a home network, to which both computing device <NUM> and content delivery server <NUM> are connected. In such an example, a non-local network of communications link <NUM> (if it exists) may not be used when accessing content from content delivery server <NUM>.

In some implementations, a user can choose a media content item to be presented using computing device <NUM> from a memory of user device <NUM> (e.g., the user can choose a video that is stored locally on user device <NUM>). Additionally or alternatively, a user can choose a media content item to be presented using computing device <NUM> that is stored on another computing device present on the same local area network (e.g., network attached storage, a memory of a personal computer on a home network, memory on a server on the same network, etc.). Such a media content item stored on a computing device connected to a local area network can be transmitted from user device <NUM> to computing device <NUM> over communications link <NUM> (or any other suitable communications link), without being transmitted through content delivery server <NUM>.

<FIG> shows an example <NUM> of a generalized schematic diagram of a system on which the mechanisms for controlling audio output as described herein can be implemented in accordance with some implementations of the disclosed subject matter. As illustrated, system <NUM> can include one or more user devices <NUM>. User devices <NUM> can be local to each other or remote from each other. User devices <NUM> can be connected by one or more communications links <NUM> to a communications network <NUM> that can be linked via a communications link <NUM> to computing device(s) <NUM>, via a communications link <NUM> to content delivery server <NUM>, via communications link <NUM> to content discovery server <NUM>, and via communications links <NUM> to other user devices <NUM>.

In some implementations, each of the user devices <NUM>, content delivery server <NUM>, content discovery server <NUM> and computing device <NUM> can be any of a general purpose device such as a computer or a special purpose device such as a client, a server, etc. Any of these general or special purpose devices can include any suitable components such as a hardware processor (which can be a microprocessor, digital signal processor, a controller, a field-programmable gate array, a complex programmable logic device, an application-specific integrated circuit, etc.), memory, communication interfaces, display controllers, input devices, etc. For example, user device <NUM> can be implemented as a smartphone, a tablet computer, a wearable computer, a personal computer, a laptop computer, a gaming console, a digital media receiver, a set-top box, a smart television, a server, etc. As another example, computing device <NUM> can be implemented as a wireless speaker, a voice command device, a device that provides a digital assistant service, a personal computer, a laptop computer, a gaming console, a smart television, a smartphone, a tablet computer, a wearable computer, a digital media receiver, a set-top box, a smart television, a server, any suitable home automation device that is connected to and/or can connect to an internal and/or external speaker, etc..

Communications network <NUM> can be any suitable computer network or combination of such networks including the Internet, an intranet, a wide-area network (WAN), a local-area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), etc. Communications links <NUM>, <NUM>, <NUM> and <NUM> can be any communications links suitable for communicating data among user devices <NUM>, computing device <NUM>, content delivery server <NUM>, and content discovery server <NUM>, such as network links, dial-up links, wireless links, hard-wired links, any other suitable communications links, or any suitable combination of such links. User devices <NUM> can discover, browse, download, stream, store, playback, transmit, and/or otherwise present content stored locally at user device <NUM> and/or stored remotely at content delivery server <NUM> using any suitable technique(s). Additionally, in some implementations, computing device <NUM> can discover, browse, download, stream, store, playback, transmit, and/or otherwise present content stored locally at computing device <NUM> and/or stored remotely at content delivery server <NUM> using any suitable technique(s).

System <NUM> can include computing device <NUM> as described above in connection with <FIG>, and can include an application installed on and/or executed by on one or more of external sound systems <NUM>, which can replicate the functions of computing device <NUM> described herein.

System <NUM> can include one or more content delivery servers <NUM>. Content delivery server <NUM> can be any suitable server for providing access to media content items in accordance with the mechanisms described herein, such as a processor, a computer, a data processing device, and/or any suitable combination of such devices.

System <NUM> can include one or more content discovery servers <NUM>. Content discovery server <NUM> can be any suitable server for facilitating discovery and/or browsing of media content items accessible on content delivery server <NUM> in accordance with the mechanisms described herein, such as a processor, a computer, a data processing device, and/or any suitable combination of such devices.

In some implementations, communications link <NUM> described above in connection with <FIG> can include communications link <NUM>, communication network <NUM>, and communications link <NUM>; communications link <NUM> described above in connection with <FIG> can include communications link <NUM> and/or <NUM>, communication network <NUM>, and communications link <NUM>; and communications link <NUM> described above in connection with <FIG> can include communications link <NUM> and/or <NUM>, communication network <NUM>, and communications link <NUM>.

<FIG> shows an example <NUM> of hardware that can be used to implement one of computing devices <NUM>, external sound system <NUM> and content delivery server <NUM> depicted in <FIG> in accordance with some implementations of the disclosed subject matter. Referring to <FIG>, computing device <NUM> can include a hardware processor <NUM>, one or more microphones <NUM>, memory <NUM>, a transmitter/receiver <NUM>, one or more audio output ports <NUM> and/or one or more speakers <NUM> which can be interconnected. In some implementations, memory <NUM> can include a storage device (such as a non-transitory computer-readable medium) for storing a computer program for controlling hardware processor <NUM>.

Hardware processor <NUM> can use the computer program to present and/or record audio using microphone(s) <NUM> and/or to interact with a user (e.g., by capturing words spoken by the user using microphone(s) <NUM> and/or by presenting audio using speaker(s) <NUM>) to allow a user to, among other things, request media content to be presented using computing device <NUM>. It should also be noted that data received through communications link <NUM>, and/or any other communications links can be received from any suitable source. In some implementations, hardware processor <NUM> can send and receive data through communications link <NUM>, communications link <NUM>, communications link <NUM>, communications link <NUM>, connection <NUM>, and/or any other communications links using, for example, a transmitter, receiver, transmitter/receiver, transceiver, or any other suitable communication device, such as transmitter/receiver <NUM>. Microphone(s) <NUM> can include any suitable number and/or type of microphones. Transmitter/receiver <NUM> can include any suitable transmitter and/or receiver for transmitting and/or receiving, among other things, media content including audio and/or video content. Audio output <NUM> can include any suitable audio port or audio ports (and/or video ports in some implementations) for outputting audio to an external sound system (e.g., external sound system <NUM>). For example, as described above in connection with <FIG>, audio output <NUM> can include a <NUM> jack that accepts a <NUM> connector, a fiber optic audio port (e.g., that accepts a TOSLINK connector), an RCA jack, a High-Definition Multimedia Interface (HDMI) port, a Universal Serial Bus (USB) port (e.g., a USB <NUM>. x port, a USB <NUM>. x port, etc.), a THUNDERBOLT port, any other suitable port and/or any suitable combination of ports. Speaker(s) <NUM> can include any suitable number and/or type of speakers for presenting audio and/or providing audio feedback to a user. In some implementations, internal speaker(s) <NUM> can be omitted and/or computing device <NUM> can output audio to any other suitable device that can output audio. For example, in such implementations, computing device can output audio to another device to which it is connected by wire and/or wirelessly, such as a standalone speaker (e.g., connected by a speaker wire), a user device having an internal speaker (e.g., a smartphone, a tablet computer, etc.), a computer having external speakers that is connected to the same local area network, a wireless speaker to which computing device <NUM> can connect (e.g., by Bluetooth connection), etc..

External sound system <NUM> can include a hardware processor <NUM>, one or more speakers <NUM>, memory <NUM>, and one or more audio inputs <NUM>, which can be interconnected. In some implementations, memory <NUM> can include a storage device (such as a non-transitory computer-readable medium) for storing a computer program for controlling hardware processor <NUM>.

Hardware processor <NUM> can use the computer program to present audio received via audio input <NUM> and/or from any other source (e.g., from an internal or external antenna that can be used to receive audio broadcast over the air).

Content delivery server <NUM> can include a hardware processor <NUM>, a display/input device <NUM>, memory <NUM>, and a transmitter/receiver <NUM>, which can be interconnected. In some implementations, memory <NUM> can include a storage device (such as a non-transitory computer-readable medium) for storing a server program for controlling hardware processor <NUM>.

Hardware processor <NUM> can use the server program to communicate with user device <NUM> and/or computing device <NUM>, as well as provide access to media content. It should also be noted that data received through communications link <NUM> or any other communications links can be received from any suitable source. In some implementations, hardware processor <NUM> can send and receive data through communications link <NUM> or any other communication links using, for example, a transmitter, receiver, transmitter/receiver, transceiver, or any other suitable communication device, such as transmitter/receiver <NUM>. In some implementations, hardware processor <NUM> can receive commands and/or values transmitted by one or more users. Display/input device <NUM> can include a touchscreen, a flat panel display, a cathode ray tube display, a projector, a speaker(s), and/or any other suitable display and/or presentation devices, and can further include a computer keyboard, a computer mouse, a microphone, a touchpad, a voice recognition circuit, a touch interface of a touchscreen, and/or any other suitable input device. Transmitter/receiver <NUM> can include any suitable transmitter and/or receiver for transmitting and/or receiving, among other things, media content including audio and/or video content.

In some implementations, content delivery server <NUM> can be implemented in one server or can be distributed as any suitable number of servers. For example, multiple servers <NUM> can be implemented in various locations to increase reliability and/or increase the speed at which server <NUM> can communicate with user devices <NUM> and/or computing device <NUM>.

In some implementations, content discovery server <NUM> can be implemented using similar hardware to content delivery server <NUM>, but can include a server program for facilitating discovery and/or browsing of media content items accessible from content delivery server <NUM>.

<FIG> shows an example <NUM> of a process for controlling audio output in accordance with some implementations of the disclosed subject matter. At <NUM>, process <NUM> can receive a request to present audio content. In some implementations, the request to present audio content can be received from any suitable source and/or can be received using any suitable technique or combination of techniques. For example, in some implementations, process <NUM> can receive the request to present audio content from a user device (e.g., user device <NUM>), with identifying information (e.g., a URL) that computing device <NUM> can use to request the audio content. As another example, in some implementations, process <NUM> can receive the request to present audio content as spoken words that are recorded by microphone <NUM> and converted into one or more instructions by hardware processor <NUM> (and/or any other suitable processor, such as a processor of a server with which computing device <NUM> can communicate to interpret spoken user instructions). In such an example, the request spoken by the user can be a request to present particular audio content (e.g., the name of an artist, an author, a title of a song or book, etc.), a request to present content using a particular application, a request to present a particular type of content (e.g., a particular genre), any other suitable request for content, and/or any suitable combination thereof. As yet another example, process <NUM> can receive a request to present content pushed to computing device <NUM> by another device (e.g., content delivery server <NUM>, user device <NUM>, etc.). In such an example, the device pushing the content can send a request to computing device <NUM> to play the content over any suitable communications link such as the Internet, a local area network (e.g., Wi-Fi), a short range wireless connection (e.g., Bluetooth), any other suitable communications link, or any suitable combination of communications links.

At <NUM>, process <NUM> can request and/or receive the audio content that was requested to be presented at <NUM>. In some implementations, process <NUM> can use any suitable technique or combination of techniques to request and/or receive the audio content requested to be presented at <NUM>. For example, process <NUM> can request the content using an address (e.g., a URL) provided by user device <NUM>. As another example, process <NUM> can request the content by determining a search query and/or instruction based on one or more words spoken by a user, and can use the search query and/or instruction to request content responsive to the one or more words spoken by the user. As yet another example, process <NUM> can receive audio content pushed by another device (e.g., user device <NUM>, content delivery server <NUM>, etc.).

At <NUM>, process <NUM> can determine whether an output to an external sound system (e.g., external sound system <NUM>) is connected. In some implementations, process <NUM> can use any suitable technique or combination of techniques to determine whether an output to an external sound system is connected. For example, process <NUM> can determine whether an audio output port (e.g., connector <NUM>) is connected to a corresponding plug. In a more particular example, process <NUM> can determine whether a <NUM> plug is inserted into the <NUM> jack. In such examples, process <NUM> can use any suitable technique or combination of techniques to determine that a plug is inserted into an output port, such as determining whether a metal contact of the plug is present in the output port. In another more particular example, process <NUM> can determine whether a particular signal is received at a contact of the output port, such as a hot plug detect signal that is output on pin <NUM> of an HDMI port and received back at pin <NUM> of the HDMI port when an HDMI sink is connected to the HDMI port (e.g., via an HDMI cable). As another example, computing device <NUM> can determine whether audio content is being output over a wireless connection that is used to transmit audio content to external sound system <NUM>.

If process <NUM> determines that there is no output to external sound system <NUM> connected ("NO" at <NUM>), process <NUM> can proceed to <NUM>, and output audio using one or more internal speakers (e.g., speaker(s) <NUM>). Otherwise, if process <NUM> determines that an output to external sound system <NUM> is connected ("YES" at <NUM>), process <NUM> can proceed to <NUM>.

At <NUM>, process <NUM> can output the audio received at <NUM> over the connection to external sound system <NUM> that is connected. In some implementations, process <NUM> can inhibit the audio being output over the connection to external sound system <NUM> from also being presented using speaker(s) <NUM>. Additionally, in some implementations, process <NUM> can output the audio over only a single output port in cases in which multiple output ports are present and connected to external sound systems (which may be the same external sound system <NUM> or different external sound systems). Alternatively, in some implementations, process <NUM> can output the audio over any output ports for which a connection is detected. In such implementations, the audio output over different output ports can be output with different timing in situations in which different external sound systems exhibit different delays.

At <NUM>, process <NUM> can record one or more audio samples using microphone <NUM>, which can be used to determine whether the audio being output at <NUM> is being presented by an external speakers (e.g., speaker(s) <NUM> of external sound system <NUM>). In some implementations, process <NUM> can record any suitable amount of audio and/or any suitable frequencies of audio for comparison with the audio output at <NUM>. For example, process <NUM> can cause the audio to be continuously recorded, and can use a particular portion of the recorded audio to compare to the audio output at <NUM> (e.g., as described below in connection with <NUM>). As another example, process <NUM> can record samples at discrete time intervals, without recording signals from sounds that occur between the time intervals (e.g., process <NUM> can capture samples every quarter second, every fifth of a second, etc.). As yet another example, process <NUM> can record audio at particular frequencies (e.g., by filtering out or otherwise discarding audio at other frequencies).

In some implementations, signals recorded at <NUM> and signals recorded in order to determine whether the user has issued voice commands are deleted, overwritten and/or otherwise disposed of when computing device <NUM> has used the signal (e.g., to determine whether audio is being played by an external speaker, whether the user has issued a voice command, etc.). Additionally, in some implementations, microphone <NUM> can be disabled when not in use and/or and signals output by microphone <NUM> can be disregarded (e.g., never recorded in memory <NUM>) when the user has not requested that the signals be recorded. For example, in some implementations, signals from microphone <NUM> can be recorded (e.g., in memory <NUM>) when audio content is being output using audio output <NUM> in order to determine whether external sound system <NUM> is presenting the audio content. As another example, signals from microphone <NUM> can be recorded (e.g., in memory <NUM>) when the user is giving a voice command in order to determine one or more instructions based on the voice command. In such an example, a user can be required to say a particular word or combination of words (e.g., a trigger word or trigger phrase) and/or perform some other recognized action, such as pressing a physical button to initiate recording of subsequent words that are to constitute a voice command. Until the trigger word or trigger phrase is recognized, signals from microphone <NUM> can be discarded.

At <NUM>, process <NUM> can determine whether the audio recorded at <NUM> matches the audio output at <NUM>. Process <NUM> can use any suitable technique or combination of technique to determine whether the recorded audio matches the outputted audio. For example, in some implementations, process <NUM> (and/or any other suitable process) can generate one or more audio fingerprints based on the recorded audio, and compare the generated fingerprints to one or more audio fingerprints of the audio being output at <NUM>. In such an example, the fingerprints of the recorded audio and/or the fingerprints of the outputted audio can be generated by computing device <NUM> and/or can be generated by a remote computer (e.g., a remote server). In some implementations, the fingerprints of the outputted audio can be generated as the audio is output. Additionally or alternatively, the fingerprints of the outputted audio can be generated prior to the content being output (e.g., in a database of known audio content), and the fingerprints of the recorded audio can be compared to fingerprints from the previously generated fingerprints.

As another example, in some implementations, process <NUM> (and/or any other suitable process) can compare the recorded audio signal to the outputted audio signal to determine whether the signals match. In a more particular example, process <NUM> can take a difference between the recorded audio signal and the outputted audio signal. In such a more particular example, if the result of the difference is below a threshold, process <NUM> can determine that the outputted signal and the recorded signal match. In some implementations, delays caused by transmitting the audio signal to external sound system <NUM>, delays caused by external sound system <NUM> and/or delays introduced when recording the audio content can cause a comparison between the audio currently being output and the recorded audio to be different regardless of whether the audio is being presented by external sound system <NUM>. In some implementations, process <NUM> can compare the recorded audio to a time-delayed version of the signal that was output to external sound system <NUM>. Process <NUM> can use any suitable technique or combination of techniques to determine the amount of delay, such as by matching a pattern of local maxima from both signals to attempt to match the timing of the signals, by comparing the recorded signal to versions of the output signal with multiple different amounts of delay, by determining the delay between the output audio and the recorded audio once (e.g., using a calibration process) and using the determined delay, etc. In some implementations, process <NUM> (and/or any other suitable process) can compensate for background noise captured in the recording using any suitable technique or combination of techniques, such as by filtering the audio signal based on the known signal (e.g., the output audio signal). Note that in most cases the recorded audio will not match the output audio exactly (e.g., due to the hardware being used to play the audio, the hardware being used to record the audio, ambient noise, the acoustics or the room, etc.), and accordingly, process <NUM> can determine whether the recorded audio is substantially similar to the output audio signal. For example, process <NUM> (and/or any other suitable process) can determine a confidence value that represents how closely the recorded audio matches the output signal. In such an example, if the confidence value is greater than a threshold value, process <NUM> can determine that the recorded audio matches the audio being output.

In some implementations, process <NUM> can alter the audio signal output at <NUM> to generate a modified audio signal. In such implementations, process <NUM> can introduce one or more signals or alterations to the existing signal to increase the likelihood that the recorded audio signal can be recognized without substantially changing how the output audio signal sounds to most human listeners. For example, process <NUM> can add a signal having a particular pattern at a frequency that can be output by most external speakers, but that is outside of the range of human hearing. As another example, process <NUM> can change the amplitude of the signal in a particular pattern such that it is unlikely to be observed by a listener, but is recognizable by process <NUM> (and/or any other suitable process). In such implementations, process <NUM> can determine whether the pattern of the inserted signal and/or the change in amplitude is present in the recorded signal to determine whether the outputted audio signal is being presented by external audio system <NUM>.

If process <NUM> determines that the recorded audio matches the outputted audio signal ("YES" at <NUM>), process <NUM> can return to <NUM> and can continue to output the audio if the output to the external sound system is connected.

Otherwise, if process <NUM> determines that the recorded audio does not match the outputted audio signal ("NO" at <NUM>), process <NUM> can proceed to <NUM>. At <NUM>, process <NUM> can output the audio signal using speaker(s) <NUM>, and can return to <NUM> to determine whether an output to an external audio system is connected. The recorded audio may not match the outputted audio signal for various reasons, such as the external sound system being powered down, a different input of the external sound system being selected, the output not being connected to the external audio system, a malfunction, etc..

In some implementations, during periods when process <NUM> is presenting audio content using speaker(s) <NUM> at <NUM>, process <NUM> can provide audio to audio output <NUM> such that, if and when external sound system <NUM> becomes configured to present audio content, process <NUM> can provide the audio content at <NUM> and inhibit presentation of the audio content by speaker(s) <NUM> at <NUM>. In some implementations, process <NUM> can provide any suitable audio to audio output <NUM> at <NUM>. For example, an audio signal outside the range of human hearing can be provided, and process <NUM> can determine, at <NUM>, whether that signal is present in audio recorded at <NUM>. As another example, the audio content received at <NUM> can be provided. In such an example, in cases in which the audio content is provided to audio output <NUM>, the audio content may be presented by both the internal speaker and the external speakers (e.g., speaker(s) <NUM> of external sound system <NUM>) with a delay between the two, and both may be present in audio recorded at <NUM>. In some implementations, process <NUM> (and/or any other suitable process) can use any suitable technique or combination of techniques to distinguish between the audio presented by speaker(s) <NUM> and speaker(s) <NUM>. For example, in some implementations, process <NUM> can subtract the signal provided to speaker(s) <NUM> from the recorded signal based on a known timing relationship between when audio is presented by speaker(s) <NUM> and when that same audio is recorded by microphone <NUM>. As another example, multiple microphones <NUM> can record sound, and can be placed in relation to speaker(s) <NUM> such that the signals from the multiple microphones can be modified and/or combined (e.g., by subtracting the output of one microphone from the output of another, by inverting the output from one microphone and adding it to another, etc.) such that the signal provided to speaker(s) <NUM> is removed. In such an example, the residual signal can be used to determine whether the audio content is present in the recorded signal as presented by external sound system <NUM>.

In some implementations, process <NUM> can be used for some audio to be presented and not for other audio to be presented. For example, in some implementations, audio that is output to interact with a user (e.g., prompts, queries, responses to queries, etc.) can always be presented using only speaker(s) <NUM>.

In some implementations, the mechanisms described herein, can include server-side software, server-side hardware, client-side software, client-side hardware, or any suitable combination thereof. For example, the mechanisms can encompass a computer program written in a programming language recognizable by hardware processor <NUM>, hardware processor <NUM>, and/or hardware processor <NUM> (e.g., a program written in a programming language, such as, Java, C, Objective-C, C++, C#, Javascript, Visual Basic, or any other suitable approaches). As another example, the applications can encompass code corresponding to one or more Web pages or Web page portions (e.g., via any suitable encoding, such as Hyper Text Markup Language ("HTML"), Dynamic Hyper Text Markup Language ("DHTML"), Extensible Markup Language ("XML"), JavaServer Pages ("JSP"), Active Server Pages ("ASP"), Cold Fusion, or any other suitable approaches).

In some implementations, any suitable computer readable media can be used for storing instructions for performing the functions and/or processes described herein. For example, in some implementations, computer readable media can be transitory or non-transitory. For example, non-transitory computer readable media can include media such as magnetic media (such as hard disks, floppy disks, etc.), optical media (such as compact discs, digital video discs, Blu-ray discs, etc.), semiconductor media (such as flash memory, electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), etc.), any suitable media that is not fleeting or devoid of any semblance of permanence during transmission, and/or any suitable tangible media. As another example, transitory computer readable media can include signals on networks, in wires, conductors, optical fibers, circuits, any suitable media that is fleeting and devoid of any semblance of permanence during transmission, and/or any suitable intangible media.

It should be understood that the above described steps of the processes of <FIG> can be executed or performed in any order or sequence not limited to the order and sequence shown and described in the figures. Also, some of the above steps of the processes of <FIG> can be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times.

It should also be noted that, as used herein, the term mechanism can encompass hardware, software, firmware, or any suitable combination thereof.

Accordingly, systems, methods, and media for controlling audio output are provided.

Claim 1:
A method for controlling audio output, the method comprising:
determining, using an audio input device of a computing device (<NUM>) to capture a portion of audio being played back by a sound system (<NUM>) connected to the computing device, whether audio content that has been transmitted to the connected sound system is being played back by the connected sound system; and
inhibiting the audio content from being provided to the connected sound system and beginning playing the audio content to be presented using an audio output device (<NUM>) of the computing device based on the determination that the audio content that has been transmitted to the connected sound system is not being played back by the connected sound system.