Patent Description:
The types of consumer electronic devices being served by set-top boxes are also varied and include devices such as televisions, computers, portable devices, electronic tablets, smart phones, and Bluetooth devices. Whether it is Bluetooth sound bars, Bluetooth headphones or other Bluetooth audio sink devices, a synchronization problem often arises with regard to the audio data being output by the Bluetooth devices and the video data being output by the video sink devices.

That is, there may be a time difference between the video and audio output from the video sink devices, and audio output from the Bluetooth devices. Many times, this type of mismatch is enough to cause noticeable and annoying lip-sync problems. The lip-sync problem is due to the fact that there are primary and secondary delays associated with the use of the Bluetooth devices. The primary delays are related to transmitting the audio data using a Bluetooth protocol and the secondary delays are related to the Bluetooth devices processing the audio data. Additional lip-sync problems can be created when using multiple Bluetooth devices because different Bluetooth devices may have different primary and secondary delays.

Using conventional techniques, there is no effective way to consistently determine and configure what delays (e.g., video and audio delays) should be used so that the video presentation matches the audio presentation for all the Bluetooth devices. Thus, there is a need to be able to consistently determine and configure what delays (e.g., video and audio delays) should be used so that the video presentation matches the audio presentation for all Bluetooth devices.

<CIT> relates to video-audio processing devices, and particularly to a video-audio processing device which performs synchronization between a video signal and an audio signal in reproduction. <CIT> relates generally to playing audio and video data, and in particular, to separately playing audio and video data in local networks.

<CIT> describes a solution that allows a user to utilize a personal portable device such as a smartphone to enjoy audio associated with a public display of video. <CIT> relates to techniques for wireless devices to communicate audio data using Bluetooth Low Energy communications.

In an embodiment described in the present application, an electronic apparatus is implemented to achieve synchronization between video data displayed on a video sink device and audio data executed on one or more Bluetooth devices.

The electronic apparatus includes an input circuit, an A/V decoder, an output circuit, a Bluetooth transceiver, and a controller. The input circuit receives audio and video (A/V) content from an A/V content provider, the A/V decoder decodes the A/V content to obtain video data and audio data, the output circuit outputs the A/V data to the video sink device, and the Bluetooth transceiver wirelessly communicates information with one or more Bluetooth devices according to a wireless protocol, wherein the information includes the audio data. The wireless protocol is a Bluetooth wireless protocol, and the electronic apparatus may be a set-top box.

In embodiments described in the present application, a method and algorithm are implemented to achieve synchronization between video data executed on a video sink device and audio data executed on one or more Bluetooth devices.

<FIG> is a diagram illustrating synchronization issues between video data and audio data when using Bluetooth audio devices (hereafter Bluetooth devices). As shown in <FIG>, there is a noticeable delay between the audio data being output by the Bluetooth device and the audio/video (A/V) data being output by the television (TV) (e.g., video sink device), after receiving A/V data from, for example, a set-top box. The A/V content to the set-top box is provided by service providers including cable television providers, satellite television providers, internet service providers, and multiple system operators.

The synchronization problem between video data and audio data when using a Bluetooth device arises because there are primary and secondary delays associated with using the Bluetooth device, which causes noticeable and annoying lip-sync problems between the video data displayed on the TV and the audio data being output by the Bluetooth device. The primary delays are related to transmitting the audio data using a Bluetooth protocol and the secondary delays are related to the Bluetooth devices processing the audio data.

For example, a user may be using a Bluetooth device (e.g., Bluetooth sound bars, Bluetooth headphones or other Bluetooth audio sink device such as a mobile phone, tablet or the like having Bluetooth capability) to listen to audio data related to video data or content (e.g., television programing and movies) being displayed on the TV. Due to the primary and secondary delays associated with using the Bluetooth device, there can be a mismatch between the audio data being listened to and the video data being displayed by the TV (i.e., lip-sync problem). Thus, there is a need to address the synchronization problem between video data and audio data when using a Bluetooth device.

<FIG> is a block diagram showing an exemplary system for achieving synchronization between video data and audio data when using a Bluetooth device.

As shown in <FIG>, the system includes a set-top box <NUM>, a video sink device <NUM>, a remote control device (e.g., an electronic remote control) <NUM>, and a Bluetooth device <NUM>. For ease of discussion, <FIG> refers to a stand-alone set-top box, but the present disclosure is not intended to be limited only to this type of set-top box and alternatively the set-top box <NUM> can be integrated directly into a consumer device such as a television, computer, or any other consumer device, such as electronically equipped appliances.

The set-top box <NUM> houses components and circuits (e.g., as shown in <FIG>) that convert the A/V content <NUM> to the set-top box <NUM> into audio and video (A/V) data that is usable by the video sink device <NUM> and the Bluetooth device <NUM>. For example, the A/V content <NUM> to the set-top box <NUM> may be provided by service providers including cable television providers, satellite television providers, internet service providers, and multiple system operators; and the A/V content may include, but is not limited to, television programing and movies. The operation of the components and circuits of the set-top box <NUM> will be discussed in more detail with reference to <FIG>.

The video sink device <NUM> may include, but is not limited to, a television, a computer, a portable device, an electronic tablet, a smart phone and other consumer electronic device capable to executing and displaying A/V data received from the set-top box <NUM>. On the other hand, the Bluetooth devices <NUM> may include, but are not limited to, Bluetooth sound bars, Bluetooth headphones or other Bluetooth devices such as a smart phone, electronic tablet, televisions, computers, portable electronic devices, or other consumer electronic devices capable of communicating data with the set-top box <NUM> using a Bluetooth protocol. Each Bluetooth device <NUM> includes components and circuits (e.g., as show in <FIG>) that decode and execute the audio data received from the set-top box <NUM>. The operation of the components and circuits of each Bluetooth device <NUM> will be discussed in more detail with reference to <FIG>.

The A/V content <NUM> received by the set-top box <NUM> is decoded to obtain the audio and the video data (A/V) data, and the A/V data is provided to the video sink device <NUM> via communication connection <NUM>. The communication connection <NUM> between the video sink device <NUM> and the set-top box <NUM> can be a wired connection. The communication connection <NUM> may include, for example, a digital media interface (DMI), high-definition multimedia interface (HDMI) or other audio/video interface for transferring video data and digital audio data from an HDMI-compliant source device. Although the communication connection <NUM> is described as a wired connection, the communication connection <NUM> may also be a wireless connection that operates in accordance with, but is not limited to, IEEE802. <NUM> protocol, a Radio Frequency For Consumer Electronics (RF4CE) protocol, ZigBee protocol, and/or IEEE802. <NUM> protocol.

The audio data obtained by decoding the A/V content <NUM> received by the set-top box <NUM> is also provided to each of the Bluetooth devices <NUM> via the communication connections <NUM>. The communication connections <NUM> between the set-top box <NUM> and each of the Bluetooth devices <NUM> are wireless connections that operate in accordance with a Bluetooth protocol. That is, the communication connections <NUM> operate in accordance with a wireless technology standard for exchanging data over short distances using short-wavelength Ultra high frequency (UHF) radio waves from <NUM> to <NUM>. For example, data can be exchanged between the step-top box <NUM> and the Bluetooth devices <NUM> via the communication connections <NUM> at a rate of approximately <NUM>-<NUM> megabits per second and at a range of approximately <NUM>-<NUM> meters. However, the data rate and distance for exchanging data between the set-top box <NUM> and the Bluetooth devices <NUM> may vary depending on the generation of Bluetooth technology being implemented.

The remote device <NUM> also communicates wirelessly with the set-top box <NUM> using a wireless communication connection <NUM>. The remote device <NUM> may be a stand-alone remote control, or any electronic device that can be implemented as a remote control for communicating data with the set-top box <NUM> using an Infrared (IR) signal or protocol sent via the communication connection <NUM>. Although the remote device <NUM> is described as a remote control device using an IR signal or protocol, it is contemplated by the invention that other remote control devices could be implemented that use protocols such as, but not limited to, Bluetooth Audio/Video Remote Control Profile (AVRCP), RF4CE, ZigBee, Wi-Fi, and Z-Wave.

After receiving the A/V content <NUM> at the set-top box <NUM>, the A/V content <NUM> is decoded by the set-top box <NUM>, and the A/V data is transmitted to the video sink device <NUM> and the audio data is transmitted to one or more Bluetooth devices <NUM>. A user of the Bluetooth device <NUM> can then watch the A/V content such as television programing and movies on the video sink device <NUM> while listening to the audio data on the Bluetooth devices <NUM>. To address the lip-sync problem between the audio data listen to on the Bluetooth device <NUM> and video data watched on the video sink device <NUM>, the set-top box <NUM> implements a manual and an automatic calibration for achieving synchronization between video data watched on the video sink device <NUM> and audio data listen to on the Bluetooth device <NUM>. A detailed discussion of the manual calibration and the automatic calibration for achieving synchronization between video data watched on the video sink device <NUM> and audio data listened to on a Bluetooth device <NUM> will be discussed in more detail with reference to <FIG>.

<FIG> is a block diagram showing an exemplary set-top box <NUM> for achieving synchronization between video data and audio data when using Bluetooth devices. As shown in <FIG>, the set-top box <NUM> includes a communication bus <NUM> through which various components in the set-top box <NUM> are connected for communicating data there between and for converting input A/V content <NUM> to the set-top box <NUM> into A/V data usable by the video sink device <NUM> and a Bluetooth device <NUM>. The components of the set-top box <NUM> include a controller <NUM>, a random access memory (RAM) <NUM>, a non-volatile memory <NUM>, a Bluetooth transceiver <NUM>, a user interface <NUM>, an A/V input/output circuit <NUM>, a tuner <NUM>, an A/V decoder <NUM>, a Wi-Fi transceiver <NUM>, and an Infrared (IR) receiver <NUM>.

The controller <NUM> may be a dedicated controller, CPU, microprocessor, etc., capable of controlling the operation of the components and circuits of the set-top box <NUM>. The RAM <NUM> may be implemented as a working memory for the controller <NUM>, and the non-volatile memory <NUM> can be provided for storage of program code, user A/V content, determined or calculated delay information, and other data. The Bluetooth transceiver <NUM> transmits data to and receives data from the Bluetooth device <NUM> in accordance with a Bluetooth protocol. Similar to the Bluetooth devices <NUM>, the Bluetooth transceiver <NUM> operates in accordance with a wireless technology standard for exchanging data over short distances using short-wavelength UHF radio waves from <NUM> to <NUM>.

Communications between the Bluetooth transceiver <NUM> and the Bluetooth devices <NUM> can be point-to-point or multipoint. Additionally, the data can be exchanged between the Bluetooth transceiver <NUM> and the Bluetooth devices <NUM> at, for example, a data rate of approximately <NUM>-<NUM> megabit per second and at a range of approximately <NUM>-<NUM> meters. However, the data rate for exchanging data and the distance for exchanging data between the Bluetooth transceiver <NUM> and the Bluetooth devices <NUM> may vary depending on the generation of Bluetooth technology being implemented in the Bluetooth transceiver <NUM> and the Bluetooth devices <NUM>.

The user interface <NUM> provides a means for inputting instructions directly to the set-top box <NUM>. The user interface <NUM> may include, but is not limited to, keys, buttons, knobs, or other similar input device that can be used to input instructions for controlling operations on the set-top box <NUM> and/or the video sink device <NUM>. The A/V input/output circuit <NUM> includes one or more connectors, such as RF connectors or Ethernet connectors. One of the connectors of the A/V input/output circuit <NUM> can be connected to a content provider, such as a multiple system operator (MSO), by terrestrial antenna, satellite dish, or wired cable. Through this connector of the A/V input/output circuit <NUM>, the set-top box <NUM> receives the A/V content <NUM> from the content provider. Additionally, one of the connectors of the A/V input/output circuit <NUM> can be used to send data to the content provider.

The tuner <NUM> can select a desired channel from the received A/V content <NUM> based on the input instruction by the user either through a button or buttons of the user interface <NUM> on the set-top box <NUM> or via IR signal received from the remote device <NUM>. The signal of the selected channel is decoded by the A/V decoder <NUM>. The A/V decoder <NUM> decodes the selected signal so that the A/V content is usable by the video sink device <NUM> and the Bluetooth devices <NUM>. The A/V input/output circuit <NUM> can also include a connector that is to be connected to the user's video sink device <NUM>, such as a television, for displaying A/V content received by the set-top box <NUM> and decoded by the A/V decoder <NUM>. Additionally, the set-top box <NUM> can provide the A/V data according to Internet Protocol Television (IPTV), in which the tuner <NUM> may be omitted.

The Wi-Fi transceiver <NUM> is, for example, a Wi-Fi WLAN interface radio transceiver, or an in-home LTE (Long Term Evolution) transceiver that outputs signals of the selected channel to a wireless user device. The wireless output from the Wi-Fi transceiver can be in place of or in addition to the wired output by the A/V input/output circuit <NUM>. The A/V content <NUM> from the service provider can be received by a separate electronic device, such as a cable modem, or a different set-top box, and communicated to the set-top box <NUM> wirelessly via the Wi-Fi Transceiver <NUM>. The IR receiver <NUM> communicates IR signals with the remote device <NUM>, and the IR signal may include data that can be used by the controller <NUM> to control operations of the components and circuits of the set-top box <NUM>. Although the set-top box <NUM> is described as implementing the use of an IR receiver <NUM>, it is contemplated by the invention that other types of receivers could be implemented that use protocols such as, but not limited to, Bluetooth AVRCP, RF4CE, ZigBee, Wi-Fi, and Z-Wave.

<FIG> is a block diagram showing an exemplary Bluetooth device <NUM>. As shown in <FIG>, each Bluetooth device <NUM> includes a communication bus <NUM> through which various components in the Bluetooth device <NUM> are connected for communicating data there between. The components of the Bluetooth device <NUM> include a Bluetooth transceiver <NUM>, a random access memory (RAM) <NUM>, a non-volatile memory <NUM>, a controller <NUM>, an audio decoder <NUM> and a speaker or speakers <NUM>. The Bluetooth transceiver <NUM> transmits data to and receives data from the Bluetooth transceiver <NUM> of the set-top box <NUM> in accordance with a Bluetooth protocol. The Bluetooth transceiver <NUM> operates in accordance with a wireless technology standard for exchanging data over short distances using short-wavelength UHF radio waves from <NUM> to <NUM>.

Similar to the Bluetooth transceiver <NUM> of the set-top box, the data can be exchanged between the Bluetooth transceiver <NUM> and the Bluetooth transceiver <NUM> of the set-top box at a rate of approximately <NUM>-<NUM> megabit per second and at a range of approximately <NUM>-<NUM> meters. However, the data rate for exchanging data and the distance for exchanging data between the Bluetooth transceiver <NUM> and the Bluetooth transceiver <NUM> of the set-top box <NUM> may vary depending on the generation of Bluetooth technology being implemented in the Bluetooth transceivers <NUM>, <NUM>.

The RAM <NUM> may be implemented as a working memory for the controller <NUM>, and the non-volatile memory <NUM> can be provided for storage of program code, audio data, ID data, and other information. The controller <NUM> may be a dedicated controller, CPU, microprocessor, etc., capable of controlling the operation of the components of the Bluetooth device <NUM>. The audio decoder <NUM> decodes the signal received from the set-top box <NUM> so that the audio data is usable by the Bluetooth device <NUM>. The audio data decoded by the audio decoder <NUM> can be converted into a corresponding sound and output by one or more speakers <NUM> of the Bluetooth device <NUM>. That is, the speakers <NUM> are audio speakers that include an electroacoustic transducer, which can convert audio signal (e.g., an electrical audio signal) into a corresponding sound.

<FIG> is flow chart showing an exemplary algorithm and method that are not part of the invention as specified by the appended set of claims. The exemplary algorithm and method are for achieving manual synchronization between video data and audio data when using Bluetooth devices according to the present invention. The algorithm and method shown in <FIG> can be implemented in the exemplary system and set-top box <NUM> shown respectively in <FIG>, <FIG> and <FIG>. For example, the method and algorithm can be implemented by a program stored in the RAM <NUM> or the non-volatile memory <NUM> and executed by the controller <NUM>, such that the controller <NUM> controls the operations of the components and circuits of the set-top box <NUM> to perform operations of the method and algorithm.

Although the detailed operation of the components and circuits of the set-top box <NUM> and the Bluetooth devices <NUM> have already been described in detail with reference to <FIG> and <FIG>, the method and algorithm for performing the manual synchronization between video data and audio data when using Bluetooth devices <NUM> shown in <FIG> will be described with reference to some of the elements in <FIG>, <FIG>, and <FIG>.

As shown in <FIG>, the manual operation for synchronization between video data and audio data when using Bluetooth devices can include the user first entering into a calibration mode, as shown in step S1. The user enters the calibration mode by making an input selection on the set-top box <NUM> either through pushing a button or buttons using the user interface <NUM> on the set-top box <NUM> or via IR signal received through the communication connection <NUM> from the remote device <NUM>.

Once the user has entered the calibration mode, in step S2 the controller of the set-top box <NUM> controls the output of a graphical tool stored either in RAM <NUM> or the non-volatile memory <NUM> to the video sink device <NUM> through the communication connection <NUM>, and the graphical tool is displayed on the video sink device <NUM>. As shown in <FIG>, which is not part of the invention as specified by the appended set of claims, the graphical tool is a moving animation <NUM> that illustrates a hammer hitting a nail. Although <FIG> illustrates a hammer hitting a nail <NUM>, the type of animation stored in the set-top box <NUM> and used for calibration is not meant to be limited to one-type of animation, and can be any moving animation illustrating contact between objects, surfaces, or combination thereof. In step S3, the controller <NUM> of the set-top box <NUM> also controls the output of audio data corresponding to the animation <NUM> to the Bluetooth device <NUM> through the communication connection <NUM>.

The user of a Bluetooth device <NUM> listens to the audio data received from the set-top box <NUM> and in step S4 determines if the moving animation <NUM> shown on the video sink device <NUM> (i.e., hammer hitting the nail) is synchronized with the audio data being listened to on the Bluetooth device <NUM>. That is, contact between the hammer and the nail in the animation <NUM> should sufficiently correspond with the audio data (e.g., sound of hammer hitting a nail) listened to by the user on the Bluetooth device <NUM>. If the audio data listened to by the user using the Bluetooth device <NUM> is not synchronized with the animation <NUM> shown on the video sink device <NUM>, then in step S5 the user can send a command signal for either increasing or decreasing the delay of the animation <NUM>. The user can send the command signal either through pushing a button or buttons using the user interface <NUM> on the set-top box <NUM> or via IR signal received through the communication connection <NUM> from the remote device <NUM>.

For example, as shown in <FIG>, the user of the Bluetooth device <NUM> can make selections <NUM>, <NUM> on the calibration animation <NUM> in order increase <NUM> or decrease <NUM> the value of the delay <NUM> by sending the command signal. Steps S3-S5 can be repeated until acceptable synchronization is achieved between the moving animation <NUM> shown on the video sink device <NUM> (i.e., hammer hitting the nail) and the audio data listened to on the Bluetooth device <NUM>.

As the delay is adjusted by the user of the Bluetooth device <NUM> (i.e., steps S3-S5), the delay value <NUM> is shown on the animation <NUM>. Once the moving animation <NUM> shown on the video sink device <NUM> (i.e., hammer hitting the nail) is determined by the user of the Bluetooth device <NUM> to be synchronized with the audio data, in step S6 the user can exit the calibration mode. The user can exit the calibration mode using a similar method to that used to enter the calibration mode (e.g., an input instruction selection on the set-top box <NUM> either through pushing a button or buttons using the user interface <NUM> on the set-top box <NUM> or via IR signal received through the communication connection <NUM> from the remote device <NUM>). In step S7, the controller <NUM> of the set-top box <NUM> determines that the value of the delay <NUM> set by the user in the animation <NUM> is the delay needed for synchronization and the delay is stored by the controller <NUM> in the RAM <NUM> or the non-volatile memory <NUM> of the set-top box <NUM>.

In step S8, the controller <NUM> of the set-top <NUM> uses the stored delay as calibration information for delaying the output of the A/V data to the video sink device <NUM> in order to synchronize video data shown on the video sink device <NUM> with the audio data listened to on the Bluetooth device <NUM>. For example, after the A/V content <NUM> is received by the set-top box <NUM>, the audio data is output to the Bluetooth device <NUM> via the communication connection <NUM> without delay and the A/V data is output to the video sink device <NUM> via communication connection <NUM>, but only after applying the delay (e.g.,. <NUM> sec shown in <FIG>) stored in the RAM <NUM> or non-volatile memory <NUM>.

By implementing the manual calibration method and algorithm described above, the user can watch the A/V content such as television programing and movies on the video sink device <NUM> while listening to the audio data on the Bluetooth device <NUM>. By applying delay stored in the RAM <NUM> or non-volatile memory <NUM> to the A/V data output to the video sink device <NUM>, the lip-sync problem between the audio data listen to on the Bluetooth device <NUM> and video data watched on the video sink device <NUM> is avoided.

<FIG> and <FIG> are flow diagrams showing an exemplary algorithm and method that are part of the invention as specified by the appended set of claims. The exemplary algorithm and method are for achieving automatic synchronization between video data and audio data when using a Bluetooth device according to the present invention. The algorithm and method shown in <FIG> and <FIG> can be implemented in the exemplary system and set-top box <NUM> shown respectively in <FIG> and <FIG>. For example, the method and algorithm can be implemented by a program stored in the RAM <NUM> or the non-volatile memory <NUM> and executed by the controller <NUM>, such that the controller <NUM> controls the operations of the components and circuits of the set-top box <NUM> to perform operations of the method and algorithm. Although the detailed operation of the components and circuits of the set-top box <NUM> and the Bluetooth devices <NUM> have already been described in detail with reference to <FIG> and <FIG>, the method and algorithm for performing the automatic synchronization between video data and audio data when using Bluetooth devices <NUM> shown in <FIG> and <FIG> will be described with reference to some of the elements in <FIG> and <FIG>.

As shown in step S10, the controller <NUM> of the set-top box <NUM> transmits audio data and a timer marker message to the Bluetooth device <NUM> via the communication connection <NUM>. The audio data and timer marker message implemented by the set-top box <NUM> can be stored in the RAM <NUM> or the non-volatile memory <NUM> of the set-top box <NUM>. The audio data can be a small amount of sound data (e.g., chirp, beep or audio clip) and the time marker message can be time information such as a time stamp or other time information that indicates the time at which the audio data is transmitted from the set-top box <NUM> to the Bluetooth device <NUM>.

Once the Bluetooth device <NUM> receives the audio data and the time mark message from the set-top box <NUM>, in step S11 the controller <NUM> of the Bluetooth device <NUM> transmits an "Audio Received" response to the set-top box <NUM>. In step S12, the controller <NUM> of the Bluetooth device <NUM> also transmits an "Audio Data Processed" response to the set-top box <NUM>, after processing the audio data received by the set-top box <NUM>.

In step S13, the controller <NUM> of the set-top box <NUM> uses the time difference between initial transmission of the audio data and receipt of the "Audio Received" response from the Bluetooth device <NUM> as an indication of "a round-trip transmission time" between the set-top box <NUM> and the Bluetooth device <NUM>. Additionally, the controller <NUM> of the set-top box <NUM> uses the time difference between the initial transmission of the audio data from the set-top box <NUM> and receipt of the "Audio Data Processed" response from the Bluetooth device <NUM> as an indication of data processing time of data by the Bluetooth device <NUM>. In step S14, the controller <NUM> of the set-top box <NUM> uses both responses (i.e., first and second responses) from the Bluetooth device <NUM> as calibration information for calculating the delay for outputting the A/V data to the video sink device <NUM>. In this case, the delay for outputting the A/V data to the video sink device <NUM> is determined using the formula: Delay = <NUM>/<NUM> (Round Trip Bluetooth Transmission Time) + (Audio Processing Time).

The delay value is stored in the RAM <NUM> or the non-volatile memory <NUM> of the set-top box <NUM>. In step S15, the controller <NUM> of the set-top <NUM> uses the stored delay as calibration information for delaying the output of the A/V data to the video sink device <NUM> in order to synchronization video data shown on the video sink device <NUM> with the audio data listened to on the Bluetooth device <NUM>. For example, after the A/V content <NUM> is received by the set-top box <NUM>, the audio data is output to the Bluetooth device <NUM> via the communication connection <NUM> without delay and the A/V data is output to the video sink device <NUM> via communication connection <NUM>, but only after applying the delay stored in the RAM <NUM> or non-volatile memory <NUM>.

By implementing the automatic calibration described above, the user can watch the A/V content such as television programing and movies on the video sink device <NUM> while listening to the audio data on the one or more Bluetooth devices <NUM>. By applying delay stored in the RAM <NUM> or non-volatile memory <NUM> to the A/V data output to the video sink device <NUM>, the lip-sync problem between the audio data listen to on the Bluetooth device <NUM> and video data watched on the video sink device <NUM> is avoided.

<FIG> is a diagram illustrating synchronization issues between video data and audio data when there are multiple Bluetooth devices have different delays. As discussed previously with reference to <FIG>, synchronization problems between video data displayed on a TV (i.e., video sink device) and audio data listened to on a Bluetooth device <NUM> arise because there is a time delay in the processing and output of the audio data (e.g., primary and secondary delays) from the Bluetooth device, which causes noticeable and annoying lip-sync problems.

As shown in <FIG>, Bluetooth device <NUM> and Bluetooth device <NUM> not only have similar primary and secondary delay issues with respect to the presentation of A/V data by the TV described with reference to <FIG>, but also have the added problem of having different primary and secondary delays with respect to each other. That is, Bluetooth device <NUM> has more of a delay in the presentation of the audio data (with respect to the A/V data presented on the TV) than Bluetooth device <NUM>. Thus, there is a need to address the additional lip-sync problems created when using multiple Bluetooth devices have different primary and secondary delays with respect to each other.

<FIG> is a flow chart showing an exemplary algorithm and method that is part of the invention as specified by the appended set of claims. The exemplary algorithm and method are for achieving automatic synchronization between video data and audio data when using multiple Bluetooth devices having different primary and second delays (i.e., hereafter referred to as "lag delay") according to the present invention. The algorithm and method shown in <FIG> can be implemented in the exemplary system and set-top box <NUM> shown respectively in <FIG> and <FIG>. For example, the method and algorithm can be implemented by a program stored in the RAM <NUM> or the non-volatile memory <NUM> and executed by the controller <NUM>, such that the controller <NUM> controls the operations of the components and circuits of the set-top box <NUM> to perform operations of the method and algorithm.

Although the detailed operation of the components and circuits of the set-top box <NUM> and the Bluetooth devices <NUM> have already been described in detail with reference to <FIG> and <FIG>, the method and algorithm for performing the automatic synchronization between video data and audio data when using multiple Bluetooth devices shown in <FIG> will be described with reference to some of the elements in <FIG> and <FIG>.

In step S16, the controller <NUM> of the set-top box <NUM> transmits audio data and a timer marker message to each of the Bluetooth devices <NUM>. The audio data and timer marker message can be stored in the RAM <NUM> or the non-volatile memory <NUM> of the set-top box <NUM>. The audio data can be small amount of sound data (e.g., chirp, beep or audio clip) and the time marker message can be time information such as a time stamp or other time information that indicates the time at which the audio data is transmitted from the set-top box <NUM> to each of the Bluetooth devices <NUM>.

Once each Bluetooth device <NUM> receives the audio data and the time mark message from the set-top box <NUM>, in step S17 the controller <NUM> of each Bluetooth device <NUM> transmits an "Audio Received" response back to the set-top box <NUM>. The controller <NUM> of the set-top box <NUM> use the time difference between initial transmission of the audio data and receipt of the "Audio Received" response from each of the Bluetooth devices <NUM> as an indication of "round-trip transmission time" between the set-top box <NUM> and each of the Bluetooth devices <NUM> via the communication connections <NUM>. In step S18, the controller <NUM> of each of the Bluetooth devices <NUM> then transmits and "Audio Data Processed" response to the set-top box <NUM>, after each Bluetooth device has processed the audio data transmitted by the set-top box <NUM>.

The controller <NUM> of the set-top box <NUM> uses the time difference between the initial transmission of the audio data from the set-top box <NUM> and receipt of the "Audio Data Processed" response from each Bluetooth device <NUM> as an indication of the data processing time of audio data by each Bluetooth device <NUM>. In step S19, the controller <NUM> of the set-top box <NUM> uses the responses (i.e., the first response "Audio Received" and second response "Audio Data Processed") from the Bluetooth device <NUM> for determining the lag delay for each Bluetooth device <NUM>. For example, the controller <NUM> of the set-top box <NUM> uses the following formula for determining the lag delay for each Bluetooth device <NUM>: Lag Delay = <NUM>/<NUM> (Round Trip Bluetooth Transmission Time) + (Audio Processing Time).

In step S20, the controller <NUM> compares the calculated lag delays and determines a delay for outputting the A/V data to the audio sink device <NUM> (e.g., TV) via communication connection <NUM> (hereafter "first delay") and a delay for outputting the audio data to each of the Bluetooth devices <NUM> via communication connection <NUM> (hereafter "second delay"). For example, in the case of having three Bluetooth devices <NUM>, assume the controller <NUM> calculates the lag delays for the three Bluetooth devices (e.g., as in step S19) based on the responses received (e.g. as in steps S17 and S18) from each of the Bluetooth devices <NUM>. In this example, the Bluetooth devices <NUM> have the following calculated lag delays: <NUM>) the first Bluetooth device has a calculated lag delay of <NUM> sec; <NUM>) the second Bluetooth device <NUM> has a calculated lag delay of. <NUM> sec; <NUM>) and the third Bluetooth device <NUM> has a calculated lag delay of <NUM> sec.

The controller <NUM> of the set-top box <NUM> compares the lag delays (e.g., <NUM> sec, <NUM> sec, and <NUM> sec), and determines the assignment of the first delay for the audio sink device <NUM> and the second delay for each Bluetooth device <NUM>. Specifically, the controller <NUM> of the set-top box <NUM>, based on a comparison of all the calculated lag delays, determines the longest calculated lag delay among the Bluetooth devices <NUM>, and then assigns a value for the second delay to each of the Bluetooth devices <NUM> by taking the difference between the longest calculated lag delay (e.g. Bluetooth device <NUM>) and the other calculated lag delays (e.g., Bluetooth devices <NUM> and <NUM>), such that the second delay for each of the Bluetooth devices <NUM> is inversely proportional to the calculated lag delay for each Bluetooth device <NUM>. For example, the Bluetooth device <NUM> having the longest lag delay will be assigned the shortest second delay by the controller <NUM>.

In this example, the controller <NUM> would determine that the Bluetooth device <NUM> with the <NUM> sec lag delay has the longest lag delay among the three Bluetooth devices <NUM>. Thus, the second delay for each of the Bluetooth devices <NUM> would be assigned by the controller <NUM> as follows:.

The controller <NUM> would also assign a first delay to the audio sink device <NUM> that is equal to the longest lag delay. That is, controller <NUM> would assign a first delay to the audio sink device <NUM> of <NUM> sec, which is equal to the lag delay of Bluetooth device <NUM> will the longest lag delay (i.e., Bluetooth device <NUM>). The lag delay values, the first delays and the second delays can be stored in the RAM <NUM> or non-volatile memory <NUM> of the set-top box <NUM> for use by the controller <NUM>.

In step S21, the set-top <NUM> uses the stored delays as calibration information for outputting the A/V data to the video sink device <NUM> and for outputting the audio data to the Bluetooth devices <NUM> in order to achieve synchronization between the video data shown on the video sink device <NUM> and the audio data listened to on all the Bluetooth devices <NUM>. For example, after the A/V content <NUM> is received by the set-top box <NUM>, the audio data is output to the Bluetooth devices <NUM> via the communication connections <NUM> in accordance with the assigned second delay and the A/V data is output to the video sink device <NUM> via communication connection <NUM> according to the assigned first delay. By using the method and algorithm shown in <FIG>, all the Bluetooth devices <NUM> can be synchronized with each other as well as with the video sink device <NUM>. In the example provided above, three Bluetooth devices <NUM> were discussed. However, the method and algorithm shown in <FIG> can be implemented with any number of Bluetooth devices <NUM> receiving audio data from the set-top box <NUM>.

In an alternative embodiment that is not part of the invention as specified by the appended set of claims, the set-top box <NUM>, instead of using responses (e.g., first and second response) from the Bluetooth devices <NUM> as described with reference to <FIG> and <FIG>, may apply a first delay to the audio sink device <NUM> based on the determination of an acceptable lip-sync tolerance range for the Bluetooth devices <NUM>. In some cases, it may not be possible to receive lag delay information (e.g., primary and secondary delays) from a Bluetooth device for determining the appropriate first and/or second delays needed to determine acceptable synchronization between the A/V data displayed on the audio sink device <NUM> and the audio data listened to the Bluetooth devices <NUM>.

In this case, the set-top box <NUM> can still provide a better listening experience for the users of the Bluetooth devices <NUM> by applying a first delay to the video sink device <NUM> that would result in the synchronization between the A/V data displayed on the audio sink device <NUM> and the audio data listened to the Bluetooth devices <NUM> (i.e., lip-sync) that falls within an acceptable lip-sync tolerance range. For example, the set-top box <NUM> may implement the use of a lip-sync tolerance range of between +<NUM> milliseconds (ms) for an acceptable leading audio signal for the Bluetooth devices <NUM> and -<NUM> for an acceptable lagging audio signal for the Bluetooth devices <NUM>. That is, as long as the audio data listened to by the Bluetooth devices <NUM> falls within the acceptable lip-sync tolerance range, the user of the Bluetooth device <NUM> would experience acceptable synchronization between the A/V data displayed by the video sink device <NUM> and the audio data listened to by each of the Bluetooth devices <NUM>.

For example, lets assume the set-top box <NUM> assigns a value of. <NUM> seconds as the first delay for the video sink device <NUM>, which still results in a lip-sync delay at the Bluetooth devices of +<NUM> on one Bluetooth device <NUM> and -<NUM> on another Bluetooth device <NUM> (i.e., after receiving the audio data from the set-top box <NUM>). Although there is a respective lip-sync delay on each Bluetooth device <NUM>, the lip-sync delays are still with the acceptable lip-sync tolerance range for the Bluetooth devices of between +<NUM> for an acceptable leading audio signal and -<NUM> for an acceptable lagging audio signal.

Thus, the users of the Bluetooth devices can watch the A/V content such as television programing and movies on the video sink device <NUM> while listening to the audio data on the Bluetooth devices <NUM> without experiences noticeable lip-sync problems. The acceptable lip-sync tolerance range and the first delay can be predetermined or selected by the user, and stored in the RAM <NUM> or non-volatile memory <NUM> for use by the controller <NUM>. The acceptable lip-sync tolerance range and the first delay can also be adjusted to accommodate different types of Bluetooth devices and Bluetooth protocols being implemented.

As noted above, the present invention can be implemented not only as an apparatus or system, but also as a method and algorithm for achieving synchronization between the A/V data presented by a video sink device and audio data executed on one or Bluetooth devices. The present invention can be implemented as a program on a non-transitory computer-readable medium for causing a computer, such as the controller in set-top box, to execute the step described in <FIG>, <FIG> and <FIG>. The non-transitory computer-readable recording medium could be, for example, a CD-ROM, DVD, Blu-ray disc, or an electronic memory device.

The present invention may be implemented as any combination of a system, a method, an integrated circuit, and a computer program on a non-transitory computer readable recording medium. The controller and any other parts of the electronic apparatuses may be implemented as Integrated Circuits (IC), Application-Specific Integrated Circuits (ASIC), or Large Scale Integrated circuits (LSI), system LSI, super LSI, or ultra LSI components which perform a part or all of the functions of the electronic apparatuses, such as set-top boxes.

Each of the parts of the present invention can be implemented using many single-function components, or can be one component integrated using the technologies described above. The circuits may also be implemented as a specifically programmed general purpose processor, CPU, a specialized microprocessor such as Digital Signal Processor that can be directed by program instructions on a memory, a Field Programmable Gate Array (FPGA) that can be programmed after manufacturing, or a reconfigurable processor. Some or all of the functions may be implemented by such a processor while some or all of the functions may be implemented by circuitry in any of the forms discussed above.

The present invention may be a non-transitory computer-readable recording medium having recorded thereon a program embodying the methods/algorithms discussed above for instructing the controller to perform the methods/algorithms. Each of the elements of the present invention may be configured by implementing dedicated hardware or a software program on a memory controlling a processor to perform the functions of any of the components or combinations thereof. Any of the components may be implemented as a CPU or other processor reading and executing a software program from a recording medium such as a hard disk or a semiconductor memory.

The sequence of the steps included in the above described algorithms is exemplary. Moreover, steps, or parts of the algorithm, may be implemented simultaneously or in parallel.

The components of the present invention can be in the form of a set-top box as in the exemplary embodiments disclosed above, or in other standalone devices, or may be incorporated in a television or other content playing apparatus, or other device or appliance. The scope of the present invention is specified by the appended set of claims.

Claim 1:
An electronic apparatus (<NUM>) for achieving synchronization between video data displayed on a video sink device (<NUM>) and audio data presented on one or more Bluetooth devices (<NUM>), the electronic apparatus (<NUM>) comprising:
an input circuit (<NUM>) that receives audio/video (A/V) content from an A/V content provider;
an A/V decoder (<NUM>) that decodes the A/V content to obtain video data and audio data;
an output circuit (<NUM>) that outputs the video and audio data to the video sink device;
a Bluetooth transceiver (<NUM>) that wirelessly communicates information with the one or more Bluetooth devices according to a wireless protocol, the information including the audio data;
a controller (<NUM>) that
outputs a time marker message and an audio clip to the one or more Bluetooth devices, and receives responses from the one or more Bluetooth devices based on receipt of the time marker message and an audio clip, the responses including a first response with information to determine a round trip Bluetooth transmission time between the electronic apparatus and the one or more Bluetooth devices, and a second response with information on an audio processing time by the one or more Bluetooth devices of the audio clip, wherein the controller uses the first and second responses from the one or more Bluetooth devices as second calibration information;
obtains the second calibration information for the one or more Bluetooth devices, the second calibration information being associated with delays relating to the Bluetooth transceiver transmitting the audio data to the one or more Bluetooth devices using the wireless protocol and the one or more Bluetooth devices processing the audio data,
determines a first delay for outputting the video and audio data to the video sink device using the second calibration information, the first delay representing a time difference between the video sink device displaying video data and the one or more Bluetooth devices outputting audio data corresponding to the video data,
controls the Bluetooth transceiver to output the audio data to the one or more Bluetooth devices, and
controls the output circuit to apply the first delay so as to output the video and audio data to the video sink device after the first delay, such that the video data displayed by the video sink device is synchronized with the audio data output by the one or more Bluetooth devices.