Patent Description:
The term streaming media describes the playback of media on a playback device, where the media is stored on a server and continuously sent to the playback device over a network during playback. For purposes of this discussion, media and/or encoded media are defined as data of a work that includes video, audio, pictures, or another type of presentation that may be displayed, played or in some other way presented by a playback device. Typically, the playback device stores a sufficient quantity of media in a buffer at any given time during playback to prevent disruption of playback due to the playback device completing playback of all the buffered media prior to receipt of the next portion of media. Adaptive bit rate streaming or adaptive streaming involves detecting the present streaming conditions (e.g. the user's network bandwidth and CPU capacity) in real time and adjusting the quality of the streamed media accordingly. Typically, the source media is encoded at multiple bit rates and the playback device or client switches between streaming the different encodings depending on available resources.

Adaptive streaming solutions typically utilize either Hypertext Transfer Protocol (HTTP), published by the Internet Engineering Task Force and the World Wide Web Consortium as RFC <NUM>, or Real Time Streaming Protocol (RTSP), published by the Internet Engineering Task Force as RFC <NUM>, to stream media between a server and a playback device. HTTP is a stateless protocol that enables a playback device to request a byte range within a file. HTTP is described as stateless, because the server is not required to record information concerning the state of the playback device requesting information or the byte ranges requested by the playback device in order to respond to requests received from the playback device. RTSP is a network control protocol used to control streaming media servers. Playback devices issue control commands, such as "play" and "pause", to the server streaming the media to control the playback of media files. When RTSP is utilized, the media server records the state of each client device and determines the media to stream based upon the instructions received from the client devices and the client's state.

In adaptive streaming systems, the source media is typically stored on a media server as a top level index file pointing to a number of alternate streams that contain the actual video and audio data. Each stream is typically stored in one or more container files. Different adaptive streaming solutions typically utilize different index and media containers. The Synchronized Multimedia Integration Language (SMIL) developed by the World Wide Web Consortium is utilized to create indexes in several adaptive streaming solutions including IIS Smooth Streaming developed by Microsoft Corporation of Redmond, Washington, and Flash Dynamic Streaming developed by Adobe Systems Incorporated of San Jose, California. HTTP Adaptive Bitrate Streaming developed by Apple Computer Incorporated of Cupertino, California implements index files using an extended M3U playlist file (. M3U8), which is a text file containing a list of URIs that typically identify a media container file. The most commonly used media container formats are the MP4 container format specified in MPEG-<NUM> Part <NUM> (i.e. ISO/IEC <NUM>-<NUM>) and the MPEG transport stream (TS) container specified in MPEG-<NUM> Part <NUM> (i.e. ISO/IEC Standard <NUM>-<NUM>). The MP4 container format is utilized in IIS Smooth Streaming and Flash Dynamic Streaming. The TS container is used in HTTP Adaptive Bitrate Streaming.

Adaptive bitrate streaming performs adequately for playback of media content on a device. However, there is currently no viable way to synchronize the playback of streamed content on two or more playback devices such that there is no discernible difference between the presentation times of the content on the two or more devices. Currently, the only way to synchronize the device is for the user to manually control the start and stop of the playback on each of the devices to synchronize the playback which is often impossible or at least very difficult to achieve. Furthermore, the streaming of Over The Top (OTT) content or other presentation time sensitive content using adaptive bit rate streaming and other streaming processes is becoming more prevalent. OTT content is content that a media content provider delivers but may not have any control of the content and can only provide the content as it is received. The best example of OTT content is a live broadcast. During a live broadcast, the content is streamed directly to the playback devices to be played shortly after it is received in real time. Due to differences in the capabilities of playback devices and the network connections of the devices, the timing of receipt and playback of the streamed content on different devices may not be synchronized. As such, the manual synchronization is even more difficult than it is for synchronizing the playback of stored content on playback devices.

Prior art document <CIT> presents a local network which controls the synchronization of distribution and playback of media content for several playback devices via a centralized reference clock.

The lack of synchronization may be a problem when one or more devices are performing the playback within close proximity of one another as the lack of synchronization may be noticeable by viewers. For example, two or more playback devices may be set up in a restaurant to view a live sporting event. If the playback is not synchronized, the user may notice the broadcast delay between devices.

Systems and methods, according to the independent claims <NUM> and <NUM> respectively, for synchronizing the playback of streamed OTT content or other time sensitive content in accordance with embodiments of this invention are disclosed. In accordance with embodiments of this invention, a method for playback of encoded media performed by a playback device be performed by the following actions. The playback device transmits a request to a network time provider. In response to the request, the playback device receives the time information from the network time provider. The playback device sets a playback clock in the device using the received time information. In order to provide a playback of a stream of content, the playback device receives stream initiation information for the stream of encoded media from a stream media provider. The stream initiation information includes a stream start time. The playback device then receives the stream of encoded media. The stream includes frames of the encoded content. The presentation time information of each of the frames of the stream is adjusted in the playback device based upon the stream initiation information.

In accordance with some embodiments of this invention the network time provider is a Network Time Protocol Server and the transmitting of the request and receiving of the time information are performed using Network Time Protocol (NTP). Furthermore, the time information and/or the start time is provided in Coordinated Universal Time in accordance with some embodiments this invention.

In accordance with some other embodiments of this invention, the network time provider is a local time server. In accordance with some of these embodiments, the local time server performs the following process to obtain time information. The local time server transmits a request for time information to a Network Time Protocol server. In response to the request, the local time server, receives the time information from the Network Time Protocol server in the local time server. The local time server then may receive the request for time information from the playback device. The local time server then generates the time information for the playback device from the timing information received from the Network Time Protocol server and transmits the time information to the playback device.

Turning now to the drawings, systems and methods for synchronizing playback of OTT content by multiple clients is provided. In particular, embodiments of this invention can use time information from a common source to set the playback clocks in the individual playback devices. The playback devices then receive start time information from a media content provider that indicate the time for the start of playback. The start time information is determined using the time information from the common source. Each device then adjusts the presentation time stamps of the frames in the received content based on the received start time information.

Turning now to the <FIG>, an adaptive streaming system in accordance with an embodiment of the invention is illustrated. The adaptive streaming system <NUM> includes a source encoder <NUM> configured to encode source media as a number of alternative streams. In the illustrated embodiment, the source encoder is a server. In other embodiments, the source encoder can be any processing device including a processor and sufficient resources to perform the transcoding of source media (including but not limited to video, audio, and/or subtitles). Typically, the source encoding server <NUM> generates a top level index to a plurality of container files containing the streams, at least a plurality of which are alternative streams. Alternative streams are streams that encode the same media content in different ways. In many instances, alternative streams encode media content (such as but not limited to video) at different maximum bitrates. In a number of embodiments, the alternative streams are encoded with different resolutions and/or at different frame rates. The top level index file and the container files are uploaded to an HTTP server <NUM>. A variety of playback devices can then use HTTP or another appropriate stateless protocol to request portions of the top level index file and the container files via a network <NUM> such as the Internet.

Time server <NUM> is a server that provides Coordinated Universal Time (UTC) or equivalent information. In the shown embodiment, the time server <NUM> is a Network Time Protocol (NTP) server or an equivalent system that may be maintained by a third party. The use of NTP allows the time server <NUM> to provide time information to the devices allow internal clocks in the individual devices that are set using the time information to be synchronized within tens (<NUM>) of milliseconds (ms) of one another.

In the illustrated embodiment, playback devices include personal computers <NUM>, CE players, and mobile phones <NUM>. In other embodiments, playback devices can include consumer electronics devices such as DVD players, Blu-ray players, televisions, set top boxes, video game consoles, tablets, and other devices that are capable of connecting to a server via HTTP and playing back encoded media. Although a specific architecture is shown in <FIG>, any of a variety of architectures including systems that perform conventional streaming and not adaptive bitrate streaming can be utilized that enable playback devices to request portions of the top level index file and the container files in accordance with embodiments of the invention.

As illustrated, some playback devices, such as user computers <NUM>, are connected to the network <NUM> via a Local Area Network (LAN) <NUM>. LAN <NUM> may include a local time server <NUM>. Local time server <NUM> receives time information from the NTP server <NUM> and generates local time information based upon the time information received from the NTP server <NUM>. Although shown as a server, one skilled in the art will recognize that any device connected to LAN <NUM> that can provide time information may be used without departing from embodiments of this invention. The local time information of local time server <NUM> may be provided to the playback devices connected to the LAN <NUM> to ensure that better time synchronization between the connected devices. The use of local time server <NUM> connected to the LAN <NUM> may be used when a synchronization of less than the <NUM> of ms achieved using network time server <NUM> is desired. Network time servers, playback devices, and local time servers in accordance with embodiments of the invention are discussed further below.

Some processes for providing methods and systems in accordance with embodiments of this invention are executed by a playback device. The relevant components in a playback device that can perform the processes in accordance with an embodiment of the invention are shown in <FIG>. One skilled in the art will recognize that playback device may include other components that are omitted for brevity without departing from this invention. The playback device <NUM> includes a processor <NUM>, a non-volatile memory <NUM>, and a volatile memory <NUM>. The processor <NUM> is a processor, microprocessor, controller, or a combination of processors, microprocessor, and/or controllers that performs instructions stored in the volatile <NUM> or non-volatile memory <NUM> to manipulate data stored in the memory. The non-volatile memory <NUM> can store the processor instructions utilized to configure the playback device <NUM> to perform processes including processes in accordance with embodiments of the invention and/or data for the processes being utilized. In other embodiments, the playback device software and/or firmware can be stored in any of a variety of non-transitory computer readable media appropriate to a specific application.

Some processes for providing methods and systems in accordance with embodiments of this invention are executed by the HTTP server; source encoding server; and/or local and network time servers. The relevant components in a server that perform the processes in accordance with embodiments of the invention are shown in <FIG>. One skilled in the art will recognize that a server may include other components that are omitted for brevity without departing from the embodiments of this invention. The server <NUM> includes a processor <NUM>, a non-volatile memory <NUM>, and a volatile memory <NUM>. The processor <NUM> is a processor, microprocessor, controller, or a combination of processors, microprocessor, and/or controllers that performs instructions stored in the volatile <NUM> or non-volatile memory <NUM> to manipulate data stored in the memory. The non-volatile memory <NUM> can store the processor instructions utilized to configure the server <NUM> to perform processes including processes in accordance with embodiments of the invention and/or data for the processes being utilized. In other embodiments, the server software and/or firmware can be stored in any of a variety of non-transitory computer readable media appropriate to a specific application. Although a specific server is illustrated in <FIG>, any of a variety of server configured to perform any number of processes can be utilized in accordance with embodiments of the invention.

In accordance with embodiments of this invention, playback devices obtain time information from a common source and use the time information to set the playback clock of the devices. The playback devices also receive playback time information from the stream content provider. The playback time information is then used to adjust the presentation time of frames in the stream based upon the playback clock. Since the playback clocks have been set based upon time information from a common source there should by less than <NUM> of ms delay between the presentations. A timing diagram of messages and information passed between devices over a network in accordance some embodiments of this invention is illustrated in <FIG>.

In <FIG>, the devices include a first playback device, a second playback device, a media server, and a network time server. The first and second playback devices are devices capable of presenting streamed encoded media content. The media server is a system that is providing the streamed content over the network to the first and second playback devices. The network time server is a time server that provides time information. In accordance with some embodiments, the time information is provided using NTP. Furthermore, the time information provides time information based upon Coordinated Universal Time (UTC).

In accordance with the timing diagram of <FIG>, the first playback device transmits a request <NUM> to the time server for time information and receives time information <NUM> from the time server in response to the request. Likewise, the second playback device transmits a request <NUM> to the time server for time information and receives time information <NUM> from the time server in response to the request. Furthermore, the media server also transmits a time information request <NUM> to the time server and receives time information <NUM> from the time server in response to the request. In accordance with some embodiments, these exchanges are performed using NTP. However, other means for performing these exchanges of messages may be used without departing from embodiments of this invention. Furthermore, one skilled in the art will recognize that devices do not need to receive the time information from the same time server as long as the time information provided by the different time servers is generated from a common source. One skilled in the art will recognize that due to the network bandwidth and configuration of the networks connected to the devices that some delay may be introduced between the time information provided to the various devices by the time server.

The first and second playback devices use the time information received to set the respective playback clocks in each playback device. The first playback device transmits a request <NUM> for OTT content or other presentation time sensitive content to the media server. The media server transmits stream initiation information for a stream of encoded media for the desired content <NUM> to the first playback device. Similarly, the second device transmits a request <NUM> for the same OTT content or other presentation for time sensitive content to the media server. The media server transmits stream initiation information for a stream of encoded media for the desired content <NUM> to the second playback device. In accordance with some embodiments, the stream initiation information includes a start time based on the UTC.

The media server then commences transmitting streams of the media content <NUM> and <NUM> to each of the first and second playback devices. Each of the first and second devices then adjusts the Presentation Time Stamp (PTS) of each frame received in the stream of media content based upon stream initiation information. In accordance with some of these embodiments, the start time in the stream will be based upon the playback clock derived from UTC and the UTC of the start time. Since the start time of the stream is derived from the UTC and the playback clock is derived from the UTC, the delay between the presentations will be in the <NUM> of ms based on the difference of time information received by the various devices.

In some embodiments, it may be desirable to have the synchronization of playback of the content include less than <NUM> of ms of delay. This may be the case where devices are providing playback of the content in close proximity to one another. A timing diagram of a system that provides for greater synchronization between playback devices in accordance with some embodiments of this invention is shown in <FIG>.

The system in <FIG> includes a first playback device, a second playback device, a local time server, a media server, and a network time server. The first and second playback devices are devices capable of presenting streamed encoded media content and each of these devices is connected to a LAN. The LAN includes a local time server. The local time server is a server that provides time information to devices connected to the LAN. In accordance with these embodiments, the time information is based upon time information received from a network time server as explained further below. The media server is a system that is providing the streamed content over the network to the first and second playback devices. The network time server is a time server that provides time information. In accordance with some embodiments, the time information is provided using NTP. Furthermore, the time information provides time information specified in terms of UTC.

In accordance with the timing diagram of <FIG>, the local time server transmits a request <NUM> for time information to the network time server and receives the time information <NUM> from the time server in response. The media server also transmits a time information request <NUM> to the time server and receives time information <NUM> from the time server in response to the request. In accordance with some embodiments, these exchanges are performed using NTP. However, other means for performing these exchanges of messages may be used without departing from embodiments of this invention. Furthermore, one skilled in the art will recognize that servers do not need to receive the time information from the same time server as long as the time information provided by the different time servers is generated from a common source. One skilled in the art will also recognize that due to the network bandwidth and configuration of the networks connected to the servers that some delay may be introduced between the time information provided to the various devices by the time server.

The local time server then uses the time information to generate time information for use by the playback devices connected to the LAN. The first playback device transmits a request <NUM> to the local time server for time information and receives time information <NUM> from the local time server in response to the request. Likewise, the second playback device transmits a request <NUM> to the local time server for information and receives time information <NUM> from the local time server in response to the request. The first and second playback devices use the time information received to set the respective playback clocks in each device. As the time information is received over a LAN from the local server, there is less delay introduced into the time information. Thus, the playback clocks of the first and second playback devices will be more closely synchronized.

To start playback, the first device transmits a request <NUM> for OTT content or other presentation time sensitive content to the media server. The media server transmits stream initiation information for a stream of encoded media for the desired content <NUM> to the first playback device. Similarly, the second device transmits a request <NUM> for the same OTT content or other presentation for time sensitive content to the media server. The media server transmits stream initiation information for a stream of encoded media for the desired content <NUM> to the second playback device. In accordance with some embodiments, the stream initiation information includes a start time based on the UTC.

The media server then commences transmitting streams of the media content <NUM> and <NUM> to each of the first and second playback devices. Each of the first and second devices then adjusts the Presentation Time Stamp (PTS) of each frame received in the stream of media content based upon stream initiation information. In accordance with some of these embodiments, the start time in the stream will be based upon the playback clock derived from UTC information received from the local time server and the UTC of the start time. Since the start time of the stream is derived from the UTC and the playback clock is derived from the UTC received from the local time server instead of the network time server, the delay between the presentations will typically be less than the <NUM> of ms of the system described above in reference to <FIG>. This is because the internal clocks of the individual playback devices are more closely synchronized due to the lack of latency between the individual playback devices and the local time server of the LAN.

A process performed by a playback device to provide synchronized playback in accordance with some embodiments of this invention is illustrated in <FIG>. The process <NUM> includes transmitting a request for time information to a time provider (<NUM>). As discussed with reference to <FIG> and <FIG>, the time provider may be, but is not limited to, a network time server and/or a local time server. The device receives time information from the time provider (<NUM>). The time information is preferably based upon the UTC. The time information is used by the playback device to set a media playback clock (<NUM>).

When the playback device plays back OTT or other presentation time content, the playback device receives stream initiation information (<NUM>). In accordance with some embodiments, the stream initiation information may be in the top level index of the requested content or provided along with some other type of administrative information such as metadata or the like. In accordance with some embodiments, the stream initiation information includes a start time based on the UTC. The playback device then receives the stream of media content. As the stream is received, the playback device adjusts the presentation time information of each frame based upon the stream initiation information and the playback clock (<NUM>). The presentation time information may be the Presentation Time Stamp (PTS) of a frame, PTS of a reference frame for a frame, or any other information that may be used to determine the time at which a frame may be presented. In accordance with some of these embodiments, the start time and the time of the playback clock are used to adjust the PTS of each frame. In particular, the difference between the start time received from the media server and the playback clock may be used to adjust the PTS of each frame. The playback of the stream then commences (<NUM>).

A process for providing stream initiation time information performed by a media server in accordance with embodiments of this invention is illustrated in <FIG>. Process <NUM> includes transmitting a request for time information to a network time provider (<NUM>). As discussed with reference to <FIG> and <FIG>, the time provider may be, but is not limited to, a network time server. The media receives time information from the time provider (<NUM>). The time information is preferably based upon the UTC. The time information is used by the media server to determine a stream initiation time.

As discussed above, the stream initiation time may be a start time that is based upon the UTC.

The stream initiation time is then used to generate stream initiation information for the stream of content (<NUM>). When a request for a stream of the content is received (<NUM>), the stream initiation information is provided to the requesting playback device (<NUM>) and the content is streamed to the playback device (<NUM>).

Claim 1:
A method (<NUM>) for playback of encoded media performed by a set of playback devices (<NUM>), comprising:
transmitting (<NUM>) a request for time information (<NUM>, <NUM>) from a set of two or more playback devices connected to a local network (<NUM>) to a network time provider for the local network;
receiving (<NUM>) the time information (<NUM>, <NUM>) from the network time provider (<NUM>) in the set of playback devices;
the method being characterized in:
setting (<NUM>) a playback clock within each playback device of the set of playback devices using the received time information wherein each playback clock in each playback device of the set of playback devices are set from time information from a common source;
receiving (<NUM>) stream initiation information for a stream of encoded media from a stream media provider (<NUM>) in each playback device of the set of playback devices wherein the stream initiation information includes a stream start time;
receiving the stream of encoded media in each playback device wherein the stream includes a plurality of frames of encoded content; and
adjusting (<NUM>) presentation time information of each of the plurality of frames of the stream in each playback device of the set of playback devices based upon the stream initiation information and time information from each respective playback clock; and
presenting (<NUM>) each of the plurality of frames in a synchronized playback using each playback device of the set of playback devices based upon the adjusted presentation time information in each of the set of playback devices.