Method and apparatus for synchronizing clocks

A system receives multiple data samples and determines time stamp values associated with each of the multiple data samples. The system identifies an earliest time stamp value and uses that time stamp value as an initial system clock value. The system may also subtract a delay factor from the earliest time stamp value to account for delay in decoding the multiple data samples. The earliest time stamp value is also communicated to an audio decoder and a video decoder. The audio and video decoders provide clock data back to a component that maintains the system clock value. If the difference between the clock data received from the decoders and the system clock value exceeds a threshold value, the system clock value is recalculated.

TECHNICAL FIELD

The systems and methods described herein relate to synchronizing a receiving clock with a source clock.

BACKGROUND

In a media distribution environment, one or more source devices communicate media content to one or more receiving devices. In this environment, it is important for the source devices and the receiving devices to have synchronized clocks. Synchronizing source and receiving clocks is important, for example, to maintain synchronization between the audio and video data presented by the receiving devices. If the audio and video data is not synchronized, the audio data will not be presented at the correct time with respect to the video data. For example, if the audio and video data is not synchronized, the movement of a person's lips presented from the video data may not correspond to the spoken word(s) presented from the audio data. The audio presentation may appear to be “ahead of” or “behind” the video presentation.

Some media distribution environments include a time synchronization protocol that allows source devices and receiving devices to exchange clock information. An example time synchronization protocol may be provided via the MPEG-2 (Moving Picture Experts Group) system layer, which defines transport streams and program streams. However, other media distribution environments do not utilize a time synchronization protocol. In particular, certain networked media playback systems do not use a time synchronization protocol. These systems need to maintain synchronization between the source device clock and the receiving device clock without the benefit of a time synchronization protocol.

SUMMARY

The systems and methods described herein support the synchronization of a receiving device clock with a source device clock by using information contained within a data stream, thereby eliminating the need for an additional time synchronization protocol. In one embodiment, the receiving device clock is synthesized using the information in the data stream. Multiple data samples are received and an earliest time stamp value is identified. The earliest time stamp value is communicated to a decoder and is used as an initial system clock value. Clock data is received from the decoder. If a difference between the clock data received from the decoder and the system clock value exceeds a threshold value; the system clock value is recalculated.

DETAILED DESCRIPTION

The systems and methods described herein permit the synchronization of a receiving device clock with a source device clock by using information contained within a data stream, thereby eliminating the need for an additional time synchronization protocol. These systems and methods synthesize a clock on the receiving device using the information in the data stream. Upon initialization of a data stream, a system time clock in the receiving device is set using information contained in the data stream. The system time clock in the receiving device is then incremented based on the receiving device's internal clock. Thus, the receiving device clock remains synchronized with the source device clock as long as the two clocks run at the same rate. If there is a discontinuity in the data stream, the system time clock in the receiving device is reset using information contained in the data stream.

As used herein, the term “receiving device” may also be referred to as a “receiver”, a “rendering device”, or a “network receiver.” Additionally, the term “source device” may also be referred to as a “transmitter” or a “media transmitter.” The term “data sample” may also be referred to as a “frame” or a “data frame.”

Particular examples discussed herein relate to a network-based media player environment. However, the systems and methods discussed herein are applicable to any environment containing any type of source device and receiving device. Additionally, these systems and methods may be applied to any type of data communicated using any communication protocol and communication medium.

FIG. 1illustrates an example environment100in which the methods and systems described herein can be implemented. A transmitter102receives data (such as audio data, video data, and other media content) from multiple sources, such as a tuner104, a file source106, and a media source110. Tuner104receives signals, such as broadcast signals, from any number of sources using various mediums. Tuner104may receive signals via a cable television link, a satellite communication link, an over-the-air broadcast, and the like. Tuner104tunes, demultiplexes, and/or encodes received signals. The tuner then provides the tuned/demultiplexed/encoded data to transmitter102. Alternatively, tuner104may communicate data to file source106, which stores the received data to a storage device108, such as a hard disk drive. File source106is also capable of retrieving data from storage device108and providing the retrieved data to transmitter102. File source106may handle any type of data, such as audio data, video data, configuration data, and the like. Storage device108can store various types of recorded content, such as music, movies and television programs.

Media source110communicates data to transmitter102via a data communication network112, such as the Internet. Media source110may generate its own data or may receive data from another device or system (not shown) coupled to media source110. For example, media source110may provide streamed audio and/or video content to transmitter102. Transmitter102may receive data from any number of tuners104, file sources106, media sources110, or other sources not shown inFIG. 1.

Transmitter102communicates data to a receiver114via a data communication network116. Data communication network116can be any type of network capable of communicating data, such as a local area network (LAN) or the Internet. Receiver114receives audio and video data from transmitter102and manages the processing of that data such that the audio and video data is presented synchronously. Receiver114includes an internal clock118that operates as discussed herein. Receiver114is coupled to an audio decoder120and a video decoder122via communication links124and126, respectively. Audio decoder120contains a digital-to-analog converter to convert received digital audio data into analog audio data. Audio decoder120generates an audio output that is provided to an audio rendering device (not shown), such as a speaker or similar device. Video decoder122contains a digital-to-analog converter to convert received digital video data into analog video data. Video decoder122generates a video output that is provided to a video rendering device (not shown), such as a television, projector, monitor, or other display device. In alternate embodiments, audio decoder120generates a digital audio signal and video decoder122generates a digital video signal.

Audio decoder120and video decoder122are responsible for decompressing a media stream (e.g., MPEG-1, MPEG-2, MPEG-4, or Windows Media) and delivering uncompressed output data to a renderer in a synchronized manner. A video renderer is the output section of a video pipeline and typically includes a video scaler and a video encoder. The video renderer is responsible for converting the digital image to the correct output signal format. For example, for NTSC (National Television System Committee) output, the video renderer converts the scaled and mixed digital output to an analog signal that conforms to the NTSC standard. For audio data, the renderer typically includes an audio mixer and a DAC (Digital-to-Analog Converter). The audio decoder outputs PCM samples that can be mixed by the audio mixer with local audio (e.g., user interface audio such as beeps and clicks) and transferred to an output DAC that takes a digital stream, converts the stream to an analog format, amplifies the analog signal and provides the signal to a speaker. Certain embodiments may omit the digital-to-analog conversion process.

Receiver114is also coupled to clock modules128and130. Clock module128is coupled to audio decoder120and clock module130is coupled to video decoder122. Clock modules128and130are reference clocks that are used by decoders120and122, respectively, to determine when to present particular data samples. Decoders120and122use clock modules128and130, respectively, to synchronize the delivery of a data sample and provide clock information (such as when to present particular frames of data) used by audio decoder120and video decoder122, respectively. For example, clock module128provides clock information to audio decoder120indicating when certain frames of audio data received on communication link124should be presented or rendered (i.e., output from audio decoder102to an audio rendering device). Similarly, clock module130provides clock information to video decoder122indicating when certain frames of video data received on communication link126should be presented or rendered (i.e., output from video decoder122to a video rendering device).

Audio decoder120is also coupled to receiver114via an audio clock data communication link132. Communication link132provides clock information associated with audio decoder120back to receiver114. Receiver114uses this clock information to determine whether audio decoder120is using the correct time for presenting audio data. Video decoder122is also coupled to receiver114via a video clock data communication link134. Communication link134provides clock information associated with video decoder122back to receiver114. Receiver114uses the received clock information to determine whether video decoder122is using the correct time for presenting video data. The operation of receiver114and related components shown inFIG. 1are discussed below with respect toFIGS. 2-5.

In a particular embodiment, tuner104is contained within a computing device along with transmitter102and any number of other devices. In other embodiments, transmitter102and tuner104are separate devices.FIG. 1shows receiver114, audio decoder120, video decoder122and clock modules128and130as separate components. However any two or more of these components may be combined into a single component. Additionally, receiver114, audio decoder120, video decoder122and clock modules128and130may all be contained in a single device, such as a networked media receiver.

Although two separate clock modules128and130are shown inFIG. 1, alternate embodiments may utilize a single clock module. If two separate clock modules128and130are used (as shown inFIG. 1), the two clock modules need to be synchronized with each other. In one implementation, this synchronization is performed by seeding clock module128with the new stream time and starting the clock in clock module128. Next, the audio decoder120is activated and the current value of clock module128is determined. Clock module130is then seeded with the current value read from clock module128. The clock in clock module130is then started and the video decoder122is activated. A similar procedure is followed each time the clock value is re-seeded.

FIG. 2is a flow diagram illustrating an embodiment of a procedure200for determining initial clock data. In the described embodiment, procedure200is implemented by receiver114. Initially, procedure200receives and buffers multiple data samples (202). Data samples may also be referred to as “frames” or “frames of data.” These data samples may include audio and/or video data. The procedure then identifies PTS (Presentation Time Stamp) values associated with the received data samples (block204). PTS values define a time at which a data sample is to be presented. PTS values are synchronized to a PCR (Program Clock Reference) value. For example, if the PCR has a current value of 120 and the PTS value is 120, the decoder should present the data sample associated with the PTS value. PTS values are sent at periodic intervals, such as every 33 ms for video data (based on 30 frames per second) and every 70 ms for audio data. PTS values may also be referred to as “time stamp values” and a PCR value may also be referred to as a “system clock value.”

Referring again toFIG. 1, audio decoder120compares the reference clock value provided by clock module128to the next data sample's PTS value. If the next PTS value and the current reference clock value provided by clock module128are approximately the same (e.g., within ½ of the data sample duration), audio decoder120will present the data sample. Similarly, video decoder122compares the reference clock value provided by clock module130to the next data sample's PTS value. If the next PTS value and the current reference clock value provided by clock module130are approximately the same (e.g., within ½ of the data sample duration), video decoder122will present the data sample.

Clock modules128and130are capable of having their reference clock values changed—e.g., seeded with a new time value based on incoming PTS values from the data stream during data stream start-up. Clock modules128and130are also capable of starting and stopping the running of their reference clocks.

FIG. 3Aillustrates example PTS values associated with five received video data samples. In the example ofFIG. 3A, the five PTS values are each separated by approximately 33 ms.FIG. 3Billustrates example PTS values associated with five received audio data samples. In the example ofFIG. 3B, the five PTS values are each separated by approximately 70 ms. The video data samples shown inFIG. 3Ado not necessarily correspond to the audio data samples shown inFIG. 3B. In a particular embodiment, the PTS values associated with audio data share the same frequency domain as PTS values associated with corresponding video data. For example, the frequency domain for MPEG-2 is a derivative of 27 Mhz in which 90 Khz is the base. Audio and video data samples contain a PTS value for the respective data sample they describe. In this embodiment, all audio data samples include a PTS value and every third or fourth video data sample includes a PTS value. The source of the data samples includes a number of PTS values to ensure that the audio decoder and the video decoder provide a synchronous delivery of data.

Referring back toFIG. 2, after identifying PTS values associated with the received data samples, any erroneous data samples are ignored (block206).FIG. 3Cillustrates an example of an erroneous PTS value. PTS value302is inconsistent with the other four PTS values. The other four PTS values are spaced apart by approximately 33 ms, which is appropriate for 30 frames/sec. video data. However, PTS value302is not consistent with this spacing and is considered erroneous. Thus, erroneous data samples are identified by finding irregularities in the time spacing of PTS values. If a particular data sample has a PTS value that is significantly different from the other PTS values, that data sample is ignored. For example, in the case of 30 frames/sec. video data, a difference in PTS values significantly greater than 33 ms indicates an erroneous value.

Procedure200continues by identifying the earliest PTS value associated with the data samples (block208). For example, inFIG. 3A, the earliest PTS value is 100 ms. Similarly, inFIG. 3C, the earliest non-erroneous PTS value is 2000 ms (the erroneous PTS value of 100 ms is ignored). By identifying the earliest PTS value, the procedure accounts for the possibility that data samples were not necessarily received in chronological order. In one implementation, the first five PTS values are identified. If one or more of the first five data samples are chronologically out of order, the procedure selects the earliest PTS value among the five values, thereby avoiding the situation where the starting PTS value is later than other received PTS values.

When seeding (or re-seeding) the clock based on incoming PTS values, the system considers PTS values from both the audio data stream and the video data stream. Although the various PTS values will have the same time domain, the PTS values may have different arrival times due to network congestion, processor overload, etc. In one embodiment, PTS values associated with several audio samples and several video samples are evaluated to identify the earliest PTS value.

After identifying the earliest PTS value, the procedure applies a delay factor to the PTS value to account for decoding delay (block210). This decoding delay occurs, for example, as receiver114, audio decoder120and video decoder122process audio and video data to generate audio output and video output. In one implementation, this decoding delay is a constant value determined based on testing of the decoding components or by estimating the delay. Alternatively, the decoding delay may vary based on current processing delays, system usage and the like. The use of a decoding delay is optional. Alternate embodiments omit block210and do not consider a decoding delay when calculating PCR values discussed below.

Procedure200continues by using the earliest PTS value minus the delay factor as a seed value for the PCR value (block212). This seed PCR value is stored in clock118contained in receiver114. The seed PCR value is then communicated to the audio decoder and the video decoder (block214). For example, receiver114communicates the PCR value to clock modules128and130, which then provide the PCR value to audio decoder120and video decoder122, respectively. Clock modules128and130regulate playback of data by audio decoder120and video decoder122, respectively. At this point, the receiver and related decoding components begin processing received data samples as discussed below.

FIG. 4is a flow diagram illustrating an embodiment of a procedure400for processing data. Initially, audio data samples and video data samples are received (block402), for example, by receiver114. The received audio data samples are communicated to an audio decoder (block404), such as audio decoder120. The received video data samples are communicated to a video decoder (block406), such as video decoder122. Procedure400continues by receiving audio clock data from the audio decoder and video clock data from the video decoder (block408). In the example ofFIG. 1, the audio clock data is received on communication link132and the video clock data is received on communication link134. The audio clock data and the video clock data identifies the time associated with the data that the audio decoder and video decoder are currently presenting. In one implementation, the audio decoder and the video decoder provide audio clock data and video clock data at regular intervals (e.g., when presenting each data sample). Alternatively, the receiver may periodically request that the audio decoder and the video decoder send their current clock data to the receiver.

The procedure continues by comparing the audio clock data and the video clock data to a receiver's internal clock (block410), such as clock118in receiver114. If the difference between the audio or video clock data and the receiver's internal clock does not exceed a threshold value (block412), then the procedure returns to block402and continues receiving audio data samples and video data samples. However, if the difference between the audio or video clock data and the receiver's internal clock exceeds a threshold value (block412), then the procedure recalculates the PCR value (block414). A difference that exceeds the threshold value indicates a discontinuity in the data stream caused, for example, by dropped frames, errors in the data stream, or processing errors. Recalculation of the PCR value is discussed below with respect toFIG. 5. In a particular embodiment, the threshold value is a constant value that is equal to ½ the video data sample rate. For example, if the video data sample rate is 30samples per second (or 30 frames per second), the threshold value is (½( 1/30)sec), which equals 16.6667 ms.

After recalculating the PCR value, the procedure communicates the new PCR value to the audio decoder and the video decoder (block416). In the example ofFIG. 1, the PCR value is communicated to the audio decoder and the video decoder via clock modules128and130, respectively. The procedure then returns to block402and continues receiving audio data samples and video data samples.

FIG. 5is a flow diagram illustrating an embodiment of a procedure500for handling discontinuities in the audio data and/or the video data. Initially, procedure500discards all data samples buffered in the receiver (block502). The procedure then receives and buffers multiple data samples (block504). Next, PTS values associated with the data samples are identified, while ignoring erroneous PTS values (block506). As discussed above with respect toFIG. 2, erroneous PTS values are not considered. The procedure then identifies the earliest PTS value associated with the data samples (block508). The delay factor (discussed above) is subtracted from the PTS value to determine a new value for the PCR (block510). The new PCR value is stored in the receiver's internal clock (block512), such as clock118inFIG. 1. The new PCR value is then communicated to the audio decoder and the video decoder (block514). The receiver then begins processing data in the manner discussed above with respect toFIG. 4. By using the procedure ofFIG. 5, playback of the data stream is disrupted temporarily, but eliminates the need to return to the beginning of the data stream and count forward to resynchronize the clocks.

FIG. 6illustrates a general computer environment600, which can be used to implement the techniques described herein. The computer environment600is only one example of a computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the computer and network architectures. Neither should the computer environment600be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the example computer environment600.

Computer environment600includes a general-purpose computing device in the form of a computer602. The components of computer602can include, but are not limited to, one or more processors or processing units604(optionally including a cryptographic processor or co-processor), a system memory606, and a system bus608that couples various system components including the processor604to the system memory606.

Computer602typically includes a variety of computer readable media. Such media can be any available media that is accessible by computer602and includes both volatile and non-volatile media, removable and non-removable media.

The system memory606includes computer readable media in the form of volatile memory, such as random access memory (RAM)610, and/or non-volatile memory, such as read only memory (ROM)612. A basic input/output system (BIOS)614, containing the basic routines that help to transfer information between elements within computer602, such as during start-up, is stored in ROM612. RAM610typically contains data and/or program modules that are immediately accessible to and/or presently operated on by the processing unit604.

Computer602may also include other removable/non-removable, volatile/non-volatile computer storage media. By way of example,FIG. 6illustrates a hard disk drive616for reading from and writing to a non-removable, non-volatile magnetic media (not shown), a magnetic disk drive618for reading from and writing to a removable, non-volatile magnetic disk620(e.g., a “floppy disk”), and an optical disk drive622for reading from and/or writing to a removable, non-volatile optical disk624such as a CD-ROM, DVD-ROM, or other optical media. The hard disk drive616, magnetic disk drive618, and optical disk drive622are each connected to the system bus608by one or more data media interfaces625. Alternatively, the hard disk drive616, magnetic disk drive618, and optical disk drive622can be connected to the system bus608by one or more interfaces (not shown).

Any number of program modules can be stored on the hard disk616, magnetic disk620, optical disk624, ROM612, and/or RAM610, including by way of example, an operating system626, one or more application programs628, other program modules630, and program data632. Each of such operating system626, one or more application programs628, other program modules630, and program data632(or some combination thereof) may implement all or part of the resident components that support the distributed file system.

A user can enter commands and information into computer602via input devices such as a keyboard634and a pointing device636(e.g., a “mouse”). Other input devices638(not shown specifically) may include a microphone, joystick, game pad, satellite dish, serial port, scanner, and/or the like. These and other input devices are connected to the processing unit604via input/output interfaces640that are coupled to the system bus608, but may be connected by other interface and bus structures, such as a parallel port, game port, or a universal serial bus (USB).

A monitor642or other type of display device can also be connected to the system bus608via an interface, such as a video adapter644. In addition to the monitor642, other output peripheral devices can include components such as speakers (not shown) and a printer646which can be connected to computer602via the input/output interfaces640.

Computer602can operate in a networked environment using logical connections to one or more remote computers, such as a remote computing device648. By way of example, the remote computing device648can be a personal computer, portable computer, a server, a router, a network computer, a peer device or other common network node, game console, and the like. The remote computing device648is illustrated as a portable computer that can include many or all of the elements and features described herein relative to computer602.

Logical connections between computer602and the remote computer648are depicted as a local area network (LAN)650and a general wide area network (WAN)652. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet.

When implemented in a LAN networking environment, the computer602is connected to a local network650via a network interface or adapter654. When implemented in a WAN networking environment, the computer602typically includes a modem656or other means for establishing communications over the wide network652. The modem656, which can be internal or external to computer602, can be connected to the system bus608via the input/output interfaces640or other appropriate mechanisms. It is to be appreciated that the illustrated network connections are exemplary and that other means of establishing communication link(s) between the computers602and648can be employed.

In a networked environment, such as that illustrated with computing environment600, program modules depicted relative to the computer602, or portions thereof, may be stored in a remote memory storage device. By way of example, remote application programs658reside on a memory device of remote computer648. For purposes of illustration, application programs and other executable program components such as the operating system are illustrated herein as discrete blocks, although it is recognized that such programs and components reside at various times in different storage components of the computing device602, and are executed by the data processor(s) of the computer.

Although the description above uses language that is specific to structural features and/or methodological acts, it is to be understood that the invention defined in the appended claims is not limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the invention.