Methods and apparatus for synchronizing transcoded and/or transrated RTP packets

Methods and apparatus for synchronizing packet streams and in particular to the synchronization of transcoded and/or transrated Real-time Transport Protocol (RTP) packet streams, e.g., transcoded and/or transrated RTP packet streams carrying audio and/or video data. In one embodiment, a packet processing device receives packets of a first RTP packet stream and a first RTCP packet stream and generates a second RTP packet stream from the first RTP packet stream and generates a second RTCP packet stream from the first RTCP packet stream. The second RTP packet stream including packets with timestamps different than packets of the first RTP packet stream. The second RTCP packet stream including NTP timestamps received in or based on the NTP timestamps of the first RTCP packet stream and associated with RTP timestamps corresponding to content in the second RTP packet stream which was generated by transrating or trancoding content in the first RTP packet stream.

FIELD

The present application relates to methods and apparatus for synchronizing packet streams and in particular to the synchronization of transcoded and/or transrated Real-time Transport Protocol (RTP) packet streams, e.g., transcoded and/or transrated RTP packet streams carrying audio and/or video data.

BACKGROUND

Synchronization, e.g., lip synchronization is the functionality of aligning audio and video streams, e.g., audio and video RTP packet streams, so that the end user is presented a coherent view. At a high level, there are two approaches to solving the problem of lip synchronization which will be discussed in connection with the transmission of Real-time Transport Protocol packet streams. The Real-time Transport Protocol and Real-time Transport Control Protocol are described in the Request For Comment (RFC) 3550 entitled “RTP: A Transport Protocol for Real-Time Applications” dated July 2003 published by the Internet Engineering Task Force. The first approach to lip synchronization includes deducting a relationship between the timestamps and arrival times of RTP packets for each stream separately and then using this information to synchronize the streams. The second approach including calculating the wall clock time for each RTP packet and synchronizing packets on both streams. This second approach requires the use of Real-time Transport Control Protocol (RTCP) and a common wall clock for the devices performing audio/video transrating and/or transcoding. A RTCP stream contains both wall clock and RTP timestamp information. This provides enough information for the receiver of the RTP packet streams and the RTCP packet streams to create a mapping from RTP time to the wall clock time. This mapping is used to find the original emitting wall clock time for each received RTP packet.

However, this second approach relies heavily on the receiver having accurate information about the original sending time for audio and video packets. The sending time of RTP packets is communicated by using RTP timestamps. RTP timestamps themselves do not directly convey information about the actual sending time. RTP timestamps are 32 bits long and indicate the sampling instance for the first octet of media in a packet indicated as a sampling period number. The sampling instant must be derived from a clock that increments monotonically and linearly in time to allow synchronization and jitter calculations. The initial sampling period number is arbitrary. As explained in the RTP: A Transport Protocol for Real-Time Applications RFC 3550, RTP timestamps from different media streams may, and in some do, advance at different rates and typically have independent, random offsets. While these timestamps may be sufficient to reconstruct the timing of a single stream, directly comparing RTP timestamps from different media is not effective for synchronization. Instead, for each medium the RTP timestamp is related to the sampling instant by pairing it with a timestamp from a referenced clock (wallclock) that represents the time when the data corresponding to the RTP timestamp was sampled. The reference clock (wallclock) is shared by all media to be synchronized. The timestamp pairs are not transmitted in every data packet, but at a lower rate in RTCP sending report (SR) packets. The sampling instant is chosen as the point of reference for the RTP timestamp because it is known to the transmitting endpoint and has a common definition for all media, independent of encoding delays or other processing. The purpose is to allow synchronized presentation of all media sampled at the same time.

As further described in the RFC 3550, when transmitting stored data rather than data sampled in real time, applications typically use a virtual presentation timeline derived from wallclock time to determine the next frame or other unit of each medium in the stored data should be presented. In such cases, the RTP timestamp would reflect the presentation time for each unit, i.e., the RTP timestamp for each unit would be related to the wallclock time at which the unit becomes current on the virtual presentation timeline. Actual presentation occurs some time later as determined by the receiver.

Drawing101ofFIG. 1illustrates an example of a timestamp at two instances in time for packets RTP-1106and RTP-2112. Line104ofFIG. 1represents time line increasing from top to bottom. Legend102ofFIG. 1shows the sampling rate and one sampling period. The sampling rate for the RTP data stream is 8 KHz which gives one sampling period equal to 0.125 msec. At time108(T0) the RTP-1(T0) packet has a timestamp110equal to 15687. At time114(T0+20 msec), the RTP-2(T0+20 msec) packet has a timestamp116calculated by adding the timestamp at T0+(T0+20 ms)/sampling period). That is the timestamp116for RTP-2(T0+20 ms)=15687+(20 ms/0.125 ms)=15847.

The actual time is found by making use of Real-time Transport Control Protocol (RTCP) packets. RTCP packets contain the sampling period number and the corresponding real time. In this way, the RTCP packets provide a binding between RTP timestamps and the absolute wall clock time. In the RTCP protocol, wall clock time (absolute date and time) is represented using the timestamp format of the Network Time Protocol (NTP), which is in seconds relative to 0 hours UTC (Coordinated Universal Time) on 1 Jan. 1900. The full resolution NTP timestamp is a 64-bit unsigned fixed-point number with the integer part in the first 32 bits and the fractional part in the last 32 bits. In the examples herein the timestamps used will be simplified for ease of explanation. Drawing201ofFIG. 2illustrates how to determine the send time of an RTP packet using information provided in an RTCP packet. As shown in the legend202, the sampling rate is 8 KHz so that one sampling period is 0.125 ms. Dashed line204represents time and it increases from top to bottom. Points214and216represent different points in time. The RTCP206packet is sent at time214and includes a NTP timestamp and a corresponding RTP timestamp as shown in box208. The RTCP206NTP timestamp (also referred to as the wall clock time)=3021157382.18 seconds (since Jan. 1, 1900). The corresponding RTP timestamp=15343.FIG. 2also illustrates a RTP-N packet210. Box212ofFIG. 2shows the RTP-N packet timestamp included in the RTP-N packet210and how to calculate the actual send time for the RTP-N packet from the RTP-N packet timestamp, the RTCP206NTP timestamp, the RTCP206RTP timestamp and the sampling period. The RTP-N packet210includes the RTP-N timestamp=15687. The actual send time is calculated as RTCP NTP timestamp/wall clock time in seconds+(RTP−N timestamp−RTCP RTP timestamp)*sampling period. The send time for the RTP-N packet210=3021157382.18+(15687−15343)*0.000125 seconds. The terms “NTP time”, “NTP timestamp”, “wall clock time”, “wall clock timestamp” and “RTCP packet timestamp value” are used interchangeably herein. Similarly the terms “RTP time”, “RTP timestamp”, and “RTP timestamp value” are also used interchangeably herein.

Transcoding refers to the functionality of changing the format of a media stream (e.g., an audio or video stream) encoded in a first format in accordance with a first codec to a second encoding format in accordance with a second codec. In some communication sessions having audio and video streams, transcoding is needed for audio, video, or both audio and video streams of the communication session and the transcoding may happen in different devices for the audio and the video. Devices that perform transcoding are called transcoders. A transcoder populates relevant fields of the RTP/RTCP packets according to the time those packets egress the transcoder. This causes the original sent time to be lost as shown in drawing301ofFIG. 3.

In the drawing301ofFIG. 3, the sending device302transmits an audio message310with a sent time=T0, e.g., an RTP packet with audio data content, and a video message316with a video sent time also equal to T0, e.g., an RTP packet with video data content. The audio packet is not transcoded so it bypasses the transcoder304and goes directly from the transcoder at step308to the receiver306at step312. This results the audio message's sent time remaining T0. That is the audio message's sent time remains unchanged. The video message316however is sent from the sending device302at step314with a sent time of T0to the transcoder304at step318. The transcoder then transcodes the video message316and at step320transmits the transcoded video message322to the receiver306with a sent time of T0+X. For example, the transcoder populates relevant fields of RTP/RTCP packets according to the time those packets egress the transcoder304. At step324the receiver306receives the transcoded video message322with the sent time T0+X. While the audio and video messages included the same sent times when transmitted by the sending device302, they now have different sent times when received by the receiver306due to the transcoding of the transmitted video but not the transmitted audio message. The audio and video message sent times and therefore the messages themselves are no longer synchronized.

Furthermore, the sending device and transcoder are using their own wall clocks which makes any sent time related adjustment in the transcoder problematic. While wall clocks are expected to be calibrated by contacting a Network Time Protocol (NTP) server, an acceptable accuracy for synchronization especially for consumer devices, e.g., browsers running on PCs, smartphones, and tablets is not achieved.

In some communication sessions transrating is applied to change the data rate of a packet stream by changing the amount of data included in each packet. The use of transrating devices sometimes referred to as transraters during a communication session result in the same type of problem described above in connection with transcoders. In most cases, transcoders also perform transrating.

It should be appreciated that there is a need for methods and apparatus that can accurately synchronize data streams such as for example audio and video streams.

It should further be appreciated that there is a need for methods and apparatus that can minimize and/or eliminate synchronization problems introduced by intermediary transcoding and/or transrating devices which are used in a communication path between a sending and receiving device.

It should further be appreciated that there is a need for methods and apparatus that improve lip synching performance for data streams with one or more intermediary devices introducing delay to audio and/or video between the sending and receiving devices.

It should further be appreciated that there is a need for methods and apparatus to accurately generate network round trip time for data streams passing through one or more intermediary devices between the network sending and receiving devices.

SUMMARY

Various embodiments, in accordance with the present invention, are directed to methods and apparatus for synchronizing data streams such as for example RTP data streams carrying audio and video content.

For example, one embodiment of the method of the present invention include the steps of (i) receiving packets of a first RTP packet stream at a packet processing device, the received packets of the first RTP packet stream including first RTP timestamp values; (ii) receiving packets of a first RTCP packet stream that corresponds to the first RTP packet stream at the packet processing device, a first packet in said first RTCP packet stream including information correlating a first RTCP NTP packet timestamp value to a RTP timestamp value of said first RTP packet stream; (iii) generating, at the packet processing device, from received RTP packets of the first RTP packet stream, a second RTP packet stream including RTP packets which are different from the RTP packets included in the first RTP packet stream, said generating a second RTP packet stream including generating second RTP timestamp values; and (iv) generating at the packet processing device, a second RTCP packet stream from the received first RTCP packets, the second RTCP packet stream including a second RTCP packet including a second RTCP packet NTP timestamp value which is associated in the second RTCP packet with a RTP timestamp value of the second RTP packet stream, the second NTP timestamp value corresponding to content in the first and second RTP streams which is the same or content in said second RTP packet stream which was generated by transrating or transcoding content in said first RTP packet stream corresponding to the second RTCP packet NTP timestamp value.

In some embodiments of the method the second RTP packets differ from the first RTP packets with respect to at least one of packet payload content or included RTP timestamp values. In some embodiments of the method, the first RTP packet stream communicates content corresponding to a time period corresponding to a range of RTCP packet NTP timestamp values. In some embodiments, the timestamp values of the first and second RTP packet stream are different and the RTCP packet NTP timestamp values included in the first and second RTCP packet streams are the same but correspond to different RTP timestamp values. In some embodiments, the packet processing device is one of a transcoder device which recodes content, a transrating device which changes the data rate used to communicate content or a device which performs transcoding and transrating on content. In some embodiments, the individual packets in the second RTP packet stream include content corresponding to different amounts of time than individual packets in the first RTP packet stream and the step of generating a second RTCP packet stream includes generating new RTCP packet NTP timestamp values for inclusion in said RTCP packet stream based on the amount of time to which content in the second RTP packet stream correspond.

In some embodiments, the method further includes steps to determine round trip time at the packet processing device. In some of such embodiments, the method includes the further steps of storing, in memory at the packet processing device, a native NTP time at which a third RTCP packet of the second RTCP packet stream was transmitted from the packet processing device; storing, in memory at the packet processing device, the NTP timestamp value include in the third RTCP packet, said stored NTP timestamp value being associated with said native NTP time of the transmission of the third RTCP packet; receiving, at the packet processing device, a fourth RTCP packet containing a NTP timestamp value equal to the NTP timestamp value of the third RTCP packet; determining, at the packet processing device, the native NTP time at which the fourth RTCP packet was received at the packet processing device; and generating a round trip time based on the stored native NTP time at which the third RTCP packet was transmitted from the packet processing device and the native NTP time at which the fourth RTCP packet was received at the packet processing device.

In some embodiments of the method, the method further includes receiving at a multi-media device the second RTCP packet stream, the second RTP packet stream, and a third RTP packet stream, the third RTP packet stream being different than the first and said second RTP packet streams; and synchronizing the playback, at the multi-media device, multi-media content included in the second and the third RTP packet streams.

Some embodiments of the present invention are directed to apparatus. For example, a packet processing apparatus such as for example a transcoder or transrater including an I/O interface configured to: (i) receive packets of a first RTP packet stream including first RTP timestamp values, and (ii) receive packets of a first RTCP packet stream that corresponds to said first RTP packet stream, a first packet in said first RTCP packet stream including information correlating a first RTCP NTP packet timestamp value to a RTP timestamp value of said first RTP packet stream; a packet generation module configured to: (i) generate from the received RTP packets of said first RTP packet stream, a second RTP packet stream including RTP packets which are different from the RTP packets included in said first RTP packet stream, said second RTP packets including second RTP timestamp values generated by said packet generation module; and (ii) generate a second RTCP packet stream from the received first RTCP packets, said second RTCP packet stream including a second RTCP packet including a second RTCP packet NTP timestamp value which is associated in said second RTCP packet with a RTP timestamp value of said second RTP packet stream, the second RTCP packet NTP timestamp value corresponding to content in said first and second RTP streams which is the same or content in said second RTP packet stream which was generated by transrating or transcoding content in said first RTP packet stream corresponding to said second RTCP packet NTP timestamp value. In another aspect of the present invention, the second RTCP packet stream includes packets with NTP timestamp values derived from the NTP timestamp values of packets of the first RTCP packet stream.

While various exemplary embodiments and features have been described, numerous additional features and embodiments are described in the detailed description which follows.

DETAILED DESCRIPTION

FIG. 4is a drawing of an exemplary packet processing apparatus in accordance with an exemplary embodiment of the present invention. Packet processing apparatus401may be implemented as a collection of separate devices or as a single apparatus or device. In some embodiments the packet processing apparatus may be implemented as a system. The packet processing apparatus401will also be referred to herein as packet processing device401. Packet processing device401may be, and in some embodiments is, implemented as an integrated circuit such as a semiconductor chip.

Exemplary packet processing device401includes I/O interfaces402, a processor408, a clock440, an assembly of modules410, e.g., an assembly of circuits, and memory412, coupled together via a bus411over which the various elements may interchange data and information. I/O interfaces402includes a plurality of interfaces including a first communications interface404and a second communications interface406. In some embodiments, the first communications interface404includes a receiver420and a transmitter422. In some embodiments, the second communications interface406includes a receiver424and a transmitter426. Memory412includes an assembly of software modules414and data/information416.

Exemplary first communications interface404couples packet processing device401to a first communications link430, e.g., first network link, from which it receives packets, e.g., RTP and RTCP packets, to be processed. Exemplary second communications interface406couples the packet processing device401to a second communications link432, e.g., second network link over which the packet processing device transmits packets that the packet processing device401has processed. In some embodiments, the packet processing device receives and processes packets from both the first and second communication links430and432and transmits processed packets over both the first and second communication links430and432. In some embodiments, the packet processing device performs transcoding and/or transrating of the data content of incoming packets. In some embodiments, the packet processing device is a transcoder. In some embodiments, the packet processing device is a transrater.

In some embodiments, one or more modules of the assembly of modules800ofFIG. 8is used in the assembly of modules410and/or414of packet processing apparatus400. The assembly of modules800ofFIG. 8may be, and in some embodiments is, used as the assembly of modules410and/or414in some embodiments of the packet processing apparatus401.

Data/information900illustrates an example of the items that may be, and in some embodiments are, stored in memory412data/information section416of packet processing apparatus401in accordance with one embodiment of the present invention. In some embodiments, one or more items of the data/information900ofFIG. 9are stored in the data/information portion416of memory412of the packet processing apparatus401ofFIG. 4.

FIG. 15is a drawing of an exemplary multi-media device1500. The Multi-media device1500may be implemented as a collection of separate devices or as a single apparatus or device. Multi-media device1500may be, and in some embodiments is, implemented as an integrated circuit such as a semiconductor chip.

Exemplary multi-media device1500includes I/O interfaces1502, a processor1508, a clock1540, an assembly of modules1510, e.g., an assembly of circuits, and memory1512, coupled together via a bus1511over which the various elements may interchange data and information. I/O interfaces1502includes a plurality of interfaces including a first communications interface1504and a second communications interface1506. In some embodiments, the first communications interface1504includes a receiver1520and a transmitter1522. In some embodiments, the second communications interface1506includes a receiver1524and a transmitter1526. Memory1512includes an assembly of software modules1514and data/information1516.

Exemplary first communications interface1504couples multi-media device1500to a first communications link1532, e.g., first network link, from which it receives packets, e.g., RTP and RTCP packets, to be processed or over which it transmits packets, e.g., RTP and RTCP packets which it generated. Exemplary second communications interface1506couples the multi-media device1500to a second communications link1532, e.g., second network link over which the packet processing device transmits packets that the multi-media device1500has processed and/or generated or over which it receives packets to process. In some embodiments, the multi-media device receives and processes packets from both the first and second communication links1530and1532and transmits processed packets over both the first and second communication links1530and1532. In some embodiments of the present invention, the multi-media device includes modules, either software or hardware or a combination of software and hardware modules, for playing back content, e.g., multi-media content, received in RTP data streams. In some embodiments of the present invention, the multi-media device includes modules, either software or hardware or a combination of software and hardware modules, for generating RTP packet data streams including multi-media content, e.g., audio and/or video content, and for generating associated RTCP packet streams. In some embodiments, the multi-media device1500includes modules for both generating and playing back RTP data streams such as for example, RTP packet streams including multi-media content and associated RTCP packet streams.

In an embodiment of the present invention, synchronization of RTP data streams is achieved by recording the original sending device's wall clock time information in an associated RTCP stream for all related RTP media streams and preserving that wall clock information even through intermediary packet processing device such as for example transcoders and/or transraters. Meanwhile, a given sending device's wall clock timestamp is kept associated with the same point in the data stream, e.g., in an audio stream, through transcoders even though transcoders may change the RTP timestamps, and may split or merge N incoming RTP messages to M outgoing RTP messages. At the ultimate destination device, or at one or more devices implementing the present invention on the media path before the ultimate destination device, the common sending device wall clock associated with all the related media streams is used to “true up” any differential delay that has accumulated between the related RTP media streams therein synchronizing the related RTP media streams. The calculated original send time for each stream is used by the playout logic of the destination device to match audio/video packets/packet ranges for simultaneous playout.

In this exemplary embodiment of the present invention, an intermediate transcoder and/or transrater relays the wall clock time effectively unchanged in RTCP messages but the transcoder/transrater generates its own RTP timestamps which are placed into outgoing RTP and RTCP messages also referred to as packets. In some cases, the transcoder/transrater generated RTP timestamps are the same as the ones it received but in the general case transcoder/transrater generated RTP timestamps are different.

In the exemplary embodiment a packet processing device such as a transcoder or transrater includes a receiving RTP packets step, a processing RTP packets step (e.g., transcoding RTP packet data content, a generating new RTP packets step with new RTP timestamp values, and a transmission step in which the newly generated RTP packets are sent by packet processing device to their next destination. In the receiving step, the packet processing device, e.g., a transcoder receives a series of RTP packets with RTP timestamp values. For example, RTP packet1with timestamp value R1, RTP packet2with timestamp value R2, RTP packet3with timestamp value R3, . . . , RTP packet N with timestamp value RN.

Operation proceeds from the packet receiving step to the packet processing step. In the packet processing step, the packet processing device, e.g., packet processing device401, applies existing/standard jitter buffer logic/error concealment and transcodes the series of received RTP packets. Operation proceeds from the packet processing step to the generating new RTP packets step.

In the generating new RTP packets step, a series of new RTP packets are generated based on the processed RTP packets. In generating each of the new RTP packets, the packet processing device sets the RTP timestamp value of each new RTP packet to the current local RTP time stamp value. For example, new RTP packet1is set to have a timestamp value equal to S1, new RTP packet2is set to have a timestamp value equal to S2, new RTP packet3is set to have a timestamp value of S3, . . . , new RTP packet N is set to have a timestamp value of SN where S1, S2, S3, . . . , SN are the current local RTP time stamp values corresponding to when each of the new RTP packets will egress the packet processing device. Operation proceeds from the new RTP packet generation step to the new RTP packet transmission step.

In the new RTP packet transmission step, the packet processing device sends each newly generated RTP packet of the new RTP packet stream to its next destination.

In the exemplary embodiment, the packet processing device, e.g., packet processing device401which may be, and in some embodiments is, a transcoder and/or transrater, receives and processes the series of RTCP packets corresponding to the RTP packets it received and processed. The packet processing device also generates and transmits a new series of RTCP packets to the same destination as it sent the newly generated RTP packets.

In one embodiment if the packet processing device, e.g., transcoder/transrater, is going to egress an RTCP message immediately, the packet processing device, e.g., transcoder/transrater, receives the RTCP message with the sender wall clock timestamp value T0(i.e., RTCP packet NTP timestamp value=T0) and RTCP RTP timestamp value=R. The packet processing device, e.g., transcoder/transrater, generates a new RTCP message based at least in part on the received RTCP message. In the newly generated RTCP message, the packet processing device, e.g., transcoder/transrater, populates the wall clock timestamp of the generated RTCP message with the value T0(i.e., RTCP packet NTP timestamp value=T0) and populates the RTP timestamp field of the generated RTCP message with the RTP timestamp S, which is the RTP timestamp to use at this moment and which corresponds to the same point in the media stream, e.g., audio stream, as the received RTP message timestamp R. The packet processing device, e.g., transcoder/transrater, then egresses the generated RTCP message by transmitting or sending the generated RTCP message on to the next destination on the media path.

If however the packet processing device, e.g., transcoder/transrater, is going to egress the received RTCP message at a different time then the following method, may, and in some embodiments is, employed. The packet processing device, e.g., transcoder/transceiver, receives the RTCP message with the sender wall clock timestamp T0(i.e., RTCP packet NPT timestamp=T0) and RTP timestamp R. The packet processing device, e.g., transcoder/transceiver, stores the time when the RTCP message was received in memory. The packet processing device, e.g., transcoder/transrater, generates a new RTCP message based on the received RTCP message and the stored reception time. When it is time to egress the newly generated RTCP message, the packet processing device, e.g., transcoder/transrater, calculates the difference or delta between the current time and the RTCP receipt time which had been stored in memory. The packet processing device, e.g., transcoder/transrater, adds the calculated delta to the received RTCP wall clock timestamp (RTCP packet NTP timestamp) and uses it as the wall clock time for the RTCP message to be egressed. The packet processing device, e.g., transcoder/transrater, populates the generated RTCP message's RTP timestamp field with the RTP timestamp value to be used at that moment. This RTP timestamp points to the same point in the media stream, e.g., audio stream, as the RTP timestamp of the received RTCP message+delta.

If a packet processing device, e.g., transcoder/transrater, needs to send RTCP messages, as discussed above, at a different pace than it is receiving the RTCP messages, the packet processing device, e.g., transcoder/transrater, can use interpolation methods to adjust the wall clock timestamps in the sent RTCP messages while still maintaining the correct reference to the media stream. That is if an RTCP message must be sent 3.31 seconds after one was received, the wall clock time in the sent RTCP message can be adjusted by adding 3.31 seconds to the RTCP packet NTP timestamp as long as the RTP sequence number is also adjusted to bind the new wall clock timestamp to a point 3.31 seconds later in the media stream.

Method501ofFIG. 5illustrates an exemplary method of adjusting the RTCP packet NTP timestamp at a transcoder502in accordance with one embodiment of the present invention. In this example transcoder502is implemented in accordance with the exemplary packet processing device401ofFIG. 4. The example will now be explained using the elements of the packet processing device401illustrated inFIG. 4. The method begins at step506when the RTCP packet504is received by the packet processing device401receiver420of the first communications interface on first communications link430. The RTCP packet504contains an RTCP packet NTP timestamp equal to 50000.234. The RTCP packet504is received at time T0. Processing proceeds from step506to step508. In step508, the packet processing device401processes the packet including storing in the data/information portion416of memory412the time T0that RTCP packet504was received by the packet processing device. Operation proceeds from step508to step510.

In step510, the packet processing device determines using the stored received time T0of the RTCP packet504that it will generate a new RTCP packet520based on the received RTCP packet504but will need to egress the newly generated RTCP packet5203.31 seconds after the RTCP packet504was received. Operation proceeds from step510to step512.

In step512, the packet processing device401generates RTCP packet520based on the received RTCP packet504and the determined amount of time that will elapse between receipt of the RTCP packet504and the egress of the RTCP packet520. The packet processing device401includes in the RTCP packet520an NTP timestamp value equal to 500003.544 and a RTP timestamp value equal to the current local RTP timestamp value. The NTP timestamp value being generated by adding the determined elapsed time from receipt of the RTCP packet504to the egress of the RTCP packet520which is 3.31 seconds to the NTP timestamp of 500000.234 of the received RTCP packet504. Operation proceeds from step512to step514.

In step514, the packet processing device401transmits the generated RTCP packet520to its next destination via second communications interface406transmitter426over second communications link432.

Round trip time is the length of time it takes from when a signal is sent to when an acknowledgement of the signal is received. When using non-native NTP timestamps for RTCP packets generated by an intermediary packet processing device such as for example a transcoder, the round trip time (RTT) calculation will be affected. Method600ofFIG. 6illustrates how to calculate RTT for standard deployments using native NTP timestamps for RTCP packets. InFIG. 6, dashed line6shows a NTP time line native to the transcoder602. In the example, the method starts at step604. In step604, the packet processing device in this example the transcoder602transmits a RTCP packet608with a RTCP packet NTP timestamp set equal to time T0which is the native NTP time of the transcoder coder602at the time the RTCP packet608is transmitted. Operation proceeds from step604to step612.

In step612, the transcoder602at its native NTP time T1, receives a RTCP packet614which includes an NTP timestamp of the last received RTCP packet having a value set to T0. Box616shows the NTP timestamp of the last received RTCP included in the RTCP packet614and the destination device's delay in reflecting this timestamp. The delay in reflecting this timestamp is TX. Operation proceeds to step618.

In step618, the transcoder602determines the round trip time, i.e., the network delay, as NTP Time T1(native NTP time RTCP packet614received)−T0(native NTP time RTCP packet608transmitted)−TX (the destination device's delay in reflecting the timestamp)

When the received RTCP packet NTP time is used by the packet processing device, e.g., the transcoder, for RTCP packets it generates based on the received RTCP packets instead of the native NTP time, the method600will not provide an accurate round trip time because the non-native NTP time was included in the transmitted RTCP packet.

The method700ofFIG. 7illustrates an exemplary method to calculate the round trip time when the packet processing device, e.g., transcoder, uses the non-native RTCP packet NTP time in accordance with an exemplary embodiment of the present invention.

The method700is performed by transcoder702which may be, and in some embodiments is, implemented in accordance with packet processing device401. Dashed line714ofFIG. 7illustrates the native NTP time for the transcoder602. The method700starts at start step703from which operation proceeds to step706.

In step706, RTCP packet704including a RTCP packet NTP timestamp equal to TA is received by the transcoder702. Operation proceeds from step706to step708.

In step708, the transcoder702transmits at native NTP time T0a RTCP packet710generated from information contained in the received RTCP packet704. The RTCP packet710includes a RTCP packet NTP timestamp equal to TA, the RTCP packet704NTP timestamp value. Operation proceeds from step708to step712.

In step712, the transcoder702stores in memory the native NPT time T0that the RTCP packet710was sent together with the corresponding NTP timestamp value sent in the RTCP packet710which is NTP timestamp value TA. Operation proceeds to step715.

In step715, the transcoder702receives the RTCP packet716at native NTP time T1. The RTCP packet716which includes the NTP timestamp of the last received RTCP packet with a value of TA as shown in box718. Box718also shows that the delay in reflecting this timestamp by the destination device is TX. Operation proceeds from step715to step720.

In step720, the transcoder702retrieves from memory the stored NTP time value of T0using the NTP timestamp value TA received in the RTCP packet716. Operation proceeds from step720to step722.

In step722, the round trip time also referred to as the network delay is calculated by the transcoder702by using the stored RTCP NTP send time T0retrieved from memory rather than the NTP timestamp TA received in the RTCP packet716. The transcoder generates the round trip time/network delay in accordance with the following equation: RTT=T1−T0−TX. The round trip time may, and in some embodiments, is stored in memory for use in other procedures or to maintain statics on the network delay for later use.

In some embodiments of the present invention the method includes receiving first RTP and first RTCP packet streams, generating a second RTP stream and second RTCP stream therefrom with the RTP stream including new RTP packets and with the RTCP stream including RTCP NTP timestamps correlating the new RTP timestamps to the same time interval as the RTCP NTP time stamps of the first RTCP stream, each RTCP NTP timestamp corresponding to a point in time. The second RTCP NTP timestamps can be the same values as the first RTCP NTP timestamps or interpolated values falling between the original RTCP NTP timestamps depending on whether or not the second RTP packets communicate content corresponding to the same amount of time as the first RTP packets which may not be the case if transrating is applied to change the data rate or transcoding is applied resulting potentially in different amounts of data per packet and possibly each packet corresponding to a different amount of time than the first RTP packets.

FIG. 10consists ofFIGS. 10A, 10B and 10Cand illustrates a method1001for synchronizing RTP packets streams. The method1001may be, and in some embodiments is, performed by packet processing device401. The exemplary method1001will be explained using the packet processing device401. The method is intended to cover the concept of receiving first RTP and first RTCP packet streams, generating a second RTP stream and second RTCP stream therefrom with the RTP stream including new RTP packets and with the RTCP stream including RTCP time stamps correlating the new RTP time stamps to the same time interval as the RTCP time stamps of the first RTCP stream, each RTCP time stamp corresponding to a point in time−the second RTCP time stamps can be the same values as the first RTCP time stamps or interpolated values falling between the original RTCP time stamps depending on whether or not the second RTP packets communicate content corresponding to the same amount of time as the first RTP packets which may not be the case if transrating is applied to change the data rate or transcoding is applied resulting potentially in different amounts of data per packet and possibly each packet corresponding to a different amount of time than the first RTP packets.

The method1001commences at start step1002shown onFIG. 10A. Operation proceeds from start step1002to steps1004and1006in parallel. While these steps are shown as occurring in parallel they may also be performed sequentially wherein the order in which the steps are performed is not important.

In step1004, packets of a first RTP packet stream are received at the packet processing device401. The received RTP packets of the first RTP packet stream include first RTP timestamp values.

In step1006, packets of a first RTCP packet stream that corresponds to the first RTP packet stream are received at the packet processing device401. A first packet in said first RTCP packet stream includes information correlating a first RTCP NTP packet timestamp value to a RTP timestamp value of said first RTP packet stream.

Operation proceeds from step1004and1006to step1008. In step1008, a second RTP packet stream is generated, at the packet processing device401, from the received RTP packets of said first RTP packet stream. The second RTP packet stream includes RTP packets which are different from the RTP packets included in said first RTP packet stream. The generation of the second RTP packet stream includes generating second RTP time stamp values for the packets of the second RTP packet stream. The generated second RTP packets1011include second RTP time stamp values of the second RTP packet stream.

Operation proceeds from step1008shown onFIG. 10Ato step1014shown onFIG. 10Bvia connection node A1012. In step1014, a second RTCP packet stream1031is generated at the packet processing device401. The second RTCP packet stream is generated from the received first RTCP packets of the first RTCP stream. The second RTCP packet stream includes a second RTCP packet NTP timestamp value which is associated in said second RTCP packet with a RTP timestamp value of said second RTP packet stream. The second RTCP packet NTP timestamp value corresponds to content in said first and second RTP streams which is the same or content in said second RTP packet stream which was generated by transrating or transcoding content in said first RTP packet stream corresponding to said second RTCP packet NTP timestamp value.

In some embodiments, step1014includes one or more of the optional sub-steps1016,1018,1021,1022,1024,1026, and1028.

In sub-step1016, the generation of the second RTCP packet stream includes the use of at least some of the first RTCP packet NTP timestamp values received in said first RTCP packet stream as second RCTP packet NTP timestamp values in said second RTCP packet stream.

In sub-step1016, during the generation of the second RTCP packet stream one of the RTCP packet NTP timestamp values from the first RTCP packet stream is included in the second RTCP packet and a new RTP timestamp value is included in the second RTCP packet, said new RTP timestamp value corresponding to a packet in the second RTP packet stream communicating data generated from a RTP packet of the first RTP packet stream which included a RTP timestamp value which is different from said new RTP timestamp value but which communicated content corresponding to said one of the RTCP timestamp values. In some of such embodiments, the individual packets in said RTP stream include content corresponding to the same length of time as individual packets in said first RTP stream.

In some embodiments, wherein the individual packets in said second RTP packet stream include content corresponding to different amounts of time then individual packets in said first RTP packet stream, optional sub-step1021is performed. In sub-step1021, the generation of the second RTCP packet stream includes generating new RTCP packet NTP timestamp values for inclusion in said second RTCP packet stream based on the amount of time to which content in the second RTP packet stream correspond. In some embodiments, the sub-step1021includes sub-step1022. In sub-step1022, said second RTCP packet NTP timestamp values generated for inclusion in said second RTCP packet stream are generated based on the content included in said second RTP packets with said second RTCP packet NTP timestamp values being generated such that a RTCP packet NTP timestamp value corresponds to the same content in either the first or second RTP content streams.

In sub-step1024, at least one of the RTCP packet NTP timestamp values of the second RTCP stream is generated using interpolation.

In sub-step1026, a RTP timestamp value to be included in a RTCP packet of the second RTCP packet stream is generated using interpolation.

In sub-step1018, a RTCP packet NTP timestamp value to be included in a RTCP packet of the second RTCP packet stream is generated using interpolation.

In some embodiments, the optional steps1047and1048are performed. In such embodiments, operation proceeds from step1014to step1047shown onFIG. 10Cvia connection node B1032. In step1047, a multi-media device, such as for example multi-media device1500shown inFIG. 15, receives said second RTCP packet stream, said second RTP packet stream, and a third RTP packet stream, said third RTP packet stream being different than said first and said second RTP packet streams. The second RTCP packets stream and second RTP packet stream having been transmitted from said packet processing device. Operation proceeds from step1047to step1048.

In step1048, the multi-media device synchronizes the playback of the multi-media content, e.g., audio and video content, included in said second and said third RTP streams. In some embodiments, the step1048includes an optional sub-step1049. In sub-step1049, the synchronizing of the playback of the multi-media content includes using the second RTCP packet timestamp values to synchronize the playback of the multi-media content included in said second and said third RTP packet stream. The third RTP packet stream including timestamps being synchronized to the wallclock used to generate the first RTCP packet NTP timestamp values.

In some embodiments of the exemplary method1001, the optional steps1034and its sub-steps10351036,1038,1039,1042and1045are performed. These steps are typically performed in embodiments in which the round trip time or network delay is determined at packet processing device

In some embodiments in which the round trip time is to be determined by the packet processing device, operation proceeds from step1014to optional step1034onFIG. 10Cvia connection node B1032. In step1034, the packet processing device determines and stores the round trip time. Optionally sub-steps10351036,1038,1039,1042and1045of step1034may be, and in some embodiments are, performed to determine and store the round trip time at the packet processing device.

In step1035, a native NTP time at which a third RTCP packet of the second RTCP packet stream was transmitted from the packet processing device is stored in memory. The native NTP time being the wallclock time of the packet processing device. Operation proceeds from step1035to optional step1036.

In step1036, the NTP timestamp value included in the third RTCP packet is stored in memory at the packet processing device. The stored NTP timestamp value being associated with said native NTP time of the transmission of the third RTCP packet. The NTP timestamp value included in the third RTCP packet is generated by the packet processing device from a NTP timestamp value included in a RTCP packet from the first RTCP packet stream. Operation proceeds from step1036to optional step1038.

In step1038, the packet processing device receives a fourth RTCP packet containing an NTP timestamp value equal to the NTP timestamp value of the third RTCP packet. Operation proceeds from step1038to optional step1039.

In step1039, the packet processing device determines the native NTP time at which the fourth RTCP packet was received at the packet processing device. In some embodiments optional step1039includes optional step1041. In step1041, the storage control module of the packet processing device identifies in the memory the stored native NTP time at which the third RTCP packet was transmitted using the NTP timestamp value received in the fourth RTCP packet. Operation proceeds from step1039to optional step1042.

In step1042, the packet processing device generates a round trip time based on the stored native NTP time at which the third RTCP packet was transmitted from the packet processing device and the native NTP time at which the fourth RTCP packet was received at the packet processing device. Operation proceeds from step1042to optional step1045. In step1045, the storage control module of the packet processing device stores in memory the generated round trip time.

In some embodiments of the method1001, the first RTCP packet NTP timestamp value indicates a first absolute time and said second RTCP packet NTP timestamp value indicates a second absolute time. In some of such embodiments, the first and second RTCP packet NTP timestamp values are the same and said first and second absolute times are the same while in some other embodiments the first and second RTCP packet NTP timestamp values are different and said first and second absolute times are different.

In some embodiments of the method1001, the second RTP packets differ from the first RTP packets with respect to at least one of packet payload or included RTP timestamp values. For example, the content of the second RTP packets may be, and in some embodiments is, content from the first RTP packets that has been transcoded and/or transrated and/or the second RTP timestamp values may be and typically are different than the first RTP timestamp values for example due to a different initial RTP timestamp value which may have been randomly generated at the packet processing device.

In some embodiments of the method1001, the timestamp values of the first and second RTP packet streams are different and the RTCP packet NTP timestamp values included in the first and second RTCP packet streams are different.

In some embodiments of the method1001, the first RTP packet stream communicates content corresponding to a time period corresponding to a range of RTCP packet NTP timestamp values.

In some embodiments of the method1001, the packets of the second RTP packet stream include content encoded using a different encoding standard then RTP packets included in said first RTP packet stream or which are encoded at a different coding rate.

In some embodiments of the method1001, the RTP packets of the second RTP packet stream include the same encoded content as packets of the first RTP packet stream but different RTP timestamp values being associated with the same encoded content.

In some embodiments of the method1001, the packet processing device is one of a transcoder device which recodes content, a transrating device which changes the data rate used to communicate content or a device which performs transcoding and transrating on content. In some of such embodiments, the packet processing device processes packets corresponding to audio packet streams, video packet streams or both audio and video packet streams.

In some of embodiments of the method1001, the packet processing device processes one of an audio packet stream or a video packet stream corresponding to a program including audio and video content but not both.

In some embodiments of the present invention, the packet processing device401ofFIG. 4is the packet processing device that implements the various steps of the method1001. In some of the embodiments, one or more modules of the assembly of modules800is included in the assembly of module410or414of the packet processing device400and perform one or more of the steps of the method1001. The assembly of modules800includes a packet generation module802, a communications interface, e.g., link interface, receiver module804, a communications interface, e.g., link interface, transmitter module806, a transcoder module808, a control module810, a transrater module820, a memory module822, a storage control module824, an interpolation module826, a RTP timestamp generation module828, a RTCP native timestamp generation module830, a NTP native time generation module832, a RTCP NTP timestamp generation module834, a determination module836, a round trip time determination module838, a round trip time generation module840, and a recode module842.

As shown inFIG. 9, the following exemplary data/information may be, and in some embodiments is, stored in the data/information section of memory416of the packet processing device400during the implementation of methods in accordance with various embodiments of the present invention: RTP packets from first RTP packet stream902, information from RTP packets included in first RTP packet stream904, first RTP timestamp values905, RTCP packets from first RTCP packet stream906, first packet from first RTCP packet stream908, information from a first RTCP packet of the first RTCP packet stream correlating a first RTCP packet timestamp value to a RTP timestamp value of said first RTP packet stream910, first RTCP timestamp values912, RTP packets from second RTP packet stream914, information from RTP packets included in the second RTP packet stream916, second RTP timestamp values918, RTCP packets from second RTCP packet stream920, information from second RTCP packets of the second RTCP packet stream922, a second RTCP packet from said second RTCP packet stream including a second RTCP NTP timestamp value which is associated with an RTP timestamp value of said second RTP packet stream924, second RTCP timestamp values926, packet processing device native NTP send time for a RTCP packet correlated to the RTCP packet NTP timestamp value included in a packet sent by the packet processing device928, and a round trip time930.

System1100ofFIG. 11illustrates an exemplary system in accordance with one embodiment of the present invention. The system1100includes a sending device1102a packet processing device1104, e.g., a transcoder, and a receiving device1106, e.g., a multi-media player. In some embodiments, the sending device1102is a multi-media device such as for example multi-media device1500ofFIG. 15. In some embodiments, the receiving device1106is a multi-media device such as multi-media device1500ofFIG. 15. In some embodiments, the packet processing device1104is implemented in accordance with the packet processing device401ofFIG. 4described above.

In system1100ofFIG. 11, the sending device in step1108generates and transmits a first RTCP packet stream1110(also referred to as RTCP packet stream1) to the packet processing device1104. Operation proceeds from step1108to step1112. In step1112, the packet processing device1104receives the first RTCP packet stream1110.

In step1114, which may take place prior to, in parallel with or after step1108, a first RTP packet stream1116(also referred to as RTP packet stream1) is generated at the sending device1102and transmitted to the packet processing device1104.

Operation proceeds from step1114to step1118. In step1118, the packet processing device1104receives the first RTP packet stream1116. The first RTP packet stream1116in this example includes video content data. In some embodiments, the first RTP packet stream includes multi-media content. In some embodiments, the first RTP packet stream includes audio content.

Steps1119and1120may be, and in some embodiments, are performed in parallel. In step1119, the packet processing device1104generates a second RTP packet stream1128(also referred to as RTP packet stream2) based on the received packets of the first RTP packet stream1116. In generating the second RTP packet stream1128, the packet processing device performs a transcoding and/or transrating operation on the packets of the first RTP packet stream. In generating the second RTP packet stream1128, the packet processing device generates new RTP timestamp values that are included in RTP packets of the second RTP packet stream1128. In step1120, the packet processing device1104generates a second RTCP packet stream1122(also referred to as RTCP packet stream2) based on the received packets of the first RTCP packet stream1110. Packets of the second RTCP packet stream1122include RTCP packet NTP timestamp values from the received packets of the first RTCP packet stream1110or based upon the received packets of the first RTCP packet stream1110and RTP timestamp values that correspond to the same instant in both the first and second RTP packet streams content. The RTP timestamp values included in the RTCP packets of the second RTCP packet stream may, and most instances are, different than the RTP timestamp values received in packets of the first RTP packet stream. The RTP timestamp values included in the packets of the second RTCP packet stream may be, but are not always, equal to RTP timestamp values included in the second RTP packet1122.

In step1121, packets of the second RTCP packet stream1122are transmitted from the packet processing device1104to the receiving device1106. In step1124, packets of the second RTCP packet stream are received by the receiving device1106.

In step1126, packets of the second RTP packet stream1128are transmitted from the packet processing device1104to the receiving device1106. In step1130, packets of the second RTP packet stream1128are received by the receiving device1106.

In step1132, which may take place prior to, in parallel with or after steps1108and1114, the sending device1102generates and transmits a third RTCP packet stream1134(also referred to as RTCP packet stream3). The third RTCP packet stream is transmitted to the receiving device1106. In this example, the third RTCP packet stream bypasses the packet processing device1104and is sent directly to the receiving device1106. As a result the RTCP packet NTP timestamp values and RTP timestamp values included by the sending device1102in the RTCP packets of the third RTCP packet stream during its generation will be unaltered when received by the receiving device1106. Operation proceeds from step1132to step1136. In step1136the receiving device1106receives the third RTCP packet stream1134.

In step1140, which may take place prior to, in parallel with or after steps11081114, and1132, the sending device1102generates and transmits a third RTP packet stream1142(also referred to as RTP packet stream3). The third RTP packet stream is transmitted to the receiving device1106. In this example, the third RTP packet stream includes audio content data. In some embodiments, the third RTP packet stream includes multi-media content data. In some embodiments the third RTP packet stream includes video content data. In this example, the third RTP packet stream bypasses the packet processing device1104and is sent directly to the receiving device1106. As a result the RTP packet timestamp values included by the sending device1102in each of the RTP packets of the third RTP packet stream during its generation will be unaltered when received by the receiving device1106. Operation proceeds from step1142to step1146. In step1146the receiving device1106receives the third RTP packet stream1142.

In some embodiments, the third RTCP packet stream and the third RTP packet stream instead of bypassing the packet processing device1104are transmitted to and received by the packet processing device1104which retransmits the third RTCP packet stream and the third RTP packet stream to the receiving device1106without altering the third RTCP packet stream or the third RTP packet stream.

In step1150, the receiving device plays back the video and audio content from the second and third RTP packet streams. The playback of the video and audio content is synchronized by the receiving device1106using the RTCP NTP timestamp values in the packets of the second and third RTCP packet streams and RTP timestamps in the second and third RTCP and RTP packet streams. In this manner, the audio and video can be played back in a synchronized manner consistent with the original timestamp values. That is RTP time stamps which may have been generated as part of transcoding will still map to the original RTCP NTP timestamp values of the original RTCP and RTP packet streams given that the mapping of media content to RTCP NTP timestamps remains consistent despite the possible change in RTP timestamp values.

Diagram1200ofFIG. 12illustrates an example of several packets from each packet stream discussed in connection with the exemplary system and method discussed in the exemplary embodiment illustrated inFIG. 11. Diagram1200ofFIG. 12illustrates several packets of the first RTCP packet stream, the first RTP packet stream, the second RTCP packet stream generated by the packet processing device1104based on the first RTCP packet stream, the second RTP packet stream generated by the packet processing device1104based on the first RTP packet stream using transcoding but without transrating, the third RTCP packet stream, and the third RTP packet stream. The various packets of the different packet streams are shown in view of a wall clock time line1202with times provided with respect to the original sender. The wall clock/NTP time1202with respect to the original sender, the RTCP NTP timestamp values, RTP timestamp values and content of the packets have been chosen for illustrative purposes and to simplify the example. The length of the packets do not represent the amount of data in the packet or the duration of the playback time to which the communicated data corresponds. The start location of the packet in connection with the time line1202indicates when it is output by its corresponding transmitting device, e.g., the packet processing device, with respect to the wall clock time of the original sending device1102.

The timeline1202represents the wall clock or RTCP NTP time with respect to the original sender which is the sending device1102. Dashed line1204represents the wall clock/NTP time of1000with respect to the original sender. Dashed line1206represents the wall clock/NTP time of1010with respect to the original sender. Dashed line1208represents the wall clock/NTP time of1020with respect to the original sender. The dashed line1210represents the wall clock/NTP time of1030with respect to the original sender. The dashed line1212represents the wall clock/NTP time of1040with respect to the original sender.

With respect to the first RTCP packet stream two RTCP packets are shown, RTCP packet11216and RTCP packet21218. RTCP packet11216includes a RTCP packet NTP timestamp value=1000 and RTP timestamp value=100. RTCP packet2includes a RTCP packet NTP timestamp value=1030 and a RTP timestamp value=400. Packets from the first RTCP packet stream are generated by the sending device1102and sent in step1108ofFIG. 11.

With respect to the first RTP packet stream which in the example discussed in connection withFIG. 11was a video packet stream, four packets of the first RTP packet stream are illustrated inFIG. 12. The four packets of the first RTP packet stream shown inFIG. 12are RTP packet11220, RTP packet21222, RTP packet31224and RTP packet41226. RTP packet11220includes RTP timestamp value100and data content1. RTP packet21222includes RTP timestamp value200and data content2. RTP packet31224includes RTP timestamp value300and data content3. RTP packet41226includes RTP timestamp400and data content4.

The RTP timestamp value100in the RTCP packet11216corresponds to the NTP timestamp value1000in RTCP packet11216but in the same units and with the same random offset as the RTP timestamp values in the first RTP packet stream. In this example, the RTP packet11220which contains content1has a RTP timestamp value=100.

With respect to the second RTCP packet stream which is generated by the packet processing device1104in step1120ofFIG. 11, two packets are shown RTCP packet11228and RTCP packet21230are shown inFIG. 12. The RTCP packet11228of the second RTCP packet stream is generated from the RTCP packet11216of the first RTCP packet stream. The RTCP packet11228NTP timestamp value is set to the value1000which is the value of the RTCP packet11216NTP timestamp value. While the RTCP NTP timestamp values in the second RTCP packet stream are consistent and fall in the range of RTCP NTP timestamp values included in the first RTCP stream (and in some instances such as the present example are the same as the values in the first RTCP stream), the RTP values are in the range of those generated by the packet processing device1104. The RTCP packet11228RTP timestamp value is5100. The RTCP packet21230of the second RTCP packet stream is generated from the RTCP packet21218of the first RTCP packet stream. The RTCP packet21230NTP timestamp value is set to the value1030which is the value of the RTCP packet21218NTP timestamp value. The RTCP packet21230RTP timestamp value is set to 5400.

The actual wall clock time NTP time value with respect to the original sender for the RTCP packet11228of the second RTCP stream is shown as approximately1013with respect to the time line1202which is the transmission time of the RTCP packet11228from the packet processing device1104. The time from1000to1013represents transmission time for the RTCP packet1216to be sent from the sending device1102to the packet processing device1104and the time to receive the RTCP packet11216, generate the RTCP packet11228based on the RTCP packet11216, and transmit the RTCP packet11228.

The actual wall clock time NTP time value with respect to the original sender for the RTCP packet21230of the second RTCP stream is shown as approximately1043with respect to the time line1202which is the transmission time of the RTCP packet21230from the packet processing device1104. The time from1030to1043represents transmission time for the RTCP packet1218to be sent from the sending device1102to the packet processing device1104and the time to receive the RTCP packet21218, generate the RTCP packet21230based on the RTCP packet11218, and transmit the RTCP packet21230.

With respect to the second RTP packet stream which is generated by the packet processing device1104in step1119ofFIG. 11, four packets are shown RTP packet11232, RTP packet21234, RTP packet31236, and RTP packet41238are shown inFIG. 12. RTP packet11232, RTP packet21234, RTP packet31236and RTP packet41238of the second RTP packet stream are generated by the packet processing device1104by transcoding but not transrating the four RTP packets1220,1222,1224and1226of the first RTP packet stream respectively.

The RTP packet11232of the second RTP packet stream is generated from the RTP packet11220of the first RTP packet stream. The RTP packet11232timestamp value is set to the value5100and the RTP packet1payload contains content1which is the same content as in RTP packet11220but has been recoded.

The RTP packet21234of the second RTP packet stream is generated from the RTP packet21222of the first RTP packet stream by the packet processing device1104. The RTP packet21234timestamp value is set to the value5200and the RTP packet21234payload contains content2which is the same content as in RTP packet21222but has been recoded.

The RTP packet31236of the second RTP packet stream is generated from the RTP packet31224of the first RTP packet stream by the packet processing device1104. The RTP packet31236timestamp value is set to the value5300and the RTP packet31236payload contains content3which is the same content as in RTP packet31224but has been recoded.

The RTP packet41238of the second RTP packet stream is generated from the RTP packet41226of the first RTP packet stream by the packet processing device1104. The RTP packet41238timestamp value is set to the value5400and the RTP packet41238payload contains content4which is the same content as in RTP packet41226but has been recoded. Note that the RTP packets of the second RTP stream communicate content which is the same as, or generated from, content in the first RTP packet stream and will map to the same RTCP NTP timestamp values despite use of different RTP timestamp values in the first and second RTP streams. Thus the bindings of RTCP NTP timestamp values to RTP timestamp values provided by the second RTCP packet stream allows content in the second RTP stream to be mapped to the original wall clock used to generate the RTCP packets of the first RTCP stream and first RTP stream.

InFIG. 12the RTP packet11232of the second RTP stream is shown as being transmitted by the packet processing device1104at time1013with respect to the sending device1102wall clock time. The RTP packet21234of the second RTP stream is shown as being transmitted by the packet processing device1104at time1023with respect to the sending device1102wall clock time. The RTP packet31236of the second RTP stream is shown as being transmitted by the packet processing device1104at time1033with respect to the sending device1102wall clock time. The RTP packet4of the second RTP stream is shown as being transmitted by the packet processing device1104at time1043with respect to the sending device1102wall clock time.

As with the transmission times of the RTCP packet11228and RTCP packet21230of the second RTCP packet stream, the transmission times of the packets of second RTP packet stream with respect to the wall clock time of the original sender as shown with respect to the time line1202represents transmission time for the RTP packets to be sent from the sending device1102to the packet processing device1104and the time to receive the RTP packets of the first RTP packet stream, generate the RTP packets of the second RTP based on the RTP packets of the first RTP stream, and transmit the RTP packets of the second RTP stream. InFIG. 12the RTP packet1of the second RTP stream is shown as being transmitted by the packet processing device1104at time1013with respect to the sending device1102wall clock time.

With respect to the third RTCP packet stream two RTCP packets are shown, RTCP packet11240and RTCP packet21242. RTCP packet11240includes a RTCP packet NTP timestamp value=1000 and RTP timestamp value=100. RTCP packet2includes a RTCP packet NTP timestamp value=1030 and a RTP timestamp value=400. Packets from the third RTCP packet stream are generated by the sending device1102and sent in step1132ofFIG. 11.

With respect to the third RTP packet stream which in the example discussed in connection withFIG. 11was an audio packet stream, four packets of the third RTP packet stream are illustrated inFIG. 12. These packets are generated and transmitted by the sending device1102in step1140shown inFIG. 11. The four packets of the third RTP packet stream shown inFIG. 12are RTP packet11244, RTP packet21246, RTP packet31248and RTP packet41250. RTP packet11244includes RTP timestamp value100and data content1. RTP packet21246includes RTP timestamp value200and data content2. RTP packet31248includes RTP timestamp value300and data content3. RTP packet41250includes RTP timestamp400and data content4. The RTCP packets of the third packet stream and the RTP packets of the third RTP packet stream are sent directly from the sending device1102to the receiving device1106and so their RTCP NTP timestamp values, RTP timestamp values and data content remains unaltered.

In this example, the sending device1102synchronizes the video stream content carried in the first RTP packet stream with the audio content carried in the third RTP packet stream through the RTCP NTP timestamp values and RTP timestamp values included in the packets of the first and third RTCP packet streams and the RTP timestamp values included in the packets of the first and third RTP streams. The RTP timestamp values reflect the sampling instant for real time sampled data (or presentation time for the next unit of stored data) of the first octet in the associated RTP data packet. For each media (audio and video) sample the RTP timestamp is related to the sampling instant by pairing it with a timestamp from a reference clock (wallclock), i.e., NTP timestamp value, that represents the time when the data corresponding to the RTP timestamp was sampled (or is to be presented). In this example, the reference clock is the sending device1102's wallclock which is used to generate the first and third NTP timestamp values. This pairing of NTP timestamp values with RTP timestamp values provides a binding between the RTP timestamps for the media (audio and video) packets to the RTCP NTP timestamp values of the wallclock of the sending device1102. The pairing of the NTP timestamp values and RTP timestamp values is contained in the packets of the first and third RTCP packet streams. This information synchronizes the content data of the different media (audio and video) streams and allows presentation of all media (audio and video) sampled at the same time (or to be presented at the same time for stored data). When the packet processing device1104transcodes the packets of the first packet processing stream and generates the packets of the second RTCP packet stream and second RTP packet stream, it generates new RTP timestamp values that it maps to the original wall clock time of the sending device1102thereby maintaining synchronization of the data.

Because the mapping of the instants in time of data sampling (or presentation time) as it relates to the sending device's wallclock has been maintained by the second RTCP packet stream NTP timestamp value to second RTP packet stream timestamp value pairings included in the packets of the second RTCP packet stream, the video and audio content data of the second and third RTP packet streams remain synchronized. The receiving device1106plays back in step1150ofFIG. 11the video of the second RTP packet stream and the audio of the third RTP packet stream in synchronization using the NTP timestamp values and RTP timestamp values of the second and third RTCP and RTP packet streams which are referenced to the sending device1102wall clock.

Diagram1300ofFIG. 13illustrates another example of several packets from each packet stream discussed in connection with the exemplary system and method discussed in the exemplary embodiment illustrated inFIG. 11. Elements with the same reference numerals inFIG. 13as those inFIG. 12are the same or similar and will not be discussed in detail in connection withFIG. 13. In the example shown inFIG. 13, the packet processing device1104performs a transrating and/or transcoding operation on the packets of the first RTP packet stream to a 1/2 data rate.

Diagram1300ofFIG. 13illustrates several packets of the first RTCP packet stream, the first RTP packet stream, the second RTCP packet stream generated by the packet processing device1104based on the first RTCP packet stream, the second RTP packet stream generated by the packet processing device1104based on the first RTP packet stream using transrating and/or transcoding to a 1/2 rate, the third RTCP packet stream, and the third RTP packet stream. The various packets of the different packet streams are shown in view of a wall clock time line1202with times provided with respect to the original sender. The wall clock/NTP time1202with respect to the original sender, the RTCP NTP timestamp values, RTP timestamp values and content of the packets have been chosen for illustrative purposes and to simplify the example. The length of the packets do not represent the amount of data in the packet or the duration of the playback time to which the communicated data corresponds. The start location of the packet in connection with the time line1202indicates when it is output by its corresponding transmitting device, e.g., the packet processing device, with respect to the wall clock time of the original sending device1102.

The timeline1202represents the wall clock or RTCP NTP time with respect to the original sender which is the sending device1102. Dashed line1204represents the wall clock/NTP time of1000with respect to the original sender. Dashed line1206represents the wall clock/NTP time of1010with respect to the original sender. Dashed line1208represents the wall clock/NTP time of1020with respect to the original sender. The dashed line1210represents the wall clock/NTP time of1030with respect to the original sender. The dashed line1212represents the wall clock/NTP time of1040with respect to the original sender.

With respect to the first RTCP packet stream two RTCP packets are shown, RTCP packet11216and RTCP packet21218. RTCP packet11216includes a RTCP packet NTP timestamp value=1000 and RTP timestamp value=100. RTCP packet2includes a RTCP packet NTP timestamp value=1030 and a RTP timestamp value=400. Packets from the first RTCP packet stream are generated by the sending device1102and sent in step1108ofFIG. 11.

With respect to the first RTP packet stream which in the example discussed in connection withFIG. 11was a video packet stream, four packets of the first RTP packet stream are illustrated inFIG. 13. The four packets of the first RTP packet stream shown inFIG. 13are RTP packet11220, RTP packet21222, RTP packet31224and RTP packet41226. RTP packet11220includes RTP timestamp value100and data content1. RTP packet21222includes RTP timestamp value200and data content2. RTP packet31224includes RTP timestamp value300and data content3. RTP packet41226includes RTP timestamp400and data content4.

The RTP timestamp value100in the RTCP packet11216corresponds to the NTP timestamp value100in RTCP packet11216but in the same units and with the same random offset as the RTP timestamp values in the first RTP packet stream. In this example, the RTP packet11220which contains content1has a RTP timestamp value=100.

With respect to the second RTCP packet stream which is generated by the packet processing device1104in step1120ofFIG. 11, two packets are shown RTCP packet11304and RTCP packet21306inFIG. 13. The RTCP packet11304of the second RTCP packet stream is generated from the RTCP packet11216of the first RTCP packet stream. The RTCP packet11304NTP timestamp value is set to the value1010which is based on the RTCP packet11216NTP timestamp value but interpolation has been used to calculate the RTCP packet11304NTP timestamp value that will map to the transrated data with the paired RTP timestamp value of 5200 which is included in the RTCP packet11304packet. While the RTCP NTP timestamp values in the second RTCP packet stream are consistent and fall in the range of RTCP NTP timestamp values included in the first RTCP stream, the RTP values are in the range of those generated by the packet processing device1104. The RTCP packet11304RTP timestamp value is5200. The RTCP packet21306of the second RTCP packet stream is generated from the RTCP packet21218of the first RTCP packet stream. The RTCP packet21306NTP timestamp value is set to the value1030which is the value of the RTCP packet21218NTP timestamp value. The RTCP packet21306RTP timestamp value is set to 5400.

The actual wall clock time NTP time value with respect to the original sender for the RTCP packet11304of the second RTCP stream is shown as approximately1023with respect to the time line1202which is the transmission time of the RTCP packet11304from the packet processing device1104. The time from1000to1023represents transmission time for the RTCP packet1216to be sent from the sending device1102to the packet processing device1104and the time to receive the RTCP packet11216, generate the RTCP packet11304based on the RTCP packet11216, and transmit the RTCP packet11304.

The actual wall clock time NTP time value with respect to the original sender for the RTCP packet21306of the second RTCP stream is shown as approximately1043with respect to the time line1202which is the transmission time of the RTCP packet21306from the packet processing device1104. The time from1030to1043represents transmission time for the RTCP packet1218to be sent from the sending device1102to the packet processing device1104and the time to receive the RTCP packet21218, generate the RTCP packet21306based on the RTCP packet11218, and transmit the RTCP packet21306.

With respect to the second RTP packet stream which is generated by the packet processing device1104in step1119ofFIG. 11, two packets, RTP packet11308and RTP packet21310, are shown inFIG. 13. RTP packet11308and RTP packet21310of the second RTP packet stream are generated by the packet processing device1104by transrating and/or transcoding to a 1/2 rate the four RTP packets1220,1222,1224and1226of the first RTP packet stream.

The RTP packet11308of the second RTP packet stream is generated from the RTP packet11220and RTP packet21222of the first RTP packet stream. The RTP packet11308timestamp value is set to the value5200and the RTP packet1payload contains content1and2from the RTP packet11220and RTP packet21222that has been transrated and/or transcoded to a 1/2 rate.

The RTP packet21310of the second RTP packet stream is generated from the RTP packet31224and RTP packet41226of the first RTP packet stream by the packet processing device1104. The RTP packet21310timestamp value is set to the value5400and the RTP packet21310payload contains content3and4which is the content from RTP packet31224and RTP packet41226but has been transrated and/or transcoded to a 1/2 rate.

As discussed in connection with the example ofFIG. 12, the RTP packets of the second RTP stream communicate content which is the same as, or generated from, content in the first RTP packet stream and will map to the same RTCP NTP timestamp values despite use of different RTP timestamp values in the first and second RTP streams. Thus the bindings of RTCP NTP timestamp values to RTP timestamp values provided by the second RTCP packet stream allows content in the second RTP stream to be mapped to the original wall clock used to generate the RTCP packets of the first RTCP stream and first RTP stream.

InFIG. 13the RTP packet11308of the second RTP stream is shown as being transmitted by the packet processing device1104at time1023with respect to the sending device1102wall clock time. The RTP packet21310of the second RTP stream is shown as being transmitted by the packet processing device1104at time1043with respect to the sending device1102wall clock time. As with the transmission times of the RTCP packet11304and RTCP packet21306of the second RTCP packet stream, the transmission times of the packets of second RTP packet stream with respect to the wall clock time of the original sender as shown with respect to the time line1202represents transmission time for the RTP packets to be sent from the sending device1102to the packet processing device1104and the time to receive the RTP packets of the first RTP packet stream, generate the RTP packets of the second RTP based on the RTP packets of the first RTP stream, and transmit the RTP packets of the second RTP stream. InFIG. 13the RTP packet1of the second RTP stream is shown as being transmitted by the packet processing device1104at time1023with respect to the sending device1102wall clock time.

With respect to the third RTCP packet stream two RTCP packets are shown, RTCP packet11240and RTCP packet21242. RTCP packet11240includes a RTCP packet NTP timestamp value=1000 and RTP timestamp value=100. RTCP packet2includes a RTCP packet NTP timestamp value=1030 and a RTP timestamp value=400. Packets from the third RTCP packet stream are generated by the sending device1102and sent in step1132ofFIG. 11.

With respect to the third RTP packet stream which in the example discussed in connection withFIG. 11was an audio packet stream, four packets of the third RTP packet stream are illustrated inFIG. 13. These packets are generated and transmitted by the sending device1102and sent in step1140shown inFIG. 11. The four packets of the third RTP packet stream shown inFIG. 13are RTP packet11244, RTP packet21246, RTP packet31248and RTP packet41250. RTP packet11244includes RTP timestamp value100and data content1. RTP packet21246includes RTP timestamp value200and data content2. RTP packet31248includes RTP timestamp value300and data content3. RTP packet41250includes RTP timestamp400and data content4. The RTCP packets of the third packet stream and the RTP packets of the third RTP packet stream are sent directly from the sending device1102to the receiving device1106and so their RTCP NTP timestamp values, RTP timestamp values and data content remains unaltered.

In this example, the sending device1102synchronizes the video stream content carried in the first RTP packet stream with the audio content carried in the third RTP packet stream through the RTCP NTP timestamp values and RTP timestamp values included in the packets of the first and third RTCP packet streams and the RTP timestamp values included in the packets of the first and third RTP streams. Even though the packet processing device1104transrates and/or transcodes the first RTP packet stream when generating the second RTP packet stream, the mapping of the data sampling times (or presentation times) between the content and the sending device1102wall clock time has been maintained during the generation of the second RTCP and RTP packet streams. Because the mapping of the instants in time of data sampling (or presentation time) as it relates to the sending device's wallclock is maintained by the second RTCP packet stream NTP timestamp value to second RTP packet stream timestamp value pairings included in the packets of the second RTCP packet stream, the video and audio content data of the second and third RTP packet streams remain synchronized. The receiving device1106plays back in step1150ofFIG. 11the video of the second RTP packet stream and the audio of the third RTP packet stream in synchronization using the NTP timestamp values and RTP timestamp values of the second and third RTCP and RTP packet streams which reference the sending device1102wall clock.

Diagram1400ofFIG. 14illustrates another example of several packets from each packet stream discussed in connection with the exemplary system and method discussed in the exemplary embodiment illustrated inFIG. 11. In the example shown inFIG. 14, the packet processing device1104performs a transrating and/or transcoding operation on the packets of the first RTP packet stream to a 2/3 rate.

Diagram1400ofFIG. 14illustrates several packets of the first RTCP packet stream, the first RTP packet stream, the second RTCP packet stream generated by the packet processing device1104based on the first RTCP packet stream, the second RTP packet stream generated by the packet processing device1104based on the first RTP packet stream using transrating and/or transcoding to a 2/3 rate, the third RTCP packet stream, and the third RTP packet stream. The various packets of the different packet streams are shown in view of a wall clock time line1402with times provided with respect to the original sender. The wall clock/NTP time1402with respect to the original sender, the RTCP NTP timestamp values, RTP timestamp values and content of the packets have been chosen for illustrative purposes and to simplify the example. The length of the packets do not represent the amount of data in the packet or the duration of the playback time to which the communicated data corresponds. The start location of the packet in connection with the time line1402indicates when it is output by its corresponding transmitting device, e.g., the packet processing device, with respect to the wall clock time of the original sending device1102. However, to make it easier to understand the 2/3 rate transrating/transcoding of the second RTP packet stream, the transmission times of the second RTCP and RTP packet streams do not reflect the transmission times of the second RTCP and RTP packets from the packet processing device1104as was done inFIGS. 12 and 13. For example, the transmission time and reception time of the first RTCP and RTP packets and the time to generate and transmit the packets of the second RTCP and RTP streams is not shown.

The timeline1402represents the wall clock or RTCP NTP time with respect to the original sender which is the sending device1102. Dashed line1404represents the wall clock/NTP time of1000with respect to the original sender. Dashed line1406represents the wall clock/NTP time of1010with respect to the original sender. Dashed line1408represents the wall clock/NTP time of1020with respect to the original sender. The dashed line1410represents the wall clock/NTP time of1030with respect to the original sender. The dashed line1412represents the wall clock/NTP time of1040with respect to the original sender. The dashed line1414represents the wall clock/NTP time of1050with respect to the original sender.

With respect to the first RTCP packet stream three RTCP packets are shown, RTCP packet11416, RTCP packet21417and RTCP packet31418. RTCP packet11416includes a RTCP packet NTP timestamp value=1000 and RTP timestamp value=100. RTCP packet2includes a RTCP packet timestamp value=1020 and a RTP timestamp value=300, and RTCP packet3includes a RTCP packet NTP timestamp value=1040 and a RTP timestamp value=500. Packets from the first RTCP packet stream are generated by the sending device1102and sent in step1108ofFIG. 11.

With respect to the first RTP packet stream which in the example discussed in connection withFIG. 11was a video packet stream, five packets of the first RTP packet stream are illustrated inFIG. 14. The five packets of the first RTP packet stream shown inFIG. 14are RTP packet11420, RTP packet21422, RTP packet31424, RTP packet41426and RTP packet51427. RTP packet11420includes RTP timestamp value100and data content1. RTP packet21422includes RTP timestamp value200and data content2. RTP packet31424includes RTP timestamp value300and data content3. RTP packet41426includes RTP timestamp400and data content4. RTP packet51427includes RTP timestamp500and data content5.

The RTP timestamp value100in the RTCP packet11416corresponds to the NTP timestamp value100in RTCP packet11416but in the same units and with the same random offset as the RTP timestamp values in the first RTP packet stream. In this example, the RTP packet11420which contains content1has a RTP timestamp value=100.

With respect to the second RTCP packet stream which is generated by the packet processing device1104in step1120ofFIG. 11, two packets, RTCP packet11428and RTCP packet21430, are shown inFIG. 14. The RTCP packet11428of the second RTCP packet stream is generated from the RTCP packet11416of the first RTCP packet stream. The RTCP packet11428NTP timestamp value is set to the value1000which is based on the RTCP packet11216NTP timestamp value. While the RTCP NTP timestamp values in the second RTCP packet stream are consistent with and fall in the range of RTCP NTP timestamp values included in the first RTCP stream, the RTP values are in the range of those generated by the packet processing device1104. The RTCP packet11428RTP timestamp value is5100. The RTCP packet21430NTP timestamp value is set to the value1030which is generated based on the value of the RTCP packet21417NTP timestamp value but modified to accurately map the wallclock time of the original sending device1102and it's correlating data from the first RTP packet stream. The RTCP packet21430RTP timestamp value is set to 5400.

With respect to the second RTP packet stream which is generated by the packet processing device1104in step1119ofFIG. 11, three packets, RTP packet11432, RTP packet21434and RTP packet31436, are shown inFIG. 14. RTP packet11432RTP packet21434and RTP packet31436of the second RTP packet stream are generated by the packet processing device1104by transrating and/or transcoding to a 2/3 rate the five RTP packets14201422,1424,1426and1427of the first RTP packet stream.

The RTP packet11432of the second RTP packet stream is generated from the RTP packet11420and RTP packet21422of the first RTP packet stream. The RTP packet11432timestamp value is set to the value5100and the RTP packet1payload contains content1and the first half of content2from the RTP packet11420and RTP packet21422that has been transrated and/or transcoded to a 2/3 rate.

The RTP packet21434of the second RTP packet stream is generated from the RTP packet21422and RTP packet31426of the first RTP packet stream by the packet processing device1104. The RTP packet21434timestamp value is set to the value5250which is determined using interpolation and the RTP packet21434payload contains the second half of content2from the RTP packet21422and content3from RTP packet31424but has been transrated and/or transcoded to a 2/3 rate.

The RTP packet31436of the second RTP packet stream is generated from the RTP packet41426and RTP packet51427of the first RTP packet stream by the packet processing device1104. The RTP packet31436timestamp value is set to the value5400which is determined using interpolation and the RTP packet31436payload contains the content4from the RTP packet41426and the first half of content5from RTP packet51427but has been transrated and/or transcoded to a 2/3 rate.

As discussed in connection with the example ofFIG. 12, the RTP packets of the second RTP stream communicate content which is the same as, or generated from, content in the first RTP packet stream and will map to the same RTCP NTP timestamp values despite use of different RTP timestamp values in the first and second RTP streams. Thus the bindings of RTCP NTP timestamp values to RTP timestamp values provided by the second RTCP packet stream allows content in the second RTP stream to be mapped to the original wall clock used to generate the RTCP packets of the first RTCP stream and first RTP stream.

With respect to the third RTCP packet stream two RTCP packets are shown, RTCP packet11438and RTCP packet21440. RTCP packet11438includes a RTCP packet NTP timestamp value=1000 and RTP timestamp value=100. RTCP packet21440includes a RTCP packet NTP timestamp value=1030 and a RTP timestamp value=400. Packets from the third RTCP packet stream are generated by the sending device1102and sent in step1132ofFIG. 11.

With respect to the third RTP packet stream which in the example discussed in connection withFIG. 11was an audio packet stream, four packets of the third RTP packet stream are illustrated inFIG. 14. These packets are generated and transmitted by the sending device1102and sent in step1140shown inFIG. 11. The four packets of the third RTP packet stream shown inFIG. 14are RTP packet11444, RTP packet21446, RTP packet31448and RTP packet41450. RTP packet11444includes RTP timestamp value100and data content1. RTP packet21446includes RTP timestamp value200and data content2. RTP packet31448includes RTP timestamp value300and data content3. RTP packet41450includes RTP timestamp400and data content4. The RTCP packets of the third packet stream and the RTP packets of the third RTP packet stream are sent directly from the sending device1102to the receiving device1106and so their RTCP NTP timestamp values, RTP timestamp values and data content remains unaltered.

In this example, the sending device1102synchronizes the video stream content carried in the first RTP packet stream with the audio content carried in the third RTP packet stream through the RTCP NTP timestamp values and RTP timestamp values included in the packets of the first and third RTCP packet streams and the RTP timestamp values included in the packets of the first and third RTP streams. Even though the packet processing device1104transrates and/or transcodes the first RTP packet stream when generating the second RTP packet stream, the mapping of the data sampling times (or presentation times) between the content and the sending device1102wall clock time has been maintained during the generation of the second RTCP and RTP packet streams. Because the mapping of the instants in time of data sampling (or presentation time) as it relates to the sending device's wallclock is maintained by the second RTCP packet stream NTP timestamp value to second RTP packet stream timestamp value pairings included in the packets of the second RTCP packet stream, the video and audio content data of the second and third RTP packet streams remain synchronized. The receiving device1106plays back in step1150ofFIG. 11the video of the second RTP packet stream and the audio of the third RTP packet stream in synchronization using the NTP timestamp values and RTP timestamp values of the second and third RTCP and RTP packet streams which reference the sending device1102wall clock.

The techniques of various embodiments may be implemented using software, hardware and/or a combination of software and hardware. Various embodiments are directed to apparatus and/or systems, e.g., communications device such as for example a packet processing device, multi-media, etc. Various embodiments are also directed to methods, e.g., a method of operating a communications device such as a packet processing device, multi-media device, transcoder, transrater, etc. Various embodiments are also directed to machine, e.g., computer, readable medium, e.g., ROM, RAM, CDs, hard discs, etc., which include machine readable instructions for controlling a machine to implement one or more steps of a method. The computer readable medium is, e.g., non-transitory computer readable medium.

In various embodiments devices described herein are implemented using one or more modules to perform the steps corresponding to one or more methods, for example, signal generation, signal transmission, signal reception, signal processing, and/or other steps. Thus, in some embodiments various features are implemented using modules. Such modules may be implemented using software, hardware, e.g., circuits, or a combination of software and hardware. In some embodiments, one or more modules are implemented by one or more processors configured to perform the modules functions. In some embodiments, one or modules are implemented in hardware specific circuits such as for example ASICs. Many of the above described methods or method steps can be implemented using machine executable instructions, such as software, included in a machine readable medium such as a memory device, e.g., RAM, floppy disk, etc. to control a machine, e.g., general purpose computer with or without additional hardware, to implement all or portions of the above described methods, e.g., in one or more nodes. Accordingly, among other things, various embodiments are directed to a machine-readable medium, e.g., a non-transitory computer readable medium, including machine executable instructions for causing a machine, e.g., processor and associated hardware, to perform one or more of the steps of the above-described method(s). Some embodiments are directed to an apparatus, e.g., a communications device such as a packet processing device including a processor configured to implement one, multiple or all of the steps of one or more methods of the invention.

In some embodiments, the processor or processors, e.g., CPUs, of one or more devices, e.g., of the communications device, e.g., packet processing device, are configured to perform the steps of the methods described as being performed by the apparatus. The configuration of the processor may be achieved by using one or more modules, e.g., software modules, to control processor configuration and/or by including hardware in the processor, e.g., hardware modules, to perform the recited steps and/or control processor configuration. Accordingly, some but not all embodiments are directed to a device, e.g., such as communications device, e.g., a packet processing device, with a processor which includes a module corresponding to each of the steps of the various described methods performed by the device in which the processor is included. In some but not all embodiments an apparatus, e.g., a communications device, e.g., a packet processing device, includes a module corresponding to each of the steps of the various described methods performed by the device in which the processor is included. The modules may be implemented using software and/or hardware. The hardware may be circuits, ASICs or other specialized or dedicated circuitry.

Numerous additional variations on the methods and apparatus of the various embodiments described above will be apparent to those skilled in the art in view of the above description. Such variations are to be considered within the scope of the invention.