Accommodating irregular timing information in streamed media data

Techniques for accommodating irregular timing information in streamed media data are described. According to some embodiments, at least a portion of a media stream that includes a video component is received, the video component including a plurality of video frames and video timing data, and wherein the media stream includes an indication of a video frame rate. A first duration of a first video frame of the plurality of video frames is determined to be different than an expected video frame duration, wherein the expected video frame duration is based at least in part on the indication of the video frame rate. A timestamp of at least one video frame of the plurality of video frames is adjusted to change the first duration.

BACKGROUND

Various protocols exist for streaming live or pre-recorded media over Internet Protocol (IP)-based networks such as the internet. Exemplary protocols include Real-time Transport Protocol (RTP), Real-Time Message Protocol (RTMP), Dynamic Adaptive Streaming over HTTP (DASH), and HTTP Live Streaming (HLS). One common aspect of each of these protocols is the conveyance of timing information to allow the recipient of the stream to correctly process the various stream components such as audio, video, captions, etc. both individually and together. For example, using the timing information, the recipient can determine when and how long to display a particular video frame relative to other video frames. The recipient can also determine which audio data is to be rendered during the display of a given video frame so that the various components remain synchronized.

DETAILED DESCRIPTION

The present disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for accommodating irregular timing information in streamed media data. Typically, streaming protocols deliver stream components such that they can be processed at a given rate. Such a rate is often advertised to the recipient as part of metadata associated with the stream, such as in header information that can apply globally to the stream or some portion thereof. For example, a stream might include metadata indicating the stream includes a 30 frame-per-second (fps) video component (e.g., each video frame spans ˜33 milliseconds) and a 40 fps audio component (e.g., each audio frame contains ˜25 milliseconds of audio). Thus, the recipient can expect to receive timing information indicating that video frames are to be processed at 33 millisecond (ms) intervals and have 33 ms durations and that audio frames are to be processed at 40 ms intervals and have 40 ms durations. Some streaming sources, however, may send timing information that deviates from a recipient's expectation under certain scenarios. One common source of such irregular timing information is mobile devices. Mobile devices, such as laptops, tablets, smartphones, and the like, often share a finite amount of compute capacity across a range of computing tasks which include capturing, compressing, and transmitting streaming media. Other computing tasks can cause the streaming task to wait for access to a shared resource, resulting in generating and sending irregular timing information. Regardless of the source, irregular timing information presents difficulties for a recipient when it comes to processing the stream. What should the recipient do when it receives a frame with a duration of 1 ms despite an advertised frame rate of 60 fps? Absent techniques for handling such irregular timing information, recipients often stitch together non-consecutive frames or insert duplicate frames or blanking periods that can result in undesirable audio or video artifacts. For example, the recipient might duplicate a single video frame for multiple frame durations resulting in a stutter in the rendered video. Similarly, the recipient might skip an audio frame resulting in audible pops or clicks in the rendered audio.

To address these and other problems, various approaches disclosed herein allow a streaming media recipient to reduce the impact of irregular timing data on viewers. According to some embodiments, a timing manager uses the irregular timing data to adjust how the streamed media is processed, deviating from the advertised rate. The timing manager can adjust the expected timestamps of future video and/or audio frames based on the irregular timing data rather than on advertised rate(s). According to some embodiments, the timing manager modifies the irregular timing data to provide a more regular cadence relative to the advertised rate. The timing manager can adjust the durations and/or timestamps of frames to more evenly space frames based on an advertised rate. According to some embodiments, the timing manager adjusts one or more parameters that govern how the different stream components are synchronized. The timing manager permits the stream components to drift further out of sync when irregular timing data is received than would otherwise be permitted when regular timing data is received.

The aforementioned approaches can be implemented anywhere from within a cloud-based service that receives, processes, and distributes media streams to downstream viewers to a computer system that both receives the stream and renders it locally to a viewer. Such approaches can be implemented alone or in various combinations as each contributes to the reduction in undesirable artifacts typically associated with irregular timing information.

FIG. 1is a diagram illustrating an exemplary environment for processing streaming media transmissions according to some embodiments. A provider network100provides users with the ability to utilize one or more of a variety of types of computing-related resources and/or services. In the illustrated environment, the provider network100that includes a media ingestion service110and a media distribution service130. The media ingestion service110, the media distribution service130, and the other illustrated components of the provider network100can be implemented as software, hardware, or a combination of hardware and software. For example, the components of the provider network100can be implemented as software programs (e.g., instructions or code) executed by one or more computer system of the provider network100.

Users can interact with the provider network100across one or more intermediate networks106(e.g., the internet) via one or more interface(s)160, such as through use of application programming interface (API) calls, via a console implemented as a website or application, etc. The interface(s)160may be part of, or serve as a front-end to, a control plane of the provider network100that includes “backend” services supporting and enabling the services offered to customers such as the media ingestion service110and/or media distribution service130. For example, a user can issue one or more commands via a computer system150and the interface(s)160to set up a portion of the computing resources of the provider network100to ingest and distribute a media stream. Further, the user can issue one or more commands to the interface(s)160to send messages to the other components of the provider network to enable and/or disable the various approaches for handling media streams with irregular timing data as well as specify any available configuration parameters (e.g., thresholds, tolerances, window sizes, and other parameters as described herein). Note that the user that configures the provider network to ingest and distribute media from a given source may be different than the user providing the source.

The media ingestion service110receives a media stream with irregular timing data from a computer system105via one or more networks106(e.g., the internet). The media ingestion service110can receive media streams from a variety of sources via various protocols such as Real-time Transport Protocol (RTP), Real-Time Message Protocol (RTMP), Dynamic Adaptive Streaming over HTTP (DASH), and HTTP Live Streaming (HLS). The media ingestion service110can further decode and encode the received media streams into different media formats (e.g., changing the form of compression used on the video and/or audio) and prepare the media for distribution using one or more protocols that may be different than the protocol via which the media stream was originally received. After processing the received stream, the media ingestion service110can send the resulting processed stream(s) to one or more data stores120for later retrieval or directly to the media distribution service130.

The media distribution service130can stream or send the output of the media ingestion service110to one or more computer systems140A-140N that render the media to viewers. The media distribution service130can be a content delivery network with multiple, geographically dispersed locations to facilitate the timely delivery of media data to users (and, in the case of streaming media to users, reduce the network distance to reduce the likelihood of transmission problems).

As illustrated, the media ingestion service110includes a timing manager112, one or more decoders114, one or more encoders116, and one or more buffers118. The media ingestion service110receives the media stream from the computer system105. The media stream can include one or more components (e.g., audio, video, captions, etc.). The decoders114can decode stream components from a compressed format such (e.g., H.264 or AAC) to an uncompressed format. The encoders can encode the uncompressed format into a compressed format. Note that in this embodiment, the media ingestion service110is performing transcoding operations. In other embodiments, the encoders116may be omitted, such as in the case where the timing manager112, decoders114, and buffers118are implemented as part of a streaming media client executed by a computer system that receives and renders a media stream locally to a viewer.

The media stream further includes metadata that includes timing information to allow the recipient to correctly process the various stream components both individually and together (i.e., keeping multiple components in sync, when present). As used herein, the term “frame” refers to a portion of one of the media components. In the case of video, a frame can correspond to a single frame of a video that is to be rendered to a viewer at some time and for some duration associated with a frame rate, while in the case of audio, a frame can correspond to one or more audio samples to be rendered to a viewer at some time and for a duration associated with a sample rate. The media stream includes (either explicitly or implicitly) metadata for each frame that governs playback timing. Such metadata can include frame timestamps or frame durations. Note that timestamps can be used to calculate durations (e.g., the difference between consecutive timestamps) and durations can be used to calculate timestamps (e.g., the sum of preceding durations in a sequence of frames).

Example timing data170includes both stream metadata180and video frame timestamp metadata190that are received or derived from timing information transmitted with the stream. Stream metadata180includes a frame rate for components of the stream illustrated here as a 30 fps (or Hz) video component. Accordingly, the recipient should expect video frames with timestamps approximately 33 ms apart. That is not the case when the stream provides irregular timing data as illustrated by the video frame timestamp metadata190. As shown, a sequence of frames having associated timestamp of 1000, 1066, 1067, 1100, and 1133, indicating the first frame has a duration of 66 ms (1000 to 1066), the second frame has a durations of 1 ms (from 1066 to 1067), and the third and fourth frames have durations of 33 ms (from 1067 to 1100 and from 1100 to 1133, respectively). Audio frame timestamps can have similar irregularities relative to their advertised frame rate.

For regular timing data, the media ingestion service110can feed the decoder114and/or the encoder116from a buffer based on the advertised frame rate. For example, if the initial timestamp is 1000 with 30 fps video, the media ingestion service110can process (with the decoder114or encoder116) a frame with a timestamp of 1000 (plus or minus some tolerance) first, then a frame with a timestamp of ˜1033, then a frame with a timestamp of ˜1066. This monotonically increasing expected timestamp is based on the advertised frame rate. The media ingestion service110can use the expected timestamp to identify the next frame for processing (e.g., decoding, encoding, rendering, etc.). If the expected timestamp is greater than a buffered frame's actual timestamp given the tolerance, the media ingestion service110can treat the frame as old and discard the frame. Conversely, if the expected timestamp is less than any buffered frame's actual timestamp given the tolerance, the media ingestion service110can cause a prior frame to be duplicated (e.g., in the case where a frame may have been dropped). Finally, if the buffer does not contain a frame within the expected tolerance, the media ingestion service110can feed the next frame in the buffer to the decoder or encoder and cause a resynchronization amongst the stream components. Such may be the case where the stream switched from one component source to another component source resulting in a changed timestamp.

The media ingestion service110can synchronize the multiple stream components to one of the components. As used herein, the primary stream component serves as the timing reference to which the other, secondary stream components are synchronized. For example, if the stream includes both audio and video components, the media ingestion service110can treat the video component as the primary component and the audio component as the secondary component thereby synchronizing the audio to the video.

The following table provides sample data to illustrate the above exemplary operations. In this example, video is the primary (“Pri.”) component and audio is the secondary (“Sec.”) component. To simplify this example, the sample data represents audio and video frames with ˜33 ms durations. In practice, the two components can have different frame durations and/or time scales (e.g., sequential video frame timestamps are 33 ticks apart while sequential audio timestamps are 50 ticks apart where video and audio ticks corresponds to one ms; sequential video frame timestamps are 33 ticks apart where one video tick is one ms and sequential audio frame timestamps are 10 ticks apart where one audio tick is 100 ms).

As shown, video frames one through seven have associated timestamps in the column labeled “Pri. Frame TS”. The expected timestamp represents the expected timestamp of the next frame given the advertised frame rate which can be used to derive the expected duration of a frame or vice versa (e.g., 33 ms expected frame durations correspond to a 30 fps frame rate). In this example, the expected timestamp is initially adjusted to zero with an offset based on the first frame's timestamp listed under the “Offset” column. Such an offset can be used to transform streamed media data that switches between different sources having different timing to a common timeline (e.g., to handle abrupt changes in timestamps). The offset is updated and sent to the audio pipeline of the media ingestion service110when it changes by some threshold amount (e.g., 60 ms) as indicated in the “Offset Update” column Here, between primary frames four and five, the timestamps change by more than the threshold. The offset is updated from −1000 to −8867 for both pipelines.

Irregular timing information can cause the video pipeline of the media ingestion service110to duplicate or skip video frames resulting in undesirable video artifacts. For example, the video pipeline can sequentially process frames that match the expected timestamp within some tolerance such as 10 ms of expected. Here, the first, second, and fourth video frames are within that tolerance, but the third frame is not. As a result, the video pipeline may duplicate the second frame or insert a blank frame while discarding the third frame as indicated by the strikethrough.

Irregular timing information can also cause the audio pipeline of the media ingestion service110to skip audio frames resulting in undesirable audio artifacts. For example, the audio pipeline can sequentially process frames that match the expected timestamp within some tolerance such as 10 ms of expected. Here, the first, third, and fourth audio frames are within that tolerance, but the second frame is not. As a result, the audio pipeline may insert a period of silence or duplicate another frame while discarding the second frame.

Beyond irregularities in the audio timestamps, synchronizing the audio to the video can also cause undesirable audio artifacts due to offset changes. For example, when the offset changes between video frames four and five, the offset for the audio pipeline is updated. Note that the offset is applied to the value in the column labeled “Sec. Frame TS” to calculate the “Adjusted Frame TS.” The audio pipeline applies the new offset to find the next audio frame having an expected timestamp of 133. In this case, the new offset causes the audio pipeline to match frame number six instead of five as the next frame in the sequence. As a result, the audio pipeline skips frame five which can result in an undesirable artifact.

FIGS. 2-4present several approaches that can be used to accommodate irregular timing information in streamed media data to reduce the occurrence of undesirable video and audio artifacts such as that result from the operations described above. With reference toFIG. 2, a timing manager200can adjust expected timestamps based on duration information received with the stream. With reference toFIG. 3, a timing manager300can adjust the timestamps of frames received with the stream. With reference toFIG. 4, a timing manager400can adjust how far out of sync a secondary component of the media stream can be relative to the primary component. In various embodiment, a user may enable and/or disable one or more of these approaches and configure associated parameters via the interface(s)160.

FIG. 2is a block diagram illustrating an exemplary streaming media processing system implementing a first approach for accommodating irregular timing information in streamed media data according to some embodiments. In particular,FIG. 2depicts an exemplary implementation of the media ingestion service110that receives a media stream with irregular timing data and outputs another media stream for storage by the data store120or streaming by the media distribution service130. As illustrated, the media ingestion service110includes a demultiplexer202, a video decoder206, a video sync stage210, a video encoder212, an audio decoder226, an audio sync stage230, an audio encoder232, and a multiplexer240. Together, the video decoder206, the video sync stage210, and the video encoder212can be considered a video pipeline, and the audio decoder226, the audio sync stage230, and the audio encoder232can be considered an audio pipeline. The various pipeline stages can be interconnected via buffers204,208,214,224,228,234as illustrated, which may be implemented using a single memory device or multiple memory devices.

In this example, the media ingestion service110receives streamed media data and transcodes it for subsequent distribution. The media ingestion service110can change the streaming format (e.g., from a TCP-based RTMP stream to an HTTP-based HLS stream) and/or the video and/or audio formats (e.g., from one level of compression to another level of the same type of compression; between compression formats, etc.). The demultiplexer202separates the components of a received media stream (e.g., audio, video) for processing by the video pipeline and the audio pipeline. For example, the demultiplexer202can extract video frames and associated metadata, including timing information, from a RTMP media stream and store that data in the buffer204. Similarly, the demultiplexer202can extract audio frames and associated metadata, including timing information, from the RTMP media stream and store that data in the buffer224. The multiplexer240combines the components of the processed media stream processed by the video pipeline and the audio pipeline. For example, the multiplexer240can combine the processed video and audio portions into another RTMP stream or one or more files for segmented delivery via an HTTP-based streaming format such as HLS or DASH.

The video decoder206and the audio decoder226respectively process the video and audio portions of the received media stream. For example, they can decompress the video and audio portions into an uncompressed format. The video encoder212and the audio encoder232respectively process the uncompressed video and audio portions of the media stream. For example, they can compress the video and audio portions into a compressed format.

The video sync stage210and the audio sync stage230attempt to ensure that the data processed by the video encoder212and the audio encoder232, respectively, match the timing information provided by the stream and that the video and audio remain in sync. When the advertised rate of video and/or audio data diverges from actual frame timestamps, frames may be dropped or offset updates between the primary and secondary component(s) can cause the undesirable artifacts, such as those described for video and audio with reference to the above table. For example, when the advertised—or expected—timing of frames (e.g., 30 fps) diverges from actual frame timestamps, the media ingestion service110may drop frames. As another example, if the video is the primary component, the video sync stage210can advertise timing changes (e.g., a new offset) to the audio sync stage230as indicated by sync message250.

To illustrate the first approach for accommodating irregular timing information in streamed media data, an example293includes plots295and297. In example293, assume a media stream has an advertise frame rate of 30 fps, the threshold for offset adjustment is 60 ms, and of the 19 frames, the durations of frames 2, 5, 6, 10, 13, 14 and 16 are approximately 1 ms while the durations of the remaining frames (excluding frame 19 because the timestamp of frame 20 is not known at this time) are approximately 33 ms. For example, the plateau between frames 2-3 on the dashed, actual line indicates that frame 3 had a timestamp approximately 1 ms after the timestamp of frame 2.

In plot295, the solid line indicates the expected timestamp of incoming frames based on the expected frame duration given an advertised frame rate, the dashed line indicates the actual timing of incoming frames (based on the delta between the timestamp of a frame and the timestamp of the next frame), and the dotted line indicates how the expected timestamp is adjusted using an updated offset when the different between the expected timing and actual timing exceeds the example offset threshold of 60 ms. Since each shortened frame is approximately 32 ms off expected and the threshold for an offset adjustment is 60 ms, every two short frames results in a synchronization event. These synchronization events can cause undesirable playback artifacts such as the duplication or dropping of frames as described herein.

To reduce or avoid synchronization events due to irregular frame timing data, the timing manager200adjusts the expected frame times based on frame durations as calculated from frame timestamps, deviating from the advertised frame rate of the media stream. The timing manager200reads a buffer (e.g., buffer204or208) prior to the video sync stage210to determine actual frame durations. Each time a frame with a shorter than expected duration is processed, the timing manager200adjusts the next expected frame time downward to prevent the expected frame times from running away relative to the actual frame times. This can be illustrated in plot297where the solid line indicates the adjusted expected timestamp of incoming frames and the dashed line indicates the actual timing of incoming frames. By adjusting the expected timestamps, the timing manager200can cause the difference between expected frame times and the actual frame times to accumulate at a lower rate resulting in fewer synchronization events and associated undesirable artifacts in playback. Note that in the case of long frames (e.g., frames with a duration longer than expected based on the advertised frame rate), the timing manager200can take no action to prevent advancing the expected frame times beyond the advertised frame rate. Instead, long frames can be handled by other components of the media ingestion service110such as by inserting duplicate frames.

FIG. 3is a block diagram illustrating an exemplary streaming media processing system implementing a second approach for accommodating irregular timing information in streamed media data according to some embodiments. In particular,FIG. 3depicts an exemplary implementation of the media ingestion service110that receives a media stream with irregular timing data and outputs another media stream for storage by the data store120or streaming by the media distribution service130. The above description of the demultiplexer202, the video and audio pipelines, and the multiplexer240with reference toFIG. 2applies equally to the similarly numbered components ofFIG. 3.

To reduce or avoid synchronization events due to irregular frame timing data, the timing manager300adjusts frame timestamps to lengthen the durations of shorter frames and shorten the durations of longer frames relative to the advertised or expected frame rate, deviating from at least some of the timing data of the media stream. Two techniques for such adjustment are illustrated in examples395and397. In example395, a sequence of six frames with a total duration of 200 ms is shown where the duration of each frame is the difference between the timestamp of the frame and the timestamp of the next frame. Assuming a video component of a media stream was advertised to have a 30 fps frame rate, frames two and five have short durations, frames one and six have long durations, and frames three and four have durations in line with the advertised frame rate. The timing manager300can inspect frame timestamps over a window of two or more frames with a step size based on the window size. Based on the total duration, the timing manager300can adjust the timestamps of the frames within the window. In example395, the timing manager300adjusts the timestamps of the six frames based on the total duration of the frames over the window.

In example397, the timing manager300matches one or more patterns of frames that can be set based on empirical evidence of irregularities in timing information in streamed media data. In a simple case, the timing manager300matches a long frame followed by a short frame and advances the timestamp of the short frame to decrease the duration of the long frame. For example, the timing manager300can match a frame having a duration above some threshold of the expected duration (e.g., 10 ms above 33 ms) followed by a frame having a duration below some threshold of the expected duration (e.g., 10 ms below 33 ms). In example397, frame one has a duration of 50 ms and frame two has a duration of 1 ms, so the timing manager advances the timestamp of frame two—the short frame—by approximately half the total duration of the two frames. In another simple case, the timing manager300matches a short frame followed by a long frame and defers the timestamp of the long frame to increase the duration of the short frame.

Whether using the technique illustrated by example395, example397, or some other technique to adjust frame timestamps, the timing manager300regularizes frame timestamps. By doing so, the timing manager300increases the likelihood that frames having timestamps that deviate from an expected timestamp will be within a tolerance around that expected tolerance, thereby reducing the likelihood of frame skips/repeats.

FIG. 4is a block diagram illustrating an exemplary streaming media processing system implementing a third approach for accommodating irregular timing information in streamed media data according to some embodiments. In particular,FIG. 4depicts an exemplary implementation of the media ingestion service110that receives a media stream with irregular timing data and outputs another media stream for storage by the data store120or streaming by the media distribution service130. The above description of the demultiplexer202, the video and audio pipelines, and the multiplexer240with reference toFIG. 2applies equally to the similarly numbered components ofFIG. 4.

To reduce or avoid re-synchronizing a secondary stream component with a primary stream component, the timing manager400adjusts how far out of sync the secondary stream component can be from the primary stream component before re-resynchronization of the secondary component to the primary component. Two techniques for such adjustment are illustrated in examples495and497. In example495, the timing manager400adjusts a tolerance between the expected secondary component frame timestamps. Such a tolerance may be used to identify the next frame in the secondary component frame sequence based on the secondary component frame rate and offset determined during processing of the primary component (e.g., the offset advertised from video sync to audio sync via sync message250). For example, if the next expected audio timestamp is 10080 ms as adjusted by the advertised offset and the tolerance is 5 ms, the audio sync stage230will process the audio frame having a timestamp of 10080+/−5 ms. If no audio frame within that range is available, processing of audio frames can experience a discontinuity resulting in undesirable audio artifacts. As a result, irregular frame timestamps of the secondary component can lead to frequent discontinuities.

To avoid frequent discontinuities, the timing manager400can monitor the number of discontinuity events402within a window406and, based on a discontinuity limit, adjust the tolerance up or down. To illustrate, window406A covers four discontinuity events402where the timestamp deltas404(e.g., the difference between the expected timestamp and the nearest timestamp) were 9, 13, 11, and 7 ms. If the discontinuity limit for that period of time is two, the timing manager400adjusts the tolerance to 9 ms so that moving forward, audio frame timestamps that are +/−9 ms of the next expected audio timestamp will not cause a discontinuity. Later in time at window406B, the window covers four discontinuity events402where the timestamp deltas404were 13, 11, 7, and 6 ms. Using the same discontinuity limit, the timing manager400adjusts the tolerance to 7 ms so that audio frame timestamps that are +/−7 ms of the next expected audio timestamp will not cause a discontinuity. In some embodiments, the timing manager400can decrease the tolerance to a tolerance floor such as +/−5 ms.

In example497, the audio sync stage230defers resynchronizing the audio (e.g., the secondary component) to the video (e.g., the primary component) until the timestamp deltas410within a window412have stabilized. In other words, rather than introducing a discontinuity immediately when no audio frame is available with a timestamp within the tolerance (e.g., 5 ms) of the next expected audio timestamp, the audio sync stage230processes the next available frame with a timestamp closest to the next expected timestamp. The audio sync stage230continues to process the next available audio frames until the timestamp deltas410reach a level of stability. For example, the audio sync stage230would process the frame marked A having a timestamp delta of ‘1’ (less than the tolerance) and then process the frame marked B having a timestamp delta of ‘−25’ even though it is greater than the tolerance. In some embodiments, if no frame is within a tolerance ceiling (e.g., 50 ms), the audio sync stage230immediately resynchronizes the audio to the video.

In particular, the timing manager400can monitor the stability of the differences between the expected frame timestamps and the nearest frame timestamps. For example, the timing manager400can monitor the variation amongst the timestamp deltas410within a window (e.g., the N most recent N frames). If the stability of the timestamp deltas within that window is within a limit, the timing manager400can trigger resynchronization of the secondary media component to the primary media component (e.g., by triggering resynchronization with the audio sync stage230). To illustrate, window412A covers five frames with timestamp deltas410(e.g., the difference between the expected timestamp and the nearest timestamp) of −25, 11, 11, 10, and 12 ms, and window412B covers five frames with timestamp deltas410of 11, 11, 10, 12, and 11 ms. When that variation is within some allotted range but outside of the tolerance, the timing manager400triggers resynchronization. Such a variation may be measured in various ways, such as based on a variance or standard deviation around a mean of the timestamp deltas410or based on each of the timestamp deltas410being offset from zero but within some tolerance. For example, the timing manager400can trigger resynchronization at time414based on the deltas410within window412B and not the deltas410within window412A due to the higher variation in window412A. As another example, the timing manager400can trigger resynchronization at time414based on the deltas410within window412B since each of the events is within the example tolerance of 5 ms but with reference to a new mean (e.g., +11 given the timestamp deltas within window412B). By deferring resynchronizations, the audio sync stage230introduces fewer discontinuities by resynchronizing either when there is a large timestamp delta410(e.g., greater than the tolerance ceiling) or when the timestamp deltas410have stabilized.

FIG. 5is a flow diagram illustrating operations of a method for accommodating irregular timing information in streamed media data according to some embodiments. Some or all of the operations (or other processes described herein, or variations, and/or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware or combinations thereof. The code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) of the operations are performed by the media ingestion service110of the other figures.

The operations include, at block502, receiving, by a media ingestion service of a provider network, at least a portion of a media stream that includes a video component and an audio component, the video component including a plurality of video frames and video timing data, the audio component including a plurality of audio frames and audio timing data, and wherein the media stream includes an indication of a video frame rate and an indication of an audio frame rate. For example, the media ingestion service110of the other figures can receive and process a multimedia stream. The multimedia stream can include various media components such as video and audio. In addition, the multimedia stream can provide indications of expected video and audio frame rates as well as frame-specific timing information (e.g., timestamps, sequences and durations, etc.).

The operations further include, at block504, determining that a first duration of a first video frame of the plurality of video frames is different than an expected video frame duration, wherein the first duration of the first video frame is a difference between a timestamp of the first video frame and a timestamp of a second video frame of the plurality of video frames, the timestamp of the first video frame and the timestamp of the second video frame based at least in part on the video timing data, and wherein the expected video frame duration is based at least in part on the indication of the video frame rate. Occasionally, the transmitter of the media stream may transmit timing data that is inconsistent with the advertised frame rates. To detect this, the media ingestion service can compare the durations of individual frames compared to an expected duration given the advertised frame rate. In some cases, the durations of individual frames can be calculated from timestamps of adjacent frames.

The operations further include, at block506, adjusting a timestamp of at least one video frame of the plurality of video frames to change the first duration. One of the many approaches to accommodating irregular timing information in streamed media data involves changing the received timing data associated with individual media frames to better match the advertised frame rate. Examples of such an approach are provided with reference toFIG. 3.

FIG. 6is a block diagram illustrating an example computer system that may be used in some embodiments, such as computer systems105,140,150, or computer system(s) used to implement the components of the provider network100. Exemplary computer system600includes one or more processors610coupled to a memory620via an interface630. Computer system600further includes a network interface640coupled to the interface630. Computer system600optionally includes one or more displays650, one or more other input/output (I/O) components660, and/or one or more accelerators665. WhileFIG. 6shows computer system600as a single computing device, in various embodiments a computer system600may include one computing device or any number of computing devices configured to work together as a single computer system600.

In various embodiments, computer system600may be a uniprocessor system including one processor610, or a multiprocessor system including several processors610(e.g., two, four, eight, or another suitable number). Processors610may be any suitable processors capable of executing instructions. For example, in various embodiments, processors610may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, ARM, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of processors610may commonly, but not necessarily, implement the same ISA.

The memory620may store instructions and data accessible by processor(s)610. In various embodiments, the memory620may be implemented using any suitable memory technology, such as random-access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions and data implementing one or more desired functions, such as those methods, techniques, and data described above are shown stored within the memory620as code625and data626. For example, the code625can include media ingestion service code627for one or more of the components of the media ingestion service112, such as code for the demultiplexer202, the video decoder206, the audio decoder226, the video sync stage210, the audio sync stage230, a timing manager implementing one or more of the approaches described herein (e.g., the timing manager112,200,300,400), the video encoder212, the audio encoder232, the multiplexer240, and/or the buffers204,224,208,228,214, and234(e.g., interface code used to store data to and retrieve data from physical memory devices). The data626can include frame data and metadata628, such as the stream metadata180, the frame timestamp metadata190, and frame data (in compressed and/or uncompressed form). Note that in some embodiments, the components of the media ingestion service112may be implemented by a plurality of networked computer systems such that one computer system performs a portion of the media ingestion service112(e.g., receiving and decoding a stream, implementing approaches for accommodating irregular timing information) while another computer system performs another portion of the media ingestion service (e.g., encoding a stream).

In one embodiment, the interface630may be configured to coordinate I/O traffic between processor610, memory620, and any peripheral devices in the device, including network interface640or other peripheral interfaces. In some embodiments, the interface630may perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., memory620) into a format suitable for use by another component (e.g., processor610). In some embodiments, the interface630may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example. In some embodiments, the function of the interface630may be split into two or more separate components, such as a north bridge and a south bridge, for example. Also, in some embodiments some or all of the functionality of the interface630, such as an interface to memory620, may be incorporated directly into processor610.

Network interface640may be configured to allow data to be exchanged between computer system600and other devices680attached to a network or networks670, such as other computer systems or electronic devices as illustrated in the other figures, for example. In various embodiments, network interface640may support communication via any suitable wired or wireless general data networks, such as types of Ethernet network, for example. Additionally, network interface640may support communication via telecommunications/telephony networks such as analog voice networks or digital fiber communications networks, via storage area networks (SANs) such as Fibre Channel SANs, or via I/O any other suitable type of network and/or protocol.

The display(s)650, such as a touch screen or liquid crystal display (LCD), convey visual information to a user, although some computer systems may not have a display at all (e.g., servers). The I/O components660provide facilities to the user for interacting with the computer system600. Such I/O components660include, but are not limited to, speakers for rendering audio, keyboards, mice, or other input devices for receiving user inputs, microphones, cameras, other sensors, etc. for collecting data, etc. The accelerators665may provide hardware acceleration for media encode or decode operations.