Source: https://patents.google.com/patent/US9237387B2/en
Timestamp: 2019-08-20 07:39:06
Document Index: 564255876

Matched Legal Cases: ['Application No. 61', 'Application No. 201080045546', 'Application No. 201080045546', 'Application No. 10822639', 'Application No. 201080016209', 'Application No. 201080045546', 'Application No. 2010080012748', 'Application No. 2010080012748', 'Application No. 2010080012748', 'Application No. 2010080012748', 'Application No. 2010080012748', 'Application No. 2010080012748', 'Application No. 2010080012748', 'Application No. 2010080012748', 'Application No. 2010080012748', 'Application No. 2010080012748', 'Application No. 201080012748', 'Application No. 201080045546', 'Application No. 201080045546', 'Application No. 201080045546', 'Application No. 1', 'Application No. 201080012748']

US9237387B2 - Low latency cacheable media streaming - Google Patents
Low latency cacheable media streaming Download PDF
US9237387B2
US9237387B2 US12/611,133 US61113309A US9237387B2 US 9237387 B2 US9237387 B2 US 9237387B2 US 61113309 A US61113309 A US 61113309A US 9237387 B2 US9237387 B2 US 9237387B2
US12/611,133
US20110080940A1 (en
2009-10-06 Priority to US24925709P priority Critical
2009-11-03 Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
2009-11-03 Priority to US12/611,133 priority patent/US9237387B2/en
2009-11-13 Assigned to MICROSOFT CORPORATION reassignment MICROSOFT CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: DUGGARAJU, KRISHNA PRAKASH, FREELANDER, JACK E., LIN, NING, LIU, LIN, ROY, ANIRBAN, BOCHAROV, JOHN A.
2011-04-07 Publication of US20110080940A1 publication Critical patent/US20110080940A1/en
2016-01-12 Publication of US9237387B2 publication Critical patent/US9237387B2/en
A low latency streaming system provides a stateless protocol between a client and server with reduced latency. The server embeds incremental information in media fragments that eliminates the usage of a typical control channel. In addition, the server provides uniform media fragment responses to media fragment requests, thereby allowing existing Internet cache infrastructure to cache streaming media data. Each fragment has a distinguished Uniform Resource Locator (URL) that allows the fragment to be identified and cached by both Internet cache servers and the client's browser cache. The system reduces latency using various techniques, such as sending fragments that contain less than a full group of pictures (GOP), encoding media without dependencies on subsequent frames, and by allowing clients to request subsequent frames with only information about previous frames.
The present application claims priority to U.S. Provisional Patent Application No. 61/249,257 entitled “LOW LATENCY CACHEABLE MEDIA STREAMING,” and filed on Oct. 6, 2009, which is hereby incorporated by reference.
A low latency streaming system is described herein that provides a stateless protocol between the client and server with reduced latency over previous systems. The server embeds incremental information in media fragments (i.e., chunks) that eliminates the usage of a typical control channel. In addition, the server provides uniform media fragment responses to media fragment requests (i.e., clients requesting the same fragment get the same response), thereby allowing existing Internet cache infrastructure to cache streaming media data. Each fragment has a distinguished Uniform Resource Locator (URL) that allows the fragment to be identified and cached by both Internet cache servers and the client's browser cache. The system reduces latency using various techniques, such as sending fragments that contain less than a full group of pictures (GOP), encoding media without dependencies on subsequent frames, and by allowing clients to request subsequent frames with only information about previous frames. Thus, the low latency streaming system provides a more scalable streaming media server without tracking client state and with an increased likelihood that clients will receive media with lower latency from a cache server local to the client.
FIG. 1 is a block diagram that illustrates components of the low latency streaming system, in one embodiment.
A low latency streaming system is described herein that provides a stateless protocol between the client and server with reduced latency over previous systems. The server embeds incremental information in media fragments (i.e., chunks) that eliminates the usage of a typical control channel. In addition, the server provides uniform media fragment responses to media fragment requests (i.e., clients requesting the same fragment get the same response), thereby allowing existing Internet cache infrastructure to cache streaming media data. Each fragment has a distinguished Uniform Resource Locator (URL) that allows the fragment to be identified and cached by both Internet cache servers and the client's browser cache. Caching reduces the load on the server and allows more clients to view the same content at the same time. The low latency streaming system receives media data in fragments from one or more encoders, creates an index of each fragment, and stores the fragments.
When switching bit rates, the client 150 simply begins requesting the new bit rate and playing back the new bit rate chunks as the client 150 receives the chunks. Unlike previous systems, the client 150 does not have to send control information to the server 105 and wait for the server 105 to adapt the stream. The client's requests may not even reach the server 105 due to a cache in between the client 150 and server 105 satisfying the request. Thus, the client 150 is much quicker to react than clients in traditional media streaming systems are, and the burden on the server 105 of having different clients connecting under various current conditions is reduced dramatically. In addition, because current conditions tend to be localized, it is likely that many clients in a particular geographic region or on a particular Internet service provider (ISP) will experience similar conditions and will request similar media encodings (e.g., bit rates). Because caches also tend to be localized, it is likely that the clients in a particular situation will find that the cache near them is “warm” with the data that they each request, so that the latency experienced by each client will be low.
CodecPrivateData=”...” />
The client manifest lists the decoding information as well as information for all the fragments that the server has archived so far. The total media fragment number and duration is only for the media fragments that the server has archived up until when the client makes the request (this allows the client to quickly build a seek bar). For each media fragment, “t” means the absolute timestamp. The client uses this value to compose the fragment URL (e.g., “Fragments(video={start time})). LookAheadFragmentCount indicates the targeted number of subsequent fragments that “TrackFragmentReferenceBox” is going to reference as described further herein. “IsLive” indicates whether the live broadcast is still going on.
In some embodiments, the ingest servers 250 comprise one or more servers dedicated to ingesting encoder media streams. An administrator or content author may create a publishing point that defines a URL at which clients of the ingest servers 250 can find a particular media element (e.g., a live event). For example, using IIS, the administrator may publish a URL “http://ingserver/pubpoint.isml.” The publishing point is used by the encoders 230 to provide new media data to the ingest servers 250 and by the origin servers 260 to request media data from the ingest servers 250. Each encoder may use a distinguished URL to connect to the ingest servers 250 so that the ingest servers 250 can detect different encodings of the same data. For example, based on the URL in the previous example, an encoder may send an HTTP POST to provide media data to the ingest server using the URL “http://ingserver/pubpointisml/Streams(stream1).” The ingest servers 250 store the received data for later retrieval by clients of the ingest servers 250 (e.g., the origin servers 260). The POST may contain various types of media formats, such as an MP4 container. An MP4 container contains various types of information, called boxes, that are typically labeled with a four-letter code, such as “ftyp” to describe the type of encoding used and “moov” to contain audiovisual data. Whether using MP4 or other container formats, the encoder may add additional boxes or information to the stream, such as a “ManifestBox” that contains a manifest describing the media element.
When the ingest servers 250 receive a request for data, the ingest servers 250 provide the data stored earlier. The ingest servers 250 may support several types of requests, including a request for encoder stream manifest that identifies the available encoder streams and a request for data from a specific stream (including portions of the stream data). The type of request may be identified by the URL of the request. For example, when the ingest servers 250 receive the URL “http://ingserver/pubpoint.isml/StreamManifest,” the ingest servers 250 return an encoder manifest that contains identifiers for each available encoder. When the ingest servers 250 receives the URL “http://ingserver/pubpoint.isml/Streams(stream1),” the ingest servers 250 send a corresponding media stream for the encoder associated with the identifier “Encoder1” in response. The response may include the MP4 data, such as the cached “ftyp,” “ManifestBox,” and “moov” boxes described above followed by media fragments in a FIFO buffer. The ingest servers 250 may also receive partial data requests (e.g., during failover scenarios) of the form “http://ingserver/pubpointisml/Streams(stream1)/StartTime(12345678),” that cause the ingest servers 250 to skip sending “ftyp,” “ManifestBox,” and “moov” boxes and to try to start from the media fragment that is closest to the specified timestamp.
Continuing in block 440, the system receives a fragment request from a client. The client may identify that fragment by using a particular URL. The URL may identify a time of the fragment as well as an encoding. For example, the URL may be of the form “http://server/event.isml/QualityLevels(1500000)/Fragments(video=20000000),” where the QualityLevels parameter is a bit rate measured in bits per second, video is the name of the track being requested, and the value following “video=” is the time position in units of 100 nanoseconds (the scale of the unit depends on the way the presentation is encoded). In some embodiments, the client may request a fragment by referencing a previous fragment, such as using a URL like “NextFragments (video=20000000)” in place of “Fragments” in the previous URL. Continuing in block 450, the system builds an incremental manifest by retrieving manifest information from the fragment data store and local index table that describes the requested fragment. The system may also include manifest information for one or more subsequent fragments in the incremental manifest as described herein.
In some embodiments, the low latency streaming system provides streams that play at different rates to clients. For example, the server may include 2×, 5×, 0.5×, and other speeds of playback. The client can switch to a stream of a different rate to provide the appearance to the user that the media is fast-forwarding (e.g., 2×) or rewinding (e.g., 0.5×). To switch, the client simply requests a different media fragment, e.g., at a different URL. The client can smoothly switch between playing media fragments at the current rate and playing media fragments at a different rate by continuing to play the particular media fragments that are received. This provides a seamless experience to the end user with little latency between the user's request and the change in the media playback. This also saves network bandwidth as the client does not download, for example, two times the data to play media twice as fast, but rather downloads a reduced size encoding of the media that is encoded at the accelerated rate.
In some embodiments, the low latency streaming system provides failover for various components of the system. For example, the system may include redundant encoders, ingest servers, origin servers, and so forth. During an encoder failover, the server may append “StartTime(nnnn)” to the encoder URL where “nnnn” is the absolute timestamp of the last fragment the server successfully received. An example of the failover URL would be “http://encoder:port/StartTime(12345678).” When using MP4 boxes, the backup encoder does not need to resend the “ftyp,” “ManifestBox,” and “moov”' boxes when it starts the stream. If encoder failover caused missing fragments, the server will return “404—File Not Found” if those fragments are requested by the client.
sending encoder configuration information to one or more encoders that includes parameters to reduce latency for one or more clients watching a live event encoded by the one or more encoders;
requesting a media manifest from each registered encoder that describes the media data available from the encoder;
receiving an encoder manifest from each encoder that describes one or more parameters of the encoder;
receiving a media fragment from an encoder;
indexing the received media fragment and adding the index information to an index table that catalogs the available media fragments;
building a server manifest that includes information regarding a media event of which the media fragments are a part by adding information regarding the received fragment to the manifest,
2. The method of claim 1 wherein the encoder configuration information includes a parameter that requests that the encoders encode video without bidirectionally encoded frames (B-frames).
a register event component configured to receive information regarding a live media event for which the system will receive encoded media data and provide the encoded media data to one or more clients watching the encoded live media event;
an index fragment component configured to create and maintain an index table of media fragments received from encoders;
a build client manifest component configured to build a manifest to satisfy a client request that includes information regarding each of the encodings available from the system and fragments stored by the system up to a time of the request; and
10. The system of claim 9 wherein the encoder interface component is further configured to send encoder configuration information to encoders that specifies parameters for reducing latency.
16. A computer-readable storage device comprising instructions for controlling a computer system to playback media on a client with low latency, wherein the instructions, when executed, cause a processor to perform actions comprising:
sending from the client a request for a chunk of media to a server over a network related to a live media event to be watched at the client, wherein the chunk comprises a uniform portion of the media available from the server to multiple clients;
playing the media data using a codec identified by the chunk metadata and hardware of the client.
17. The device of claim 16 wherein the request includes an indication that the client is request a chunk following a chunk previous received by the client.
18. The device of claim 16 wherein the received chunk contains no bidirectionally encoded frames (B-frames).
19. The device of claim 16 wherein the client plays the media data with lower latency without using a decoding buffer for decoding bidirectionally encoded frames (B-frames).
20. The device of claim 16 wherein playing the media data comprises combining data in a chunk without an intraframe (I-frame) with data in a chunk with an intraframe (I-frame).
US12/611,133 2009-10-06 2009-11-03 Low latency cacheable media streaming Active 2030-04-22 US9237387B2 (en)
US24925709P true 2009-10-06 2009-10-06
US12/611,133 US9237387B2 (en) 2009-10-06 2009-11-03 Low latency cacheable media streaming
PCT/US2010/051695 WO2011044285A2 (en) 2009-10-06 2010-10-06 Low latency cacheable media streaming
EP10822639A EP2486705A4 (en) 2009-10-06 2010-10-06 Low latency cacheable media streaming
CN201080045546.9A CN102577272B (en) 2009-10-06 2010-10-06 Low latency cacheable streaming media
US20110080940A1 US20110080940A1 (en) 2011-04-07
US9237387B2 true US9237387B2 (en) 2016-01-12
ID=43823144
US12/611,133 Active 2030-04-22 US9237387B2 (en) 2009-10-06 2009-11-03 Low latency cacheable media streaming
US (1) US9237387B2 (en)
EP (1) EP2486705A4 (en)
CN (1) CN102577272B (en)
WO (1) WO2011044285A2 (en)
US10123040B2 (en) 2016-08-30 2018-11-06 Qualcomm Incorporated Intra-coded video frame caching for video telephony sessions
US9930379B2 (en) 2012-01-31 2018-03-27 Comcast Cable Communications, Llc System and method for data stream fragmentation
US20140078911A1 (en) * 2012-09-17 2014-03-20 Fluke Corporation Method and apparatus to determine the amount of delay in round trip latency for a connection where the tcp traffic does not contain multi-packet responses or may not be transaction oriented traffic.
JP6059820B2 (en) * 2013-07-18 2017-01-11 コニンクリーケ・ケイピーエヌ・ナムローゼ・フェンノートシャップ Low-latency streaming
MX358670B (en) * 2013-07-17 2018-08-31 Sony Corp Content provision device, content provision method, program, terminal device, and content provision system.
AU2015202846B2 (en) * 2014-06-27 2016-07-21 Foxtel Management Pty Ltd A method and system for streaming one or more broadcast events
EP3466081A1 (en) * 2016-06-01 2019-04-10 Amazon Technologies Inc. Catching up to the live playhead in live streaming
DE102016116555A1 (en) * 2016-09-05 2018-03-08 Nanocosmos Informationstechnologien Gmbh A method for transmitting real-time-based digital video signals in networks
EP3393129A1 (en) * 2017-04-21 2018-10-24 Alcatel-Lucent España, S.A. Multimedia content delivery with reduced delay
US20190036763A1 (en) * 2017-07-28 2019-01-31 Skitter, Inc. System and method for providing fault tolerant streaming of segmented content and cache coherency on multi-hosted origin systems
WO2002017644A1 (en) 2000-08-21 2002-02-28 Nokia Corporation Video coding
2009-11-03 US US12/611,133 patent/US9237387B2/en active Active
2010-10-06 EP EP10822639A patent/EP2486705A4/en active Pending
2010-10-06 CN CN201080045546.9A patent/CN102577272B/en active IP Right Grant
2010-10-06 WO PCT/US2010/051695 patent/WO2011044285A2/en active Application Filing
"First Office Action and Search Report Issued in Chinese Patent Application No. 201080045546.9", Mailed Date: Dec. 17, 2013, 13 Pages.
"Fourth Office Action and Search Report Issued in Chinese Patent Application No. 201080045546.9", Mailed Date: Jun. 12, 2015, 14 Pages.
"HTTP Streaming", retrieved at <<http://www.peerapp.com/Data/Files/HTTP-Video-Streaming-Solution-Brief.pdf>>, Sep. 25, 2009, pp. 1-6.
"Office Action Issued in European Patent Application No. 10822639.0", Mailed Date: Nov. 14, 2014, 5 Pages.
"Second Office Action Issued in Chinese Patent Application No. 201080016209.7", Mailed Date: Aug. 18, 2014, 10 Pages.
"Smooth Streaming Architecture | Alex Zambelli's Streaming Media Blog", Published on: Feb. 10, 2009, Available at: http://alexzambelli.com/blog/2009/02/10/smooth-streaming-architecture/.
"Third Office Action Issued in Chinese Patent Application No. 201080045546.9", Mailed Date: Feb. 10, 2015, 6 Pages.
Chinese Patent Application No. 2010080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Amendment dated Apr. 23, 2014, 17 pages (including English summary).
Chinese Patent Application No. 2010080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Amendment dated Jan. 7, 2014, 5 pages (including English summary).
Chinese Patent Application No. 2010080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), First Office Action dated Oct. 29, 2012, 11 pages (including English Translation).
Chinese Patent Application No. 2010080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Notice on Reexamination dated Mar. 13, 2015, 9 pages (including English Translation).
Chinese Patent Application No. 2010080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Office Action dated Feb. 12, 2014, 8 pages (including English Translation).
Chinese Patent Application No. 2010080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Office Action dated Oct. 31, 2013, 7 pages (including English Translation).
Chinese Patent Application No. 2010080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Response to First Office Action dated Feb. 28, 2013, 14 pages (including English Translation of claims).
Chinese Patent Application No. 2010080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Response to Office Action dated Sep. 13, 2013, 12 pages (including English Translation of claims).
Chinese Patent Application No. 2010080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Second Office Action dated Jul. 11, 2013, 6 pages (including English Translation).
Chinese Patent Application No. 2010080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Summary of Notice on Reexamination dated Mar. 13, 2015, 1 page.
Chinese Patent Application No. 201080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Request for Reexamination dated Sep. 17, 2014, 9 pages.
Chinese Patent Application No. 201080045546.9 (Foreign counterpart application to U.S. Appl. No. 12/611,133), Office Action dated Dec. 17, 2013, 13 pages (including English translation).
Chinese Patent Application No. 201080045546.9 (Foreign counterpart application to U.S. Appl. No. 12/611,133), Response to Fourth Office Action, Mailed Date: Jul. 23, 2015, 17 Pages.
Chinese Patent Application No. 201080045546.9(Foreign counterpart application to U.S. Appl. No. 12/611,133), Notice of Allowance dated Nov. 23, 2015, 4 pages (including English Translation).
European Patent Application No. 1 0822639.0 (Foreign counterpart application to U.S. Appl. No. 12/611,133), Amendment dated May 12, 2015, 9 pages.
European Patent Application No. EP 10 75 3888.6 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Extended European Search Report dated Nov. 16, 2012, 7 pages.
European Patent Application No. EP 10 75 3888.6 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Response to Extended European Search Report dated Feb. 15, 2013, 19 pages.
European Patent Application No. EP 10 82 2639.0 (Foreign counterpart application to U.S. Appl. No. 12/611,133), Extended European Search Report dated Feb. 26, 2013, 7 pages.
European Patent Application No. EP 10 82 2639.0 (Foreign counterpart application to U.S. Appl. No. 12/611,133), Response Extended European Search Report dated Aug. 5, 2013, 14 pages.
Hsueh, et al., "A Packet-Based Caching Proxy with Loss Recovery for Video Streaming", retrieved at <<http:// ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1185637&isnumber=26604>>, 2002, pp. 6.
Nelson, David, "Windows Media Server or Web Server?", retrieved at <<http://learn.iis.net/page.aspx/454/windows-media-server-or-web-server/>>, May 15, 2008, pp. 9.
Office Action Issued in Chinese Patent Application No. 201080012748.3 (Foreign counterpart application to U.S. Appl. No. 12/405,220), Mailed Date: Jun. 13, 2014, Filed Date: Mar. 9, 2010, 11 Pages.
PCT Application No. PCT/US2010/026710 (Foreign counterpart application to U.S. Appl. No. 12/405,220), International Search Report mailed Sep. 27, 2010, 11 pages.
U.S. Appl. No. 12/405,220, Amendment dated Apr. 28, 2014, 13 pages.
U.S. Appl. No. 12/405,220, Amendment dated Mar. 8, 2011, 19 pages.
U.S. Appl. No. 12/405,220, Amendment dated Sep. 30, 2011, 11 pages.
U.S. Appl. No. 12/405,220, Final Rejection dated Mar. 31, 2011, 33 pages.
U.S. Appl. No. 12/405,220, Non-Final Rejection dated Jan. 27, 2014, 25 pages.
U.S. Appl. No. 12/405,220, Non-Final Rejection dated Oct. 8, 2010, 27 pages.
U.S. Appl. No. 12/405,220, Notice of Allowance dated Aug. 4, 2014, 8 pages.
U.S. Appl. No. 12/405,220, Terminal Disclaimer dated Apr. 28, 2014, 5 pages.
Wu, et al., "Streaming Video over the Internet: Approaches and Directions", retrieved at <<http://people.cs.ubc.ca/˜krasic/cpsc538a-2005/papers/wustreaming.pdf>>, Mar. 2001, pp. 282-300.
US20110080940A1 (en) 2011-04-07
WO2011044285A3 (en) 2011-09-09
CN102577272A (en) 2012-07-11
WO2011044285A2 (en) 2011-04-14
CN102577272B (en) 2016-03-16
EP2486705A2 (en) 2012-08-15
EP2486705A4 (en) 2013-03-27
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BOCHAROV, JOHN A.;DUGGARAJU, KRISHNA PRAKASH;LIU, LIN;AND OTHERS;SIGNING DATES FROM 20091027 TO 20091030;REEL/FRAME:023512/0955