Source: https://patents.justia.com/patent/20110252118
Timestamp: 2019-06-26 01:45:44
Document Index: 86693332

Matched Legal Cases: ['Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 61']

US Patent Application for REAL-TIME OR NEAR REAL-TIME STREAMING Patent Application (Application #20110252118 issued October 13, 2011) - Justia Patents Search
Justia Patents Accessing A Remote ServerUS Patent Application for REAL-TIME OR NEAR REAL-TIME STREAMING Patent Application (Application #20110252118)
Methods and apparatuses provide real-time or near real-time streaming of content, specified in one or more playlists, using transfer protocols such as an HTTP compliant protocol. In one embodiment, a method can execute a user application on a client device to present media files and to control presentation of the media files. The method can further run a media serving process on the client device to retrieve a playlist specifying the media files and a media source at which the media files are available, to retrieve the media files from the media source, and to decode the media files retrieved. The media serving process can call the user application to process a custom URL in order to obtain an object referred to by the custom URL.
This application claims the benefit of the filing dates of the following U.S. provisional applications: U.S. provisional application 61/321,767 filed on Apr. 7, 2010 (Docket No. P7437Z8) and U.S. provisional application 61/351,824 filed on Jun. 4, 2010 (Docket No. P7437Z9), and this application hereby incorporates by reference herein those provisional applications. This application is also related to the following patent applications:
(1) Application No. 61/142,110 filed on Dec. 31, 2008 (Docket No. P7437Z);
(2) Application No. 61/160,693 filed on Mar. 16, 2009 (Docket No. P7437Z2);
(3) Application No. 61/161,036 filed on Mar. 17, 2009 (Docket No. P7437Z3);
(4) Application No. 61/167,524 filed on Apr. 7, 2009 (Docket No. P7437Z4);
(5) Application No. 61/240,648 filed on Sep. 8, 2009 (Docket No. P7437Z5);
(6) Application No. 61/288,828 filed on Dec. 21, 2009 (Docket No. P7437Z6); and
(7) Application No. 61/320,213 filed on Apr. 1, 2010 (Docket No. P7437Z7). All of these U.S. provisional applications are incorporated herein by reference to the extent that they are consistent with this disclosure.
The present U.S. patent application is related to the following U.S. patent applications, each of which is incorporated herein by reference to the extent they are consistent with this disclosure:
(1) application Ser. No. 12/479,690 (Docket No. P7437US1), filed Jun. 5, 2009, entitled “REAL-TIME OR NEAR REAL-TIME STREAMING;”
(2) application Ser. No. 12/479,698 (Docket No. P7437US2), filed Jun. 5, 2009, entitled “VARIANT STREAMS FOR REAL-TIME OR NEAR REAL-TIME STREAMING;”
(3) application Ser. No. 12/479,732 (Docket No. P7437US3), filed Jun. 5, 2009, entitled “UPDATABLE REAL-TIME OR NEAR REAL-TIME STREAMING;”
(4) application Ser. No. 12/479,735 (Docket No. P7437US4), filed Jun. 5, 2009, entitled “PLAYLISTS FOR REAL-TIME OR NEAR REAL-TIME STREAMING;”
(5) application Ser. No. 12/878,002 (Docket No. P7437X), filed Sep. 8, 2010, entitled “VARIANT STREAMS FOR REAL-TIME OR NEAR REAL-TIME STREAMING TO PROVIDE FAILOVER PROTECTION;” and
(6) application Ser. No. 12/968,202 (Docket No. P7437×2), filed Dec. 14, 2010 entitled “REAL-TIME OR NEAR REAL-TIME STREAMING WITH COMPRESSED PLAYLISTS.”
In one embodiment, a system can search for content based upon a date and time. For example, in one implementation, timestamped tags are created, and each of the timestamped tags can be associated with a particular media file. The timestamp in a timestamped tag indicates a beginning date and time of the associated media file. Note that the media file may contain its own internal timestamps. A playlist file can be created with one or more timestamped tags. The playlist file can be distributed and made available for searching by date and time using the date and time in the timestamped tags. In one embodiment, the timestamped tags can use a format known as ID3.
In one embodiment, a method can execute a user application on a client device to present media files and to control presentation of the media files. The method can further run a media serving process on the client device to retrieve a playlist specifying the media files and a media source at which the media files are available, to retrieve the media files from the media source, and to decode the media files retrieved. The user application can be configured to communicate with one or more servers through a custom URL or custom protocol or both even though the media serving process is not configured to process the custom URL or custom protocol. The custom URL or custom protocol can specify or provide a decryption key for decrypting encrypted content in the media files.
Other methods are described herein and systems for performing these methods are described herein and machine readable, non-transitory storage media storing executable instructions which when executed can cause a data processing system to perform any one of these methods are also described herein.
FIG. 12A is a flowchart depicting a method according to one embodiment of the present invention for controlling the creation and distribution of playlists.
FIG. 12B shows a timeline of how, in one embodiment, playlists can be transmitted or otherwise distributed using, for example, a method as in FIG. 12A.
FIG. 13 is a method, according to one embodiment of the invention, for controlling playback at a client device.
FIG. 14A shows a flowchart depicting a method, in one embodiment, for adaptively determining an amount of minimum overlap based upon connection speed or connection type. FIGS. 14B, 14C, and 14D show another aspect of an embodiment which uses an overlap for switching between streams.
FIG. 15 is a flowchart depicting another method according to one embodiment of the present invention.
FIG. 16A shows a flowchart that depicts a method according to one embodiment for using the timestamped tags to create a playlist file.
FIG. 16B shows a flowchart that depicts a method according to one embodiment for using the timestamped tags in a playlist file to search for media files.
FIG. 17A shows an example of software architecture to allow a media serving daemon to interact with a user application. FIG. 17B shows an example of a software architecture which can use a custom URL technique, and FIG. 17C is a flowchart showing an example of a method to use a custom URL technique. FIG. 17D shows a flowchart that provides an example of a method performed by an application that uses a custom URL technique; and FIG. 17E is a flowchart that shows an example of a method performed by a player service or Operating System or both.
FIG. 18 illustrates a block diagram of an exemplary API architecture usable in some embodiments of the invention.
FIG. 19 shows an exemplary embodiment of a software stack usable in some embodiments of the invention.
FIG. 20 shows an example of a software architecture for allowing one device to control or to initiate playback of media on another device.
FIG. 21 shows an example of a software architecture employing calls between different components of the software in the software architecture on the two different devices shown in FIG. 20 and FIG. 21.
FIGS. 22A and 22B show an example of a method according to one embodiment of the present invention which may be employed to allow two devices, such as those shown in FIG. 20, to interact and provide remote playback on one device which is initiated by another device.
EXT-X-DISCONTINUITY
The EXT-X-TARGETDURATION tag can indicate, in one embodiment, the approximate duration of the next media file that will be added to the presentation. It can be included in the playback file and the format can be:
where “seconds” indicates the duration of the media file. In one embodiment, the actual duration may differ slightly from the target duration indicated by the tag. In one embodiment, every URI indicating a segment will be associated with an approximate duration of the segment; for example, the URI for a segment may be prefixed with a tag indicating the approximate duration of that segment. In another embodiment, the EXT-X-TARGETDURATION tag can specify the maximum media file duration; the EXTINF duration of each media file in the playlist file should be less than or equal to the target duration, and this tag (which specifies the maximum media file duration) can be specified just once in the playlist file and it applies to all media files in the playlist file, and its format can be:
#EXT-X-TARGETDURATION:
where “s” is an integer indicating the target duration in seconds.
where “number” is the sequence number of the URI. If the playlist file does not include a #EXT-X-MEDIA-SEQUENCE tag, the sequence number of the first URI in the playlist can be considered 1. A media file's sequence number is not required to appear in its URI in one embodiment, and in one embodiment, a playlist can contain only one EXT-X-MEDIA-SEQUENCE tag. In one embodiment, the sequence numbering can be non-sequential; for example, non-sequential sequence numbering such as 1, 5, 7, 17, etc. can make it difficult to predict the next number in a sequence and this can help to protect the content from pirating. Another option to help protect the content is to reveal only parts of a playlist at any given time.
#EXT-X-KEY:METHOD=<method>[,URI=“<URI>”] [,IV=<IV>]
The METHOD parameter specifies the encryption method and the URI parameter, if present, specifies how to obtain the key and the IV (Initialization Vector), if present, specifies an initialization vector used in the encryption method (e.g. with the key).
An encryption method of NONE indicates no encryption and if NONE is indicated then, in one embodiment, the URI and IV parameters should not be present. Various encryption methods may be used, for example AES-128, which indicates encryption using the Advance Encryption Standard encryption with a 128-bit key and PKCS7 padding [see RFC3852]. A new EXT-X-KEY tag supersedes any prior EXT-X-KEY tags.
The EXT-X-PROGRAM-DATE-TIME tag can associate the beginning of the next media file with an absolute date and/or time and can include or indicate a time zone. In one embodiment, the date/time representation is ISO/IEC 8601:2004. The value of the date and time in this tag can provide an informative mapping of the timeline of the media to an appropriate wall-clock time, which may be used as a basis for seeking, for display or other purposes, content for playback based on a date and time. In one embodiment, if a server provides this mapping, it should place an EXT-X-PROGRAM-DATE-TIME tag after every EXT-X-DISCONTINUITY tag in the playlist file. The tag format can be:
The EXT-X-ALLOW-CACHE tag can be used to indicate whether the client may cache the downloaded media files for later playback. This tag can appear anywhere in the playlist file in one embodiment but, in one embodiment, should appear only once in the playlist file. The tag format can be:
In one embodiment, if a playlist contains the final segment or media file then the playlist will have the EXT-X-ENDLIST tag. This tag can appear, in one embodiment, anywhere in a playlist file, and in one embodiment, it can occur only once in the playlist file.
where the following attributes may be used. An attribute of the same type, in one embodiment of this tag, should not appear more than once in the same tag. The attribute BANDWIDTH=<n> is an approximate upper bound of the stream bit rate expressed as a number of bits per second. In one embodiment, the attribute BANDWIDTH can be an upper bound of the overall bitrate of each media file, calculated to include container overhead that appears or will appear in the playlist. The attribute PROGRAM-ID=<i> is a number that uniquely identifies a particular presentation within the scope of the playlist file. A playlist file may include multiple EXT-X-STREAM-INF URIs with the same PROGRAM-ID to describe variant streams of the same presentation and these variant playlists can contain additional EXT-X-STREAM-INF tags. Variant streams and variant playlists are described further in this disclosure (e.g. see FIGS. 9A-9D). The attribute CODECS=“[format][,format]*” can be used to specify a media sample type that is present in a media file in the playlist file, where each format specifies a media sample type; in one embodiment, valid format identifiers can be those in the ISO File Format Name Space defined by RFC 4281. The attribute RESOLUTION=<N>×<M> can specify a resolution of video within the stream, where N is the approximate encoded horizontal resolution of video within the stream, which can be expressed as a number of pixels, and M is the approximate encoded vertical resolution.
The EXT-X-DISCONTINUITY tag indicates an encoding discontinuity between the media file that follows it and the one that preceded it. The set of characteristics that MAY change is:
number and type of tracks
timestamp sequence
The EXT-X-VERSION tag indicates the compatibility version of the playlist file. The playlist file, its associated media, and its server should, in one embodiment, comply with all provisions of the most-recent version of this document describing the protocol version indicated by the tag value.
#EXT-X-VERSION:<n>
where “n” is an integer indicating the protocol version.
A playlist file, in one embodiment, can contain no more than one EXT-X-VERSION tag. A playlist file that does not contain an EXT-X-VERSION tag should, in one embodiment, comply with version 1 of this protocol. If the playlist file has this tag then its value, in one embodiment, should be the lowest protocol version with which the server, playlist file and associated media files all comply.
Each media file URI in a playlist file identifies a media file that is a segment of the original presentation (i.e., original media content). In one embodiment, each media file is formatted as a MPEG-2 transport stream, a MPEG-2 program stream, or a MPEG-2 audio elementary stream. The format can be specified by specifying a CODEC, and the playlist can specify a format by specifying a CODEC. In one embodiment, all media files in a presentation have the same format; however, multiple formats may be supported in other embodiments. A transport stream file should, in one embodiment, contain a single MPEG-2 program, and there should be a Program Association Table and a Program Map Table at the start of each file. A file that contains video SHOULD have at least one key frame and enough information to completely initialize a video decoder. A media file in a playlist MUST be the continuation of the encoded stream at the end of the media file with the previous sequence number unless it was the first media file to appear in the playlist file or if it is preceded by an EXT-X-DISCONTINUITY tag. Clients SHOULD be prepared to handle multiple tracks of a particular type (e.g. audio or video) by choosing a reasonable subset. Clients should, in one embodiment, ignore private streams inside Transport Streams that they do not recognize. The encoding parameters for samples within a stream inside a media file and between corresponding streams across multiple media files SHOULD remain consistent. However clients SHOULD deal with encoding changes as they are encountered, for example by scaling video content to accommodate a resolution change.
If the encryption method is AES-128, AES-128 CBC encryption, for example, may be applied to individual media files. In one embodiment, the entire file is encrypted. Cipher block chaining is normally not applied across media files in one embodiment. The sequence number of the media files can be used as the IV or the IV can be the value of the IV attribute of the EXT-X-KEY tag as described above. In one embodiment, the server adds an EXT-X-KEY tag with the key URI to the end of the playlist file. The server then encrypts all subsequent media files with that key until a change in encryption configuration is made.
The client device can request a playlist file in operation 370. As discussed above, the playlist file may be retrieved utilizing a URI provided to the client device. In one embodiment, the playlist file includes listings of variant streams of media files to provide the same content at different bit rates; in other words, a single playlist file includes URIs for the media files of each of the variant streams. The example shown in FIG. 3B uses this embodiment. In another embodiment, the variant streams may be represented by multiple distinct playlist files separately provided to the client that each provides the same content at different bit rates, and a variant playlist can provide a URI for each of the distinct playlist files. This allows the client device to select the bit rate based on client conditions.
#EXTM3U #EXT-X-TARGETDURATION:10 #EXTINF:5220, http://media.example.com/entire.ts #EXT-X-ENDLIST
#EXTM3U #EXT-X-TARGETDURATION:8 #EXT-X-MEDIA-SEQUENCE:2680 #EXTINF:8, https://priv.example.com/fileSequence2680.ts #EXTINF:8, https://priv.example.com/fileSequence2681.ts #EXTINF:8, https://priv.example.com/fileSequence2682.ts
#EXTM3U #EXT-X-MEDIA-SEQUENCE:7794 #EXT-X-TARGETDURATION:15 #EXT-X-KEY:METHOD=AES-128,URI=“ https://priv.example.com/key.php?r=52” #EXTINF:15, http://media.example.com/fileSequence7794.ts #EXTINF:15, http://media.example.com/fileSequence7795.ts #EXTINF:15, http://media.example.com/fileSequence7796.ts #EXT-X-KEY:METHOD=AES-128,URI=“ https://priv.example.com/key.php?r=53” #EXTINF:15, http://media.example.com/fileSequence7797.ts
#EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1280000 http://example.com/low.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2560000 http://example.com/mid.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=7680000 http://example.com/hi.m3u8 #EXT-X-STREAM-INF:PROGRAM- ID=1,BANDWIDTH=65000,CODECS=“mp4a.40.5” http://example.com/audio-only.m3u8
#EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000 http://ALPHA.mycompany.com/low/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000 http://BETA.mycompany.com/low/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000 http://ALPHA.mycompany.com/mid/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000 http://BETA.mycompany.com/mid/prog_index.m3u8
#EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1280000 http://example.com/low.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2560000 http://example.com/mid.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2560000 http://example1.com/mid-redundant2.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=7680000 http://example.com/hi.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=7680000 http://example2.com/hi-redudant2.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=7680000 http://example3.com/hi-redudant3.m3u8 #EXT-X-STREAM-INF:PROGRAM- ID=1,BANDWIDTH=65000,CODECS=“mp4a.40.5” http://example.com/audio-only.m3u8
FIGS. 12A and 12B show one embodiment of a server timing model for the transmission of succeeding playlists when additional media files will be added (e.g., when the current playlist being transmitted does not contain an EXT-X-ENDLIST tag). If a current playlist does not contain the final media file of a presentation, then a data processing system or server can make a new version of the playlist that contains at least one new media file URI. FIGS. 12A and 12B show one embodiment of a server timing model for ensuring that the new playlist with the new media file URI will be available for transmission to client devices in a manner continuous with the previous version of the playlist. This model may, for example, be used when media files, specified in the playlist, are allowed to be short in duration (e.g. only a few seconds long). In one embodiment, by setting a maximum media file duration for each media file and by setting a minimum amount of a playlist duration based upon the maximum media file duration, a server or other data processing system can ensure a continuous distribution or transmission of the content to client devices even when each media file is only a few seconds in duration.
Referring now to FIG. 12A, operation 1201 can be used to establish a target duration as a maximum media file duration of each media file in a playlist if an endlist tag is not present in a next playlist file as determined in operation 1200. Operation 1201 can be performed by a data processing system which is dividing a stream of data into multiple media files and storing those multiple media files as individual files. The process of dividing the stream can utilize the target duration (e.g. the target duration of the current playlist file) to ensure that each media file specified in the playlist file is less than the target duration (or is less than the target duration plus or minus a small period of time). The data processing system which generates a playlist can also ensure that the duration of the playlist file can be at least a multiple of the target duration as shown in operation 1203. In one embodiment, the multiple can be three target durations (or some other multiple of the target duration) which is used as a minimum of a playlist duration, wherein the duration of a playlist is defined by the cumulative durations of the media files specified within the playlist. A system (e.g. a server) that generates a playlist can comply with the minimum duration of the playlist by ensuring that each playlist specify at least a sufficient number of media files to satisfy the minimum duration; for example, if the minimum duration is 3 target durations, then each playlist should include at least 3 target durations.
Operation 1205 can also be used as a further mechanism to ensure that a consistent and continuous stream is made available from a data processing system such as a server which is transmitting the media files. This further mechanism can reduce the amount of polling or pulling, by a client device, to determine whether there are changes to the playlist. In operation 1205, a server can be set up such that there is an earliest time and a latest time for the server to transmit the next playlist file. The earliest time and the latest time can be used as a time window that is based on or relative to the time that the previous playlist file (which immediately precedes the new playlist file) was made available. The earliest time can, for example, be based upon a time when an immediately previous playlist was first made available for transmission (but not necessarily have been transmitted) from the server. The latest time can, for example, also be based upon a time when that immediately previous playlist was first made available for transmission from the server (but not necessarily have been transmitted). For example, in one embodiment the earliest time may be specified as a time that is no earlier than a first predetermined percentage (e.g. one-half) of the target duration (e.g. the target duration set in operation 1201) from when the previous playlist file was first made available for transmission, and the latest time can be set to be no later than a second predetermined percentage (e.g. one and a half times) of the target duration from when the immediately previous playlist file was first made available for transmission from the server. The time of when the playlist file was first made available for transmission could be, in one embodiment, the time of creation of the playlist file (that time being recorded by a file system on the server). This example is shown in FIG. 12B which includes a timeline 1211. Target duration 1213 is a portion of the playlist duration 1215 which represents the duration of an immediately previous playlist that was first made available by one or more servers at time 1209 which is the time at which the previous playlist file was first made available for transmission. The media files specified in that playlist can begin their transmission at nearly time 1209. According to the server timing model shown in FIG. 12B, a server should not transmit the next playlist file until the earliest time 1217 which is one-half of a target duration after time 1209, and the server should not make available the next playlist file any later than time 1219 which has been specified to be one and a half target durations after time 1209 in the example shown in FIG. 12B. This server timing model can be used to ensure that playlist files are made available to client devices to provide the client device with enough time to retrieve media files specified in the playlist and to then present those media files consistently and continuously without stalls in the presentation of the content during playback. In one embodiment, these server timing models can be used when the content is a transmission of a live event and a stream of data from the live event is being divided into multiple media files and then those multiple media files are transmitted in near real time relative to the live event to client devices that receive the multiple media files shortly after they were divided out of the stream of data of the live event, such as a baseball game, etc.
FIG. 13 shows an embodiment of a method which may be used to avoid stalls in playback at a client device, particularly when a client device is presenting, in near real-time, a live event and when the client device is presenting content which is near the current end (being the most recent in time) of a live event. For example, if the live event is a baseball game, a user of a client device may prefer to watch only the most recent events in the game rather than beginning to watch the game from the very beginning of the game. If a user desires to watch only the most recent events of a game that is in progress, the user may seek to set playback to start from a point beginning in the last 10 or 15 seconds from the end of the available media stream. Problems or delays in a network can suddenly cause the data to become unavailable and can prevent new data from becoming available, and hence in a very short period of time, the client device can run out of content to present when a user has set a client device to operate in this mode. The method of FIG. 13 can be employed in order to mitigate the chances of this happening by enforcing a rule at a client device that playback is required to start at a start point which is at least a period of time (for example, 30 seconds) before an end of the current playlist file. For example, if a playlist file has 5 media files specified within it (each media file being 10 seconds long), then one implementation of this rule may be to enforce a start point to be no later than the third media file in the sequence of five media files specified in the playlist. Referring now to FIG. 13, operation 1301 can be used to determine whether or not an endlist tag or marker is present in the playlist. If such an endlist tag is present, then the method of FIG. 13 can stop as no new content will be added to the playlist, so there is no need to enforce the rule in operation 1303 in one embodiment. On the other hand, if there is no endlist tag present in the playlist, then a rule can be enforced at a client device which requires a start point to be at least a period of time before an end of the playlist file. The period of time can be specified based upon target durations of the media files. For example, in one embodiment, the client device can be required to start from a media file that is more than three target durations from the end of the playlist file.
Another aspect of the present invention relates to methods which can be used when switching between streams from two playlists (e.g. two variant streams) or other switching between two sets of media files. An example of a method for switching between streams from two different playlists has been provided in conjunction with FIGS. 9A, 9B, 9C, and 9D. In that method, an overlap in time between the two streams can be used to ensure a consistent and continuous playback such that a switch or transition between the streams can be seamless. As shown in FIG. 9D, the overlap 955 represents a period in time in which media content from both streams is stored at a client device and capable of being played back at the client device, thereby allowing a seamless switch between the two streams. In one embodiment, the overlap may be a minimum number which never varies and is set within the client device. While this embodiment can work well, there can be times when the overlap can be unnecessarily too long. In other words, the overlap can prevent a switch or transition from occurring even though a device is ready to make the transition. For example, when switching from a lower resolution to a higher resolution, an unnecessarily long overlap can force the user to watch the lower resolution presentation for a period of time when the higher resolution presentation is already available and ready to be presented. Higher speed connections can, for example, provide the ability to quickly develop an overlap which can be shorter than an overlap required for a lower speed connection or type of connection. In an embodiment according to FIG. 14A, a client device can adapt to the connection speed or connection type and modify the minimum overlap required based upon the connection speed or connection type. For example, if the connection speed or type is fast then the minimum overlap can be reduced relative to a minimum overlap required for a lower connection speed or connection type. As conditions change (e.g. the client device loses a 3G connection and must rely upon a 2G or slower connection), then the minimum overlap can be changed. Hence, the client device can adapt the minimum overlap based upon the connection speed or type. Referring now to FIG. 14A, in operation 1401, a client device can determine a speed of or type of connection. Referring back to FIG. 9D, it can be seen that a second stream of data from a second playlist is a new source of data which is being received while the client device also receives the stream from a first playlist. At this time, the client device can determine a speed of connection or a type of connection in order to determine, in operation 1403, a minimum amount of overlap required based upon the current connection speed or connection type. As conditions change, this minimum overlap can be adapted based upon the changing conditions, such as wireless connections to cellular telephone towers, WiFi basestations, etc. This may be particularly advantageous when the client device is moving through a wireless cellular telephone network or other data network. After establishing that the minimum overlap for the current condition exists, then the client device can, in operation 1405, switch or transition from the stream from the first playlist or the old source to the new source which may be the stream from the second playlist. An example of this transition has been provided in connection with the description associated with FIGS. 9A-9D.
FIGS. 14B, 14C, and 14D show another aspect of how an overlap between two streams (such as the overlap described and shown in conjunction with FIGS. 9A-9D or the overlap described in conjunction with FIG. 14A). The method shown in FIGS. 14B, 14C and 14D may be implemented with an adaptively derived overlap (which was described in conjunction with FIG. 14A) or this method may be used with a fixed overlap which does not change. The method depicted in FIGS. 14B-14D can begin with the downloading of media files from the “old stream” 1410 (e.g. which can be a lower resolution video downloaded at a first speed which is slower in bit rate than a second speed of future downloads for the new stream 1414). The old stream 1410 has been downloaded as indicated by the hash marker 1411 and it is currently being presented, on a client device, to a user at playback point (e.g. playback head position at) 1412; the already downloaded content in old stream 1410 beyond the current playback point 1412 is buffered content that is available should the connection become faulty. The client device can then read a playlist file for the new stream 1414 and determine from the playlist file the content “blocks,” such as blocks 1416 and 1415, before even downloading the content of those blocks; for example, the playlist file for the new stream can indicate, at least approximately, the locations in time of the content blocks 1416 and 1415 relative to old stream 1410. This determination can allow the client device to conservatively decide to download first block 1415 for the new stream 1414 by requesting and retrieving one or more media files for block 1415, and FIG. 14C shows the result of that download (block 1415A has hash marks to show that this block has been downloaded). The playback position has progressed in time to a new location (still within the leftmost block of old stream 1410). In this instance the downloading of block 1415 was fast enough that the playback position did not leave that leftmost block of old stream 1410. Block 1415 was selected conservatively in case the download took longer so that playback could at least be switched around block 1415A. At the point depicted in FIG. 14C, the client device can check how much time is left between the overlap provided by block 1415A and the current point of playback (shown by 1412 in FIG. 14C). If there is enough time given the connection speed, the client device can download the block or segment 1416 which is the block previous to the current overlap, and then the client device can repeat the check to determine how much time is left between the overlap provided by just downloaded block 1416A (shown in FIG. 14D after it has been downloaded as indicated by the hash marks) and the current point of playback (shown by 1412 in FIG. 14D). If, as in the case of the example shown in FIG. 14D, the download of 1416A happens quickly, then the client device can move the point of overlap backward in time, reducing the time it will take to switch between the streams (and hence allowing a switch within block 1416A); on the other hand, if there are delays in downloading 1416A such that the switch cannot occur within block 1416A, then the client device can use block 1415A as an overlap that could be used to cause the switch to occur within block 1415A.
Another aspect of the present invention can utilize an attribute defining a resolution of images. This attribute can allow a client device to decide that it should not switch resolutions or otherwise switch streams based upon the attribute. For example, a client device can decide that it is already playing the maximum resolution which it can display and that there is no point in downloading a higher resolution which may be available to the device through a data network.
FIG. 15 shows an example of a method in one embodiment for utilizing such an attribute. In operation 1501, a playlist file can be received by a client device, and the client device, in operation 1503, can determine from the playlist file that an attribute exists within the playlist file which defines the resolution of images available to the client device. Based upon that attribute, the client device can, in operation 1505, determine whether to retrieve another playlist file or to retrieve a media file associated with that attribute. By providing the resolution attribute, a client device can intelligently decide how to process the data in the playlist. Moreover, the client device can make decisions about the retrieval of data which can prevent unnecessary downloads, and this can, in turn, minimize the amount of data traffic on the network.
An embodiment of the invention can allow a system to search for content based upon a date and time. For example, a user may want to see a home run hit on Apr. 9, 2009 at about 5 PM or may want to see another event on a date and approximate time. An embodiment of the invention can provide this capability by timestamping, through the use of an EXT-X-PROGRAM-DATE-TIME tag that is associated with the beginning of a corresponding media file; the tag can be associated with its corresponding media file by having the tag appear before that media file in a playlist file. A system, such as a server, can store one or more playlists which can be retrieved (e.g., downloaded) by a client device and used to search for a date and time to find a desired media file; alternatively, a client device can request (e.g., through a date and time search request) the server to search through the one or more playlists to identify one or more media files that match the date and time search request, and the server can respond by identifying the one or more media files. In one embodiment, the tag indicates a substantially precise beginning of the media files, and timestamps within the media file can be used to find a playback point with finer granularity in time. For example, a tag's timestamp can indicate the media file began on Apr. 9, 2009 at 5:03 PM, and the timestamps (or other indicators of time) within a media file can specify time in increments of minutes or seconds, etc. after 5:03 PM to allow a device to begin playback (through a selection of a playback start point) at, for example, 5:06 PM or 5:05:30 PM.
FIG. 16A shows a flowchart that depicts a method according to one embodiment for using the timestamped tags to create a playlist file. The method can be performed by a server implemented with processing logic including software, hardware, firmware, or a combination of any of the above. In some examples, the server is provided by a media provider, such as MLB.
At box 1610, processing logic creates timestamped tags and associates each of the timestamped tags with one media file. The timestamp in a timestamped tag indicates a beginning date and time of the associated media file. Details of some embodiments of timestamped tags have been discussed above.
At box 1620, processing logic creates a playlist file with one or more timestamped tags (e.g., EXT-X-PROGRAM-DATE-TIME tag), each of which is associated with a particular media file. Note that the media file itself has internal timestamps as well. At box 1630, processing logic may distribute the playlist so that the playlist file is available for searching by date and time using the date and time in the timestamped tags. In some embodiments, the playlist is stored in a repository, from which client devices may download the playlist.
FIG. 16B shows a flowchart that depicts a method according to one embodiment for using a playlist file created with the timestamped tags. The method can be performed by a client device implemented with processing logic including software, hardware, firmware, or a combination of any of the above. The client device may be used by individual consumers, subscribers, or viewers of the media associated with the playlist file to access and play the media.
At box 1650, processing logic receives a user request for a segment of a program beginning at a particular date and time. For example, the user may request a fourth inning of a baseball game that begins at 8:15 pm on Apr. 6, 2010, instead of the entire baseball game. In response to the user request, processing logic downloads one or more playlist files associated with the program from a media server at block 1652. At block 1654, processing logic searches the playlist files downloaded using the date and time in the timestamp tags inside the playlist files for the date and time stamps closest to the date and time of the segment requested. Then processing logic subtracts its date and time from the date and time of the segment requested at block 1656. This produces a duration. Processing logic then walks forward through the subsequent media file durations in the playlist file until processing logic locates a target media file about that much duration after the datestamped media file at block 1657. Processing logic then downloads this target media file at block 1658, as it is the best guess about which file contains the requested segment.
In some embodiments, all media files between the datestamped one and the target one are part of a single encoding, that is, no discontinuity tag in between them. If they are, processing logic can subtract media file timestamps in the datestamped file from those in the target file to get precise durations, which allows the location of the requested date and time precisely.
Using the dates and times in the timestamped tags in the playlist files, processing logic does not have to download all media files of the entire program in order to search through the media files to find the requested segment. Because the client device does not have to download all media files of the entire program when the user does not request the entire program, significant savings in bandwidth can be achieved. Furthermore, many typical media files contain only arbitrary timestamps, which often start at zero. Thus, the dates and times of the timestamped tags discussed above may associate the arbitrary timestamps in the media files with a real date and/or time. Using the timestamped tags, the client device can locate the playlist element containing a particular date and/or time more efficiently than scanning through each media file.
One embodiment of the invention allows insertion of timed metadata into a media stream in an ID3 format. The media stream may include video and/or audio data encoded in a predetermined format. For example, the media stream may include video and audio data encoded in MPEG-2 developed by the Moving Pictures Expert Group (MPEG), which is international standard ISO/IEC 13818. Broadly speaking, metadata includes information on data in the media stream, and timed metadata referred to metadata associated with a particular time (e.g., the time at which a goal was scored). Note that timed metadata may change over time. The timed metadata may be inserted into the media stream in a predetermined format for storing metadata, such as ID3 format. In some embodiments, the video data may be divided into a sequence of frames. Timed metadata of the video data may also be divided into containers associated with the sequence of frames. Each container may store both timed metadata of a corresponding frame and the time associated with the corresponding frame. Alternatively, each container may store both timed metadata of a corresponding frame and frame number of the corresponding frame. In some embodiments, the timed metadata of a frame may include a set of predetermined information of the frame. For example, the timed metadata may include location information (e.g., global positioning system (GPS) data) of the location at which the corresponding frame of video data was recorded.
In one embodiment, the following describes how ID3 metadata can be carried as timed metadata in MPEG-2 Transport Streams (see ISO/IEC 13818-1:2007 Information Technology—Generic Coding of Moving Pictures and associated audio information: systems which is hereinafter referred to as “the MPEG-2 standard”) as used by the HTTP live streaming protocol described herein. Metadata can be carried in transport streams according to section 2.12 of the MPEG-2 standard. The metadata can be carried in an elementary stream (PES), rather than, for example, in a carousel. ID3 metadata is self-describing and needs no configuration information, so the provisions for metadata decoder configuration data do not need to be used. The metadata stream can be in the same program as the main program material (i.e. the audio/video content).
Tables S1, S2, S3 and S4 provide one embodiment of syntax that can be used. In the syntax tables below, the syntax structure (left column) is shown with only the outline that is in effect, and the names of fields. This means that ‘if’ blocks for which the condition is false are omitted, for clarity. The MPEG-2 standard can be consulted for the complete syntax, the field sizes, and the acceptable values. The right column indicates, in a line with each field name, the value needed in this context, or contains an explanation of that line.
Summary of the Code-Points Used
ID3 defines both a format and a semantic, and so the same registered format_identifier can be used for both metadata_format_identifier and metadata_application_format_identifier. The registered value for these, at the registration authority SMPTE Registration Authority (see http://www.smpte-ra.org), is “ID3” (I D 3 space, or 0x49 0x44 0x33 0x20) (assignment pending). To indicate a registered value is used, the fields metadata_format and metadata_application_format can take the values 0xff and 0xffff respectively. In one embodiment, this metadata can be carried in a private stream, not a stream formatted as metadata Access Units (MAUs) as defined in 12.4 of the MPEG-2 standard. The stream_id value used for the stream is therefore private_stream_id—0, 0xbd, as specified in 2.12.3 of the MPEG-2 standard. The stream_type is set to 0x15, indicating carriage of metadata in a PES stream, as specified in 2.12.9.1 of the MPEG-2 standard. Since only one metadata stream is normally carried, the metadata_service_id is normally set to 0; however, any suitable value can be used to distinguish this metadata stream from others, if needed.
The format and content of the metadata descriptors is documented in sections 2.6.58 to 2.6.61 of the MPEG-2 standard.
Descriptor Loop of the PMT for the Program
To declare the presence of the metadata stream, a metadata_pointer_descriptor (2.6.58 of the MPEG-2 standard) can be placed in the PMT, in the program_info loop for the program. The metadata can be in the same program as the main program (audio/video) content. In one embodiment, the use of this descriptor to refer to another program is not supported.
TABLE S1 Syntax Value
Metadata_pointer_descriptor ( ) { descriptor_tag 0x37 - Metadata_pointer_descriptor tag descriptor_length - the length of the descriptor metadata_application_format 0xFFFF if (metadata_application_format==0xFFFF) { metadata_application_format_identifier ‘ID3 ’ (0x49 0x44 0x33 0x20) } metadata_format 0xFF if (metadata_format==0xFF) { metadata_format_identifier ‘ID3 ’ (0x49 0x44 0x33 0x20) } metadata_service_id - any ID, typically 0 metadata_locator_record_flag 0 MPEG_carriage_flags 0 reserved 0x1f if (MPEG_carriage_flags ==0|1|2){ program_number - program number of the program whose es descriptor } loop contains the metadata_descriptor }
The elementary stream also can be declared in the loop of elementary streams, in the program map (section 2.4.4.8 of the MPEG-2 standard.
TABLE S2 Syntax Value
stream_type 0x15 reserved 0x7 elementary_PID - pid of the elementary stream carrying the metadata reserved 0xf ES_info_length - length of the elementary stream info descriptor loop, including the metadata_descriptor
Descriptor Loop of the PMT for the Elementary Stream
To declare the format of the metadata stream, a metadata descriptor (2.6.60 of the MPEG-2 standard) can be placed in the PMT, in the es_info loop for the elementary stream.
TABLE S3 Syntax Value
Metadata_descriptor ( ) { descriptor_tag 0x38 - Metadata_descriptor tag descriptor_length - the length of the descriptor metadata_application_format 0xFFFF if (metadata_application_format==0xFFFF) { metadata_application_format_identifier ‘ID3 ’ (0x49 0x44 0x33 0x20) } metadata_format 0xFF if (metadata_format==0xFF) { metadata_format_identifier ‘ID3 ’ (0x49 0x44 0x33 0x20) } metadata_service_id - any ID, typically 0 decoder_config_flags 0 DSM-CC_flag 0 reserved 0xf
PES Stream Format
ID3 metadata can be stored as a complete ID3v4 frame in a PES packet, including a complete ID3 header stream. The ID3 tag can start immediately after the PES header; this PES header can contain a PTS (PTS_DTS_flags to ‘10’). The PTS can be on the same timeline as the audio and video frames. The data_alignment bit can be set to 1. The PES header can contain a PES_packet_length that is non-zero. If an ID3 tag is longer than 65535 bytes, it can have more than 1 PES header. The second and following PES headers can have data_alignment set to 0, and the PTS_DTS flags set to ‘00’ (and hence no PTS). The PES header can be formatted as documented in 2.4.3.7 of the MPEG-2 standard.
TABLE S4 Syntax Value
PES_Packet ( ) { packet_start_code_prefix 0x00 0x00 0x01 stream_id 0xbd - private_stream_id_1 PES_packet_length - the length of the packet, which must not be 0 if (...) { - a large test which is true in this case ‘10’ ‘10’ PES_scrambling_control 0 PES_priority 0 data_alignment_indicator 1 for the packet containing start of the ID3 header, else 0 copyright 0 original_or_copy 0 PTS_DTS_flags if (data_alignment==1) ‘10’ else ‘00’ ESCR_flag 0 ES_rate_flag 0 DSM_trick_mode_flag 0 additional_copy_info_flag 0 PES_CRC_flag 0 PES_extension_flag 0 PES_header_data_length - the length of the data; padding may be used } }
The metadata stream can be incorporated into a transport stream in the same way as audio or video is. For example, that means that in a transport_packet( ) (see 2.4.3.2 of the MPEG-2 standard) the payload_unit_start_indicator can be set to 1 only when a PES header follows. (The PES header, in turn, can indicate whether the start of the ID3 data follows, or whether that has been divided into multiple PES packets, as noted in the previous paragraph).
In one embodiment of the invention, processing of media files (e.g., retrieved of playlists and retrieved of media files specified in the playlist and decoding of the content in the media files) can be done separately, from a user interface that presents and controls the media being presented. For example, a user application, such as an application for watching live events (e.g., as Major League Baseball (MLB) application for watching baseball games) or other streams can provide the user interface for presenting and controlling (e.g., receiving a selection of a media file) the presentation while another software process (e.g., a software process that serves media such as a daemon for serving media, which can be referred to as “mediaserverd”) can retrieve playlists and retrieve and decode media files. In some cases, the media files can be encrypted, and the encryption can be controlled by the user application (e.g., the MLB application); for example, a user application can install a client certificate (for example, an X.509 certificate to provide authentication and chain of trust, and revocability) into their keychain (either persistently or in memory only) that can be used to answer a server challenge when an HTTP Secured Sockets Layer (SSL) connection is made to download a key that can be used to decrypt the media's content. In other cases, a playlist can contain URLs for one or more keys that use a custom URL scheme that is used by the user application or a server that interacts with the user application; in this case, a user application can register URL protocol handlers for these custom URL schemes that can be invoked to obtain a key (such as a new key), and this can allow a user application to transport keys out of band (e.g., hidden in their application binary), or obtain a key from a server using a private protocol that is understood by both the user application and the server that interacts with the user application, but is not understood by other systems.
FIG. 17A shows one embodiment of software architecture to allow a media serving daemon to interact with a user application. The architecture includes a media serving daemon (“mediaserverd”) 1710 and an exemplary user application, Event Media Provider (EMP) application 1720, both executable in processes running on a client device, such as, for example, a smart phone, a personal digital assistant, a desktop computer, a laptop computer, a tablet device, etc. One embodiment of the client device may be implemented using electronic system 800 shown in FIG. 8. In some embodiments, both mediaserverd 1710 and EMP application 1720 share the same privileges with respect to memory control, memory space, memory allocation, file system control, and network control. As such, mediaserverd 1710 may access data that EMP application 1720 can access. Likewise, mediaserverd 1710 is prohibited from accessing data that EMP application 1720 cannot access.
In some embodiments, EMP application 1720 further includes a core media stack 1721, which is a customized software stack for accessing a networking stack 1723, which in turns accesses an URL protocol handler, EMP handler 1725. EMP application 1720 can register EMP handler 1725 for a custom URL scheme that can be invoked to obtain one or more keys. Thus, EMP application 1720 can transport keys out of band (e.g., hidden in the application binary).
In general, mediaserverd 1710 and EMP application 1720 can interact with each other to download and playback media files for live streaming content from a content provider's application, which is EMP in the current example. Playback can be done in mediaserverd 1710 on the client device. In some embodiments, mediaserverd 1710 can download keys for decryption of media files, and if this fails, mediaserverd 1710 may ask EMP application 1720 to download the key from a content provider server, which is EMP server 1730 in the current example. EMP application 1720 running on the client device can sign up to get one or more keys. In one embodiment, EMP application 1720 may have signed up and obtained the keys prior to downloading the media files. Details of some embodiments of the interactions between mediaserverd 1710 and EMP application 1720 are discussed below to further illustrate the concept.
Referring to FIG. 17A, EMP application 1720 in one embodiment sends a playlist with at least an URL and a key to mediaserverd 1710 (1). Using the key, mediaserverd 1710 attempts to access a media source provided by EMP at the URL and to download media files specified in the playlist from the media source. The media files may be encoded or encrypted to prevent unauthorized viewing of the content of the media files. If mediaserverd 1710 fails to download the media files, or it fails to decode or decrypt the media files downloaded (2), mediaserverd 1710 reports the failure to EMP application 1720 (3).
In response to the failure report from mediaserverd 1710, EMP application 1720 uses its core media stack 1721 to access networking stack 1723 in order to request a new key (4), which in turns accesses EMP handler 1725 for the new key (5). EMP handler 1725 connects to EMP server 1730 over a network (e.g., Internet) to request the new key from EMP server 1730 (6). In response to the request, EMP server 1730 sends the new key to EMP handler (7). Then EMP handler 1725 passes the new key to core media stack 1721 (8), which then passes the new key to mediaserverd 1710 (9).
When mediaserverd 1710 receives the new key from core media stack 1721, mediaserverd 1710 may try to download the media files again using the new key and then decode the media files downloaded using the new key (10). Alternatively, if the media files were successfully downloaded previously, but mediaserverd 1710 failed to decrypt the media files, then mediaserverd 1710 may try to decrypt the media files previously downloaded using the new key. If mediaserverd 1710 successfully downloads and decodes the media files using the new key, then EMP application 1720 may present the decoded media files on the client device.
FIGS. 17B and 17C show another embodiment in which processing of media files (e.g. the retrieval of playlists and the retrieval of media files identified in the playlist and the decoding of encrypted media files) can be done by a player service separately from a user application (e.g. “AppX’) that presents and controls a user interface that presents content from the processed media files. The separation between the processing of media files and the control of the user interface allows a content provider to create a unique user interface and present that user interface through an application created by or for the content provider, and it also allows the content provider to use custom URLs or custom protocols, that can be hidden or difficult to reverse engineer, in order to protect the content. The custom URLs or custom protocols can be controlled by the content provider's application (e.g. “AppX”) and by the systems (e.g. a server controlled by the content provider or agents of the content provider) that interact with the content provider's application. FIG. 17B shows an example of a software architecture on a client device 1750 such as, for example, a smart phone, a personal digital assistant, a desktop computer, a laptop computer, a tablet device, an entertainment system, or a consumer electronic device, etc. The client device can be, for example, the system shown in FIG. 8. The client device 1750 can interact and communicate through a network 1752 (e.g. the Internet or a telephone network, etc.) with one or more servers 1753. The one or more servers can store and transmit the playlists (e.g. playlist 1754) and the media files referred to in the playlists, and these servers can be controlled by the content provider that provides the application (e.g. AppX) so that the application and the servers are designed to work together using custom protocols to ensure that the content is protected or to provide greater flexibility in controlling the distribution of the content, etc.
Client device 1750 includes an operating system (OS) 1756 that can include a player service 1757 although, in another embodiment the player service can be provided separately from the OS. The OS 1756 can maintain a registry 1755 which can be used to store information that is registered by applications, such as AppX; this information, stored in the registry, can include information that shows a relationship between a custom URL and the application that uses that custom URL so that the player service or the OS can call the application that uses that custom URL in order to obtain an object (e.g. a decryption key) from that custom URL. In other words, the registry allows the OS or the player service to identify an application to call by using a custom URL found in a playlist to look up the associated application (which can be identified by an identifier associated with the application) in the registry. The custom URL can be specified, in one embodiment, by the EXT-X-KEY tag, and the player service can be configured to accept, as parameters of that tag, URLs that are specified as one of: http; https; and registered identifiers (such as identifiers that have been registered in a registry such as registry 1755). Client device 1750 can include one or more user applications, such as AppX 1751 (or, for example, a Major League Baseball (MLB) application or other applications that provide a user interface for streaming content obtained from one or more playlists, such as the playlists described herein). These applications can be provided by the entities that provide the content (e.g. MLB provides the content, the baseball games, that are streamed to a client and presented in the user interface of MLB's application that is executing on the client device), or these applications can be provided by application developers that create user interfaces for players for general use with content created by others.
The client device 1750 and the one or more servers 1753 can operate according to the method shown in FIG. 17C to resolve a custom URL that is not recognized by a player service, such as player service 1757. An application, such as AppX 1751 can, when installed or later, cause a custom URL to be registered in a registry, such as registry 1755, in operation 1761 in FIG. 17C. In one embodiment, the application can, as part of its installation, make a call to OS 1756 to cause its one or more custom URLs to be stored in registry 1755 along with an identifier that associates these one or more custom URLs with the application. After installation, the application can be launched and used by a user, which can occur in operation 1763 when the user makes a selection in the application to cause the application to present a selected HTTP stream. In response to this input, the application in operation 1765 calls (in call 2 of FIG. 17B) player server 1757 to present (e.g. display) the HTTP stream. The player service, in operation 1767, retrieves (call 3) the playlist specified by the user's input and determines that the playlist includes a custom URL that is not recognized or supported by the player service; in the case of FIG. 17B, the custom URL is for a decryption key that is used by the player service to decrypt media files referred to in the playlist; upon determining that the playlist includes a custom URL, the player service calls OS 1756 (call 4) to cause registry 1755 to be examined to determine the application that should be requested to resolve or use the custom URL. In operation 1769, OS 1756 determines that the custom URL in the playlist is to be resolved by AppX 1751 and OS 1756 in turn calls (call 5) AppX 1751 to cause AppX to retrieve the object (in this case a decryption key to be used to decrypt content for presentation by AppX) using the custom URL. In operation 1771, AppX 1751 can receive the call from OS 1756 and in response can determine the URL to use and can call the OS (call 6) to retrieve the object (in this case a decryption key) using the URL determined by AppX 1751. Then in operation 1773, OS 1756 receives the object and passes it to AppX 1751 which in turn passes the object to player service 1757 to allow the player service to use the object to process the playlist or the media files or both (as in operation 1775). In an alternative embodiment of operation 1773, the OS 1756 can pass the object, once received, directly to player service 1757.
FIG. 17D shows an example of a method that can be performed by an application, such as AppX, in order to use a custom URL. This method can be used with the software architecture shown in FIG. 17A. In operation 1780, an application, such as AppX, can cause its one or more custom URL(s) to be registered; this registration can be either at installation of the application or when the application is first launched. After the application has been launched, it can receive a user input such as a selection of an HTTP stream (in operation 1781), and in response, the application can call (e.g. through an API) a player service to cause the presentation of the HTTP stream. If the playlist for that HTTP stream includes a custom URL that is not recognized or cannot be processed by a player service, then the application can receive a call (in operation 1783), from either the player service or the operating system, to cause the application to resolve the custom URL (e.g. appx://appx.com/key) registered by the application. In operation 1784, the application can then resolve the custom URL in response to the call in operation 1783; the resolution can involve a predetermined, proprietary (to the application) scheme that can involve determining a legitimate URL based on, for example, the application's privileges (e.g. level of protections or cost of the application), the content sought, date and time of request to view the content, etc. The application can, after resolving the custom URL, call the OS or a network stack to request the object (e.g. decryption key) represented by the custom URL from a remote server that is coupled to a network (e.g. the Internet). The OS or network stack can obtain the object (by, for example, sending the resolved URL to the server which responds with the object), and cause the object to be passed to the player service (either directly or through the application); if the object is passed directly to the player service, then operation 1785 can be performed by components other than the application.
FIG. 17E shows an example of a method that can be performed by one or both of a player service and an OS on a device in order to use a custom URL. This method can be used with the software architecture shown in FIG. 17A. In operation 1790, a player service can receive a call from an application (e.g. AppX) to present an HTTP stream, and in response in operation 1791, the player service can, using data specified in the request, retrieve a playlist as described in this application. Then, in operation 1792, the player service can determine that the playlist includes a custom URL and can then, in operation 1793, call the OS to cause the OS to examine a registry, such as registry 1755, to determine if the custom URL has been registered; alternatively, the player service could itself call a service to examine the registry. The result of examining the registry can determine the application that registered the custom URL (or can determine that there is no such registration, in which case the player service can present an error message that the HTTP stream is not available). If the application is identified in the registry for the custom URL, then, in operation 1794, the player service or the OS calls the application and passes the custom URL to the application to cause the application to resolve the custom URL. After the application resolves the custom URL, it causes the object (e.g. a decryption key) to be obtained, and, in operation 1795, the player service receives the object (either through the application or through a network stack or OS component). After receiving the object, the player service can, in operation 1796, process the media file referred to in the playlist by, for example, retrieving each of the media files and decoding them using the object and presenting them.
Another embodiment of the present invention can employ a distributed system which employs at least two devices to provide playback using the playlists and streaming techniques described herein. FIGS. 20, 21, 22A, and 22B provide an example of such a distributed system in which one device can initiate a remote playback on another device which can drive or provide an output to a TV or other presentation device such as a stereo system, etc. Referring to FIG. 20, a first device 2001 can be configured to communicate with a second device 2003 through, for example, a wired network or wireless network or a direct connection, such as a wired connection. In one embodiment, the first device can be a tablet computer or a laptop computer or a smart phone, and the second device can be a set top box or audio/visual controller, such as the Apple TV from Apple Inc. of Cupertino, Calif. The first device, in one embodiment, can be an iPad from Apple Inc. of Cupertino, Calif. Each of the devices 2001 and 2003 can be an electronic system, such as the electronic system described in conjunction with FIG. 8 and can include one or more processors and memory and peripheral devices, such as input or output or input/output interfaces. In one embodiment, each of the devices 2001 and 2003 can include a wireless network interface which allows the devices to participate in a WiFi wireless network. In one embodiment, the connection 2005 can be a connection to a WiFi network, and in other embodiments, other wireless connections or wired connections may be utilized to couple together the two devices 2001 and 2003. The device 2003 includes an output to a presentation device, such as a TV or stereo system, and this output provides the remote presentation of the media which can be initiated by selecting the media through an app or application, such as app 2007 on the device 2001. The presentation device can be separate from the device 2003 when the device 2003 is a set top box, and in an alternative embodiment, the presentation device (e.g. a TV) can include device 2003.
Device 2001 can include a plurality of software components including, for example, application or app 2007 which may be, for example, an application such as the Major League Baseball app used to present baseball games on the device 2001 or a Netflix application or app used to present movies or other content through the streaming techniques described herein. Device 2001 also includes a media server 2009 which can be similar to the player service 1757 shown in FIG. 17B or the media server daemon 1710 shown in FIG. 17A. The device 2001 can also include a remote playback service 2011 which is a software component that provides an interface between the first and the second device for allowing remote playback of content selected or initiated at the first device through the second device which then outputs the content to a presentation device, such as a TV, which can be coupled to the output of the second device 2003. The remote playback service can be activated or selected by a user of the first device to cause it to communicate with the second device to cause remote playback through the second device onto the presentation device, such as a TV, which is coupled to the output of the second device. In one embodiment, the remote playback service can be part of the AirPlay facility provided on iPads or iPhones from Apple Inc. The device 2001 can also include an operating system 2013 which provides the standard and known services of an operating system to allow the device to function with one or more applications as is known in the art.
The second device can also include a media server 2017 which can be similar to media server or player service 1757 or be similar to the media server daemon 1710 in FIG. 17A. When a remote playback mode is selected, in one embodiment, the media server 2017 on the second device retrieves and processes playlists, specified by the first device, for display to an output of the second device. In this case, the first device acts to control (e.g. start, stop, pause, fast forward, select different media, etc.) the playback of the second device which is retrieving and processing the media in the playlist and is also processing the playlist as described herein. In effect, the first device can act, in this remote playback mode, as a remote control for the second device and can present (e.g. display) a user interface, on a display coupled to the first device, for controlling the playback on the second device; an example of such a user interface is described in U.S. Provisional Application No. 61/378,893, filed on Aug. 31, 2010 (Docket No. P7437Z10), which provisional application is incorporated herein by reference. In one embodiment, even in this remote playback mode, the first device can execute multiple applications concurrently (e.g. a web browser application, an email application, a book reader application, and other applications) while also executing an application or other software to act as a remote control for the second device. Further, the second device 2003 also includes an operating system 2019 and a remote playback daemon 2015. In one embodiment, the remote playback daemon 2015 is a daemon which is a background process which is launched at startup of the second device and which is configured to watch for certain commands such as a “play” URL command or a “play” URL call and then which is configured to provide that call to a media server, such as the media server 2017. The daemon can operate as a background process and launch itself at startup without requiring any interaction from the user of the second device 2003. It will also be understood that the app 2007 can be similar to the app 1751 of FIG. 17B.
In an alternative embodiment, the media server and the remote playback portions of each device may be combined together so that there is no need for an exposed API call between the combined objects.
FIG. 21 shows an example of how the software architecture shown in FIG. 20 can be used with a series of calls and returns to provide for the use of custom URLs in the context of the distributed system shown in FIG. 20 in which playback can be initiated from a first device and can be provided remotely by the second device to an output of the second device to a presentation device, such as a TV or stereo system or a combination of a TV and a stereo system. The calls and the returns of the calls are shown as numbered circles. For example, the app in first device 2001 makes a call to play a URL to a media server, and this call is labeled as “1” within the circle shown in FIG. 21. A further discussion of the interaction of the software components and the calls between those components will now be provided in conjunction with the method shown in FIGS. 22A and 22B.
Referring now to FIGS. 22A and 22B, a method can begin in operation 2025 in which a connection is established between two devices, such as two wireless networking devices, and one of the devices can be set up to allow it to play and control media on or through the second device. In one embodiment, the first device can be a tablet computer or laptop computer or other data processing system which includes an application, such as app 2007, which a user can use to select content to be presented remotely through the second device or on the second device. In one embodiment, the first device may include an output such as a display to also display the same media while it is being presented through the second device on a presentation device such as a TV or another display of the second device. In one embodiment, a user of the first device can select a remote playback mode, such as an AirPlay mode on an iPad and, in doing so, set up the first device to allow it to control and play media on the second device; similarly, this setup can also establish the connection between the devices. It will be appreciated that the sequence of operations 2025 and 2027 can be reversed and that the connection can be established after selecting the media for playback after the application has been launched. In operation 2027, the system launches an application either at user request or automatically by the first device and receives a request to play back a video or other media. For example, a user could launch a Netflix application on an iPad and then select, within the Netflix application, a particular video or other content to play back. In response to the selection of a particular video or other media to play back, the app on the first device, in operation 2029, calls the media server on the first device to play a URL for a playlist. This is shown as call 1 in FIG. 21. In response, the media server on the first device determines whether the remote playback mode has been selected or enabled. If it has not, then playback occurs through the first device in the manner described in FIGS. 17A-17E if a custom URL needs to be resolved. On the other hand, if remote playback mode has been selected or enabled, then the media server on the first device calls the remote playback service, through call 2 on FIG. 21. This call, in effect, passes the URL, which may be a URL for the playlist or a URL for a key in the playlist, to a remote playback daemon 2015 on the second device. This call is shown as call 3 in FIG. 21. In operation 2031, the remote playback daemon, such as remote playback daemon 2015, receives the URL from the first device and calls the media server on the second device to play the URL; this call is shown as call 4 in FIG. 21. In response, media server 2017 on the second device determines that the URL is a custom URL that the media server 2017 cannot process. In one embodiment, a custom URL is any URL that is not a standard URL scheme such as an I-ITTP scheme or an HTTPS scheme or a file scheme, or a DAV scheme. The custom URL scheme will be understood by the application in terms of how to decode or process the custom URL to retrieve the data or content pointed to by the URL.
Then in operation 2035, the media server on the second device, such as media server 2017, calls the remote playback daemon, such as remote playback daemon 2015, on the second device to request resolution of the custom URL. In response, the remote playback daemon calls the remote playback service, through call 6 shown in FIG. 21. Then in operation 2039, the remote playback service, such as remote playback service 2011 on the first device, receives the request to resolve the custom URL and calls, in call 7, the media server, such as media server 2009 on the first device to resolve the custom URL. In turn, in operation 2041, the media server 2009 receives the custom URL and calls the app, such as app 2007 (after, in one embodiment, consulting the registry to determine the appropriate app to call), in order to resolve the custom URL. The call to the app is shown as call 8 in FIG. 21. Next, in operation 2043, the app resolves the URL and returns the content of the custom URL, such as a key or a playlist or other content pointed to by the custom URL, and this content is returned to the media server on the first device and is shown as return 9 in FIG. 21. Then in operation 2045, the media server on the first device returns the content of the custom URL to the remote playback service, such as remote playback service 2011, which in turn returns the content of the URL to the remote playback daemon on the second device which then returns the content of the URL to the media server on the second device. At this point, in operation 2049, the media server on the second device can use the content of the URL, which can be a key or a playlist, to process the playback process, and this process can repeat as necessary whenever the media server 2017 encounters a custom URL. In which case, when encountering a custom URL, it will repeat the process by beginning with the call 5 shown in FIG. 21 and begin with operation 2035 in FIG. 22A which will continue through the resolution of the custom URL call, ending again with operation 2047.
It will be appreciated that in operation 2047, if the content of the URL is a key, then the key will be used by the media server on the second device to decrypt the media file and to present the media file through the output of the second device, and if the content is the playlist, the media server on the second device will use the playlist as described in the rest of this disclosure to retrieve and then decode media files specified by the playlist and to present the media file through the output of the second device. If keys within that playlist are also specified by custom URLs, then the process will again be repeated, beginning in operation 2035 and ending in operation 2047 as described above. When the first device is in the remote playback mode, the media server 2017 on the second device retrieves and processes the playlist that can be specified by the first device, and the media server 2017 retrieves and processes and decodes and presents the media files in the playlist, resulting in a presentation of media on an output of the second device.
FIG. 18 is a block diagram illustrating an exemplary API architecture, which may be used in some embodiments of the invention. As shown in FIG. 18, the API architecture 1800 includes the API-implementing component 1810 (e.g., an operating system, a library, a device driver, an API, an application program, software or other module) that implements the API 1820. The API 1820 specifies one or more functions, methods, classes, objects, protocols, data structures, formats and/or other features of the API-implementing component that may be used by the API-calling component 1830. The API 1820 can specify at least one calling convention that specifies how a function in the API-implementing component receives parameters from the API-calling component and how the function returns a result to the API-calling component. The API-calling component 1830 (e.g., an operating system, a library, a device driver, an API, an application program, software or other module), makes API calls through the API 1820 to access and use the features of the API-implementing component 1810 that are specified by the API 1820. The API-implementing component 1810 may return a value through the API 1820 to the API-calling component 1830 in response to an API call.
It will be appreciated that the API-implementing component 1810 may include additional functions, methods, classes, data structures, and/or other features that are not specified through the API 1820 and are not available to the API-calling component 1830. It should be understood that the API-calling component 1830 may be on the same system as the API-implementing component 1810 or may be located remotely and accesses the API-implementing component 1810 using the API 1820 over a network. While FIG. 18 illustrates a single API-calling component 1830 interacting with the API 1820, it should be understood that other API-calling components, which may be written in different languages (or the same language) than the API-calling component 1830, may use the API 1820.
The API-implementing component 1810, the API 1820, and the API-calling component 1830 may be stored in a machine-readable non-transitory storage medium, which includes any mechanism for storing information in a form readable by a machine (e.g., a computer or other data processing system). For example, a machine-readable medium includes magnetic disks, optical disks, random access memory; read only memory, flash memory devices, etc.
In FIG. 19 (“Software Stack”), an exemplary embodiment, applications can make calls to Services 1 or 2 using several Service APIs and to Operating System (OS) using several OS APIs. Services 1 and 2 can make calls to OS using several OS APIs.
1. A machine readable non-transitory storage medium storing executable instructions that when executed by a data processing system cause the system to perform a method comprising:
executing a user application on a client device to present media files and to control presentation of the media files; and
running a media serving process on the client device, separate from the user application, to retrieve a playlist specifying the media files and a media source at which the media files are available, to retrieve the media files from the media source, and to decode the media files retrieved and to provide decoded content from the media files to the user application.
2. The medium as in claim 1, wherein the media serving process and the user application share the same privileges with respect to memory control, memory space, memory allocation, filesystem control, and network control.
3. The medium as in claim 2 wherein the user application provides a user interface to control the presentation and communicates with the media serving process through an Application Programming Interface (API) and wherein the user application and the media serving process are different software processes.
4. The medium as in claim 2 wherein the media serving process retrieves media files which are decoded using a key processed by or retrieved and processed by the user application.
5. The medium as in claim 2 wherein the user application installs a client certificate which is used to answer a server challenge when a connection is made to download a decryption key.
6. The medium as in claim 2 wherein the playlist contains URLs for decryption keys that use a custom URL scheme.
7. The medium as in claim 3 wherein the media serving process calls, through the API, the user application when the media serving process fails to load or decode a media file so that the user application retrieves one or more keys which are returned to the media serving process.
8. The medium as in claim 3 wherein the media serving process retrieves media files which are decoded using a key processed by or retrieved and processed by the user application.
9. The medium as in claim 8 wherein the user application installs a client certificate which is used to answer a server challenge when a connection is made to download a decryption key.
10. The medium as in claim 9 wherein the playlist contains URLs for decryption keys that use a custom URL scheme.
11. A machine implemented method performed by a data processing system, the method comprising:
12. The method as in claim 11, wherein the media serving process and the user application share the same privileges with respect to memory control, memory space, memory allocation, filesystem control, and network control.
13. The method as in claim 12 wherein the user application and the media serving process communicate through an API and wherein the user application and the media serving process are different software processes.
14. The method as in claim 12 wherein the media serving process retrieves media files which are decoded using a key processed by or retrieved and processed by the user application.
15. The method as in claim 12 wherein the user application installs a client certificate which is used to answer a server challenge when a connection is made to download a decryption key.
16. The method as in claim 12 wherein the playlist contains URLs for decryption keys that use a custom URL scheme.
17. The method as in claim 13 wherein the media serving process retrieves media files which are decoded using a key processed by or retrieved and processed by the user application, and wherein the user application installs a client certificate which is used to answer a server challenge when a connection is made to download a decryption key and wherein the playlist contains URLs for decryption keys that are retrieved through a custom URL scheme.
means for executing a user application on a client device to present media files and to control presentation of the media files; and
means for running a media serving process on the client device, separate from the user application, to retrieve a playlist specifying the media files and a media source at which the media files are available, to retrieve the media files from the media source, and to decode the media files retrieved and to provide decoded content from the media files to the user application.
19. The system as in claim 18, wherein the media serving process and the user application share the same privileges with respect to memory control, memory space, memory allocation, filesystem control, and network control.
20. A machine readable non-transitory storage medium storing executable instructions that when executed by a data processing system cause the system to perform a method comprising:
running a media server process on the client device, separate from the user application, to retrieve a playlist specifying the media files and a media source at which the media files are available, to retrieve the media files from the media source, and to decode the media files retrieved;
receiving, by the media server, a URL in the playlist, which URL refers to data to be used by the media server to decode at least one of the media files;
calling, by the media server, the user application to process the URL to obtain the data to be used by the media server;
receiving the data in response to the user application processing the URL to obtain the data;
decoding at least one of the media files using the data.
21. The medium as in claim 20 wherein the data is a decryption key.
22. A user application stored on a machine readable non-transitory medium that performs those portions of claim 20 performed by the user application.
23. A media server stored on a machine readable non-transitory medium that performs those portions of claim 20 performed by the media server process.
24. The medium as in claim 21 wherein the user application uses a custom URL that is used to protect content provided through the user application.
25. The medium as in claim 24 wherein a registry stores a relationship between the custom URL and the user application, and wherein the media server checks the registry to call the user application when the media server is unable to decode a media file.
26. The medium as in claim 25 wherein the custom URL is specified by an EXT-X-KEY tag.
27. The medium as in claim 25 wherein the user application is authorized by a provider of the media files.
28. The medium as in claim 24 wherein the user application installs, into a registry, a relationship between a custom URL, which is used to retrieve one or more decryption keys for the media files, and the user application, and wherein the user application installs the relationship into the registry when the user application is first installed or first launched.
29. The medium as in claim 28 wherein the relationship points the media server to the user application which uses the custom URL.
30. The medium as in claim 24 wherein the custom URL is not supported by the media server.
31. The medium as in claim 24 wherein the custom URL is used to retrieve the decryption key which is provided to the media server to decode the media files in the playlist.
32. The medium as in claim 31 wherein a resolution of the custom URL by the user application depends upon at least one of (a) the user application's level of privilege relative to the content in the media files; (b) the content in the media files; and (c) a date or time or both of the request to present the content in the media files.
33. The medium as in claim 32 wherein the decryption key is returned to the media server through the user application.
34. A machine readable non-transitory storage medium storing executable instructions that when executed by a data processing system cause the system to perform a method comprising:
executing a user application on a client device, the user application being configured to control presentation of the media files; and
receiving, by the media server, a URL in or for the playlist, which URL refers to data to be used to process at least one of the media files;
calling, by the media server, the user application to process the URL to obtain the data to be used by a remote media server;
receiving the data in response to the user application processing the URL to obtain the data; and
transmitting the data to the remote media server.
35. The medium as in claim 34 wherein the URL is a custom URL for one of the playlist or a decryption key for at least one media file in the playlist and wherein the user application uses the custom URL to protect content provided through the user application.
36. The medium as in claim 34 wherein a registry stores a relationship between the custom URL and the user application, and wherein the media server examines the registry to call the user application when the media server is unable to process the URL.
37. The medium as in claim 36 wherein the remote media server is part of a set top box and is configured to retrieve and process a playlist and is configured to produce a message, which is transmitted back to the media server, to request a resolution of the custom URL and wherein the transmitting of the data to the remote media server provides the resolution of the custom URL.
38. A device for providing an output to a presentation device, the device comprising:
a remote playback component configured to allow the device to be set up for control by another device;
a media server component, the media server component being configured to retrieve and process a playlist specifying media files and to retrieve and decode media files specified in the playlist, and the media server component being configured to produce a message to resolve a custom URL which either represents the playlist or data within the playlist, and the remote playback component being configured to pass the message to the another device;
an output to provide video data or audio data or both video data and audio data to the presentation device, the output being coupled to the media server component which provides the video data or audio data or both;
an input/output (I/O) interface coupled to the remote playback component, the I/O interface being configured to pass the message, from the media server component, to resolve a custom URL and the I/O interface being configured to pass a resolution of the custom URL to the media server component to allow the output to provide the video data or audio data or both and wherein an application on the another device resolves the custom URL and provides the resolution of the URL through the I/O interface.
Publication number: 20110252118
Patent Grant number: 8892691
Inventors: Roger Pantos (Cupertino, CA), Alan Tseng (Baltimore, MD), William May, JR. (Sunnyvale, CA), James David Batson (Sunnyvale, CA)
Application Number: 13/082,324
Current U.S. Class: Accessing A Remote Server (709/219); Application Program Interface (api) (719/328)
International Classification: G06F 15/16 (20060101); G06F 9/44 (20060101);