Source: https://patents.google.com/patent/US8521709B2/en
Timestamp: 2019-09-15 06:04:25
Document Index: 539609030

Matched Legal Cases: ['§119', 'art 100', 'art 100', 'art 100', 'art 100', 'art 300', 'art 310', 'art 300', 'art 300', 'art 300', 'art 300', 'art 310', 'art 300', 'art 300', 'art 300', 'art 300', 'art 400', 'art 410', 'art 400', 'art 400', 'art 400', 'art 400', 'art 500', 'art 510', 'art 500', 'art 500', 'art 500', 'art 500', 'art 500']

US8521709B2 - Methods for preloading media assets - Google Patents
Methods for preloading media assets Download PDF
US8521709B2
US8521709B2 US11/925,460 US92546007A US8521709B2 US 8521709 B2 US8521709 B2 US 8521709B2 US 92546007 A US92546007 A US 92546007A US 8521709 B2 US8521709 B2 US 8521709B2
US11/925,460
US20080104121A1 (en
2006-10-31 Priority to US85593806P priority Critical
2007-10-26 Application filed by Jellyvision Lab Inc filed Critical Jellyvision Lab Inc
2007-10-26 Priority to US11/925,460 priority patent/US8521709B2/en
2007-10-30 Assigned to THE JELLYVISION LAB, INC. reassignment THE JELLYVISION LAB, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: FRAM, STEVEN, GOTTLIEB, HARRY N., JACOVER, EVAN
2008-05-01 Publication of US20080104121A1 publication Critical patent/US20080104121A1/en
2013-08-27 Publication of US8521709B2 publication Critical patent/US8521709B2/en
230000036316 preload Effects 0 Abstract Claims Description 33
A method for preloading media assets during playback execution of a multimedia experience is disclosed. The method includes providing a default preloading scheme. The method also includes receiving a custom instruction to preload at least one media asset associated with a multimedia experience and, in response to the custom instruction, loading the at least one media asset so as to override the default preloading scheme. The method further includes playing the least one media asset after loading the at least one media asset.
The present patent document claims the benefit of the filing date under 35 U.S.C. §119(e) of Provisional U.S. Patent Application Ser. No. 60/855,938, filed Oct. 31, 2006, which is hereby incorporated by reference.
Multimedia technologies have become more and more ubiquitous. A multimedia experience refers to the use of media in an interactive environment. The media generally include one or more types of information content, including for example, text, audio, graphics, animation, and video. During a multimedia experience, the media are presented to an end-user according to a logical sequence that can be affected by the end-user's actions.
In order to avoid playback interruptions during a multimedia experience, it is known to provide algorithms to determine which media assets to request in advance so that they will be ready for playback if they are needed. The retrieval of data in advance (while other playback activity takes place) is referred to as “preloading.” In general, preloading attempts to ensure that media playback proceeds smoothly and without interruption when media resources are not immediately available. A preloading scheme is, by definition, a set of algorithms that implement rules for determining what media assets to request and when to request them.
In some environments, such as the Macromedia Flash® authoring environment, a multimedia experience proceeds along one or more linear timeliness For linear multimedia experiences, a preloading scheme can anticipate with certainty before the start of the multimedia experience which media will be played back, and in what order. In a multimedia experience that contains conditional branching, i.e., a non-linear multimedia experience, making such advance predictions are difficult. Additionally, often only a part of the total amount of media referenced by a non-linear multimedia experience will be played back during any one playback session. Thus, to request a transfer of all media referenced by the multimedia experience would be inefficient and, in scenarios where bandwidth expenses exist, unnecessarily costly. In order to avoid playback interruptions, a default preloading scheme (as described further below) can be implemented that seeks to strike a balance between requesting too much media that is never ultimately played and requesting enough media to ensure that what actually gets played is available the moment it is needed.
This patent discloses tools, methods and systems for preloading media assets as part of an interactive multimedia experience. The tools, methods and systems include providing a default preloading scheme. The tools, methods and systems also include receiving a custom instruction to preload at least one media asset associated with a multimedia experience and, in response to the custom instruction, loading the at least one media asset so as to override the default preloading scheme. The tools, methods and systems further include playing the least one media asset after loading the at least one media asset.
FIG. 1 is an illustration of a flowchart representing a sample interactive multimedia experience using a known default preloading scheme;
FIG. 4 is an illustration of a flowchart representing a sample interactive multimedia experience and a “Properties Window” for displaying and editing a preloading scheme according to one embodiment of the present invention;
Turning now to the drawings, FIG. 1 is an illustration of a flowchart 100 representing a sample interactive multimedia experience using a default preloading scheme. The flowchart 100 comprises a plurality of cells (cells 110-150), some of which are rectangular-shaped and others of which are diamond-shaped. The rectangular-shaped cells are referred to herein as “simple cells,” and the diamond-shaped cells are referred to herein as “conditional cells.” A simple cell is a cell that contains a single branching link to a single cell. For example, in the flowchart 100 shown in FIG. 1, simple cell 110 branches to conditional cell 101 and simple cell 120 branches to simple cell 130. Unlike a simple cell, a conditional cell contains multiple branching links to multiple cells. The branching links are associated with conditions necessary for that branching link to be followed. For example, in the flowchart 100 shown in FIG. 1, conditional cell 101 branches to simple cells 120 and 122. The branch to simple cell 122 is followed if the condition following the decision implemented by conditional cell 101 is true, while the branch to simple cell 120 is followed if the condition is false.
The cells in a flowchart can contain any suitable content. For example, as in a traditional flowchart drawn on paper, a cell can contain text. A cell can also contain instructions that are implemented by the application (or by another application) when the cell is “played.” For example, a cell can contain instructions to trigger a piece of media, gather user input, generate visual or audio information, send/receive information or media to a database, process data, perform a calculation, or perform other functions, such as describing how and when media should be played. Examples of media assets include, but are not limited to, digital audio/video objects representing still art, animation, full-motion video, graphics, on-screen text, music, sound effects, or voiceover dialog. Different cells can have different durations. For example, some cells can be played after a previous cell initiates its function, while other cells can be played after a previous cell finishes its function. (The timing of a cell can also be based on a time X before or after the beginning or end of a cell, and one cell can be played simultaneously with another cell. Timing can also be based off of a common clock. Other timing mechanisms can be used.) A cell can also provide selection choices to a user and evaluate which choice was selected. For example, when conditional cell 101 in FIG. 1 is played, the user is prompted to make a decision and input a response using a user interface device, such as a keyboard, mouse, microphone, remote control, or any other type of device. Conditional cell 101 also determines whether the user input results in “true” or “false” condition.
D represents the total size in K of the downstream media to preload for the two levels of cells beneath L; and
C represents the total size in K of the media at L and the smallest of the media one level beneath L
D>C×100/17,
D>5.88×C.
As a result, a lag could occur under the conditions described above if the size D of the downstream media is greater than about six times the total size C of the media at L and the smallest of the media one level beneath L.
The following table displays possible scenarios with different media durations for the cells of the flowchart shown in FIG. 1 corresponding to whether or not a lag would occur:
cell 110 cell 120 cell 122 cell 130 cell 132 cell 134 Playback
Senario (sec.) (sec.) (sec.) (sec.) (sec.) (sec.) C × 5.88 D lag?
1 10 8 12 10 9 13 105.84 52 FALSE
2 150 150 160 170 180 170 1764 830 FALSE
3 10 10 10 10 10 700 117.6 740 TRUE
4 10 5 10 30 30 30 88.2 105 TRUE
Method 1: Manual Override
According to one embodiment of the present invention, a multimedia experience can override the default preloading scheme, or provide its own preloading scheme in the absence of a default preloading scheme, by signaling the application and/or runtime engine to preload individual target media assets once playback reaches a specific location L in the multimedia experience. For example, FIG. 3 illustrates a flowchart 300 representing a sample interactive multimedia experience. In this example, the application uses a default preloading scheme that normally requests media assets two levels downstream from a currently playing node in the flowchart 310. Since the flowchart 300 contains cells that do not reference media, the default preloading scheme only pays attention to downstream cells that reference media assets. Intervening non-media cells are ignored when determining which downstream cells to preload. Thus, according to the default preloading scheme, the runtime engine would automatically preload media assets referenced by cells 320, 322, 330, 332 and 334 when playback execution reaches cell 310.
According to the present embodiment, the designer of the multimedia experience can program cells in the flowchart 300 with instructions to preload media assets referenced by individual target cells in the flowchart 300 regardless of the location of the target cells. This is referred to as a “manual override” of the default preloading scheme. As shown in FIG. 3, for example, the cell 310 is programmed with instructions to preload media assets for the target cell 348 when execution reaches the cell 310 even though the target cell 348 is more than two levels downstream from the cell 310.
When the designer selects the cell 310, the application allows a user to edit the content of the cell 310, including instructions to preload media assets referenced by target cells. For example, the application can display a graphical view of the flowchart 300 in a first display region and a textual and/or graphical view of the content of the cell 310 in a second display region. The second display region receives user input for editing the content of the cell 310. As used herein, the term “display region” refers to an area of display on one or more display devices (e.g., computer monitors). The first and second display regions can be fixed or movable and can be non-overlapping or can overlap each other. Additionally, the first and second display regions can be alternately shown (one than the other) in response to a command from the application and/or in response to a command from the user.
In the present embodiment, the application displays a “Properties Window” 330 in the second display region when the designer selects the cell 310 as shown in FIG. 4. The “Properties Window” 330 displays the content of the cell 310, including preloading instructions, and receives user input for editing the content of the cell 310. The “Properties Window” 330 may display preloading instructions separately or jointly with other cell content. In this example, the designer has specified that the cell 310 will request the media assets referenced by the target cell 348 for preloading. The designer can specify the target cell 348, for example, by typing in the cell number using a keyboard or by speaking the number of the cell into a microphone or by selecting the target cell from a list of cells in the flowchart 310.
It will be recognized that other means can be used to program selected cells in the flowchart 300 with instructions to override the default preloading scheme. For example, a designer may edit the content of selected cells in a second display region referred to as an “Editor Pane,” which is arranged alongside the first display region and displays a textual view of the content of selected cells. Alternatively, the cells themselves may display content and receive user input to allow a designer to edit the content of a selected cell in-place within the selected cell. It is also possible to provide a third display region for displaying and editing the preloading instructions for all of the cells in the flowchart 300 at once. Likewise, a configuration file may be provided that stores the preloading instructions separately from the multimedia experience represented by the flowchart 300, and the user may edit the configuration file to program cells in the flowchart 300 with preloading instructions.
Method 2: Cell-based Customization
According to another embodiment of the present invention, a multimedia experience can override the default preloading scheme, or provide its own preloading scheme in the absence of a default preloading scheme, by signaling the application and/or runtime engine to preload all media assets falling within a programmable distance V from a specific location L in the multimedia experience. For example, FIG. 5 illustrates a flowchart 400 representing a sample interactive multimedia experience. In this example, the application uses a default preloading scheme that normally requests media assets two levels downstream from a currently playing node in the flowchart 410. Thus, according to the default preloading scheme, the runtime engine would automatically preload media assets referenced by cells 420, 422, 430, 432 and 434 when playback execution reaches cell 310.
According to the present embodiment, the designer of the multimedia experience can program a selected cell in the flowchart 400 with instructions to preload media assets referenced by target cells in the flowchart 400 falling within a distance V from the selected cell. This is referred to as “cell-based” customization. As shown in FIG. 5, for example, the cell 410 is programmed with instructions to preload all media assets for target cells 440-450 falling three levels downstream from the cell 410. A designer can select and edit the content of the cell 410 to program the cell 410 with instructions to override the default preloading scheme as described above. During playback, the multimedia experience will signal the application and/or runtime engine to preload media assets referenced by the target cell 440-450 when execution reaches the cell 410. In this way, the designer can override the default number of levels of cells for preloading specified by the default preloading scheme. Once execution proceeds past the cell 410, the preloading scheme reverts to the default preloading scheme.
A designer may find “cell-based” customization to be particularly useful when the media assets downstream from the cell L are of sufficiently short duration. In that case, some of the media assets downstream from the cell L beyond the number of levels specified by default preloading scheme may not be fully retrieved before those media assets should begin playback. For example, in the flowchart 400, if the media assets referenced by the cells 420-434 are sufficiently short in duration, then the media assets of cells 440-450 may not be fully retrieved before they should begin playback. Thus, the default preloading scheme may not preload a sufficient number of levels of cells to ensure smooth playback. “Cell-based” customization permits the designer to signal the runtime engine to preload more levels of downstream cells than the default preloading scheme, but only while execution is at cell L.
In addition, “cell-based” customization may also be desirable when the media assets downstream from a particular cell L are sufficiently large in size. In that case, when execution reaches one of the child cells of cell L, there will be sufficient time to load the media assets for any grandchild cells during playback of the child cell. For example, if the media assets referenced by the cells 420-450 in the flowchart 400 are sufficiently large, then there will be sufficient time to load the media assets for cell 432 when execution reaches cell 422. “Cell-based” customization allows the designer to signal the application and/or runtime engine to preload media assets for fewer levels of downstream cells than specified by the default preloading scheme while execution is at cell L. In the example above, the designer can specify that media assets referenced by all cells one level downstream from cell 410 should be preloaded, i.e., cells 420 and 422. In this way, the designer can optimize bandwidth usage without sacrificing playback quality by eliminating the unnecessary preloading of the two grandchildren cells 430 and 434 that refer to large quantities of media that ultimately never get played.
Method 3: Global Preload Customization
According to another embodiment of the present invention, a multimedia experience can override the default preloading scheme, or provide its own preloading scheme in the absence of a default preloading scheme, by signaling the application and/or runtime engine to apply a different preloading scheme globally for the entire multimedia experience. For example, FIG. 6 illustrates a flowchart 500 representing a sample interactive multimedia experience. In this example, the application uses a default preloading scheme that normally requests media assets two levels downstream from a currently playing node in the flowchart 510. Thus, according to the default preloading scheme, the runtime engine would automatically preload media assets referenced by cells 520, 522, 530, 532 and 534 when playback execution reaches cell 510.
According to the present embodiment, the designer of the multimedia experience can program all of the cells in the flowchart 500 to preload media assets falling within a global distance G from a currently playing node so as to globally change the number of levels of cells to preload specified by the default preloading scheme. This is referred to as “global preload” customization. As shown in FIG. 6, for example, each cell in the flowchart 500 is programmed with instructions to preload all media assets for target cells falling within a global distance G of one level downstream from the current cell whereas the default preloading scheme requests media assets two levels downstream. During playback, when execution reaches the cell 510, the multimedia experience will signal the application and/or runtime engine to preload media assets referenced by the target cells 520, 522 falling one level downstream from the cell 510. A designer can program the cells in the flowchart 500 with instructions to override the default preloading scheme as described above.
In addition, the “global preload” customization can further include dynamically determining during playback different values for the global distance G based on threshold values for the size of media assets referred to by the cells falling within the global distance G. Such thresholds may be specified by the designer. For example, in addition to configuring the flowchart 500 with a global preloading scheme for the flowchart 500, including a global distance G of one level downstream, as discussed above, the designer of the multimedia experience may specify at least one size threshold value T for the global preloading scheme. During playback, the application and/or runtime engine will adjust the value of the global distance G such that the size of media assets referred to by the cells falling within the global distance G is less than the size threshold value T. For example, the application and/or runtime engine may increase the value of the global distance G to two levels downstream when execution reaches the cell 510 if the total size of the media assets referenced by the target cells 520, 522, 530, 532 and 534 is less than the size threshold value T. In this way, the number of levels of downstream cells for preloading is adjusted during playback of the multimedia experience so as to optimize bandwidth usage without sacrificing playback quality.
1. In a non-transitory computer readable storage medium having stored therein data representing instructions executable by a programmed processor for supporting a preloading of media assets associated with a multimedia experience, the storage medium comprising instructions operative for:
providing an interactively programmable executable flowchart representing a multimedia experience;
receiving a user input indicating a distance V from a location L in the multimedia experience represented in the flowchart; and
in response to receiving the user input indicating the distance V from the location L, programming the flowchart with at least one instruction to preload when playback execution of the multimedia experience reaches the location L at least one media asset falling within the indicated distance V from the location L.
2. The non-transitory computer readable storage medium of claim 1 wherein the at least one instruction is configured to override a predefined scheme for preloading media assets falling within a fixed distance from the location L during playback execution of the multimedia experience.
3. The non-transitory computer readable storage medium of claim 2 wherein the distance V is static.
4. The non-transitory computer readable storage medium of claim 1 wherein the distance V is dynamic.
5. In a non-transitory computer readable storage medium having stored therein data representing instructions executable by a programmed processor for supporting a preloading of media assets associated with a multimedia experience, the storage medium comprising instructions operative for:
receiving an input indicating a global distance G for a global scheme for preloading media assets; and
in response to receiving the input indicating the global distance G, programming the flowchart with the global scheme for preloading media assets, including at least one instruction to preload when playback execution of the multimedia experience reaches each of a plurality of locations in the multimedia experience media assets falling within the global distance G from the each of the plurality of locations in the multimedia experience.
6. The method of claim 5 wherein the at least one instruction is configured to override a default scheme for preloading media assets during playback execution of the multimedia experience.
7. The non-transitory computer readable storage medium of claim 5 wherein the global scheme is configured to dynamically vary the global distance G during playback execution of the multimedia experience based on a playback property of the multimedia experience.
8. The non-transitory computer readable storage medium of claim 7 wherein the playback property includes a total size of a set of media assets falling within the global distance G from the each of the plurality of locations in the multimedia experience.
9. The storage medium of claim 7 wherein the playback property includes a likelihood of reaching one of a set of media assets falling within the global distance G from the each of the plurality of locations in the multimedia experience.
US11/925,460 2006-10-31 2007-10-26 Methods for preloading media assets Expired - Fee Related US8521709B2 (en)
US85593806P true 2006-10-31 2006-10-31
US11/925,460 US8521709B2 (en) 2006-10-31 2007-10-26 Methods for preloading media assets
US20080104121A1 US20080104121A1 (en) 2008-05-01
US8521709B2 true US8521709B2 (en) 2013-08-27
ID=39331622
US11/925,460 Expired - Fee Related US8521709B2 (en) 2006-10-31 2007-10-26 Methods for preloading media assets
US (1) US8521709B2 (en)
US6973639B2 (en) 2000-01-25 2005-12-06 Fujitsu Limited Automatic program generation technology using data structure resolution unit
2007-10-26 US US11/925,460 patent/US8521709B2/en not_active Expired - Fee Related
"Using Macromedia Flash Asset Xtra (Part I)", 2006, http://hi.baidu.com/anycai/blog/item/998219d193aebbd3572c848b.html. *
Office Action regarding U.S. Appl. No. 11/957,066 dated Oct. 13, 2010.
Shelldrake Developer, http://www.shelldrake.com/Developergaqs.htm, 3 pages (2000).
US20080104121A1 (en) 2008-05-01
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GOTTLIEB, HARRY N.;JACOVER, EVAN;FRAM, STEVEN;REEL/FRAME:020038/0703