System and method for controlling actions within a programming environment

This patent discloses tools, methods and systems for dynamically visualizing, controlling and modifying the timing of actions that are implemented as part of an interactive multimedia experience. The tools, methods and systems include a timing region that dynamically reflects the timing of at least some of the cells along a selected single path in the multimedia experience and allows a designer to play through the cells and to change the timing of actions associated with the cells.

This patent is related to co-pending, Provisional U.S. Patent Application Ser. No. 60/875,179, titled, “Methods for Identifying Actions in a Flowchart,” filed on Dec. 14, 2006; and U.S. patent application Ser. No. 10/038,527, titled, “Method for Identifying Cells in a Path in a Flowchart and for Synchronizing Graphical and Textual Views of a Flowchart,” filed on Jan. 2, 2002. The entire contents of these related patent applications are incorporated herein by reference for all purposes.

BACKGROUND

A multimedia experience refers to the use of media in an interactive environment. The media generally include one or more types of information content, including for example, text, audio, graphics, animation, and video. During a multimedia experience, the media is presented to an end-user according to a logical sequence that can be affected by the end-user's actions.

A multimedia experience can be modeled as a flowchart that defines the logical sequence for playback of the multimedia experience. Such a flowchart generally consists of a sequence of linked cells that directly or indirectly reference media assets to be played by an application in a predefined order. The selection of which media referenced in the cells will be played can depend, in part, upon the choices made by end-users during each playback session of the multimedia experience. Flowcharts can be prepared by hand using pencil and paper or can be prepared electronically using a computer. Some software applications require a user to build a flowchart by drawing graphical shapes and then typing text into each graphical shape.

Often, during a playback session, the media referenced by several cells are presented to the end user in a coordinated manner, either simultaneously or in short succession of one another. In these situations, one cell is considered to represent playback of primary media off of which related cells representing secondary media are timed. In order to illustrate the dependencies between coordinated media, it is known to display the secondary cells on a side branch connecting these cells to the primary cell. The side branch may be visually distinct from other branches in the flowchart. This approach is disclosed, described and presented in related U.S. patent application Ser. No. 10/038,527, titled, “Method for Identifying Cell in a Path in a Flowchart and for Synchronizing Graphical and Textual Views of a Flowchart,” filed on Jan. 2, 2002.

The timing and flow of the experience and the user's interaction therewith can be difficult and time consuming to establish. It is desirable to interact and modify the timing of the experience along a particular path within the experience in a fast and dynamic manner to produce and provide a multimedia experience. In order to produce and provide a desirable multimedia experience, there exists a need for tools, methods and systems to facilitate the modification and interaction with components of a multimedia experience in an efficient manner.

SUMMARY

This patent discloses tools, methods and systems for dynamically visualizing, controlling and modifying the timing of actions that are implemented as part of an interactive multimedia experience. The tools, methods and systems include a timing region that dynamically reflects the timing of at least some of the cells along a selected single path in the multimedia experience and allows a designer to play through the cells and to change the timing of actions associated with the cells.

DETAILED DESCRIPTION

Disclosed and described herein are tools, systems, methods and embodiments that may be implemented using software and/or hardware components. For example, the one embodiment can be implemented with a software application stored on a memory and (i.e., computer-readable program code) executable by a processor of a general-purpose computer. Alternatively, some or all of the functionality of the application can be implemented with application-specific hardware components. For simplicity, the term “application” shall be used herein to refer generally to the entity (be it software and/or hardware) used to implement the embodiments described below. The term “tool” shall be used interchangeably with the term “application.”

I. Multimedia Creation

Turning now to the drawings,FIGS. 1A and 1Bdepict a display or graphical user interface (GUI) generated by one embodiment of an exemplary application or tool. The GUI illustrated inFIGS. 1A and 1Bincludes two display regions or windows100,200. As used herein, the term “display region” or “display window” refers to an area of the GUI generated and/or provided on one or more display devices, e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), etc. Each display region100,200can be a separate window or the display regions100,200can be different areas in a single window. The first and second display regions100,200can be fixed or movable and can be non-overlapping (as shown inFIG. 1) or can overlap each other. Additionally, the first and second display regions100,200can be alternately active or in focus (one then the other) in response to a command from the application and/or in response to selection by the user. The first display region100may be referred to herein as the “Structure Window,” and the second display region200may be referred to herein as the “Editor Pane.”

The first display region100provides or projects an exemplary flowchart150representative of a multimedia experience. As used herein, the term “flowchart” refers to a series of linked graphical icons, symbols or cells. The lines in the flowchart show how the cells are connected and/or interconnected. Moreover, the position and arrangement of the cells within the flowchart define the order in which they are traversed in operation. The Structure Window, i.e., the first display region100, is so referred to because it displays the layout, i.e., the structure, the cells and the connecting lines that comprise the multimedia experience represented by the flowchart150. The first display region100can display the exemplary flowchart150. Alternatively, a more complex, detailed and/or extended version of the flowchart150may be displayed in the first display region100as shown inFIGS. 1A and 1Bby utilizing, for example, a scroll bar to translate up or down or through the screen to reveal portions of the extended flowchart that may not always be visible within the first display region100.

The flowchart150includes a plurality of icons or cells (cells102to122). As described in greater detail further below, the cells102to122represent media content and user interactivity and the links connecting the cells represent the flow of the multimedia experience through time. The shape and/or configuration of the cells102to122represent the function or type of cell. For example, the rectangular-shaped are referred to as “simple cells” and the diamond-shaped are referred to as “conditional cells.” The simple cells each contain a single link or branch that connects to another cell. For example, the simple cell108connects or branches to the simple cell110which, in turn, connects or branches to the conditional cell112. The conditional cell112, unlike the simple cells, contains multiple connections or branches links to multiple cells. The branching links are associated with conditions necessary for that branching link to be followed. For example, the conditional cell112branches to the simple cells114and116via a branch132and a branch134, respectively. The branch132to the cell114is followed if the condition “ready to order” is true, while the branch134to the cell116is followed if the condition is false. It will be understood that the terms “simple cell” and “conditional cells” as discussed above, describe the flowchart connections or connectivity associated with the different types of cells. These descriptive labels are not intended to describe or limit the functionality of the individual cells, but instead describe how the cells connect or interact with other cells to define the experience. Moreover, that the graphical symbols representative of the cells, etc., described herein are intended as illustrative rather than limiting and that, according to the present disclosure, other types of symbols or representation can be used to illustrate the cells in a flowchart.

The cell114, in turn, connects to a reference cell118. The reference cell118includes or stores information or a value provide by the user or another input in another portion of the multimedia experience. For example, in the exemplary embodiment generally shown inFIG. 1B, the user may have been queried earlier in the multimedia experience to choose between “blue” and “red” as a favorite color. The results of this previously inputted selection is stored in the reference cell118as the value of the variable “g.favorite_color.” If the user previously indicated that the preferred color was “blue,” upon reaching the reference cell118the experience would proceed to the media cell120(media cells are described in greater detail further below). Alternatively, if the user previously indicated that the preferred color was “red,” upon reaching the reference cell118and the experience would proceed to the simple cell122. The direction of the user to either media cell120or simple cell122can occur automatically because the information necessary to control the direction or flow of the experience has previously been provided and stored. Thus, depending on the user's previous selection of “blue” or “red,” the reference cell118connects to the media cell120and announces “I think you'll like our blueberry pies” while playing an image of a pie carousel that began 0.37 (S+0.37) seconds after the announcement started, or connects to simple cell122and announces “Our strawberries are excellent right now.” Utilizing a reference cell, the flowchart150can automatically and conditionally select between a branch132aand a branch132bbased on the value of the stored variable “g.favorite_color.” In this way, the flow and experience provided by the flowchart150can seem, from the user's perspective, to be customized or tailored to their tastes and/or desires.

It should be noted that a cell can be a “grouped cell” and contain two or more individual cells (simple cells or conditional cells). The group cells may be utilized to simplify the display of a flowchart representing a multimedia experience by grouping multiple, typically related, cells together in a single graphical construct or cell. In addition to simple and conditional cells, a group cell can contain other group cells (i.e., groups cells can be nested) and other types of cells, such as “go-to” cells, “alias” cells, and other types of cells are described below.

Go-to cells can be used to direct or route the steps or operation of the flowchart150from one cell to another. For example, the cell106could be a go-to cell that directs the flow of the multimedia experience to the conditional cell112. In this example, the action would skip over, i.e., not execute or experience the element or actions associated with the skipped cells108and110. Alternatively, the go-to cell of the previous example could be utilized in place of a branching link to direct the action or flow of the multimedia experience from the cell106to the simple cell108. In this way, go-to cells can be utilized to organize the flowchart150to improve the readability and usability of the overall information presentation. Alias cells can be used to mimic the function/operation of another cell. Thus, if the application designer or programmer decides to reuse sections or elements of the flowchart150, the alias cell provides an effective tool or pointer for implementing this course of action.

Generally, it will be understood that the cells within the exemplary flowchart150, or any other flowchart or construct that may be designed in accordance with the teachings provided herein, can contain any suitable content, action or information. For example, each cell can be configured to: (i) contain a text message for presentation to the user; (ii) prompt or query the user for information and/or an input; (iii) evaluated provided information or input based on a pre-programmed logic or set of conditions, etc. In one embodiment, the conditional cell112can prompt the user to input “Yes” or “No” using a user interface device, such as a keyboard, mouse, microphone, remote control, telephone, or any other type of device. The conditional cell112also determines whether the user input is “Yes” or “No.” If the user input is “Yes,” the branch132leading to cell114is followed, and the application runs the script “Great. Let's Proceed.” If the user input is “No,” the branch134leading to cell116is followed, and the application runs the script “No problem. Take your time.”

According to the present embodiment, a cell can also contain actions that are implemented by the application, or by another program that is linked, embedded or otherwise connected to the application, in a coordinated manner, either simultaneously or in short succession of one another, when the cell is activated or played. For example, a cell can contain coordinated instructions to trigger a piece of media, gather user input, generate visual or oral information, send/receive information or media to a database, process data, perform a calculation, or perform other functions, such as describing how and when media should be played. Examples of media assets that may be linked or connected to the cell include, but are not limited to, digital audio/video objects representing still art, animation, graphics, on-screen text, music, sound effects, and voiceover dialogue. The term “media cell” shall be used herein to refer generally to a cell containing one or more actions whose timing is coordinated. Conceptually, a media cell acts as a container for actions, or references to actions, that trigger media playback. A media cell can contain multiple actions for executing media playback that are implemented by the application (or by another application) in a coordinated manner when the cell is played. In particular, each media cell includes at least one primary action and possibly one or more secondary action whose timing is coordinated with the primary action.

The primary action of a media cell preferably triggers playback of media that conveys a comprehensive message at a particular moment of the experience, such as audio or video of an actor speaking. A secondary action preferably triggers playback media that illustrates, reinforces and/or supports the comprehensive message being delivered through the primary action, such as text displaying all or a subset of the script performed by the actor and/or graphics that help illustrate the message of the actor's script. Preferably, the timing of a secondary action of a media cell is based on a time X before or after the beginning or end of the primary action of that media cell. Different actions in a cell can trigger playback of media that execute simultaneously with each other. It will be understood that the timing of a secondary action can be coordinated with the timing of a primary action using other timing mechanisms. For example, the primary and secondary actions could be timed off of a common clock, the secondary action could be timed off of another secondary action that is timed off of the primary action, etc. Thus, each media cell contains one or more actions whose timing is coordinated. Moreover, the primary and secondary actions may be initiated to coincide with each other, e.g., to show an image and play a sound clip, or may be activated sequentially. Referring to the flowchart150illustrated inFIG. 1A, the exemplary media cell104includes one primary action124and two secondary actions126and128.

In this way, the application distinguishes between media cells generally representing application flow and the coordinated actions performed by a cell when it is “played.” In particular, the media cell is a compound cell where several discrete and timed events can occur when flow reaches the cell during playback. When playback reaches a particular cell, the application and/or a runtime engine will generate a signal to invoke the cell's primary action, for example, for executing playback of a media asset referenced by the cell. The application and/or runtime engine will also generate signals to invoke any of the cell's secondary actions based on timing coordinated with the timing of the primary action. Thus, media cells may function as placeholders for actions to be executed within the overall progression of the application at runtime. Coordinated actions, such as those corresponding to the presentation or modification of media content during runtime flow, can be modeled internally in a media cell as separate objects themselves.

The exemplary cells102to122, when run or played during the execution of the steps or actions represented by the flowchart150, provide an interactive multimedia experience for the user. In this exemplary embodiment, the interactive multimedia experience consists of spoken or verbal scripts accompanied by on-screen text and graphics, and is executed and proceeds according to the logical flowcharted sequence that is established by the programmer and affected or interacted with by the user's actions. The cells102to122represent distinct sequences, portions or moments within the overall progression of the interactive multimedia experience and are arranged in the flowchart150in the order in which they are to be executed in operation.

For example, when the exemplary flowchart150executes, the media cell104initiated the primary action124which, in turn, plays an audio file such as, for example, a .wav file, a mp3 file, a .m4p file, etc., to greet the user with a recorded voice announcing “Top of the Morning.” Subsequently, the media cell104executes the secondary action126and128in a timed manner relative to the primary action124. In this embodiment, the secondary actions126and128include sound effects selected to enhance and augment the primary action and/or the interactive multimedia experience for the user. The secondary action126is timed to execute and begin playing a “Leaves Falling” sound effect one and a half (1.50) seconds before the end completion of the primary action124. Similarly, the secondary action128is timed to execute and begin playing a “Birds Chirping” sound effect zero point five (0.50) seconds before the completion of the primary action124. Thus, the “Birds Chirping” sound effect initiates one (1.00) second after the initiation of the “Leaves Falling.”

The cell106, upon completion of all actions associated with the cell104, executes an audio file which may, for example, be the sound of footsteps or alternatively may voice the phrase “Walking Down the Main Path” as its primary, and in this instance only, action. The cell108, in turn, executes an audio file that voices the phrase “Welcome.” Similarly, the cell110executes an audio file voicing the question “Are You Ready to Order?” The application, in turn, pauses or waits until the user responds to this query. The application proceeds to the conditional cell112, and depending on the user's response, an audio file voicing the phrase “Great. Let's Proceed.” is executed or played at the cell114or an audio file voicing the phrase “No Problem. Take Your Time.” is executed or played at the cell116. In this way, the multimedia experience can be designed to provide an interactive conversation interface that can be used to communicate ideas and information in an audio and/or visual environment, such as interactive computer games, commercials, guided tours, auctions, stories, and news, health, or financial services. The interface can be implemented with wired or wireless equipment that includes both audio and video output devices (such as a home computer or television) as well as with equipment that includes only an audio output device (such as a conventional telephone) or only a video output device. It is important to note that these embodiments can be used in flowchart applications that are unrelated to an interactive multimedia experience. Accordingly, a specific type of cell should not be read into the claims unless explicitly recited therein.

The secondary actions126and128of the media cell104may be displayed in a variety of manners such as, for example, in separate columns, as icons, in a tabbed format, etc. In one embodiment, the secondary actions126and128associated with the media cell104may be stacked or vertically aligned according to the chronological order in which they are to be executed. For example, the media for the secondary actions126and128begin execution in the same order in which they are displayed in the flowchart150shown inFIG. 1A. Additional secondary actions, media cell properties, media cell characteristics or available information may be selectively revealed of hidden utilizing a minimize and restore button130positioned adjacent to the media cell104and the secondary actions126and128.

The secondary actions126and128associated with the media cell104include timing values that control when, before the conclusion of the primary action124, each secondary action is to be initiated. For example, the timing value for the secondary action126, which is identified as “E−1.50.” indicates that the media associated with the “Leaves Falling” secondary action126begins playing one and a half seconds (1.50) before the conclusion or end (E) of the “Top of the Morning” primary action114. Similarly, the timing value identified as “E−0.50” indicates that the secondary action128executes zero point five (0.50) seconds before the end (E) of the primary action124. Alternatively, the timing of a secondary action can be based on a time X after the end of a primary action, in which case the timing value can be indicated as “E+X”, where “+X” indicates additional time or a pause between the end of the primary action124and the initiation of the secondary action. In another embodiment, if the timing of a secondary action is based on a time “X” before the start (S) of the primary action, the timing value can be indicated as “S−X”, where “S” is the start identifier and “−X” indicates the span of time before the primary action is initiated. In yet another embodiment, if the timing of a secondary action is based on a time X after the start of a primary action, the timing value can be indicated as “S+X” and follows the same convention discussed above.

The content, actions and/or information contained within the cells along a particular path of the flowchart150may be organized and displayed in the second display region200(the Editor Pane). The additional display of this information, typically associated with the primary actions, allows the user to read through the content and information provided within a single path of the flow chart150regardless of the cells in the other paths. For example, the information presented within the second display region200in the exemplary embodiment shown inFIGS. 1A and 1Bincludes the text associated with the primary actions from cells104to114,118and120. In this exemplary embodiment, the second display region200contains a textual view250of the cells102to114,118and120in the order in which the primary actions and/or information appear in the flowchart150. Thus, the embodiment illustrates, the information along a single path that includes the common flowchart path defined between102and112and branches132and132a(which includes cells114,118and120) are displayed in the second display region200.

The textual view250generally shows a “description” of the primary action of a cell, e.g., the text that is contained in the primary action, the line of dialogue that will be voiced by the primary action, the description of the animation or of the special effect (SFX) that will be played by the primary action, etc. The content displayed in the textual view250can also contain other cell “properties,” such as: (i) the font of displayed text; (ii) the name of the actor who will provide the media content; (iii) the cell's timing relative to other cells; (iv) the volume at which audio content of the cell should be played; (v) the frames per second of animation and other information. Cell properties can also be displayed in a third display region (not shown). It should be noted that the textual view250is not limited to text characters, this view could include the graphics or icons of pictures, movies, media clips, etc. that may be displayed/connected to the primary action of the cell. For example, the text of the cell can also be supported with a graphics button or thumbnail image next to the text that indicates the type of media in the cell, e.g., a button with a little horn indicating a SFX, and is clickable to open a properties window of the cell, to playback the media, or to perform other functions.

According to the present embodiment the primary action and the related secondary actions are displayed in separate areas within the media cell, although it should be understood that the application may implement other methods or alternate embodiments to identify and to differentiate the secondary actions of a media cell from the primary action of the cell. For example, the secondary actions can be displayed in the first display region adjacent or proximate the media cell according to their chronological order of execution. The positioning of the secondary actions relative to the media cell would identify their child-parent relationship with the primary action of the media cell. Visual cues can also be provided, for example, links connecting each the secondary actions to the primary action, so as to visually reinforce the child-parent relationship between the secondary action and the primary action and their membership in the same group.

In another alternate embodiment, the textual view in the second display region can show both a description of the primary action and the descriptions of the secondary action in the order of their execution. For example, the textual view can display the primary action in a first column and the secondary action next to the primary action in a second column. In yet another alternate embodiment, the secondary actions of the media cell can be displayed in a fourth display region separate from the first and second display regions, the fourth display region being associated and/or visually linked with the media cell.

II. Multimedia Control

It will be understood that in order to generate a smoothly functioning multimedia experience, it may be necessary to review, edit, adjust and tweak the timing of individual cells104to122of the flowchart150, including the primary action and secondary actions associated with each of these cells. Moreover, it will be understood that the cells106to122may contain greater or fewer secondary actions than the illustrated cell104based, in part, on the complexity and desirability of these actions in the overall layout and design of the interactive multimedia experience. Often, the programmer or designer of the multimedia experience represented by the flow chart150will require changes in the characteristics such as the timing, order and flow of the cells associated with a particular path of the flowchart150. In order to minimize the time necessary to test the application flow along each of the separate paths (see, for example, the different paths along the branches132and134and the branches132aand132b) within the flowchart150, the programmer may dynamically visualize and adjust the timing of the cells along each single path and their associated primary and secondary actions.

FIG. 2illustrates a timing region or control window300constructed in accordance with the teaching and disclosure presented herein. In this embodiment, the timing region300provides a visualization of the timing for the cells102to112, including the primary and secondary actions124,126and128of the cell104. For example, the timing region300includes numerous timing panes302to310that correspond to the cells102to110which, in turn, form part of a single path of the flowchart150. Each of the timing panes302to310visually represents the playing or initiation of audio or video files and/or the presentation of text or interactive controls represented by the cells102to110, as a function of time, e.g., along a time axis t. In the example illustrated inFIG. 2, the timing pane304displays a waveform324of the audio file to be played by the primary action124of the cell104. The waveform324states the phrase “Top of the Morning.” The runtime or length of time that the primary action124takes to complete is represented as a function of the time axis t and is shown in the duration window D. The timing pane304may further include a “play head” or visual indicator of the playback of, for example, audio that is represented by a waveform340, discussed in below. The playhead, in one embodiment may be a vertical line that sweeps from left to right during playback or execution of the primary action124within each of the timing panes302to310. The playhead can allow the user to simultaneously visualize what is being heard, making it clear precisely when each of the secondary actions126and128will be executed relative to the audio.

The timing pane304further includes action indicators or cues326and328that correspond, respectively, to the secondary actions126and128of the cell104. The action indicators326and328are textual and/or graphic representations of the secondary action126and128associated with the cell104. The action indicator326marks the initiation of the secondary action126which, in the present example, plays an audio file that represents the sounds of leaves falling. Similarly, the action indicator328marks the initiation of the secondary action128which, in the present example, plays an audio file that represents the sounds of birds chirping. It will be understood that an action indicator could represent any type of secondary action and additional secondary actions and corresponding action indicators may be associated with each of the timing panes in the timing region300representing the cells in the flow chart150.

Each of the actions124,126and128associated with the cell104, and their respective timing and interaction, is graphically and interactively represented by the waveform324and action indicators326and328within the timing pane304. As previously discussed, the action indicators326and328display to the programmer or designer of the multimedia experience the timing of the secondary actions126and128relative to the primary action124. The action indicators326and328can be selected, manipulated and adjusted with respect to the time axis t to modify or change the timing of the secondary actions126and128relative to playback of the audio associated with the primary action124, as represented by the waveform324.

For example, by selecting one of the action indicators326and328, the designer can drag or move the selected action indicator along the time axis t relative to the timing pane304. By changing the position of the action indicators326and328in this way, the designer can make immediate, dynamic adjustments to the timing of the secondary actions126and128relative to the primary action124within the timing region300. Also, the programmer or designer of the multimedia experience may select one of the action indicators326and328and adjust its edge timing332within the timing region300by entering numerical time values into an input field334and indicating whether the event should occur relative to the start (S) or end (E) of the corresponding primary action.

In addition to representing the timing of secondary actions, action indicators can also represent the timing of other cells. For example, the cell110and the conditional cell112of the flowchart150are respectively represented by the timing pane310and the action indicators336and338. The cell110as its primary, and in this example its sole action, presents the user with the question “Are you ready to order?” By turning to the timing pane310, the possible responses, e.g., “Yes” and “No,” associated with the cell112are presented to the user as the cell110begins to play. In this example, the action indicator336represents the “Yes” response and the action indicator338represents the “No” response. Depending upon the user-selected response, the flowchart150proceeds along the branch132or the branch134as shown relative to the conditional cell112inFIGS. 1A and 1B. The action indicators336and338can be selected, manipulated and adjusted as described above to modify or change the timing of the presentation of the “Yes” action indicator336and the “No” action indicator338by the cell112relative to the execution of the cell110.

The programmer or designer can, in turn, by selecting a play button330playback the cells of the flowchart150associated with each of the timing panes within the timing region300. The designer can observe the flow of time relative to the waveforms displayed in the timing panes and the action indicators associated with the timing panes. Other audio or execution controls, for example, the controls330aand330bmay be utilized to rewind or reverse (330a) or pause (330b) the playback of the waveforms associated with each of the timing panes within the timing region300. In this way, it is possible to dynamically modify and iteratively evaluate the timing for the cells, including their primary and secondary actions, associated with the timing panes within the timing region300. This process affords or provides a substantial time savings during design, production and testing of interactive multimedia experiences.

In operation, the application or the programmer or designer of the multimedia experience selects a single path of the flowchart150and one or more representative timing regions are automatically generated for cells, including conditional cells and/or reference cells, along the selected single path. For example, the experience designer may select the single path that includes the cells102to112, the cell114along the branch132and the cell120along branch132aof the flowchart150.FIGS. 3 and 3Aillustrate timing regions300and300A that may be generated for this selected single path. The timing region300provides a visualization of the timing for the cells102to112, while the timing region300A provides a visualization of the timing for the cells114and120. In the present embodiment, two timing regions300and300A are generated because the cell112requires user input and provides a user break point in the application flow along the selected path. Since the timing of the cell114relative to the cell112is indeterminate, two timing regions are generated. Alternatively, a single timing region could be generated for the single path described above, which concatenates or links the timing regions300and300A. In this way, a single timing region could define a graphical representation of all of the cells along the selected single path, including the cells102to112,114and120, as a like number and arrangement of timing panes and action indicators.

Also, it will be understood that the application or programmer may select a different single path that is of interest and the timing region(s) will automatically update to reflect the sequence of media for cells associated with the newly selected single path. For example,FIGS. 3A and 3Billustrate timing regions300A and300B that may be displayed when different single path are selected, including the branch132and the branches132aand132b, respectively. As previously discussed, the experience may playback either the cell120or the cell122in response to information previously provided with respect to the reference variable g.favorite_color.

FIG. 3Aillustrates the timing region300A that may be displayed when the selected single path includes the cells102to112, branches132and132a. Stated another way, along this selected path, the user has previously indicated a preference for the color blue. The timing pane344corresponds to the media cell120and displays the waveform340of the audio file to be played by the primary action associated with the cell120, which as previously discussed informs the user that “I think you'll like our blueberry pie.” An action indicator342shows that the pie carousel graphic associated with secondary action of the cell120is displayed zero point three-seven (0.37) seconds after the start of the primary action. Similarly,FIG. 3Billustrates the timing region300B that may be displayed when the selected single path includes the cells102to112, branch132and sub-branch132b. Stated another way, along this selected path, the user has indicated a preference for the color red. A timing pane348displays the waveform346for the audio file associated with the cell122, which as previously discussed, informs the user that “Our strawberries are excellent right now.” The action indicator342can be selected, manipulated and adjusted with respect to the time axis t to modify or change the timing of the pie carousel graphic associated with cell120when the single path corresponding to the user preference for the color blue is selected. The programmer or designer can thus dynamically modify and iteratively evaluate the flow of the possible sequences of media, audio files, textual prompts, etc., for cells along different single paths of the flowchart150.

FIG. 4illustrates an alternate embodiment of a timing tool400or region that could be incorporated into the flowchart150. In this embodiment, the size and relative location of each cell to the remaining cells in the flowchart150is indicative of timing, duration and/or primary and secondary actions associated with each cell. A cell402, in this exemplary embodiment defines a timing cell or an interactive multimedia timing cell, includes the elements and teachings disclosed in connection with the timing regions300,300A and300B. For example, the cell402can include a primary action404, a waveform406representative of the media, action, etc. associated with the cell402, and secondary actions408to412. Thus, as the experience proceeds from the start indicator to the cell402, a playhead414or progress indicator, begins to translate or track along the waveform406to visually indicate the progress or play of the media, action, etc. In this example, the flowchart, individual cells, etc. are vertically aligned, thereby indicating that progress within the experience is measured from the top of the flowchart towards the bottom of the flowchart. In other words, the time axis t proceeds in the direction indicated by the arrow.

The vertical length of the cell402further indicates that the action or waveform associated with the cell402plays or executes for a greater length of time that the waveform418associated with the smaller cell418. As discussed above in connection with the timing regions300,300A and300B, the secondary actions408to412are actions, events, media files, text, etc. which are displayed or executed relative to the action associated with the primary action404. In this embodiment, the secondary actions408to412are analogous to the action indicators discussed above. For example, the secondary action408may represent text to be presented during the execution of the experience. The time at which the secondary action408is displayed can be adjusted graphically by selecting and shifting the representative icon or action indicator relative to the waveform406. Thus, as the cell402is activated, the playhead414translates along the waveform406and upon reaching the secondary action408, the text, media, etc. associated with the particular secondary action is displayed or executed.

The cell416includes the waveform418associated with a primary action420. The cell416abuts or contacts the cell402in a contiguous fashion. The relative location of the cells402to416indicates that the experience proceeds without pause from cell402to cell416. Similarly, a gap422between the cell416and the cell424indicates a pause or dwell between the execution of the two cells416and424. Thus, the relative timing between the cells402,416and424can be quickly and easily adjusted by changing the relative spacing between these cells. Moreover, the relative size of each of these cells, i.e., the size difference between cell416and cell402, indicates that length of time necessary to execute or complete the actions associated with that cell. In other words, the cell416, e.g., a smaller sized cell, executes for a shorter amount of time than the cell402, e.g., a larger sized cell.

It will be understood that the exemplary techniques and the timing region for controlling the timing of a multimedia experience that includes media cells having multiple actions can be implemented in multimedia experiences where additional actions associated with a parent cell, group of cells, etc., are graphically presented as child cells along branches or offshoots of the parent cell. In this instance the timing region can be configured to display one or more of the elements, properties and actions associated with a given parent cell in a timing pane before displaying a child cell and the properties associated therewith as action indicators.

The second display region or Editor Pane200can be associated or utilized with the timing tool400to allow properties, such as timing, text, graphics, etc. to be directly edited without having to directly interact with the icons or graphics associated with the cells402,416and424.

It will be understood that the application can be equipped with various functionality to allow it to facilitate the construction of the media assets scripted by the writer and for providing the programming necessary to fully render the interactive multimedia experience on a given platform. As noted above, cells can contain actions to be implemented, including instructions to play a media asset such as an audio file or a video file. When a writer is scripting the content of the interactive multimedia experience, those media assets may not yet exist. The application can sort the various pieces of desired media based on the talent that is necessary to create the media or on other criteria for sorting. For example, the actions in a media cell can be divided into music tasks, animation tasks, art tasks, programming tasks, SFX tasks, writing tasks, video tasks, and performance tasks. In this way, artists used to create the media can be assigned a list of tasks that need to be performed. When each of the media assets is created by the artists and inserted into designated “slots” in a database, the application can assemble the completed media for playback. The slots in the database can be created before or after the media assets are created. Because an interactive multimedia experience can have thousands of assets and a non-linear structure, it is preferred that the application maintain a database to track the media assets.