VISUAL STORY ENGINE

A system and method for creating a navigable content having a narrative structure and behaviors configured to allow a consumer to dynamically and non-linearly control many aspects of a narrative such as plot, transition, speed, story beats, media, delay, and the like, is described. The method and system also provides authoring tools to dynamically edit source material that may be adjusted and changed by a consumer during a viewing of the navigable content.

DETAILED DESCRIPTION

Embodiments are directed to creating a content narrative and presentation system that allows a consumer virtually, in real time, to dynamically and non-linearly navigate the narrative and in a manner that allows the consumer to control many aspects of the narrative such as plot, transition, speed, story beats, media, delay, and the like. In one embodiment, a navigable story structure100is configured to provide an interactive experience with a consumer (e.g., reader, user, viewer, student, buyer, participant, etc.). For example, the consumer while viewing the story structure100may decide to interactively and dynamically change the type of content, the story speed, the narrative path, the media used, transitions between parts of the narrative, and the like.

In one embodiment, the story structure100is a configuration of a seed or base story110and a collection of one or more distinct threads120. In some embodiments, story structure100maintains a “stack of threads” referred to herein as a “thread stack”130. The thread stack130includes some or all of the threads120that make up the current active story. The thread stack130is configured to allow the base story110to be dynamically and non-linearly changed by a consumer. For example, as illustrated inFIG. 1, the story structure100may include a base story110, and story threads such as main thread120, character backstory thread130, and alternative ending thread140.

A consumer may manipulate Story110in order to create a non-linear or personalized version of the story110. For example, media components such as video, text, audio, images, and the like, may be dynamically added by pushing additional threads120onto the thread stack130. Subsequently such media components can be removed or rearranged by popping them from the thread stack130as described herein. In some embodiments, threads120can be streamed from remote URLs or placed behind pay walls providing flexibility in how the content is distributed.

In an embodiment, the threads120are composed of one or more ordered display views140, which are in turn each composed of one or more panels150. The panels150are views that are part of at least a portion of the display views140. Panels150may include one or more layers160, ordered or unordered, that extend between the back and the front of the panels150. The layers160may include any number of different media or content such as embedded behaviors, clear content, movie content, text content, image content, meta-data content, computer code, bar codes, color content, vector graphics, and the like.

Layers160include one or more behaviors. The states of the behavior contain visual attributes such as the size, position, color, pointers to image or movie media, etc. that determine how the layer will be rendered to screen at any given moment. The states also contain flow attributes to step back, step forward, jump within a thread, or to jump to a completely different thread of the narrative as described further herein. Additional attributes determine the nature of the branching such as if the narrative should return and restore the calling thread when the jump thread is completed as described herein.

Layers160may also have an editing queue associated with them. For example, when a behavior state assigns a new media pointer (URL), another preempt attribute controls if the video stream should switch immediately or if the new video should be added to the editing queue. The benefit of such an editing queue is that the video transitions can be made seamless if the two video streams connect at the transition point. “Customized Music Videos” and some of the other examples rely on the editing queue concept as described herein.

As an example, as illustrated inFIG. 1, the story structure100includes the base story110and three threads120: a main thread122, a back-story thread124, and an alternate thread126. The threads120are composed of display views140, which in this illustration include a first display view142. The first display view142includes three panels150: a first panel152, a second panel154, and a third panel156. The first panel152as illustrated includes a number, N, of layers160. The layers160may include any number of content. In this example, the content may be a stream of images and corresponding audio content. When layer N162is selected, the content is loaded into the panel for display to the viewer via a display device such as a tablet device, mobile telephone, video display, video projector, and the like.

In addition to having bounds (position and size) and optionally something to draw, layers160also may act as the primary building blocks of viewer interaction. As described herein, consumers may interact with the layers160using virtually any input device or system. For example, for devices having a touch screen, layers160may respond to touch and gesture events such as single tap selection, pinch zoom and dragging. These touch events may be used to trigger a change in the state of one or more of the layers160.

As illustrated inFIG. 2, in order to keep track of these states, a layer160may contain one or more Finite State Machines (FSM) to form a Layer Finite State Machine (LFSM)200. LFSM200controls consumer interactions with a narrative such that the states of the behavior determine both the visual appearance and flow of the narrative. LFSM200includes a series of layer states210. At least some of the layer states210includes two parts: a partial list of properties that govern the appearance or behavior of the layer in some way, and a list of event triggers to which the layer160is responsive. The triggers may be actuated by consumer action or gestures, clock or movie time, spatial relationships between layers, state transitions within other layers, etc. Triggers also have access to a global sandbox that can include personal information about the consumer and their interaction history with the current or previous narratives. This information can be used as input to conditionals that can also trigger state transitions and so influence narrative flow.

In one embodiment, in order to support multiple, overlapping behaviors LFSM200may be used. Unlike a FSM where generally attributes are captured in a state, the LSFM200provides the author with the ability to set attributes between a locked and unlocked state. Locked attributes are essentially unaffected by state transitions. The resulting behaviors are therefore more modular. In some embodiments, behaviors are “composited” to get the final overall state.

By way of illustration,FIG. 2shows layer states210including an initial state212, a movie A state214, a movie B state216, and a done or end state218. Initial state212, movie state A214, movie state B, and done state218, each include a property and event trigger. For example, initial state212includes first property220triggered by first event trigger230, movie state A includes a second property222triggered by second event trigger232, movie state B includes a third property224triggered by a third event trigger234, and done state218includes a fourth property226triggered by a fourth event trigger236.

Illustratively, layer160may be configured to transition with respect to properties for each of the states210in response to at least one of the first event trigger230, second event trigger232, third event trigger,234, and/or fourth event trigger236. For example, the layer160would change with respect to the first property220in response to a first event trigger230, the layer160would change with respect to the movie A property222, in response to a second event trigger232, the layer160would change with respect to the movie B property224in response to a third event trigger234, and/or the layer160would change with respect to the done property226in response to a fourth event trigger236. Stated differently, in some embodiments when layer160transitions into a particular state such as initial state212, movie A state214, movie B state216, and/or done state218, the layer's160appearance and/or behavior will change based on the properties defined for those states, or combinations thereof. Further, from that point on the layer160will respond to event triggers associated with those states.

In some embodiments, multiple LFSMs200in a layer160may be configured to affect one or more of the properties associated with the layer160. Further, in some embodiments a story110may include a global set of properties that can be accessed and modified by LFSMs200as well.

In an embodiment, event triggers may include at least two different types of event triggers. For example, the event trigger types may include intrinsic triggers, automatic triggers, touch based expression evaluation of layer global property triggers, panel event triggers, or triggers responsive to changes in the state of another layer's LFSM200. In some instances, event triggers may include specific arguments to determine if the trigger's conditions are met, for example “time” may be used for duration triggers. For example, a first event trigger230is illustrated as a “panel entry” event trigger type that is responsive to a panel data output, such as a touch panel control signal. Triggers may also be configured to contain a target state. After an event has successfully triggered, the LFSM200will transition to the target state.

As illustrated inFIG. 2, LFSM200may be configured to allow a consumer to modify how a movie may be played in response to inputs from a consumer via, for example, an input panel device, some of which are described herein. As illustrated, in response to a consumer's input, the layer160responds to “panel inputs” causing the movie to toggle between movie A state214and movie B state216, which may represent different scenes of a movie, or entirely different movies. When a “panel exit” is received from a viewer, the LFSM200is returned to its initial state.

Visual Story System Narrative Navigation

Referring back toFIG. 1andFIG. 2, in one embodiment, a thread stack120may be configured to allow an author to create and/or view a dynamic non-linear story110. For example, consider the case of a single thread linear story110. When the consumer begins, the initial thread122(often called “Main”) is first pushed on the thread stack130causing the display view140to be delivered to the screen, e.g. first display view142, first panel152, and N layer162. As the consumer advances, the output (e.g. read head or viewing index point) moves to the next panel, e.g., second panel154. On reaching the last panel152on a display view140, e.g., third panel156, the output advances to the next display view140. This continues until the last display view140of the last panel150is reached at which point the output cannot advance any further.

In another embodiment, the LFSM200may be used to move from the linear narrative described above to a non-linear narrative. For example, in addition to layer properties220,222, etc., a layer's state may also contain navigation properties that specify how the narrative will progress if that particular state is triggered. In addition to linear navigation commands such as moving forward or back in the narrative, the state may contain properties to jump to a specific location that may be another display view and panel within the same thread or an entirely different thread. For example, a LFSM trigger such as232may cause the narrative to digress from story thread Main (122) to Character Back Story thread124. Additional properties may give further clues on how to achieve the narrative transition. For example, whether the associated thread, such as story thread124, will transition back to the current thread, such as story thread122, on completion and whether story thread122will be restored. If the narrative jumps to a new story thread, such as story thread124, it is pushed onto the thread stack130. In this way, the dynamic structure of the narrative can be expanded and modified.

For example,FIG. 3illustrates a sample list of jump scenarios300that may be associated with each layer160for developing or viewing a non-linear story110. In this illustration, the scenarios included “jump” scenarios310, thread operations320, and affect to the image output retrieval point340. Jump scenarios310use various jump properties of the thread120in order to jump to different panel play positions (e.g. read head positions). By way of example, panel152could have a text layer160called “Next” to explicitly move to panel154in the story110.

In one embodiment, a jump property includes three parts: a thread name, a display view name or number and a panel name or number. For example, an argument may be written as: (AlternateEnding”, 1, 1), which indicates, “alternate ending, first display view142, and first panel152”. Once additional threads120are pushed on the stack130, the point at which the media is read (i.e. the index point) may be automatically transitioned between threads120if possible when asked to move forward and back. For example, presuming the thread stack130contains two threads130(Main, Extra Features). The read point will advance from (Main, last display view, last panel) to (Extra Features, first display view, first panel).

By way of example, scenarios300illustrate variations of where the read point342may be moved given a jump property of a layer160. This may be illustrated as follows: using “Jump within thread A” scenario312, when a layer160is triggered, for a jump property, using the jump within thread A scenario312, several thread operations are executed. Here, the thread A342has an index read point344positioned above a first index section of thread A342. After the jump, the read point has moved from the first index section of thread A342to a second read point above a second index section of media A342.

In this illustration, thread B350is pushed onto the thread stack130and the “jump from end of thread A to start of thread B” scenario314is invoked. This jump property allows the read point to move from the end of one thread120, e.g. thread A322, to the added thread, e.g., thread B350. For example, using the “jump from end of thread A to start of thread B” scenario314, the read point344jumps from a third index point of thread A342, which is toward the end of the thread A play index, to a fourth play index point of thread B350, which is near the starting index point of media B350.

The “jump from middle of thread A to middle of thread B” scenario316jump property allows the read point to move from about the middle of one thread120, e.g. thread A322, to about the middle of an added thread, e.g., thread B350. This jump property is configured to leave a “trim tail’ on the thread being jumped from, e.g., thread A322, and leaves a “trim head” on the thread being jumped to, e.g. thread B350. For example, using the “jump from middle of thread A to middle of thread B” scenario316, the read point344jumps from a fifth index point of thread A342which is toward the middle of the thread A play index, to a sixth play index point of thread B350, which is near the starting index point of thread B350. The index portion of media A342left (not read) would be the “trim tail”. The index portion of media B350that is skipped would be the “trim head” portion.

The “jump from thread A to thread C” scenario318property allows the read point to move from an index point on one thread, e.g., thread A342to another pushed thread120, e.g., thread B344. For example, using the “jump from thread A to thread C” scenario318, the read point344jumps from a seventh index point of thread A342which is within the index of the thread A play index, to an eighth play index point of thread C352, which within the index of thread C352.

FIG. 4AandFIG. 4Billustrate other thread jump properties: restore current thread and return from target thread. These properties allow a consumer to dynamically jump or digress. For example, as illustrated inFIG. 4A, for a “do not restore current thread” scenario, if the layer160includes a “do not restore current thread” instruction and a “do not return from target thread” instruction, when the consumer initiates a jump from thread A342to thread B350, the read point would move from a first read index point on thread A342to a first read index point of thread B350. When the read point reaches the end of thread B350, there would be no change. In other words, the read output would remain at the end of thread B. If the consumer navigates back to thread A342, the read point would jump back to about the same location as the consumer jumped from. This mode would allow the consumer to observe another panel and then return to the narrative from about where they left off.

As illustrated inFIG. 4B, for a “return from current thread” scenario, a digression mode may be invoked by the consumer. For example, if the consumer invokes a jump from thread A342to thread B350, the jump property may be set to automatically digress, or in this case set the read point to the beginning of thread B, let thread B play to the end, and then automatically jump back to the point the consumer left off at in thread A342. The consumer may then navigate within thread A to continue the story110. Additional properties, such as transitioning to multiple associated story threads, may provide additional information about how the thread assembly130is structured.

Almost infinite variations of movement within and between content may be accomplished using the above scenarios. For example, defining jumps in this way allows authors to model a wide variety of non-linear behaviors including a “table of contents page”, “choose your own adventure”, and for example, stories110personalized based on global properties about the consumer, footnotes, or digressions.

Visual Story System

Embodiments provide a Visual Story System (VSS)500as shown inFIG. 5. In one embodiment, the VSS500includes a story reader510and a Visual Story Engine (VSE)520. The story reader510is configured to receive input from a consumer (e.g., reader, user, viewer, student, buyer, participant, etc.), and display a story110to the consumer. Story reader510includes a display medium, e.g., display screen, as well as user interface elements, which may include the display screen itself (e.g., touch screen) and/or additional interface elements such as mouse, keyboard, pen, etc. The story reader510is a “user interaction” interface that allows the consumer to both view, as well as interact with the story110in a dynamic way. As way of an illustration, the story reader510may be used by someone to view content, view and modify content, modify a story thread130, and the like. For example, a consumer can use the story reader510to view and interact with a movie, comic book, electronic book (e.g., ebook), multimedia presentation, and the like. The story reader510may also be configured to allow a consumer to directly interact with the story110in a dynamic way as described further herein, through interpretations of user gestures, device motions, and the like.

The story reader510interfaces with the VSE520via a gesture handler512and a screen renderer514. The gesture handler512is configured to handle the gestures by the reader input, typically responsive to movement of the consumer's hands and fingers. In one embodiment, the gesture handler512may receive one or more signals representative of one or more finger gestures as known in the art such as swipes, pinch, rotate, push, pull, strokes, taps, slides, and the like, that are used as LFSM triggers such as232,234within the story110being viewed. For example, given a dynamic story110configured to be changed by the consumer, a consumer may use finger gestures interpreted by the gesture handler512to change the story's plot, timing, story beat, outcome, and the like.

The screen renderer514is configured to receive media assets516such as audio, video, and images, controlled by the VSE520for display to the viewer via story reader510. The screen renderer514may be used to send visual updates to the story reader510responsive to or based on processing done by the VSE520. The screen renderer514may also be used to generate and drive the screen layout. For example, consider the case where a consumer is watching a multimedia presentation. The screen renderer514receives display updates and layout instructions from the VSE520in response to the viewer's input, and the layout instructions received from the VSE520with respect to the needs of the presentation. For example, as described above with regard toFIG. 1andFIG. 2, the presentation may include panels150having layers160containing data such as still images, video segments, audio cues, screen transitions, image transition effects, and the like, that may be used by the VSE520in a manner to drive the screen renderer514to present the multimedia in a dynamic way to the consumer.

In one embodiment, the VSE520includes a narrative navigator522, layer finite state machine200, state attributes526, thread structure120, thread definitions528, and the thread stack130. The narrative navigator522is configured to receive and process the navigation signals from the gesture handler512. In response to the navigation signals, the narrative navigator522drives changes to the narrative with regard to plot, transitions, media play, story direction, speed, and the like. For example, a consumer may configure the narrative navigator522to change the plot of the story from a first plot to a second plot using a swipe gesture. For example, referring toFIG. 1-3, the VSE520may be at an initial state212. Upon receiving a gesture from a consumer to move the narrative from the initial state212to a movie A state214, the VSE520in response to a trigger gesture, may move the narrative from the initial state212to a movie state A214, using for example, “Jump within thread A” scenario312as illustrated inFIG. 3.

FIG. 6andFIG. 7illustrate a story navigation editor600which is a user interface (UI) used to create a story110for use with the VSE520The story navigation editor600includes a story outline610. The story outline has tabs for editing atomic story threads120such as tabs614,616and618or a tab612to view all threads and their relationships at once. Once a thread tab is selected, the author is presented with a thumbnail and hierarchical list of all display views140within the thread. Nested within the hierarchical list are all of the panels and layers160associated with the display views.

The story navigation editor600further includes a media output section630configured to display media assets516. The media output section630may be configured to act as display to work in conjunction with VSE520. For example, once the story110is associated with threads120and the thread stack130, and the triggers and behaviors of the layers160are created, the media output section may be used to “play” the story110to the consumer for viewing and interaction therewith.

The story navigation editor600also includes layer editor section640. The layer editor section640includes a layer tab642used to edit the property and content of layers, for example, layers160. The layer tab642exposes properties648that an author may use when creating a story. The properties include specifying a layer type, position, size, path, duration of layer, and the like. In an example, the layer tab642may be used to position a layer within a specified position of a panel to allow the author to artistically size the layer160, place the layer160within the panel160, and set the duration of a media clip.

The layer editor section640also includes a template tab644, which is used to save layer templates for use with creating dynamic stories110. In some embodiments, templates can be created at the layer160, panel150, screen or thread granularity. A template may be created by removing some or all of the media pointers from the layer160, while maintaining the structure and behaviors. In one aspect, if a layer160is disembodied from the rest of the story structure, it's possible to create dangling layer connections and narrative jump points. In order to “apply” the template the author may provide new or additional media pointers to resolve the dangling layer and narrative jump connections. Bootstrapping narratives with templates can be significantly faster than authoring narratives from scratch at the expense of arbitrary creative control. Since the templates contain the layers160, media asset pointers516, behaviors, and triggers, consumers may author narratives with their own content by binding media assets to the media assets pointers without requiring an authoring tool. The layer editor section640also includes an assets tab646. The assets tab is used to associate media assets516with one or more layers.

Referring toFIG. 7, the story navigation editor600includes a thread editor710to set the state and trigger of the thread120. For example, thread editor710has a state input/output interface712and an associated trigger input/output interface714. In one embodiment, the state input/output interface712has a trigger connector716connecting one input/output point of the state input/output interface712to an input/output point on the trigger input/output interface714, and another input/output connector718connecting another input/output point of the trigger input/output interface714to an input/output point of the state input/output interface712. In this embodiment, thread editor710may be connected to any number of state or trigger input/output points in order to achieve the desired behavior. For example, as illustrated, upon receipt of a “tap” signal a “tapped timer” behavior will be invoked placing thread120into a default timer state.

FIG. 8illustrates an example of an input of a media asset516processed by story navigation editor600. In this illustration, media asset516is a video asset used to play to audiences on a screen in a theater810. An instantiation of story navigation editor600is displayed on a computer monitor820. Once processed by story navigation editor600using the VSS500described herein, a navigable story110is resized as needed and displayed on another display device814, such as a tablet, mobile phone, computer screen, and the like. Navigation widgets816,818may be displayed based on the input trigger of the layer160to navigate the story110. For example, as illustrated, “swipe”816and “stars” are navigation widgets in this particular instantiation of the story110.

FIG. 9is a high-level block diagram of a method900to create a navigable story structure100according to one embodiment. Method900starts at step910. Method900moves to both912to define the narrative structure, and step914to digitize the media assets516. At step912an author creates and defines a narrative structure. For example, as described herein, method900receives a base story110and story threads120to form a narrative structure. In step920, media files are generated for use in the narrative structure. Once the narrative structure has been defined, in step916it is determined whether the author has more screens to author with respect to the narrative structure. If there are more screens to author, method900moves to step918to create the layers160. At step922, input is received from the author to define the behavior as described herein with respect to LFSM200. Method900returns to step916to determine if there are more screens to process. Once all the screens have been processed, method900moves to publish the narrative at step924. At step926, the narrative structure is received and the story structure100is generated as described herein. In step928, the story structure100is transferred to a device for display and manipulation by an author at step930. If there are changes to make to the story structure110, at step934method900moves to step932to make changes to the story structure100, then moves back to step924to publish the modified narrative structure. If at step934there are no changes, method900ends at step940.

FIG. 10is a block diagram of computer system1000according to an embodiment of the present invention that may be used with or to implement VSS500. Computer system1000depicted inFIG. 10is merely illustrative of an embodiment incorporating aspects of the present invention and is not intended to limit the scope of the invention as recited in the claims. One of ordinary skill in the art would recognize other variations, modifications, and alternatives.

In one embodiment, Computer system1000includes a display device1010such as a monitor, computer1020, a keyboard1030, a user input device1040, a network communication interface1050, and the like. In one embodiment, user input device1040is typically embodied as a computer mouse, a trackball, a track pad, wireless remote, tablet, touch screen, and the like. User input device1040typically allows a consumer to select and operate objects, icons, text, video-game characters, and the like that appear, for example, on the monitor1010.

Embodiments of network interface1050typically include an Ethernet card, a modem (telephone, satellite, cable, ISDN), (asynchronous) digital subscriber line (DSL) unit, and the like. In other embodiments, network interface1050may be physically integrated on the motherboard of computer1020, may be a software program, such as soft DSL, or the like.

In one embodiment, computer system1000may also include software that enables communications over communication network1052such as the HTTP, TCP/IP, RTP/RTSP, protocols, wireless application protocol (WAP), IEEE 802.11 protocols, and the like. In alternative embodiments of the present invention, other communications software and transfer protocols may also be used, for example IPX, UDP or the like.

Communication network1052may include a local area network, a wide area network, a wireless network, an Intranet, the Internet, a private network, a public network, a switched network, or any other suitable communication network. Communication network1052may include many interconnected computer systems and any suitable communication links such as hardwire links, optical links, satellite or other wireless communications links such as BLUETOOTH, WIFI, wave propagation links, or any other suitable mechanisms for communication of information. For example, communication network1052may communicate to one or more mobile wireless devices1002via a base station such as wireless transceiver1072, as described herein.

Computer1020typically includes familiar computer components such as a processor1060, and memory storage devices, such as a memory1070, e.g., random access memory (RAM), disk drives1080, and system bus1090interconnecting the above components. In one embodiment, computer1020is a PC compatible computer having multiple microprocessors. While a computer is shown, it will be readily apparent to one of ordinary skill in the art that many other hardware and software configurations are suitable for use with the present invention.

Memory1070and disk drive1080are examples of tangible media for storage of data, audio/video files, computer programs, and the like. Other types of tangible media include floppy disks, removable hard disks, optical storage media such as CD-ROMS and bar codes, semiconductor memories such as flash memories, read-only-memories (ROMS), battery-backed volatile memories, networked storage devices, and the like.

This example demonstrates using the VSS500to create multimedia graphic novels. This approach is termed as “reverse animatics”. Since panels150may have layers that are static images as well as movies and audio media, such media can be combined together in creating a multimedia experience. Viewer actions such as swipes create state transitions that navigate the viewer through the multimedia story.

This example demonstrates using the VSS500to create interactive visual books. Building on the multimedia graphic novel idea described above, layers160with behaviors can be embedded into individual panels160that cause specific visual elements to transition or be revealed; provide puzzle or gesture tasks that have to be solved to advance the narrative; and mini-games involving the story characters and environment.

This example demonstrates using the VSS500to create personalized story elements. Assuming the user has the ability to create their own images, movies or audio media via html5 or other applications (external to the VSS500), these elements are brought in at the appropriate time in the story by simply replacing the media asset516of a layer160by the corresponding user generated asset. Any behaviors defined on that layer160are still active since only the media pointer attribute has been changed. This provides a very flexible way to personalize the storytelling.

This example demonstrates using the VSS500to author interactive behind the scenes data. DVDs and websites often provide a behind the scenes look at movies, music, architecture, etc. The format for these videos typically involve the artist or creator being interviewed with appropriate cut aways to visual representations of the finished product, supporting artifacts or other visual representations of what the interviewer is referring to. In one embodiment, icon layers160appear over the main interview video layer160at the appropriate time. The viewer can make a choice to “cut away” to this supporting material and stay with it as long as they like. The main interview video can either be paused during this time, continue as voice over or continue to play as a picture-in-picture layout. A viewer can even bring up multiple representations that can play along side each other and the primary video stream.

This example demonstrates using the VSS500to compare multiple time coherent visual streams. When creating visual or diagnostic media, there are often multiple representations that provide a progression towards final result. An example for animation involves the story, layout, animation and final rendered reels. An example for medicine involves physician updates, CTs, MRIs, Contrast studies, etc. Although these individual representations can be of different lengths, it is possible to put them in synch by storing a canonical timestamp within individual samples of each stream. Once this is done, VSS500may be configured to present all the multiple versions with the ability to interactively switch between them or even bring up multiple versions alongside each other for comparison.

This example demonstrates using the VSS500to generate customized music videos. In one embodiment, a music video consisting of multiple shots is processed by VSS500. Some of the shots may contain close ups on the individual musicians, others may contain the band on stage, yet other may contain scenes of the crowd, etc. The VSS may process the shots to generate a presentation of these raw clips to the viewer. In some embodiments, by tapping on a specific clip or type of clips, the viewer can queue up a “live” edit list that determines how the music video will playback. Embodiments also provide the viewers with an option to insert clips of themselves into the music video sequence.

This example demonstrates using the VSS500to generate interactive video ads. Interactive ads include those ads generated by the VSS500where a buyer can tap on a product to get additional information about it or to even change or customize the product to match buyer's interest. One embodiment uses a behavior defined on the main product video layer160. In response to a tap, the behavior would transition to the appropriate state (based on when and where the buyer tapped). The target state in turn would jump to an appropriate product thread that would match the buyer's interest.

This example demonstrates using the VSS500to generate personalized video ads. Similar to the example above, however, the trigger on the main product video layer's behavior could be conditionals that evaluates buyer attributes such as age, sex, geographic location, interests, etc. and jumps to the appropriate product thread120.

This example demonstrates using the VSS500to generate social networking hooks within video streams. Tapping on a product or person presents the user with an option to tweet or post on a social network website a pre-authored, editable message accompanied by the visual image or video. Optionally, when the user is watching a video stream, they would be showed annotation anchors initiated by their friends or networks. These anchor points would be stored in an online database that would be accessed and filtered at viewing time based on the user and video clip. The result of the database query would be turned into overlay layers160that are displayed at the appropriate time in the video stream.

This example demonstrates using the VSS500to generate adaptable video lessons. The main video lesson is broken up into multiple video clips. These video clips are re-constituted into a linear thread120with multiple screens that present each video clip in sequential order. At the end of a clip a new screen is inserted that asks the student specific questions to test understanding. If understanding is verified the narrative moves forward, however, if the student fails the test, they are taken back to the previous lesson screen or even digressed to a related thread that expands the specific topic in slower and greater detail.

This example demonstrates using the VSS500to switch between multiple multi-capture visual streams. Sports and live events are often captured with multiple video streams that are in synch. In our approach the video layer160presenting the video stream can be switched by pressing button layers160, which in turn cause the main video layer160to have a state transition that sets the video layer to the appropriate type or camera. As the layer's video transitions to a new stream, VSS500is able to preserve the time synch using the layer's time code attribute. In another variation, VSS500may use personalized information about the viewer, such as their affinity for a particular player, to preferentially switch to streams that match their interest when the alternate streams have low activity or saliency.

This example demonstrates using the VSS500to create a video blog. Bloggers can use a simple web form to provide a name for the post, meta tags and upload media assets that correspond to a fixed, pre-determined blog structure and look. This information gets populated within a story template to create the finished narrative. In one embodiment, VSS500allows readers to leave their comments to the post in the form of text, audio or video formats.

This example demonstrates using the VSS500to create a customizable television show. This embodiment builds on the video blogging embodiment described herein. Several lifestyle, reality and shopping shows follow a standard format. As an example, consider a classic reality television show where startup companies may pitch their company to a panel of judges. Embodiments of VSS500provide tools for competitors to upload information about their startup using a standardized web form. Via templates, each startup pitch gets converted to a show segment. At viewing time different pitches can be sandwiched between a standard show open and close creating a customized viewing experience. This embodiment allows viewers to watch the show at their own frequency—someone watching the show often would see the latest pitches, others watching less frequently would see the strongest pitches since their last viewing. Also, the show could be tweaked based on the viewer's personal preferences and geo location, which can be incredibly valuable for shopping shows.

This example demonstrates using the VSS500to create targeted political canvasing. Often constituents are mostly concerned with what a candidate thinks about the specific issues most relevant to them. Ideally a candidate would target their message to each individual constituent. Unfortunately this is simply not practical. In one embodiment, a message can at least be personalized. The candidate would first record their position on a large number of key issues as well as a generic opening and closing statement. When a constituent accesses the message, the VSS500would queue up the right set of relevant issues based on their demographic information. This would be implemented as a video layer behavior that uses the global sandbox to implement conditionals that queue up the position clips that are likely to have the most resonance with the viewer. In another variation, VSS500may use the same approach to create messages of varying lengths that maybe most appropriate to the viewing venue. For example, a streaming video ad would be just 30 s while someone coming to the candidate's web site would see a 5 m presentation.

This example demonstrates using the VSS500to allow an author to create a “choose your own adventure books or video”. This embodiment builds on the “Interactive Visual Books” embodiment described herein. An explicit viewer choice or the outcome of puzzles, gesture tasks or mini-games can determine branching in the narrative flow ultimately leading to completely different story outcomes. In this embodiment, the viewer is presented with a linear view and doesn't need to think about navigating in a complex non-linear space.

This example demonstrates using the VSS500to allow an author to create a virtual tour guide. At the start of a museum or facility tour, participants would be handed a tablet. The tablet would track the participant's location using Bluetooth or GPS. As they get to key locations, the VSS500would present the viewer with specific media that provides additional context about the location. The viewer may also use the tablet screen to get an annotated overlay to the physical space.

This example demonstrates using the VSS500to allow an author to collaborate on a story. Stories110are at the heart of large budget films, TV shows and game productions. Narrative scenario planning is at the heart of an even broader set of activities such as marketing and brand campaigns. Generally there is a team of storyboard artists and creative personnel collaborating on a project. At regular intervals the storyboards are shared in the form of a story reel/linear presentation for comments with an even larger group of decision makers. Over time the story may have multiple versions that could be active till a decision is made on a final version. Also, story versions are often spliced from different versions to combine the best elements. In one embodiment, the VSS500is configured to use the thread based, nonlinear narrative structure to store different story versions. Using behaviors and layer interaction, VSS500provides the mechanism to pick between different versions. The VSS500can also provide feedback/annotation tools that integrate note creation right within the story review. Notes maybe viewed/heard (alongside storyboard presentations) by other collaborators on the team with permission controls to modulate access.

This example demonstrates using the VSS500to allow an author to generate a social story cluster. Authors contribute real life or fictional stories. Story panels150are tagged or auto-tagged with specific keywords when appropriate/possible. Tagged keywords can include location, time, famous people & events, emotions, etc. Readers enter the story cluster through a specific thread120that is shared with them by friends or relatives. In navigating through the story110the reader comes to a panel with tagged keywords. Before presenting this panel, the system checks its database for panels in other story threads120with a matching keyword. If a match is found, the current panel is presented to the reader with an option to digress to the alternate story thread. If they decide to follow this new thread, the current thread120is pushed so they can return to it later. In another embodiment, VSS500blurs the line between readers and authors. As a reader is going through a story, they may have a related story of their own to share. The VSS500would allow them to switch to an authoring mode where they create their own story thread. In an embodiment, a permanent bidirectional link may be created between the original thread120and new threads120.

This example demonstrates using the VSS500to allow an author to generate customized views with eye tracking. This builds on the examples of “Personalized Video Ads”, “Customizable TV Shows” and “Targeted Canvassing” described herein. In one embodiment, by incorporating eye tracking as a way to determine the viewers interest elements in the video stream. For example, in a travel video the viewer is initially presented with many different locations either simultaneously (as multiple video layers on the screen) or sequentially. Based on the eye direction, eye darts and frequency of blinks we can establish a correlation to interest in specific locations. Once this is established, the behavior can jump to a thread120of that location.

This example demonstrates using the VSS500to allow an author to generate social, Multi-POV variables. These are the story equivalent of massive, multi-player games. When viewers begin the story110they are assigned a “player” identity, which represents their point of view (POV) within the story. As the story progresses, players maybe asked to make choices that can lead to further refinement of their identity and role in the story110. While the over all story's plot is shared by all players, the specific version of the story110they experience and the information they have is determined by the player's identity. For example, we could have a future world that is undergoing social unrest and revolution. Player would take on the identity of politicians, rebels, soldiers, priests, etc. in this future world. A soldier who makes choices in story navigation that reveals a sympathetic bias towards the rebels may get an identity refinement that may take them on a story path of a double agent. Certain global events—a massive explosion in the kingdom or the defection of a King's General—would be shared knowledge experienced by everyone, however, specific events and information leading up to these global events maybe known only by certain players. In a further enhancement, players may take an image of their identity or some secret document from the story world into their social network (real) world. Alternatively a player may bring a photo or a talisman from their social world into the story world where it may take on specific narrative significance.

This example demonstrates using the VSS500to allow an author to customize ecommerce and merchandising transactions. Insertion of web panels150within the narrative creates a seamless transition from content to point of sale. This embodiment creates a distinct use case for brands looking to tie marketing content with sales. A few examples: 1) a video blog by a well-known fashion blogger would allow the user to tap on various articles of clothing she is wearing and link directly to a webpage where the clothing item can be purchased; 2) an interactive episode of a popular cartoon could insert links to merchandising pages where stuffed toys and videos can be purchased; 3) interactive political applications may be created to profile candidates during elections and would not only allow the user to jump to web pages that dive into detail on various issues, but also include a direct link to a donation page.

Exemplary embodiments are described herein, including the best mode known to the inventors. Variations of those embodiments may become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate, and the inventors intend for the embodiments to be practiced otherwise than as specifically described herein. Accordingly, all modifications and equivalents of the subject matter recited in the claims appended hereto are included as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed unless otherwise indicated herein or otherwise clearly contradicted by context.