Patent Description:
In particular, the present invention relates to the interactive creation of original audiovisual content inspired by listening to music and achieved through instant assembling of pre-synchronized video clips by means of a number of novel assembling methods also designed to preserve synchronization of the video clips to a common soundtrack; wherein availability of the video clips, beside depending from a previous authoring process, can also depend on reward criteria, possibly based on real-time analysis of user's interactive performance and ability in spotting key-images. For this reason, the present invention is not only useful and practical in the field of video editing applications, but above all it is useful and practical in the field of audiovisual applications offering a gaming experience.

Over the past two decades, the concept of nonlinear video clips editing, along with uninterrupted progress of the underlying enabling technologies, has revolutionized the world of professional and amateur video making, providing powerful and reliable tools to select, assemble, refine, add effects, pre-visualize and finalize video clips. As a consequence of this, a plethora of both mobile and desktop hardware platforms and software applications are available today, providing advanced video editing tools and functions to a wide audience of users in many application contexts.

Most of these known solutions implement offline editing of available content (i.e. a type of editing not performed simultaneously with the creation of content) through a consolidated operational paradigm according to which a certain selection of video clips (possibly, subject to normalization of the aspect ratio and appropriate redefinition of the duration) can be freely assembled in the desired order within a timeline, or through the use of a variety of transition effects.

The provisional result can be pre-visualized (typically with reduced resolution and quality) at the desired frame rate and refined through further processing steps or adjustments of various parameters and, finally, the result of this process can be finalized obtaining a full quality video in a variety of digital formats (both compressed and without compression).

Some known video editing systems also provide real-time (often limited) editing capabilities, such as the instant insertion of a clip, or may provide the possibility of recording and inserting live content.

Simplified versions of the known video editing systems are also used for the creation of slideshows in which the author can specify which images to include and an audio file to be reproduced as a soundtrack. In this type of application, the visualization order and duration of each image can be predetermined, randomly assigned or specified by the user during a video editing session that often requires a long and repetitive process.

Some known systems and methods for sampling audio-musical backing tracks (such as <CIT>) enable the assembling of various music video clips in real time by concatenating them manually. In this case, the video clips assembling method does not offer any reward mechanism based on user performance, nor does it support any assembling method based on the pre-synchronization of video clips with each other and with an audio track, and nor does it support any assembling method based on the off-sync insertion of pre-indexed segments of video clips associated to a tag.

Some known systems and methods (such as <CIT>) enable the recording of a number of video clips, each associated to a corresponding thumbnail, to be used for on-the -fly assembling process by tapping on the thumbnails. Even in this case, the video clips assembling method does not offer any reward mechanism based on user performance, nor does it support any assembling method based on the pre-synchronization of video clips with each other and with an audio track.

Some known systems and methods (such as <CIT>) enable gesture-based video clips editing through a preview display showing the resulting in-progress video. However, these systems rely on conventional non-linear editing approaches which do not offer any reward mechanism based on user performance, nor does it support any assembling method based on the pre-synchronization of video clips with each other and with an audio track and assembling method based on the off-sync insertion of pre-indexed video clips segments associated to a tag. In light of the above, none of the known systems described above is aimed at enabling a video creation process based on an impromptu user performance comparable to a gaming experience.

An object of the present invention is to devise a system and a method for the instant assembling of video clips through user performance, where user performance can be analyzed in real-time and, based on this analysis, a reward method can unlock access to, or even suggest, additional video clips.

Moreover, another object of the present invention is to devise a system and a method for instant assembling of video clips through user's interactive performance where each single video-making session results to be unique, being based exclusively on user's instant visual-acoustic perception and consequent gestural interaction, as well as on the availability of specific video clips which in turn can depend on the unique interaction pattern performed by the user.

All available video clips, any audio content, any performances already prepared and various additional information, are specified by a data-pack that can be transferred to the device, purpose of the invention, through a number of modes which include, but are not limited to computer networks, or through local authoring procedure. The aforementioned data-pack complies with a multimedia representation format called PBVF (Performance Based Video Format), which formally defines the types and data structures necessary to support the functionalities of this system and method for the reproduction, recording and sharing of audiovisual content.

In an embodiment of the system for the instant assembling of video clips through user performance, said device further comprises a speaker configured to reproduce any audio content, said I / O unit being configured to access said audio content in local or through said computer network, said processing unit and said memory being configured to record said audio content in combination with said video clips, comprising said plurality of video clips segments extracted from said plurality of video clips.

In one embodiment of the invention, the system for instant assembling of video clips through performance further includes:.

In an embodiment of the invention, the system for instant assembling of video clips through user performance further comprises a web authoring and publishing platform on the cloud server, configured for the creation of a data-pack that defines said plurality of video clips, said possible audio content, said sequence of manual assembling commands and other functions, and for the publication of said data-pack.

In an embodiment of the invention, the system for instant assembling of video clips through user performance further comprises a local authoring procedure on said device, configured for the creation of a data-pack that defines said plurality of video clips <NUM>, said any audio content <NUM>, said sequence of manual assembling commands <NUM>, <NUM>, <NUM> and other functions.

In the present invention, the system for instant assembling of video clips through user performance comprises a method for enabling a hidden portion <NUM>, <NUM> of said plurality of video clips and of said audio content, wherein this hidden portion <NUM>, <NUM> is made accessible according to reward criteria based on the real-time analysis of said sequence of manual assembling commands.

In an embodiment of the system for the instant assembling of video clips through user performance according to the invention, the user interface means of the device are configured according to an operating mode called TAG_BASED, with related commands and associated natural user interface, which allows the selective insertion of video clips segments extracted from said plurality of video clips, previously indexed during the data-pack authoring procedure, through manual annotation or through automatic image analysis method.

In an embodiment of the system for the instant assembling of video clips through user performance according to the invention, said user interface means of said device are configured according to an operating mode called TAIL_REPEAT, with relative commands and associated graphic layout, which allows to repeat a tail segment of a video clips, currently displayed and recorded, possibly altering the order of the frames.

In an embodiment of the invention, the system for instant assembling of video clips through user's performance can operate without any audio content for the purpose of assembling said plurality of video clips extracted from said plurality of video clips through user performance, the display order of the video clips being defined by said sequence of manual assembling commands.

In one embodiment of the invention, the method for instant assembling of video clips through user performance further includes the steps which consist of:.

In one embodiment of the invention, the method for instant assembling of video clips through performance further includes the steps which consist of:.

In one embodiment of the invention, the method for instant assembling of video clips through user performance is based on a previously authored data-structure, hereafter referenced as data-pack, to determine the number and the kind of available video clips along with the audio track these video clips are synchronized to and the number and type of available assembling algorithms and related interface layouts, according to the steps of:.

In one embodiment of the invention, the method for instant assembling of video clips through user performance further includes the steps of:.

In the present invention, the method for instant assembling of video clips through performance comprises a procedure for enabling a hidden part of said plurality of video clips and of said audio track, made accessible according to reward criteria based on the real-time analysis of said sequence of manual assembling commands.

Further characteristics and advantages of the invention will become more evident from the description of some preferred but not exclusive embodiments of the system and method for instant assembling of video clips by means of user performances according to the invention, illustrated as non-limitative examples with the aid of the attached drawings, in which:.

The present invention relates to a system and method to assemble a plurality of video clips by means of a set of manual controls which allow the user to instantly select which of the available video clips <NUM>-<NUM><NUM>-<NUM>, and video clips segments <NUM> is to be appended, simultaneously with the reproduction of an optional audio track. Video clips availability may depend on reward criteria based on specific algorithms for real-time user performance evaluation.

In this system and method, the sequence of manual assembling commands, which constitutes the interaction performed by the user, is represented univocally through a sequence of indexes which allow a complete reconstruction of the sequence of frames displayed during user performance and the reproduction of the said final assembled video <NUM>, as well as to export this final assembled video to the most popular digital video formats for sharing.

This univocal representation of the sequence of assembling manual commands, i.e. the interaction performed by the user, complies with the specifications of an original multimedia format, also part of the present invention, which formally defines the number and types of video clips, the audio track, any previously defined performance, the user interface and its appearance, the audiovisual effects applicable to the flow of images, as well as any reward criteria based on the evaluation of user interaction pattern and other mechanisms detailed below, which determine the availability of extra video clips otherwise not viewable.

In this system and method, the interaction performed by the user, that produces a new and unique sequence of frames <NUM>, is made through a natural user interface that allows instant choice of the video clips to be displayed at any given time starting from a series of options displayed through the graphical interface, in order to facilitate the creative process.

The available video clips, the content or audio track, and the effect of some of the commands present in the user interface, are defined by a data-package that can be transferred to the device according to the present invention through a multiplicity of modes that can include connections to Wi-Fi or mobile networks, Bluetooth or cable point-to-point connections.

The aforementioned data-package complies with a multimedia representation format called PBVF (Performance Based Video Format), as mentioned before part of the present invention, which formally defines the types and data structures necessary to support the features offered for reproduction, recording and sharing of audiovisual content.

In general, a performance recording phase begins with the display of one of the available video clips <NUM>-<NUM><NUM>-<NUM>, and the simultaneous reproduction of the optional associated audio track.

The system for the instant assembling of video clips through user performance according to the present invention is configured to allow the reproduction of an audiovisual content defined in the PBVF multimedia format, described in greater detail later in this document, i.e. the generation of an original video clips through a user performance based on the use of a natural user interface and the simultaneous listening of an optional audio track, which determines the immediate assembling of multiple video clips available in a PBVF data-pack.

With reference to <FIG>, the system for the instant assembling of video clips according to the invention includes a local electronic device <NUM>, operated by the user performing the performance, comprising at least the following local hardware components:.

In one embodiment, the system for the instant assembling of video clips according to the invention can further comprise the following remote hardware components:.

In one embodiment, the system for the instant assembling of video clips according to the invention can further comprise the following remote hardware component:.

With reference to <FIG>, the device <NUM> of the system for the instant assembling of video clips according to the invention includes the following software modules:.

As part of the present invention, the process of reproducing or creating a new final assembled video <NUM> strictly requires the availability, through a computer network, such as the Internet, of a data-pack in PBVF format.

Alternatively, this data-pack can be created through the procedure schematically illustrated in <FIG> and <FIG> and described in greater detail later in this document, in particular in the section "Local Authoring of PBVF data-packs".

Each data-pack typically includes one or more preview or preview images of its content, one or more performances, i.e. a defined sequence of commands related to a list of video clips determining a final assembled video <NUM>, a variable number of video clips <NUM>-<NUM> and an optional content or audio track <NUM>, possibly available in streaming mode on a web remote server. Starting from a data-pack, the user can choose whether to play the audiovisual content in the Performance Player <NUM> operating mode via the command interface shown in <FIG>, or record an original PBVF video in the Performance Recorder <NUM> operating mode through the command interface shown in <FIG>, <FIG> and <FIG>.

The command interface shown in <FIG>, relating to the operating mode Performance Player <NUM>, includes a screen or display <NUM>, a command <NUM>, for example with the "play" or "start" label, to start playback of the video performance, and the video performance <NUM> itself.

This operating mode allows only the passive reproduction of a final assembled video <NUM>, resulting from the assembling of multiple video clips <NUM>-<NUM><NUM>-<NUM> according to a performance previously defined in the data-pack which does not require any user interaction. All said video clips <NUM>-<NUM> and audio <NUM>, accessible or hidden, can be managed in streaming mode, by a special processing and networking architecture represented schematically in <FIG>, <FIG> and <FIG>.

If you choose to play a video in PBVF format, the following list includes the main operating steps required by the procedure, relating to the Performance Player <NUM> operating mode, as shown in the schematic representation in <FIG>:.

The command interface shown in <FIG>, <FIG> and <FIG>, relating to the Performance Recorder <NUM> operating mode, includes a screen or display <NUM>, a command <NUM>, for example with the "start" label, to start recording the video performance, a preview <NUM> of the same video performance, and a set of additional commands related to their respective functions.

The commands relating to the basic operating mode <NUM> and <NUM>, always present in the user interface (<FIG>, <FIG> and <FIG>), are configured to change the current display by instantly switching to the next video, or to the previous one, storing simultaneously with each new controls the index associated with the last video clips viewed, the duration of playback of this content and any audio / visual effects applied, such as for example a zoom <NUM> on a particular part of the video.

The selection of a new content to be concatenated instantly can also take place through a series of additional commands <NUM>-<NUM>, <NUM>-<NUM> present in different configurations of the user interface of the aforementioned device and relating to three different types of special functions.

The commands relating to the first type of special functions <NUM>-<NUM> (<FIG>), <NUM>-<NUM> (<FIG>) are associated with hidden video clips, specified in the data-pack in PBVF format, and made accessible to the user according to reward criteria based on the real-time evaluation of the performance currently performed by the user, such as the temporal consistency between the timing of the commands interactively operated by the user and the rhythm of the audio track, or even the correspondence between the sequence of the selected clips and a reference sequence defined on the data-pack itself.

Other reward criteria may take into account previous performances or points earned by the user during activities supported by the authoring and publishing web platform <NUM>, or the knowledge of a password provided by the creator of the data-pack enabling a cheat mode.

Another reward criterion (<FIG>) may consist in the visualization of a particular segment of a video clip <NUM>-<NUM> or a particular segment of hidden video clip <NUM>-<NUM> or audio segment to be searched and tapped on <NUM>-<NUM> during the recording performance among the available video clips. Further reward criteria may be available, or can be defined, extending the previously mentioned examples.

The commands relating to the second type of special functions <NUM>-<NUM>, always present in the user interface (<FIG>, <FIG> and <FIG>), are associated with specific segments of the video clips <NUM>-<NUM> previously indexed during the data-pack authoring phase through manual procedure or through an automatic method of analyzing the image, which is not the subject of the present invention and can be implemented on a plurality of both desktop and mobile processing devices.

The commands relating to the third type of special functions <NUM>-<NUM> (<FIG>) change the display order of the current video clips <NUM>-<NUM> , with reference to the last segment displayed.

If one chooses to generate an original PBVF video through a user performance, the following list includes the main operating steps required by the procedure, relating to the Performance Recorder <NUM> operating mode, as shown in the schematic representation in <FIG>:.

The user interface means <NUM> of the present invention belong to the category of natural interfaces which includes, but is not limited to, touchscreen devices, gesture recognition systems and similar with variable configuration of available functions and graphic layout, according to specific specifications in the PBVF data-pack.

These different command and control configurations enable alternative methods for the instant selection of the video clips to be assembled during the performance, which belong to four basic types of user interface means <NUM>:.

All the configurations of the user interface means <NUM> have in common a set of functionalities and relative basic commands, among which the commands <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> which correspond to the BASIC mode. In addition to this, based on the specifications contained in the PBVF data-pack, additional commands that enable the specific functions of the other modes may be available in various combinations. All the functions relating to each of the aforementioned modes, as well as the individual commands associated with them in the user interface, are detailed below.

The BASIC video clip assembling method comprises the commands <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> of the user interface means <NUM>, shown in <FIG>, <FIG> and <FIG>.

The REWARD operating mode comprises the commands <NUM>, <NUM>, <NUM>, <NUM>,<NUM>-<NUM> of the user interface means <NUM>, illustrated in <FIG>,<FIG> and <FIG>.

The TAG_BASED operating mode based on an off-sync assembling method, enables the insertion of pre-indexed segments of video clips selected according to their content, within the basic video clips assembling flow of the pre-synchronized video clips.

The TAG_BASED operating mode includes the commands <NUM>, <NUM>, <NUM>, <NUM>, <NUM> (in short, 50X where X = <NUM>,<NUM>,<NUM>,<NUM>,<NUM>) of the user interface means <NUM>, illustrated in <FIG>, <FIG> and <FIG>.

GOTO_TAG_X 50X: jump from the current frame i of the current video clips j to frame <NUM> of the first video excerpt associated with the 50X command (operation exemplified in <FIG>) and related explanatory icon. The video excerpts, previously indexed either through manual annotation techniques or through automatic logic of visual content analysis, are univocally defined by a video clip_index, a segment_start index, a segment_end index and an icon exemplifying the content displayed on the interface, where all these data are specified in the PBVF data-pack. In the case of multiple video excerpts available for the same command 50X and related explanatory icon, each time the command 50X is operated through the same content icon, this action triggers the passage from the current frame i of the current video clips j to the frame <NUM> of the next available video segment associated to that icon. Whenever all the video excerpts related to the same content icon have already been displayed, the next available video excerpt will be the first in the list specified in the PVFB data-pack. If the video excerpt corresponding to the 50X command has ended, but not the main audio track, the visualization of the previous video clips is resumed from current frame i+<NUM>+(excerpt_duration in frames) corresponding to the time elapsed during the execution of the 50X command, thus preserving the audiovisual synchronization.

The TAIL_REPEAT operating mode appends a tail portion of the current video clip, to the video clip itself.

The mode called TAIL_REPEAT comprises the controls <NUM> and <NUM> of the user interface means <NUM>, illustrated in <FIG>.

The AUDIO / VIDEO PROCESSING mode comprises the command <NUM> of the user interface means <NUM>, illustrated in <FIG>, <FIG> and <FIG>.

CLOSE-UP or ZOOM <NUM> - a crop-zoom of the current video clips is performed, compatibly with the constraints of the video format, based on the screen coordinates (x, y) of the point selected by the user. These coordinates become the geometric center of a rectangular region with the same aspect ratio as the current video clips but with a side size equal to <NUM>% of the same. This region is enlarged by interpolation by replacing the total image with a detail. In the event that, based on the point selected, the rectangular region to be enlarged was partially outside the frame, it will be translated in the appropriate direction to make it fall within the limits of the same frame.

The PBVF (Performance Based Video Format) multimedia format, as also mentioned part of the present invention, is a data structure capable of specifying a set of information required for the operation of the present invention.

This information is represented, in the PBVF format, through an ordered list that can include a variety of data types, including:.

Below is an example table of the ordered list of the PBVF multimedia format, which as mentioned is part of the present invention.

Below is a formal definition of the ordered list of the PBVF multimedia format, which as mentioned is part of the present invention. PBVFFormat ::= SET {
sources SEQUENCE OF Source,
audio OPTIONAL Audio,
cover OPTIONAL Content,
performances OPTIONAL SEQUENCE OF Performance,
defaultPerformance OPTIONAL INTEGER,
videoSegments OPTIONAL SEQUENCE OF VideoSegment,
specialKeys OPTIONAL SEQUENCE OF SpecialKey
}
Source ::= SEQUENCE {
sourceType SourceType,
content Content,
offset OPTIONAL INTEGER
}
<IMG>
Audio ::= SEQUENCE {
content Content
offset OPTIONAL INTEGER
}
Content ::= CHOICE {
local OCTET STRING,
remote IA5 String
}
<IMG>
<IMG>
<IMG>
<IMG>
<IMG>.

As part of this PBVF format, "Performance" is defined as an ordered list (sequenceOf) of triplets, in which:.

A PBVF data-pack constitutes a single instance of the data structure defined by the format itself, which can be used both as a basis from which to record a new original performance, and as a formal descriptor of the characteristics of an audiovisual content, to be reproduced passively. As such, a PBVF data-pack can be created by third parties through an authoring process on the web platform, or be produced locally on the device <NUM> according to the invention, through this system and method as described in the following section.

The local authoring software component Data-pack Creator <NUM> of the present invention is configured for the creation, directly on the device <NUM>, of data-packs compliant with the PBVF format, starting from a list of a source videos and audio track available locally and through an original procedure which includes the following steps, illustrated in <FIG> :.

The data-packs thus obtained can be enabled for the reproduction of default Performance <NUM>, or for the recording of new unique performance starting from the video clips <NUM>-<NUM> added with this PBVF data-pack authoring procedure.

<FIG> illustrates a schematic representation of a possible control interface in the assembling phase (also shown in <FIG>) of segments of a sourcevideo during the local authoring of a PBVF data-pack, comprising a screen or display <NUM>, a command <NUM> to start playback of a selected source video, a command <NUM>, for example with the "start selection" label, to Start/Stop master audio track reproduction <NUM> and current video segment in-point/end-point setting during the reproduction of current source video.

As mentioned above, the selection and addition of new video clips to be added to the PBVF data-pack is based on an original procedure which includes the following steps, illustrated in <FIG>:.

In summary, the essence of the present invention consists in offering the user a tool for generating an original final assembled video <NUM> by instant assembling of video clips while listening to an audio track. The final assembled video is the result of an interactive selection of segments of video clips extracted from various video clips available in a data-pack, operated by the user through commands available in the user interface, which determines a final assembled video sequence of segments of video clips <NUM> named Performance.

In practice, it has been found that the invention fully fulfills the intended aim and objects. In particular, it has been seen how the system and the method for instant assembling of video clips through user performance thus conceived allow to overcome the qualitative limits of the known art, as they allow to obtain better results than those obtainable with known solutions and/or similar results at lower cost and with higher performance.

An advantage of the system and of the method for the instant assembling of video clips through user performance according to the present invention consists in the fact that they foster and allow a creative and music-inspired approach to the making of the videos, being designed for this purpose as regards the features offered, operational paradigms and user interface.

Another advantage of the system and method for instant assembling of video clips through user performance according to the present invention consists in the fact that they provide specific means and methods for the creation of an original video through the real-time assembling of video clips operated through a user performance.

A further advantage of the system and method for the instant assembling of video clips through user performance according to the present invention consists in the fact that user performance can be evaluated according to a Reward method, possibly allowing access to additional audiovisual content.

Furthermore, an advantage of the system and of the method for the instant assembling of video clips through user performance according to the present invention consists in the fact that every single recording session made by the user is unique, being based exclusively on instant visual-acoustic perception, by the same user, on a gestural interaction that is never perfectly repeatable and which can affect the availability of audiovisual content.

Although the system and method for instant assembling of video clips through user performance according to the invention have been designed in particular for field of audiovisual applications offering a gaming experience, they can still be used, more generally, for the field video editing and digital advertising applications, configuring the PBVF as a new type of interactive video ads format.

Claim 1:
A system for instant assembling of video clips through user performance, comprising a device (<NUM>) operated by a user, wherein said device (<NUM>) comprises:
- user interface means (<NUM>) configured for input and output interaction with said user;
- a processing unit (<NUM>) and a memory (<NUM>) configured for creating a video content item (<NUM>) by assembling one after the other a plurality of video segments extracted from a plurality of source video content items (<NUM>-<NUM>) specified in a data pack; and
- an I/O unit (<NUM>) configured for access to said data pack, said plurality of source video content items (<NUM>-<NUM>) specified in a data pack, and an optional audio content item (<NUM>) specified in said data pack, locally or via a telematic communication network;
said user interface means (<NUM>) are configured to detect a sequence of manual assembling commands (<NUM>-<NUM>, <NUM>-<NUM>) performed by said user, and to play one after the other said plurality of video segments extracted from said plurality of source video content items (<NUM>-<NUM>), the display order of the video segments being defined by said sequence of manual assembling commands (<NUM>-<NUM>, <NUM>-<NUM>);
said processing unit (<NUM>) and said memory (<NUM>) are configured to record one after the other said plurality of video segments extracted from said plurality of source video content items (<NUM>-<NUM>), resulting in said video content item (<NUM>), the recording order of the video segments being defined by said sequence of manual assembling commands (<NUM>-<NUM>, <NUM>-<NUM>);
where the detection of a manual assembling command (<NUM>-<NUM>, <NUM>-<NUM>) induces the transition from a first source video content item (<NUM>-<NUM>), currently displayed and recorded to a second source video content item (<NUM>-<NUM>),
characterized in
further comprising a logic system for enabling a hidden portion of said plurality of source video content items (<NUM>-<NUM>) and of said optional audio content item (<NUM>) specified in said data pack, made accessible according to reward criteria based on real-time analysis of said sequence of manual assembling commands (<NUM>-<NUM>, <NUM>-<NUM>).