Abstract:
A method is described for recording a video. The method comprises receiving one or more user content and then providing a 3D virtual world and a virtual camera having one or more parameters. The optimal 3D flight path of the virtual camera is then determined based on the user content and the virtual camera is then allowed to travel along the optimal 3D flight path and to record the video.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention relates to a method for making a content sensitive video. 
       BACKGROUND 
       [0002]    Software products have been developed for the instant insertion of multimedia content into 3D virtual worlds and for a video recording that produces instantaneously videos, ready for global broadcasting. The video recording is achieved by passing through the 3D virtual world using a virtual camera that is controlled manually. Presently, there are no alternatives for making this form of video recording aside from manual methods. 
         [0003]    Existing software products allow a user to:
       Browse a catalogue of images/videos/audio;   Use the image/video editor to apply image/video processing tools;   Use the video composer to create videos with special effects, transitions, and trimming;   Send the video to the 3D world, manually define flight path, add localized sounds and navigate through the 3D world and record the navigation.       
 
         [0008]    Disadvantages arise in that present conventional methods of making a video recording is not automatic and that the manually defined flight path may not be suitable for, and not catered to the multimedia content provided by the user. Further, manual designation of the flight path is tedious and subject to error. 
         [0009]    Therefore, the object of the invention is to provide a solution that overcomes the above disadvantages or at least provides a novel method for making a video. 
       SUMMARY OF INVENTION 
       [0010]    According to an embodiment, a method is described for recording a video. The method comprises receiving one or more user content and providing a 3D virtual world and a virtual camera having one or more parameters. The optimal 3D flight path of the virtual camera is then determined based on the one or more user content. The virtual camera is then allowed to travel along the optimal 3D flight path and to record the video. 
         [0011]    In another embodiment, the method further comprises the step of modifying the one or more parameters of the virtual camera based on the one or more user content. 
         [0012]    In another embodiment, the step of determining the optimal 3D flight path comprises providing a plurality of 3D flight paths and extracting a plurality of attributes from the one or more user content. Each of the plurality of 3D flight paths are then optimized based on the plurality of attributes. An aesthetic function is then used to calculate an aesthetic quality of each of the optimized 3D flight paths. The optimized 3D flight path with the highest aesthetic quality is the selected as the optimal 3D flight path. 
         [0013]    In another embodiment, the one or more user content comprises one or more multimedia content and the step of optimizing each of the plurality of 3D flight paths comprises selecting one or more segments on the 3D flight path and optimizing the one or more segments based on the plurality of attributes. The 3D flight path is then deformed based on the one or more optimized segments. 
         [0014]    In another embodiment, the step of optimizing the one or more segments comprises deriving a plurality of recommended deformations for the one or more segments based on the plurality of attributes. The recommended deformation is then selected using dynamic programming and the one or more segments are then deformed based on the selected recommended deformation. 
         [0015]    In another embodiment, the one or more user content comprises an audio track. 
         [0016]    In another embodiment, a method is described for creating a new flight path. The method comprises providing a first flight path and a second flight path and concatenating the first flight path to the second flight path to create a new flight path. 
         [0017]    In another embodiment, a method is described for creating a new flight path. The method comprises providing a first flight path and a second flight path, extracting a subpath from the first flight path and concatenating the subpath to the second flight path to create a new flight path. 
         [0018]    In another embodiment, a computer program is described for instructing a computer to perform any of the methods as described herein. 
         [0019]    In another embodiment, a computer readable medium is described having the computer program as described herein. 
         [0020]    The invention will now be described in detail with reference to the accompanying drawings. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0021]    In order that embodiments of the invention may be fully and more clearly understood by way of non-limitative examples, the following description is taken in conjunction with the accompanying drawings in which like reference numerals designate similar or corresponding elements, regions and portions, and in which: 
           [0022]      FIG. 1  is a diagram illustrating a virtual camera on a flight path in a 3D virtual world; 
           [0023]      FIG. 2  is a flow chart illustrating a method to record a video; 
           [0024]      FIG. 3  is a diagram illustrating a relevant discrete point, the preceding discrete point and the subsequent discrete point; 
           [0025]      FIG. 4  is a diagram illustrating two relevant discrete points, and their preceding discrete points and subsequent discrete points. 
       
    
    
     DETAILED DESCRIPTION 
       [0026]    Referring to the drawings,  FIG. 1  shows a flight path  101  of a virtual camera  102  in a 3D virtual world. A particular 3D virtual world can have a plurality of flight paths  101  tailored or associated to it. Virtual camera  102  has a field of view  105 . This field of view  105  is what the virtual camera  102  “sees” and is what the virtual camera  102  records at any one-time. In the 3D virtual world, there are 3D objects, and the surface on a 3D object where multimedia content  104  (which includes images and videos) is displayed is called a screen  103 . 
         [0027]    When moving along the flight path  101 , one or more screens  103  may enter the virtual camera&#39;s  102  field of view  105 . The screens  103  that come into the virtual camera&#39;s  102  field of view  105  in a flight path  101  is henceforth referred to as the screens  103  along the flight path  101 . As the virtual camera  102  travels along flight path  101  and when the screens  103  come into the virtual camera&#39;s  102  field of view  105 , the virtual camera  102  will record the multimedia content  104  which are displayed on the screens  103 . Virtual camera positions  106 ,  107  and  108  are the positions along the flight path  101  where the virtual camera  102  will be momentarily stationary such that it can record the multimedia content  104  on the screens  103 . 
         [0028]    In accordance with a preferred embodiment of the invention,  FIG. 2  shows a method for producing a 2D video. In Step  201 , a software program is provided such that a user can use the software program to upload multimedia content (images and videos) and audio tracks. Alternatively, the software program also provides a database of multimedia content and audio tracks for the user to choose from. 
         [0029]    In step  202 , the software program extracts the attributes of the uploaded or selected multimedia content (images or videos) and audio tracks. For images, the software program runs image processing techniques to extract attributes of the images. Examples of image processing techniques include Face detection, Character Recognition, Geometric shape detection, natural texture labeling. Attributes of the image can be the content of the image for example, the image has “2 faces that is 30×40 pixels wide” and “3 lines of text that is 12 pixels per inch”. For videos, the software program uses video analytical tools to break the video down to a series of frames (akin to images) and extract attributes of these frames. Other attributes of the video like the length of video, and video frame rate are also extracted. 
         [0030]    The software program uses audio and pitch analysis tools to extract audio signal attributes like pitch and tempo. These audio signal attributes can be used by the software program to trigger special effects in the 3D virtual world. The software program can access an entire database of special effects that corresponds to each audio signal attribute. For example, if the audio track is a Mozart piece with a slow tempo and low pitch, the software program will insert butterflies into the 3D virtual world. If the audio track is a rock track with fast tempo and high pitch the software program will insert fireworks into the 3D virtual world. 
         [0031]    In step  203 , the software program provides a database of 3D virtual worlds for the user to select from. Examples of 3D virtual worlds can be a New York City virtual world and a football stadium virtual world. A search function can be provided to the user such that a user can enter his search criteria and the software program will return a filtered list of 3D virtual worlds. For example, a user can enter the word “stadium” into the search function, and the software program returns a basketball stadium 3D virtual world, a football stadium 3D virtual world etc. The 3D virtual worlds that are provided by the software program to the user to select from may also be dependent on the multimedia content uploaded/selected by the user. For example, if the multimedia content consists of 3 images, the software program may only shortlist to the user 3D virtual worlds which have 3 screens. Alternatively, the software program may also modify an existing 3D virtual world by adding or deleting screens to tally the number of screens with the number of multimedia content uploaded by the user, and shortlist this modified 3D virtual world to the user. 
         [0032]    In step  204 , the software program provides a plurality of virtual cameras for the user to select from. The appearance and parameters of the virtual cameras can mimic actual video camera models from Sony, Canon etc. so that users can choose to use virtual cameras that they are accustomed to. The virtual cameras can also have infrared or thermal options such that the filming of the 2-D video is seen to be in the infrared or thermal spectrum. 
         [0033]    In step  205 , the software program filters or shortlists a plurality of possible flight paths out of a database of flight paths. The shortlisted flight paths can include the flight paths tailored to the selected 3D virtual world. The shortlisted flight paths can also include flight paths that are not tailored to the selected 3D virtual world but tailored to other 3D virtual worlds. The shortlisted flight paths can also contain flight paths (out of the shortlisted flight paths) that have been modified by the software program based on certain criteria. For instance, if the number of screens in a shortlisted flight path exceeds the number of multimedia content uploaded by a user, the shortlisted flight path may be modified such that the virtual camera travels to only the screens that display the multimedia content. 
         [0034]    In an instance where the number of screens in a shortlisted flight path is less than the number of multimedia content uploaded by a user, the shortlisted flight path may be modified such that the flight path reroutes back to screens it has already passed such that different multimedia content can be displayed on the same screen across a period of time. To illustrate this with an example, the number of screens in a flight path is two (Screen 1 and 2) and the number of multimedia content uploaded by a user is three (Image 1, 2 and 3). Image 1 and 3 will be displayed on Screen 1 at different points in time and Image 2 will be displayed on Screen 2. The flight path will therefore comprise the virtual camera travelling towards Screen 1 displaying Image 1, and thereafter to Screen 2 displaying Image 2. The flight path will then result in the virtual camera travelling back to Screen 1 displaying Image 3. If the multimedia contents are sequential power point slides, the display order of the multimedia content should be adhered to. 
         [0035]    For each flight path, the virtual camera will have a set of default virtual camera parameter values (such as speed, pan and tilt) at different regions of the flight path. In step  206 , the virtual camera parameters can be adapted based on the 3D virtual world chosen. The pan and tilt of the virtual camera should be changed such that the screens would come into the field of view of the virtual camera. The speed of the virtual camera should also be changed in light of the positions of the screens. For example, at the regions of the flight path where the screen is partially inside or inside the virtual camera&#39;s field of view, the speed of the virtual camera is slowed down and the virtual camera may take a stationary position at the front of the screen. 
         [0036]    The time taken in the flight path should adhere to the duration of the selected audio track. Therefore, the speed of the virtual camera as it travels through the flight path can be modified such that the time taken in the flight path adheres to the duration of the selected audio track. The virtual camera parameters are also modified in light of the extracted attributes of the multimedia content. If the extracted attributes of the multimedia content is that it contains images of small text, the zoom of the virtual camera may be increased. 
         [0037]    In step  207 , the software program deforms the discrete points of the shortlisted flight paths based on the extracted attributes of the images and videos. The software program does this by first selecting discrete points on the shortlisted flight paths for the deformation.  FIG. 3  illustrates this. In  FIG. 3 , there is selected 3D virtual world  300 , shortlisted flight path  301 , and multimedia content  302  shown on screen  303 . The software program would select the most relevant discrete points along the shortlisted flight path  301  for deformation i.e. points on the shortlisted flight path  301  where the screen  303  is in the field of view of the virtual camera  304 . This is because these discrete points with have the most impact on the filming by the virtual camera  304  of the multimedia content  302  that will be shown on the screens  303 . 
         [0038]    In this illustration, discrete point  305  is chosen for deformation. The segment of the flight path (can be in non-linear or linear) between the preceding discrete point  306  and the subsequent discrete point  307  is analyzed for deformation. The software program performs this analysis by first accessing a table of predefined attributes with magnitudes and their corresponding recommended deformations as shown in Table 1 below. 
         [0000]    
       
         
               
             
               
               
               
               
             
           
               
                 TABLE 1 
               
             
             
               
                   
               
               
                 Table of predefined attributes with magnitudes and  
               
               
                 their corresponding recommended deformations 
               
             
          
           
               
                   
                 Predefined  
                   
                 Recommended  
               
               
                   
                 Attribute 
                 Magnitude 
                 Deformation 
               
               
                   
                   
               
               
                   
                 A 
                 1.0 
                 
                           
                 
               
               
                   
                 A 
                 0.9 
                 
                           
                 
               
               
                   
                 A 
                 0.8 
                 
                           
                 
               
               
                   
                 • 
                 • 
                 • 
               
               
                   
                 B 
                 1.0 
                 — 
               
               
                   
                 B 
                 0.9 
                 
                           
                 
               
               
                   
                 B 
                 0.8 
                 
                           
                 
               
               
                   
                 • 
                 • 
                 • 
               
               
                   
                 • 
                 • 
                 • 
               
               
                   
                 Z 
                 • 
                 • 
               
               
                   
                   
               
             
          
         
       
     
         [0039]    Table 2 below shows the predefined attributes. 
         [0000]    
       
         
               
             
               
               
               
             
           
               
                 TABLE 2 
               
               
                   
               
               
                 Predefined Attributes 
               
               
                   
               
             
             
               
                   
               
             
          
           
               
                   
                 Predefined Attribute A  
                 Size of face is 30 × 50 pixels 
               
               
                   
                 Predefined Attribute B  
                 Size of face is 40 × 60 pixels 
               
               
                   
                 Predefined Attribute C 
                 Size of text is 12 pixels per inch 
               
               
                   
                 . 
                 . 
               
               
                   
                 . 
                 . 
               
               
                   
                 Predefined Attribute Z 
                 . 
               
               
                   
                   
               
             
          
         
       
     
         [0040]    For each predefined attribute, and at a particular magnitude, there is a recommended deformation for the segment of the flight path between the preceding discrete point  306  and the subsequent discrete point  307 . Magnitude is a number between 0 and 1, and is a gauge of the extent of how close the extracted attribute of the image or video matches the predefined attribute. The predefined attributes, magnitudes and corresponding recommended deformations are trained data that are stored in a database which the software program can access. 
         [0041]    Based on the extracted attributes from multimedia content  302 , the magnitude of the predefined attributes for the segment of the flight path between the preceding discrete point  306  and the subsequent discrete point  307  are determined. With the magnitudes, the recommended deformations for each predefined attribute are obtained by referencing Table 1. This is shown in Table 3 below. 
         [0000]    
       
         
               
             
               
               
               
               
             
               
               
               
               
             
           
               
                 TABLE 3 
               
             
             
               
                   
               
               
                 Recommended deformations for each  
               
               
                 predefined attribute for a segment 
               
             
          
           
               
                   
                 Predefined 
                   
                 Recommended 
               
               
                   
                 Attribute 
                 Magnitude 
                 Deformation 
               
               
                   
                   
               
             
          
           
               
                   
                 A 
                 0.8 
                 
                           
                 
               
               
                   
                 B 
                 0.8 
                 
                           
                 
               
               
                   
                 C 
                 0.5 
                 
                           
                 
               
               
                   
                 D 
                 0.1 
                 
                           
                 
               
               
                   
                 • 
                 • 
                 • 
               
               
                   
                 Z 
                 • 
                 • 
               
               
                   
                   
               
             
          
         
       
     
         [0042]    The recommended deformations are then normalized (averaged) 1/ZΣrecommended deformation to obtain a plurality of normalized deformations for the segment of the flight path between the preceding discrete point  306  and the subsequent discrete point  307 . Each normalized deformation for the segment is assigned a local aesthetic score (between 0 and 1). The process is then repeated for each relevant discrete point. 
         [0043]    In step  208 , the shortlisted flight path is optimized. This optimization is done by selecting a normalized deformation with the best overall score. The overall score for each normalized deformation is calculated by weighting the local aesthetic score of a normalized deformation with a transition score. A transition score (between 0 and 1) is the score given to the suitability of two consecutive normalized deformations. The concept is that the overall deformation of the flight path must be considered as opposed to just considering the local (segment) deformations in isolation.  FIG. 4  shows a 3D virtual world  400 , a shortlisted flight path  401 , and screens  402 ,  403  and  410 . In this case, relevant discrete points  405  and  408  are selected for deformation. Therefore, the segment of the flight path between discrete points  404  and  406  will be analyzed based on the multimedia content displayed on screen  402 , the segment between discrete points  407  and  409  will be analyzed based on the multimedia content displayed on screen  403 , and the segment of the flight path between discrete points  411  and  413  will be analyzed based on the multimedia content displayed on screen  410 . 
         [0044]    In step  209 , an optimal aesthetic function is used to calculate the aesthetic value of the optimized shortlisted flight paths, and the optimized shortlisted flight path with the highest aesthetic value is chosen as the optimal flight path. The optimal aesthetic function uses a dynamic programming technique (Bellman) where a weighted Direct Acyclic Graph (DAG) is constructed. In the DAG, the vertices are the local deformation hypothesises {X 11 , X 12 , . . . X 1n , X 21  . . . X 2n , . . . X k,n } associated, respectively to the segment to be deformed {X 1 , X2, . . . . X k } and the edges linking two consecutive hypothesises (X in , X jn ) are weighted by the likelihood of the transition. 
         [0045]    In step  210 , the virtual camera travels on optimal flight path to record the 2D video. 
         [0046]    One skilled in the art can appreciate that one advantage conferred by the described invention is that it provides a relatively simple means of producing unique videos. That is, users with two different sets of multimedia content using the same 3D virtual world will obtain two different videos (for example, an end-user, an advertiser, or an event organizer). 
         [0047]    One would also appreciate that an online provisioning of the described invention would allow near immediate production and rendering of a video construction for customers such as MTV or advertisers at a very low cost. A functional production engine takes multimedia content and a 3D virtual world and automatically performs customized rendering of the flight path. In alternative embodiments, the production technology could be provided in an online-only version, where finished productions are offered for sale, a stand-alone version that is sold to end users, or a combination stand-alone version and on-line version that can be expanded through the use of information collection to generate metrics and statistics associated with, for example, tool use, that would provide recommendations and the like and allow for optimized tool offering and other enhancements. Metrics and statistics and other feedback generated from, for example, pre-completed productions with standard path assignments in combination with user feedback could also be used to improve product quality, could be offered in connection with other information, such as demographic information, or the like to generate marketing information, or the like. 
         [0048]    Another advantage is that the optimization of the flight path is automatically done according to intelligent, content sensitive, parameters such that meaningful videos can be produced. In an embodiment, the inventive exemplary method and apparatus described herein be used for 3D PowerPoint presentations (companies), 3D consumer electronics ads (Blackberry video), 3D event promotions (MDA, TechVenture 2010), music promotions (Viacom, MTV, Sony, EMI), books promotions (Amazon.com), concerts ticketing (Guns and Roses video) and sports events (World Cup video). Still further, the invention can also be used for daily changes of advertisements and for user-generated advertisements. Globally, implementations in accordance with embodiments can accelerate the productions of high-impact videos for event promotion, online visibility increase and immediate, impactful, viral marketing. For example, in fast paced countries such as Singapore, which is in the process of going all 3D for many advertising, promotional and other content, the methods described herein can enable users to produce their own customized videos, with their own customized 3D paths and customized moods, computed from the analysis of the multimedia content that they upload in the 3D world. Embodiments described herein can also be used in technology for e-Advertisers, web, e-cards, YouTube, Picassa, PowerPoint, or the like. 
         [0049]    Another aspect of the invention is how new flight paths are created to build up the pool of flights paths in the database. One way is to concatenate existing flight paths together to create new flight paths. Another way is to extract sub-paths from existing flight paths and then concatenate the sub-paths to an existing path to create new flight paths (referred to as splicing). The optimized flight paths can also become new flights paths in the database. 
         [0050]    There are instances where a 3D virtual world is modified. In such circumstances, the flights paths in the database that are tailored or associated to the modified 3D virtual world must also be modified accordingly. For example, if in the 3D virtual world, a new virtual building is created, the associated flight paths must be altered such that they do not “go through” the new virtual building but “go around” it. 
         [0051]    While exemplary embodiments pertaining to the invention have been described and illustrated, it will be understood by those skilled in the technology concerned that many variations or modifications involving particular design, implementation or construction are possible and may be made without deviating from the inventive concepts described herein.