Abstract:
The present invention pertains to an apparatus and method for adding a graphic element, such as a subtitle to selected locations of the frames in a 3D movie. The authoring tool receives a depth map indicating the position of various objects in the frames of 3D content along a Z-axis. The authoring device then designates a position for at least one additional graphic element in at least some of the frames, these positions being determined in relation either to the positions of the objects or the position of the screen along said Z-axis. An encoder uses parameters from the authoring tool to reauthor the 3D movie by adding the graphic content to the positions designated by the parameters.

Description:
RELATED APPLICATIONS 
     This application claims priority to U.S. provisional application Ser. No. 61/371,901 filed Aug. 9, 2010. This application is a continuation-in-part of U.S. application Ser. No. 13/173,671 filed on Jun. 30, 2011, entitled METHOD AND APPARATUS FOR GENERATING ENCODED CONTENT USING DYNAMICALLY OPTIMIZED CONVERSION and claiming priority to U.S. Provisional patent application Ser. No. 61/360,088 filed on Jun. 30, 2010, all applications being incorporated herein by reference in their entirety. 
    
    
     BACKGROUND OF THE INVENTION 
     A. Field of Invention 
     This invention pertains to a method and apparatus of enhancing a 3D movie by adding to it new content, such as subtitles, at a particular depth. 
     B. Description of the Prior Art 
     In the past, 3D movies or other episodic 3D content were recorded optically on film and then distributed in analog formats. However, advances in digital signal processing make it possible to provide such content in digital formats which have several well known advantages over the analog formats. 
     In the present application, the term ‘digital 3D movie’ is used to refer to 3D movies, episodic content, or other 3D multi-media content recorded, produced and/or converted into a digital format. This includes content produced in 2D and then post-produced from 2D to 3D, as well as rendered from 3D animation systems. 
     Digital 3D digital movies may be produced by converting the analog content (e.g., content recorded on film or analog video tape) into a desired digital format. Alternatively, digital video cameras are used to generate content in a raw digital format, which is then edited and converted into a format suitable for viewing or distribution. Different venues or distribution channels are used for distributing the digital 3D movie electronically, including:
     A. Digital communication channel to movie house for theatrical presentation;   B. Satellite transmission to subscribers, such as DirecTV;   C. Blu-ray Disc;   D. DVD;   E. Various hand-held devices such as Nintendo 3DS® receiving streamed media over a wi-fi network or other private or public channels.   

     Each of these venues require typically its own format(s). For example, a satellite communication channel for digital projection in a movie theater the digital 3D movie in a JPEG 2000 format. 
     A Blu-ray 3D Disc normally uses MVC formatted files. 
     A DVD may require MPEG-2 format files. 
     Each of these digital 3D movie distribution channels may use different forms of 3D encoding, resolution, compression, and/or authoring. For example, theatrical presentation may require 4K resolution using JPEG2000 compression with ‘burned in’ subtitles for digital theatrical exhibition. A Blu-ray disc may require an image with 1920×1080 p HD resolution using MVC compression with subtitles presented in a ‘one plane+offset’ manner for Blu-ray Disc. DirecTV may require a frame compatible spatial compression (e.g. ‘side-by-side’) and AVC compression with ‘burned in’ subtitles for DirecTV satellite broadcast. A small, hand held device may require a 800×240 pixel image (400×240 for each eye) resolution using Mobiclip compression with ‘burned in’ subtitles for Nintendo 3DS autostereo (no glasses required) viewing. 
     Moreover, as previously mentioned, digital 3D movies in some of the formats may incorporate additional content, e.g., menus, subtitles in one or more languages, etc., using an appropriate authoring process. 
     Typically, once a digital 3D movie is made and recorded in a raw format, it is converted into a format suitable for a particular distribution channel, such as the channels described above. Depending upon the order in which a digital 3D movie is prepared for distribution, e.g. theatrical, Blu-ray Disc, DVD, satellite broadcast, there may be relevant 3D information determined by encoding or authoring software and/or an operator on a frame by frame, group of frames, or scene by scene basis and recorded in a respective log. For example, a Z-axis depth map of whole frames&#39; content, or of objects of interest within frames, may be determined when preparing to position subtitles or other graphics in 3D space over the background video. 
     One problem with the existing techniques described above is that they are not used systematically to add new information, such as subtitles and/or other relevant graphics, to an existing 3D movie 
     SUMMARY OF THE INVENTION 
     The present invention pertains to an apparatus, including an authoring tool, for adding a graphic element to a 3D movie. The authoring tool receives a depth map indicating the position of various objects in the frames of the 3D movie along a Z-axis. The authoring device then designates a position for additional graphic elements in each frame, these positions being determined in relation either to the positions of the objects or the position of the screen along at least said Z-axis. An encoder uses parameters from the authoring tool to reauthor the 3D content by adding the graphic element at the positions designated by the parameters. The graphic element may include subtitles and other graphic elements. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  shows a block diagram of a system constructed in accordance with the present invention; 
         FIG. 2  shows a frame of a scene of a movie captured in 3D; 
         FIG. 3  shows a depth map with a first title position for the frame of  FIG. 2 ; and 
         FIG. 4  shows a depth map with a second title position for the frame of  FIG. 2 . 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     The present invention provides a solution for the above-described problem and it operates as follows. More particularly, the present invention presents a system and method for adding subtitles and other relevant graphics to a 3D movie. It should be understood that for the purposes of brevity and clarity, in the following description the term “subtitles” is intended not only standard subtitles (for example, dialogue for a foreign movie) but other relevant graphic content as well. 
     Referring to the  FIG. 1 , an uncompressed digital 3D movie  60  is analyzed and encoded by a first 3D video encoder  10  using a first set of encoding coefficients  10 A. For example, the digital 3D movie could be in a 1920×1080 p uncompressed image sequence for each left/right eye at the same native resolution as the source material (i.e. 1920×1080 p). 
     As part of the original analysis, a Z-axis depth map is generated either based on the content of whole frames or based on objects of interest within the frames. For example, as shown in  FIG. 2 , a frame of scene of a movie captured in 3D may include three trees A, B and C disposed at different distances from the viewer, as well as a cloud D. As part of the encoding process, the encoder generates a Z-axis depth map. This depth map may be generated in various different forms. One typical technique is to generate a gray-scale version of the frame in which each object of interest is assigned a gray scale value dependent on its distance from the viewer. However, for the sake of clarity, the depth map presented in  FIG. 3  as a linear axis, with the relative positions of objects of interest being represented by arrows A-D. Arrow S represents the position of the screen. 
     In one embodiment of the invention, the Z-axis depth map is generated by the encoder  10  and stored in a log  20  together with other compression or encoding parameters generated during the conversion process. The encoder also generates a first 3D encoded content which may be stored, recorded or distributed to customers via various distribution channels. The first 3D encoded content does not have subtitles. 
     The invention further includes an authoring tool  22  used to add subtitles to the 3D movie. For this purpose, the authoring tool  22  receives the subtitles and title position parameters indicative of where the subtitles should be positioned in the relevant frames. Preferably, subtitles can be positioned in 2D and 3D space with high quality while maintaining 3D viewing comfort. For this reason, not only 2D X and Y positions but also the position of the subtitles in the Z axis is fairly important. Various criteria or approaches can be used to accomplish this goal. Once this position is selected, then the authoring tool analyzes the Z depth map for each frame, allocates the position of the subtitles T and then generates appropriate reauthoring control signals for superimposing the subtitles on the 3D frames at the designated location. The following is a partial list of some possible criteria dictating the position of the subtitles on the Z-axis. 
     a. At a fixed distance d 1  in front of the foremost object of interest. For example in  FIG. 3 , the subtitles are positioned at T disposed at distance d 1  in front of tree A. 
     b. At a fixed distance in front of a particular object, unless, of course, the object or the subtitles are obstructed by other objects. For example, in  FIG. 4 , the subtitles are positioned at a distance d 2  in front of tree B. 
     c. At a fixed distance from the screen S. 
     Moreover, the subtitles are positioned by the authoring tool not only along the Z-axis, but also anywhere along the X- and Y-axes. This position can be done either manually or systematically, by taking into consideration the positions of the other objects of interest. 
     A further consideration in determining the position of subtitles pertains to the actual screen size for which the 3D content is intended for. Generally, 3D content is originally composed and mastered for a specific screen size. However, when content created for one screen size is subsequently prepared, distributed, and viewed on a different screen size some qualitative anomalies can occur, e.g. miniaturization. Therefore when performing subsequent encoding of 3D movies by adding graphic subtitles, it may be desired to optimize the subtitles based on the target screen size. The original stereoscopic content (left/right video image pairs and/or left/right graphic overlay pairs) is fixed in its composition. The only parameter available during the re-encoding or re-authoring process is the horizontal shift, hereafter denoted as “offset,” between the left/right video image pairs or left/right graphic overlays. 
     For the purposes of describing this invention, it is assumed that the original offset corresponding to the original 3D content targeted for an initial screen size is zero. When that same 3D content is to be re-authored for a different target screen size a change in the offset value can be made. The value of the offset is a function of the original screen width (W 1 ), the new target screen width (W 2 ), the interoccular distance between the intended viewer&#39;s eyes e (e=2.5 inches for average human adult), and the horizontal resolution of the intended screen size (HorRes). 
     It is presently estimated that
 
Offset=(1− W 2/ W 1)× e ×( HorRes/W 2)
 
     This offset is the total horizontal shift required for the left and right stereoscopic images which will result in a shift of the subtitles along the Z-axis. Preferably, each image should be shifted in the respective direction, and therefore, each image should be shifted by ½ of the offset. 
     It should be understood that the above formula is a very basic formula and that it could be refined by using additional information that may be available to the authoring tool, including, but not limited to the intended viewer&#39;s distance to the screen, maximum 3D disparity desired, original set geometry used when acquiring the 3D content (e.g., distance of each object to camera), original camera parameters used to generate the 3D content (e.g. lens focal length, horizontal offset from optical axis), etc. 
     Returning to  FIG. 1 , the depth map and other information is received from the log  20  by the authoring tool  22 . The authoring tool also receives the subtitle text and text positioning information (unless it is already preprogrammed in the authoring tool). If it is desired to tailor the position of the subtitles along the Z-axis to compensate for the intended screen size, then the authoring tool also receives the original screen size, the intended screen size and other information necessary for the shift along the Z-axis. 
     The authoring tool then generates a new set of parameters  23  for creating a second encoded 3D content. 
     A second encoder  70  (or the original encoder  10 ) receives either the original 3D uncompressed video master  60  or the first encoded content  12  and, using a second set of encoding coefficients  70 B to generate a second 3D encoded content with subtitles. This second encoded content is distributed by standard distribution. 
     In an alternate embodiment, the authoring tool generates encoding coefficients  70 B defining the position(s) of the subtitles and the text for the subtitles are then provided to the encoder  70 . 
     The second set of coefficients  70 B may include an original set of predetermined parameters, and/or some or all of the first encoding coefficients  10 A, and/or other parameters derived by the first encoder and stored in the log  20 . Importantly, the second set of coefficients  70 B also include the parameters from the authoring tool  22  for superimposing the subtitles as discussed above. 
     Depending on the actual devices used as encoders  10  or  60 , the parameters from the authorizing device may not be directly compatible with the encoder  70 . In this case, the parameters  23  are fed to a mapping/processing element  24  for mapping the parameter into corresponding coefficients. 
     The parameters from the authoring tool ( 22 ) place the subtitles in the appropriate planes in 3D space for the specific release channel(s) at issue, e.g. Blu-ray Disc, or satellite broadcast, or Nintendo 3DS distribution. 
     The authoring tool is preferably one such as Sonic Solutions&#39; Scenarist that has controls to set or calculate where to position the subtitles or other relevant graphics over the background video in 3D space using the depth map info depending upon the intended distribution channel. 
     In the embodiment described above, the depth map and other information derived from the log  20  is used to generate the parameters. Alternatively, if the log is not available, either the first 3D encoded content  12  or the 3D uncompressed video master  60  is analyzed by a separate analyzer to derive the information required by the authoring tool  22  and the optional mapping/processing element  24 . 
     In any case, the subtitles can be positioned in the X, Y and Z 3D space, either using one set of pre-set parameters for the entire duration of the video content, or dynamic positioning can be managed by setting different parameters for different scenes (or other relevant groups of frames), or manual positioning can be used to override pre-set parameters in exceptional cases. For example, prior to assigning a position for a title for a given scene, the apparent distance between relevant objects and the viewer is monitored during the scene and the shortest distance is determined. The position for the subtitle is set for that scene to be a predetermined, e.g., 2%, less than this shortest distance. 
     In this manner, video frames with subtitles or other relevant graphics are efficiently processed dynamically using the depth map obtained from the log of an original encoder or from a separate analyzer. 
     Numerous modifications may be made to the invention without departing from its scope as defined in the appended claims.