Method and system for generation of captions over stereoscopic 3D images

Management of graphical overlays for use in stereoscopic video where the graphical overlay is positioned relative to depths associated with objects showing in the video. A disparity offset processor may be configured to facilitate generating overlay mapping information as a function of object depth maps in order to facilitate positioning one or move graphical overlays relative to corresponding objects.

TECHNICAL FIELD

The present invention relates to managing graphical overlays for stereoscopic video, such as but not necessarily limiting to managing position of the graphical overlays relative to objects showing within the video.

BACKGROUND

FIG. 1schematically illustrates display of a caption10over a two-dimensional (2D) image12included as part of a television program showing within a television screen14. The illustration depicts a situation in which the caption10may be used to provide a textual description of a dialogue or other audio taking place within the television program. The caption10is shown to be at the side of the image12for exemplary purposes. The caption10may be directly over the image12to prevent viewing of the covered portion of the image12. This process is commonly referred to as closed captioning when the caption10is used to textually describe audio events taking placing within the television program. The particular positioning of the caption10within the television screen14may be defined according to an x-axis and y-axis of the screen14in which the caption10is to appear, i.e., x and y values may be used represent a placement location of the caption10respectively within the x-axis and y-axis of the screen14, such as the placement defined according to ANSI-CEA-708, which is hereby incorporated in its entirety. The caption10is then added to the video frames used to render the image12at the location specified with the 2D coordinates. This type of 2D placement coordinate may work well for placing the caption10relative to 2D images but is problematic when used to place the caption10near 3D images.

The rendering of 3D images is typically accomplished in a stereoscopic manner by rendering separate left and right viewpoint images such that the images from each viewpoint appear independently to each eye as a 3D object. Since a caption10added according to the 2D coordinate system will be added to part of the left viewpoint portion of the frame and part of the right viewpoint image of the frame, the 3D television displays the left and right viewpoint images independently such that only the portion of the caption10within each viewpoint is displayed at the same time. This essentially creates a 3D image that overlaps the two portions, rendering the closed caption text illegible.

DETAILED DESCRIPTION

FIG. 2illustrates a caption20displayed over a three-dimensional (3D) image22shown within a display24in accordance with one non-limiting aspect of the present invention. The caption20may be generated such that it appears to be parallel with or in front of the 3D image22. This presentation is believed to be advantageous in that it allows a viewer to see the caption without the caption disrupting the presentation of the 3D image22. (The caption20is shown to be at a side of the image for exemplary purposes. The present invention fully contemplates an area behind the caption20including 3D images.) The caption20may be redrawn to appear in each of the left and right viewpoint portions of each video frame prior to output such that the entire caption appears regardless of whether the left or right viewpoint is currently showing.

FIG. 3illustrates a 3D video source30configured to generate the caption20in accordance with one non-limiting aspect of the present invention. The operation of the source30is described with respect to supporting output of 3D images to the display24of the type that relies on a stereoscopic presentation of left and right viewpoints, such as that provided by a settop box (STB), Blue-Ray player, etc. The source30may be configured to support generating the caption20during playback of a television program or other video based image. While the present invention is predominately described with respect to generating the caption over 2-view stereoscopic 3D television images, the present invention is not intended to be so limited and fully contemplates generating the caption20relative to any other type of 3D image, including multiview, autostereoscopic 3D images.

The source30may include a demodulator32to demodulate signals received from a service provider (not shown), disc/media player, or other source of content. The service provider may be a multiple system operator (MSO) or other entity that provides subscription based services to one or more subscribers, such as but not limited to a cable, satellite, or broadcast television service provider; a telephone/telephony service provider; and/or a high-speed data service provider. The source30also applies to an arrangement where the demodulator32may be configured to operate with some form of packaged media or removable memory element such as a BD player in which the demodulator function may differ depending upon the source of the content. The captions20generated according to the present invention may be adapted to the service associated with each service provider and the user interface devices necessary to access the same. In the case of supporting television based signaling, the demodulator32may be a tuner or other device configured to demodulate signals received over the particular communication medium of the television service provider, i.e., wireline (cable) or wireless (satellite, broadcast) mediums.

A demultiplexer34may be included downstream of the demodulator32to demultiplex the signals output from the demodulator32. Television signals may be transported according to any number of communication protocols and standards. Moving Pictures Expert Groups (MPEG) is one standard that may be used to facilitate transmission of television based video. MPEG defines transportation of multiple element elementary streams (ESs) within a single transport stream (TS). In the case of supporting MPEG or some other multiplexed communication strategy, the demultiplexer34may be configured to demultiplex one or more of the ESs included in the TS output from the demodulator. For illustrative purposes, only ESs associated with audio, video, and captions are shown even though the system30may be configured to demultiplex and process other ESs.

An audio decoder36may be included to process the audio signals for output to a speaker38. A video decoder40may be included to process the video signals for output to a combiner42. A caption decoder44may be included to process caption signals for output to a graphics generator46. The graphics generator46may be configured to generate textual or other graphical representation to be included as part of the caption20. In the case of closed captioning, the graphics generator46may be configured to generate text that matches audio sounds being conveyed during the television program. This graphical representation may be based on corresponding data included with the caption signals transported to the source30, i.e., based on data included in the caption ES of the TS such as that defined by ANSI/CEA-708. The graphics generator46may also be configured to generate graphical elements, text, advertisement, logos, and other types of graphical icons according to the design of user-interface and other application software in the source30.

The combiner42may be configured to combine the output of the graphics generator46with the video output from the video decoder40. A driver48may then interface the video with the display used to render the 3D image. Depending on the configuration of the display24, the device driver48may be required to output the resulting video such that each frame includes a particular orientation of left and right viewpoint images, i.e., the device driver may be required to output the video according to a spatial reduction technique (side-by-side, above-and-below, checkerboard, etc.), in temporal reduction technique, or some other spatial reduction technique. The device driver48or the combiner42may include a 3D pre-formatting element (not shown) to facilitate the processing and reformatting of left and right viewpoint images as transmitted into different formats of spatially multiplexed or temporally multiplexed video frames as required by the display24.

The combiner42may be configured to add/combine the caption output from the graphics generator46to each of the left and right viewpoint images included within each video frame output to the display24.FIG. 4illustrates a video frame50output from the combiner42after redraw of the caption20in accordance with one non-limiting aspect of the present invention. The video frame50in the example is configured to operate with the display24requiring a side-by-side spatial reduction of left and right viewpoint images52,54. The caption20is redrawn to appear in both of the left and right viewpoint images52,54based on a 2D coordinate placement56specified within the caption20information included with the TS, i.e., with the caption ES. A 2D image frame58included in the upper portion ofFIG. 4to illustrate placement of the caption10if the caption10were to be added to 2D video being output to a 2D display.

The placement of the captions20redrawn in accordance with the present invention may be adjusted relative to the 2D placement coordinates56used with the 2D video frame. For the purpose of one non-limiting aspect of the present invention, it is assumed that the captions20used were created using the present (2D) caption standards, without any special ability to convey placement other than in the 2-dimensional axis (X:Y), no Z-axis data is available in the caption stream. To create the appearance shown inFIG. 2where the caption20appears to be in front of the 3D image22, the left viewpoint caption20′ is shifted to the right and the right viewpoint caption20″ is shifted to the left. The caption20′ redrawn within the left viewpoint image52may be assigned a corresponding first placement location60and the caption20″ redrawn within the right viewpoint image54may be assigned a corresponding second placement location62. The graphics generator46may use these placements60,62when generating the caption video output to the combiner42for combination with the video output from the video decoder40. The amount by which each caption20′,20″ is shifted may be selected depending on a forward end66(seeFIG. 2) of the image plane so that the resulting caption20appears to be in front of the 3D images22. This determination may be based on the characteristics of the display24and its operating capabilities. Optionally, the graphics generator46and/or the combiner42may be configured to assess these characteristics and to select an appropriate adjustment to the placement locations60,62.

The adjustments made by the present invention may be understood relative to the x and y coordinate values56typically used to define placement of the caption20within the 2D image frame58. The x and y values56associated with that caption20may be used to define of an x-axis and y-axis placement location60,62for a window or other feature used to display the caption20. In accordance with the present invention, these x and y values60,62may be adjusted to re-position a copy of the caption20′,20″ within the left and right viewpoint, spatially reduced video frames52,54so that the resulting caption20appears to a viewer to be in front of a screen plane68.FIG. 5illustrations this relation as defined relative to a z-axis (positive parallax and negative parallax) as measured from the screen plane68relative to the viewer. The parallax of the caption20may be increased by controlling the relative offset of each of the redrawn captions20′,20″ to each other, i.e., the depth may increase (caption moves closer to viewer) as the captions20′,20″ are shifted farther away from the predefined 2D coordinates56.

The 2D coordinates56used to define placement of the caption10relative to a 2D image (seeFIG. 4) may be defined relative to a 100 unit coordinate system such that placement of the caption10approximately corresponds with an x value of 50 and a y value of 100. This would place the caption10at a top, center of the screen when output at the screen plane. Since the 3D video frame is used to generate separate full-images for each of the left and right viewpoints, the same 100 unit coordinate system is reproduced within each half of the side-by-side reduction. (The size of the caption20′,20″ may be corresponding adjusted to reflect the halving necessary to support the illustrated spatial reduction.) In a temporally reduced system, the reproduction of the same coordinate system within the illustrated halves would not be necessary since the entire frame would be used to represent one of the left or right viewpoint images52,54.

Regardless of whether left and right viewpoint images52,54are temporally, spatially or otherwise reduced according to the needs of the output device, the placement of the captions20′,20″ within each of the left and right viewpoint images52,54may be shifted relative to each other in accordance with the present invention to adjust the resulting appearance of the caption20relative to the produced 3D images22. As shown inFIG. 4, the x values are the only adjusted value and each value is adjusted to produce a difference of 10 units between each caption depending on whether a negative or positive parallax of 10 units along the z-axis is desired. The amount of parallax may be selected by the graphics generator46based on the parallax of the 3D images being shown. This may include the graphics generator fixing the parallax through an entire program and/or continuously/dynamically adjusting the parallax with changes in parallax of the 3D images being shown, adjusting the parallax according to user inputs, and/or adjusting the parallax according to other messages send to the graphics generator.

FIG. 6illustrates a system100for managing graphical overlays in accordance with one non-limiting aspect of the present invention. The system100may be configured to facilitate positioning or otherwise controlling insertion of graphical overlays to be added to stereoscopic/3D video. The system100is predominately described with respect to stereoscopic 3D video that outputs the video as altering left and right images, however, the present invention is not necessary so limited and fully contemplates the use of other stereoscopic video arrangements, such as but not necessary limited to other spatial and/or temporal reduction techniques (e.g., side-by-side, above-and-below, checkerboard, etc.). The images comprising the stereoscopic video may be delivered as a plurality of image frames associated with a corresponding one of a left video input112and a right video input114. The left video input112and the right video input114may be generated with a demodulator, a demultiplexer and/or some other combination suitably configured to retrieve the stereoscopic video from a particular transmission medium.

A set-top box (STB)116is shown to facilitate processing the left video input112and the right video input114for output to a display118. Optionally, the formatting requirement of the display118may be determined in the manner described in U.S. patent application Ser. No. 12/502,434, the disclosure of which is hereby incorporated by reference in its entirety. The STB116is described for exemplary non-limiting purposes to demonstrate the use of the present invention with stereoscopic video transmitted over a cable television median, a broadcast television medium, an optical television medium, a satellite television medium or other medium where a device processes the stereoscopic video prior to output to the display118. The STB116or the capabilities/components associated therewith may be included within a standalone device connected to the television using a High-Definition Multimedia Interface (HDMI) cable or other suitable connection. While this exemplary standalone configuration is shown, the present invention fully contemplates integrating the illustrated capabilities/components with the display118, i.e., with a television, computer, tablet, a cellular phone or other device having the display or otherwise configured to facilitate interfacing the stereoscopic video with a viewer or with another device designed to transmit the stereoscopic viewer with a graphical overlay to a viewer.

A disparity detection processor120may be configured to capture samples of the left video input112and the right video input114. The sampling may correspond with the disparity detection processor120capturing the individual images or frames comprising the left video input112and the right video input114as the corresponding images are being pass-through for further processing for output to the display118.FIG. 7illustrates an exemplary left image122andFIG. 8illustrates an exemplary right image124as captured with the disparity detection processor120in accordance with one non-limiting aspect of the present invention. The left and right images122,124may include a plurality of objects where the corresponding objects (e.g.,126a) in the left image122are positioned slightly offset from the corresponding object (e.g.,126b) in the right image126in order to generate the desired parallax. The disparity detection processor120may include image segmentation software, edge assessment capabilities or other object detection capabilities sufficient to facilitate individually identifying the plurality of objects included within each of the left and right images122,124. The disparity detection processor120may be configured to continuously capture left and right images at a speed corresponding with the speed at which the left and right images are being transmitted through the left video input112and right video input124, e.g., at 30 images/frames per second (or at faster or slower speeds).

The disparity detection processor120may be configured to generate disparity maps as monochromatic images or other exemplifications sufficient to represent relative disparity between the objects included within the captured images.FIG. 9illustrates one such monochromatic image128generated for one of the captured images where the disparity map128is represent according to a color-coded gray scale. The gray scale may be used to differentiate relative depth of the objects according to a color value assigned to each pixel of the monochromatic image. For exemplary non-limiting purpose, the disparity detection processor120may be configured to assign each image pixel a value between 0 and 255 to indicate depth of corresponding object, e.g., a value closer to 0 (lighter) may be used to indicate the object is closer (negative parallax) to the viewer and a value closer to 255 (darker) may be used to indicate the object is farther from the viewer (positive parallax). Of course, the present invention is not necessarily limited to the use of monochromatic images as the disparity maps128and fully contemplates other processes for generating disparity information sufficient to represent depth, parallax or other values indicative of relative positioning of objects within an image/frame.

A filter130may process the disparity maps output from the disparity detection processor120. The filter130may be configured to facilitate scaling, smoothing and/or averaging of the disparity maps in order to mask distortions and/or to scale the underlying disparity information (e.g., color-based values) to a uniform range. The filtered disparity maps may be output to a disparity first-in-first-out (FIFO) buffer132or other suitable time-delaying feature. The disparity FIFO buffer132may be timed relative to a left video input FIFO buffer134and a right video input FIFO buffer136to facilitate timing processing of the disparity maps relative to delivery of the corresponding images within the left and right video inputs112,114. The FIFO buffering may be used to ensure a graphical overlay generated with a graphics processor138for a particular image is positioned within the corresponding image being output. This may include timing delivery of the graphical overlay with a left composite buffer140and a right composite buffer142configured to perform final processing of left and right video inputs112,114prior to output to the display118. One non-limiting aspect of the present invention envisions the FIFO buffers132,134,136buffering video for multiple seconds in order to facilitate managing insertion of the graphical overlay according the contemplated process.

A feedback loop144may optionally be included to facilitate feedback of a preceding disparity map to the filter130. The filter130may process one or more preceding disparity maps relative to a current disparity map in order to generate a filtered disparity map.FIG. 10illustrates a filtered disparity map146in accordance with one non-limiting aspect of the present invention. The filtered disparity map146is shown to be slightly fuzzier that the un-filtered disparity map128shown inFIG. 9. The fuzziness associated with the filtered disparity map146may include less defined object edges resulting from the objects slightly shifting position between image/frames (e.g., objects may slightly change position from the preceding disparity map to the subsequent/current disparity map). The less defined objects may also correspond with smoothing and/or scaling introduced with the filter130to influence the granularity at which color changes are reflected, which can be beneficial in minimizing distortion and/or rapid changes in depth position. The preceding disparity map fed back to the filter130may itself have been previously filtered such that it corresponds with a filtered disparity map. Optionally, rather than feeding back a filtered disparity map, and un-filtered or raw disparity map (e.g.128) for the preceding image may be fed back. The use of an un-filtered disparity map may be beneficial in avoiding tolerance accumulation and noise within the filtered disparity maps output from the filter130.

The filtered disparity map may be provided to a disparity offset processor148. The disparity offset processor148may be configured to facilitate compositing a graphical overlay generated with the graphics processor138to the left composite buffer140and the right composite buffer142in order to facilitate positioning the graphical overlay within desired portions of the left images and right images being output to the display118. The disparity offset processor148may be configured to generate insertion instructions, positioning instructions or other information sufficient to achieve the desired positioning of the graphical overlay with the left composite buffer140and the right composite buffer142. The disparity offset processor148may facilitate positioning the graphical overlay relative to any one or more of the objects identified within the filtered disparity map, i.e., the disparity offset processor may provide instructions sufficient to facilitate positioning the graphical overlay in front, behind or at any other depth of the stereoscopic video output. In this manner, the present invention contemplates facilitating 3D depth-based positioning of the graphical overlay relative to any one or more objects shown within the output 3D video. The depth of the graphical overlay may be controlled by shifting the graphical overlay added to the left composite buffer140relative to the graphical overlay added to the right composite buffer142such that the parallax effect achieves the desired depth relative to the desired object.

The graphics processor138may be configured to select one or more graphical overlays to be composited within the displayed stereoscopic video. The graphics processor138may include a network interface (not shown) to facilitate receiving overlay related instructions from an overlay controller, an advertisement controller or other device sufficient to identify the appropriate graphical overlay. The graphics processor138may identify a channel or other identifying information associated with the stereoscopic video tuned to by the STB116in order to identify the desired graphical overlay. Optionally, user preferences, history or other parameters may be identified to facilitate selection of the graphical overlay. The present invention is not intended to be necessary limited to the type of graphical overlay being composite within the stereoscopic video such that virtually any type of alphanumeric representation, image, caption, media or other informational conveying means may be used to form the graphical overlay. The present invention, for example, contemplates generating multiple graphic overlays for the stereoscopic video, such as but not necessary limited to facilitating displaying advertisements simultaneously with closed captioning, rolling text, banner ads and the like.

The disparity offset processor148may facilitate delivering the one or more desired graphical overlays to the left composite buffer140and the right composite buffer142, optionally with the format or representation of the graphical overlay sent to the left composite buffer140and the right composite buffer142differing according to desired 3D effects. The disparity offset processor148may be configured to time delivery of the particular graphical overlays relative to the buffering provided with the disparity FIFO buffers132,134,136in order to ensure the positioning information generated from a particular filtered disparity map is used to position the graphical overlay relative to the corresponding left image and right image being received at the left and right composite buffers140,142for output to the display118. In the event the system100is being used to facilitate playback of real-time video, the video may be buffered by an amount of time sufficient to facilitate the contemplated management of the graphical overlay insertion such that the resulting stereoscopic video output to the display118may be delayed in time relative to the real-time occurrence of the stereoscopic video. Optionally, the time delay induced with the buffering may include coordination with secondary devices used to facilitate interactions with the stereoscopic video in order to avoid occurrence of spoilers, i.e., to prevent applications executing on the secondary device from executing operations in time with the real-time video instead of the buffered video.

FIG. 11illustrates a flowchart170of method for managing graphical overlays in accordance with one non-limiting aspect of the present invention. The method may be embodied in a computer-readable medium, and/or computer program product, having non-transitory instructions stored thereon, which are operable with a processor or other logically executing device, to facilitate the contemplated graphical overlay management. While the method is predominately described with respect to leveraging use of the system100illustrated inFIG. 6to facilitate the insertion of graphical overlays within stereoscope video, this is done for exemplary non-limiting purposes as the present invention fully contemplates its use with other types and configurations of systems and its use in managing graphical overlays for other types of media besides stereoscopic or 3D video. As described in more detail below, one aspect of the contemplated method relates to locally processing video to assess objection position, depth, movement, etc. and capitalizing on the locally generated information to facilitate management of graphical overlays.

The method is predominately described with reliance on graphically processing performed with the disparity offset processor when148. The processing is illustrated with respect to the disparity offset processor148generating or otherwise mapping depth information gleaned from disparity maps output from the disparity detection processor120. This may include assessing object depth relative to overlay depth in order to calculate desired positioning (x, y, z) for the graphical overlay(s) within the left and right images. The description is provided with respect to insertion of a single graphical overlay within a single image for exemplary purposes as the present invention fully contemplates the disparity offset processor148performing similar processing for any number of graphical overlays and images, optionally simultaneously. The description is also provided with respect to the disparity offset processor148generating maps, lines and other graphically orientated references to demonstrate calculation of parameters used to manage overlay insertion without necessarily intending to limit the scope and contemplation of the present invention. The illustrated features are shown merely to demonstrate information being collected, calculated and/or otherwise processed with the disparity offset processor148when managing graphical overlay insertion. The disparity offset processor148need not necessarily generate such mappings in order to achieve the results contemplated by the present invention.

Block172relates to buffering video. The video may be buffered for a period of time sufficient to enable the processing of captured images and the subsequent insertion of a graphical overlay within the actual images from which the captured images were taken. The video may be buffered with the described disparity, left and right FIFO buffers132,134,136or with other suitable buffering devices, such as but not necessarily limited to a digital video recorder (DVR), personal video recorder (PVR), network DVR, etc. Timestamps, image identifiers or other frame-based identifiers may be assessed or generated to differentiation particular images from each other. The images being buffered may correspond with those transmitted according to Moving Pictures Expert Group (MPEG) or other suitable image/frame transmission protocols. While the buffering is shown to be achieved with the disparity, left and right FIFO buffers132,134,136, the buffers132,134,136need not necessarily be standalone buffers and instead may be incorporated into other components of the STB116or other device through with the stereoscopic video is processed for output to the display118.

Block174relates to identifying the graphical overlay desired for insertion within the stereoscopic video. The graphical overlay identification may include configuring a size, shape, appearance and other parameters for the graphical overlay. Such formatting of the graphical overlay may be tailored to the output capabilities of the output device and/or the stereoscopic operation requirements of the display. The identification of the graphical overlay may also include identifying one or more objects relative to which the graphical overlay is to be displayed. With respect to the illustrations shown inFIGS. 7 and 8, certain graphical overlays may be desired for positioning relative to one or the cones126and other graphical overlays may be desired for positioning relative to the mask152and/or the background lattice154. The identification of the object relative to which the graphical overlay is to be displayed may beneficial in allowing advertisements to be positioned relative to particular products, to facilitate displaying player information relative to particular players and/or to facilitate any number of location specific conveyance of information. Optionally, a generic object position may be specified, such as a generic specification that the graphical overlay appear in front of a nearest object within an image, behind a farthest object within an object or another location that is not tied to specific type of object.

Block176relates to determining object depth/positioning information for the object relative to which the graphical overlay is to be displayed. The method is predominately described with respect to facilitating positioning of the graphical overlay in front of the nearest object. The object depth mapping described below may correspond with identifying the nearest object within each image as a function of the disparity maps. This is done without necessarily intending to limit the scope and contemplation of the present invention as similar depth mapping may be generated for any one or more objects besides the nearest object.FIG. 12illustrates a graphical representation180of information generated to assess object depth in accordance with one non-limiting aspect of the present invention. The object depth information is shown graphically, however as noted above, the present invention is not necessary limited to generating such graphical representations and fully contemplates calculating similar information without the necessity of rendering the corresponding graphical representation.

The method is predominately described with reliance on graphically processing performed with the disparity offset processor when148. The processing is illustrated with respect to the disparity offset processor148generating or otherwise mapping depth information gleaned from disparity maps output from the disparity detection processor120. This may include assessing object depth relative to overlay depth in order to calculate desired positioning (x, y, z) for the graphical overlay(s) within the left and right images. The description is provided with respect to insertion of a single graphical overlay within a single image for exemplary purposes as the present invention fully contemplates the disparity offset processor148performing similar processing for any number of graphical overlays and images, optionally simultaneously. The description is also provided with respect to the disparity offset processor148generating maps, lines and other graphically orientated references to demonstrate calculation of parameters used to manage overlay insertion without necessarily intending to limit the scope and contemplation of the present invention. The illustrated features are shown merely to demonstrate information being collected, calculated and/or otherwise processed with the disparity offset processor148when managing graphical overlay insertion. The disparity offset processor148need not necessarily generate such mappings in order to achieve the results contemplated by the present invention.

The graphical representation180illustrates a first object depth line (solid line) to reflect object depth for a nearest object included within the filtered disparity maps (seeFIG. 10) input to the disparity offset processor148. While only the first depth line is illustrated, similar object depth lines may be included for any additional objects of interest or objects identified to be associated with the particular graphical overlay. The first depth line may be characterized as a depth map for the nearest object appearing within the filtered disparity maps. The first depth line is shown to be divided into a plurality of segments corresponding with one second intervals of time such that a first segment182, a second segment184and a third segment186are illustrated. Each of the segments182,184,186may be used to map the depth of the first object across a plurality of images occurring during the corresponding time interval. In the event the images are processed for output to the display at30images per second, each of the segments182,184,186would comprise depth information generated from 30 images, i.e. 30 filtered disparity maps.

Optionally, the first depth line may be segmented into the illustrated first, second, and third segments182,184and186using other markers or references besides the number of frames and/or elapse time. The first depth line, for example, may be segmented based on scene changes and/or other events occurring within the stereoscopic video. The disparity detection processor120or other device feature having capabilities to assess object positioning may be configured to identify scene changes to occur when a sudden or abrupt change in object depth occurs, such as that occurring at the one second interval between the first segment182and the second segment184. The partitioning of the first object depth line, and thereby the first, second and third segments182,184,186, or additional segments, may be beneficial in facilitating placement of the graphical overlay in a manner that tracks scene changes, such as to ensure a smooth transition or movement of the graphical overlay in anticipation of up-coming scene changes. In this manner, the scene changes may be identified before the corresponding scene changes are actually output to the display118such that the partitioning may occur in response to scene changes and before output of the corresponding video.

The depth of the first object is illustrated along a vertical depth axis188. Vertical elevational changes in the first object depth line indicate relative movement of the first object across each of the graphed images (e.g., 90 images/frames if output at 30 frames/second). The first object depth line is shown to experience sharp changes in elevation at a first boundary and a second boundary (vertical dashes), which may be attributed to a change in camera angle or other action in the stereoscopic video resulting in the nearest object (first object) becoming nearer to the view, such as the above noted scene changes. The first and secondary boundaries are shown to correspond with the one second interval and the two second interval for exemplary purposes as the boundaries may vary according to other intervals. Optionally, the boundaries may be based on object movement, such as according to scene change recognition where object movements are compared to a threshold associated with scene changes. The length between boundaries may vary depending on the number of scene changes such that more frequent scene changes may produce shorter lengths between boundaries than infrequent scene changes.

A first overlay depth line (dashed line) is shown to illustratively represent positioning of the graphical overlay relative to the first object depth line. The elevation of the first graphical overlay may be selected by the disparity offset processor148or otherwise implemented such that the overlay position for each frame may be related to a correspond position along the first overlay line, thereby defining positioning of the first graphical overlay for the corresponding image frame. In this manner, like the first object depth line, the first overlay depth line may be used to characterize an overlay depth map for the graphical overlay. The first overlay depth line may include a first portion192, a second portion194and a third portion196corresponding with each of the first segment182, the second segment184and the third segment186. The first, second and third portions192,194,196may be shaped differently from the corresponding first, second and third segments182,184,186.

The shaping is shown to correspond with linear sections having a consistent slope from a beginning (left side) of the corresponding portion192,194,196to an ending (right side) of the corresponding portion192,194,196. The first, second and third portions192,194,196need not necessarily be configured with the consistent slopes per portions192,194,196and instead may include other shapes. Optionally, the overlay depth line, or individual portions192,194,196, may be shaped with less undulations or less severe changes than the corresponding first, second and third segments182,184,186, i.e., with a non-linear but smoother shape. The first overlay depth line may be generated after measuring the entire length of the first depth line, i.e., after the illustrated three seconds or other buffering interval has elapsed. This may be done in order to shape the first overlay depth line in anticipation of upcoming frames to ensure the individual portions192,194,196are gradually sloped from an ending of the preceding portion192,194,196to a beginning of a following portion192,194,196.

The anticipation-based shaping of the first overlay depth line may be beneficial in preventing sudden changes in the depth appearance of the overlay to a viewer, e.g., preventing the overlay from suddenly moving toward or away from the viewer. The disparity offset processor148may be configured to shape each portion192,194,196such that the beginning and endings thereof are sloped relative to the nearest appearing object within the corresponding beginning and ending frame and/or the ending of the first object depth line in the preceding portion192,194,196and the beginning of the object depth line in the succeeding portion192,194,196. The first overlay depth line may be shaped to ensure the sloping defined between the beginning and ending of the corresponding portion192,194,196is sufficient to ensure the line remains above any peak or undulation within the corresponding segment182,184,186, which may include adjusting line elevation. Optionally, in the event sudden changes or frequent changes in the overlay depth are acceptable, the first overlay depth line may be shaped to more closely track the shape of the corresponding first object depth line, such as by setting an offset value or slight elevational difference between the portions192,194,196and the corresponding segments182,184,186and otherwise allowing the overlay to follow movement of the closet object.

Returning toFIG. 11, Block200relates to determining depth and/or other positioning information to facilitate compositing the graphical overlay with the left and right composite buffers140,142. The first overlay depth line may be used to identify depth defining information for the graphical overlay relative to each of the output images. Additional information may be included to facilitate x, y, z positioning within the output images, i.e., to facilitate positioning the graphical overlay in front of the nearest object at a bottom, top, right, left or other portion of the stereoscopic video. The ability of the present invention to generate depth and other positioning information allows the graphical overlay, or multiple graphical overlays, to be positioned to appear within a desirable portion of the output video.FIG. 13illustrates a graphical representation210of positioning information generated to facilitate positioning a first graphical overlay relative to a first overlay depth line while simultaneously positioning a second graphical overlay relative to a second overlay depth line. Block202relates to generating the necessary insertion instructions, positioning parameters, timing controls, etc. needed to properly manage insertion of the desired one or more graphical overlays within the output stereoscopic video.

As supported above, the present invention relates to a solution for generating captions (or graphics) over spatially multiplexed stereoscopic 3D images. This may include supporting caption placement within a system that relies on transmission of separate left and right viewpoints to construct a stereoscopic image. One solution proposed by the present invention is to redraw the text twice within each of the two sub-pictures, once for the left-eye half and again for the right-eye half of the image. Now when the two half images are processed by the 3D display processor they both contain the full text information for each eye, making them fully readable again. In this solution, when the captions are placed at the screen plane (zero parallax) there is no problem for portions of the image with positive parallax, however, when the captions are placed at the screen plane that intersect portions of the picture with negative parallax, there may be a depth conflict (visual paradox), which may negatively influence the 3D effect. The use of captions in this way may negatively influence the 3D effect and any extended exposure to this type of depth conflict may cause headaches and eyestrain. One solution proposed by the present invention is to render the captions in Z-space so that they appear to float in front or behind of any elements of the stereoscopic content. This may be accomplished by shifting the generated graphical (or text) elements in opposite directions for each half of the multiplexed stereoscopic image.

The text (or graphic overlay) that appears on the left-eye view may be shifted horizontally to the right while the text (or graphic overlay) for the right-eye view may be shifted to the left an equal amount away from the assigned target location. The degree or magnitude of this offset may be proportional to the resolution of the screen and the projected size of the image. The exact value may be adjusted with a user-control for the most comfortable viewing, while still minimizing the edge conflicts with any portion of the content that experiences negative parallax. Alternatively, a separate depth signal may be provided with the caption stream, which may be used by the display generator to control the off-set of the respective left and right text images, and/or data associated with multiple 2D coordinates specified for different placement locations may be processed to generate a desired z-depth according to relative differences in the specified 2D placement locations.

The present invention may be advantageous in that it may enhance the delivery a high-quality stereoscopic 3D experience to those viewers who chose to utilize the on-screen display of closed-captions during the program. Another non-limiting aspect of the present invention contemplates providing “open-captions” on a separate program stream that could be selected by the viewer where the caption text has been properly placed in the 3D space in advance by the programmer and delivered as a separate file. While this alternate method may be effective for stored content, it may less applicable to live programming and it may cost more to support transmissions of a duplicate stream.

One non-limiting aspect of the present invention contemplates actively managing graphical overlay placement in 3D-space to avoid depth space conflicts with underlying 3D video content by using modified real-time disparity detection from left and right view source material.

The generation of depth map data from stereoscopic pairs may be created in accordance with the present invention to facilitate converting disparity data associated with objects in a stereo pair to corresponding depth map. The present invention leverages the use of real-time depth map conversion along with storage and display processing to control placement of locally generated graphics in Z-space over live 3D programming and other types of stereoscopic video/media to avoid depth conflicts.

In one non-limiting aspect of the present invention, the stereo-pairs are submitted to the disparity detection processor, which generates a monochromatic image representing relative disparity between the objects. Optionally, this type of disparity map can be further smoothed and filtered to mask distortions and scaled to a uniform range which can be considered the depth map of the image. A new depth map frame may be produced for every video frame in the stereo content sequence and is sent to FIFO frame buffers. This multi-tap delay line may be used to offer several seconds of cumulative delay for the video path and the depth maps. The depth map path may include a feedback loop back to the scaling, smoothing and averaging processor where the current depth map can be compared with previous maps, so it can be averaged over time to remove any abrupt depth transitions that may occur due to edits in the programming stream.

The locally generated graphics for composting over the stereo content may be generated by the graphics processor and sent to the disparity offset processor. The disparity offset may be derived in real-time by extracting the values of the depth map for intended x,y coordinates of the graphical object and used to control the disparity of the stereo pairs of the graphical object when it is sent to the compositor. This may be done to ensure that the locally generated object will never be placed in depth behind an object in the 3D content for which it is to appear in front. The time averaging of the depth map may be used to ensure that the movements of the graphical object in depth space will be smooth, without any abrupt shifts as the background video changes with scene changes as shown below:

One non-limiting aspect of the present invention contemplates using a metadata approach where the graphical overlay position is defined prior to receipt of the stereoscope video. The metadata approach may be somewhat problematic as its predefined positioning, particularly with live content, may result in the graphical overlay automatically moving to forward and back as scenes change in the background without regard to actual movement of objects due to an inability to accurate ascertain object depths ahead of time. In the event object depth may be determined ahead of time, the metadata approach, in order to provide rich data necessary to enable placement anywhere on the screen at the appropriate depth, would incur a significant bandwidth penalty to deliver the large amount of data needed to generate the appropriate object depth information. Additionally, some most legacy systems in the transmission path may be incompatible with such metadata signals and would block this data from reaching the final display device, e.g., an HDMI cable is one example where this metadata would be interrupted.

Since the metadata approach can be visually disturbing and/or overly data intensive, one non-limiting aspect of the present invention contemplates locally generating graphical overlay position information through use of a multi-frame storage processing that averages time-domain transitions. This look-ahead buffer may be used to adjust the depth placement in anticipation of coming scene changes at a much reduced tracking rate, making it easier to view or read compared with the object that moves rapidly in Z-space. Accordingly, one non-limiting aspect of the present invention requires no separate transmission of depth metadata, but instead depends upon the local generation of this depth data as calculated in real-time from the left and right video signals by the final display device. This approach may preferred because the locally generated depth data can be filtered, processed and optimized by the same system which is generating the graphical overlays. In the metadata approach, the data will be filtered by the programmer without regard to the type or nature of the specific graphical overlay being inserted.