Patent Description:
According to the prior art, it is known to associate a depth map with an image representing a scene, the image being either a synthesis image representing a virtual scene or an image of a real scene shot by a camera. It is also known to use the depth information comprised in the depth map to perform visibility tests, i.e. to determine which fragment of the scene is hidden by another fragment of the scene according to a camera field of view. A depth map typically corresponds to an array of N lines and M columns of elements, also called pixels or texels, a single depth information being associated with each element of the depth map. The finite resolution of the depth map leads to some approximations when determining if a first fragment of the scene is hidden by (or in the shadow of) a second fragment of the scene, the depth of which with regard to the camera field of view being stored in an element (also called pixel or texel) of the depth map. The resulting aliasing is known as bias aliasing. This aliasing may particularly occur when the first and second fragments belong to the same surface of an object of the scene as viewed from the camera field of view associated with the depth map.

As to overcome at least partially this bias aliasing artifact, it is known to use a bias value that a fragment is considered occluded or in shadow when its distance to the camera field of view associated with the depth map is above the depth stored in an element of the depth map associated with this fragment plus the bias. However, the introduction of a bias value may lead to another aliasing known as peter-panning artifact. <NPL> teaches storing depth derivatives to use in shadow computation.

The purpose of the invention is to overcome at least one of these disadvantages of the prior art.

More specifically, the purpose of the invention is to establish depth information associated with one or more elements of a depth map with a better precision than in the state of the art.

The invention will be better understood, and other specific features and advantages will emerge upon reading the following description, the description making reference to the annexed drawings wherein:.

The invention will be described in reference to a particular embodiment of a method for enriching the content of a first element of a depth map. The depth map advantageously corresponds to an array of L lines and M columns of elements, L and M being integers greater than <NUM>, and is associated with a scene which is rendered in one or more images. The number of elements comprised in the depth map corresponds for example to the number of pixels of the image(s) of the scene. The depth map is advantageously enriched with one or more first information representative of the variation of depth in the first element in the space of the depth map.

The additional first information thus enables to retrieve, if needed, the depth associated with any point of the first element whereas, in depth map according to the prior art, the depth associated with any point of an element of a depth map is the same and corresponds to the depth of the closest visible fragment of the scene associated with the first element, or with its centre.

<FIG> shows a scene <NUM> rendered in an image <NUM> with an associated depth map <NUM>, according to a particular and non-limitative embodiment. The image <NUM> represents the scene <NUM> as viewed from a given point of view (also called camera field of view, not represented on <FIG>) and the depth map is generated according to another point of view <NUM> (i.e. another camera field of view). The scene corresponds for example to a virtual scene and comprises several virtual objects, i.e. a first object <NUM> and a second object <NUM>. The objects <NUM> and <NUM> are modelled according to any method known to those skilled in the art, for example by polygonal modelling, in which the model is assimilated with a set of polygons (mesh elements) each defined by the list of summits and edges that compose it, by NURBS (Non uniform rational basic spline) type curve modelling in which the model is defined by a set of curves created via control vertices, by modelling by subdivision of surfaces. By virtual object is understood any virtual representation (obtained by modelling) of an object (real or fictitious) composing a real environment / real scene (for example the ground, a house or a house front, a person, a car, a tree, that is to say any element composing an environment such as a part of a house, a street, a town, the countryside, etc.) or an imaginary element. Each object <NUM>, <NUM> of the scene <NUM> is characterized by a surface covering it, the surface of each object having reflectance properties (corresponding to the proportion of incident light reflected by the surface in one or several directions) that are specific to it.

The depth map <NUM> comprises n elements <NUM>. 10n, n being an integer greater than <NUM> that defines the resolution of the depth map <NUM>, the resolution being for example equal to 512x512 pixels, 1024x1024 pixels or 4096x4096 pixels. A depth information is advantageously associated with each element of the depth map <NUM>. This depth information corresponds to the distance between the point of view <NUM> and the closest visible fragment of the scene <NUM> along a viewing direction passing through an element of the depth map, this element of the depth map being associated with the closest visible fragment of the scene. By taking the element 10p' as an example, the depth information associated with this element 10p' corresponds to the distance between the point of view <NUM> and the fragment P' <NUM> of the scene <NUM> along the viewing direction 100p' having as origin the point of view <NUM> and passing through the element 10p', advantageously passing through the centre of the element 10p'. The fragment P' <NUM> corresponds to the first element of the scene crossed by the viewing direction 100p' when starting from the point of view <NUM>. The depth information is associated with the centre of the corresponding element of the depth map. A first information representative of the variation of depth inside an element of the depth map <NUM> is advantageously further associated with each element of the depth map, this first information being established as described with regard to <FIG>.

The image <NUM> comprises m pixels <NUM>. <NUM>, m being an integer greater than <NUM> that defines the resolution of the image <NUM>. Advantageously, m is different from n, for example n is greater than m or m is greater than n (the resolution of the depth map is for example 512x512 pixels, 1024x1024 pixels or 4096x4096 pixels whereas the resolution for the image <NUM> is for example 1024x768 pixels, 1280x720 pixels or 1920x1200 pixels). According to a variant, m is equal to n, both depth map <NUM> and image <NUM> having the same resolution. Attributes are advantageously associated with each pixel of the image <NUM>, the attributes comprising for example color information (for example RGB information) and/or the translucent character of the fragment of the scene associated with a pixel of the image. A fragment advantageously corresponds to a surface element associated with a point of the scene <NUM>, the size of which being equal to the size of a pixel of the image <NUM> that may be displayed to represent the scene <NUM> on a display screen. A fragment of the scene <NUM> becomes a pixel in the image if the fragment is visible from the point of view associated with the image <NUM>. For clarity purpose, an element (for example a point) of the scene <NUM> will be called a fragment when positioning in the space of the 3D scene (the world space) and the same element visible from the point of view associated with the image <NUM> will be called pixel when positioning in the space of the image <NUM>. A fragment visible from the point of view associated with the image <NUM> and the corresponding pixel in the image thus refers to one and a same element of the scene <NUM> and may be mixed up in the rest of the description. Advantageously the fragment is defined by a set of data grouping together one or several of the following data items:.

As to determine whether a fragment of the scene is visible from the point of view of the image <NUM>, the well-known z-buffer method (also known as z-buffer algorithm) is used in association with one or more depth maps having the same point of view as the image <NUM> and having a structure similar to the depth map <NUM>. The one or more depth maps having the same point of view as the image <NUM> and being used in the z-buffer algorithm are also called z-buffer(s). According to this method and by taking the pixel 11p of the image <NUM> as an example, the depths of the fragments of the scene located along the viewing direction 110p having as origin the point of view of the image <NUM> and passing through the centre of the pixel 11p are compared, and the fragment P <NUM> having the smallest depth (i.e. the shortest distance from the point of view along the viewing direction 110p) is the one whose attributes are associated with the pixel 11p. According to a variant, the well-known painter's algorithm is used for solving the visibility problem as to determine which fragment of the scene is visible for each pixel of the image <NUM> from the point of view of the image <NUM>.

According to the example of <FIG>, the point of view <NUM> advantageously corresponds to the light source of the scene <NUM> and the depth map <NUM> is used to determine which fragment of the scene <NUM> is lit by the light source and which fragment of the scene is in shadow. According to this example, the depth map <NUM> may also be called a shadow map. Then, as to determine whether the fragment P <NUM> (corresponding to the pixel 11p of the image <NUM>) is lit or in the shadow of an object of the scene <NUM>, the fragment P <NUM> is projected into the depth map along a viewing direction linking the point of view <NUM> and the fragment P <NUM>. The distance ∥LP∥ between the point of view <NUM> and the fragment P <NUM> is compared to the depth information ∥LP'∥ associated with the centre of the element 10p' corresponding to the projection of the fragment P <NUM> into the depth map <NUM>. If ∥LP∥ is greater than ∥LP'∥, then the fragment P <NUM> is in the shadow of the fragment P' <NUM> (i.e. the fragment P <NUM> is not directly lit by the light source L). If ∥LP∥ is less than or equal to ∥LP'∥, then the fragment P <NUM> is directly lit by the light source L. In an advantageous way, the depth information with which the distance ∥LP∥ is compared is retrieved from the depth information associated with the centre of the element 10p' and from the first information representing the variation of depth inside the element 10p'. This enables to determine with a better precision whether the fragment P <NUM> is lit or not by another fragment of the scene, especially when the projection point of the fragment P' <NUM> into the depth map does not coincide with the centre of the element 10p' while belonging to the element 10p'. This enables to correct some defects related to the finite resolution of the depth map <NUM>. Determining whether a fragment if lit enables to associate video information (e.g. RGB data) with the considered fragment in harmony with the character lit or not lit of the fragment when rendering the scene in the image <NUM>, i.e. when rendering the pixels of the image <NUM> associated with the considered fragment. The rendering of the pixels of the image is then more precise with less or no artefact when compared to rendering method using depth maps of the prior art.

According to a variant, the scene <NUM> is a real scene shot by a camera device generating the image <NUM> and advantageously one or more other images according to different points of view as to generate a stereoscopic image of the scene. According to this variant, a depth map is generated according to the same point of view as the image <NUM>, the depth map and the image <NUM> being associated with each other. Depth maps associated with the other images of the scene may also be generated. The depth information associated with the centre(s) of the element(s) of the depth map is for example generated via a depth sensor associated with the camera. According to another example, the depth map is generated by comparing two images of the scene <NUM> acquired according to two different points of view (for example a left image and a right image of the scene), a disparity information being retrieved by comparing on a pixel basis the two images for making a match between the pixels of each view by pairs of two, the disparity information corresponding to the difference in pixels according to the horizontal axis between the location of a pixel in the left image and the location of its corresponding pixel in the right image. The depth map according to this example is called a disparity map. The disparity map associated with the image <NUM> is representative of the disparity between image <NUM> (corresponding for example to a left image of a stereoscopic image) and another image of the stereoscopic image (for example the right image), another disparity map may be associated with the right image, this another disparity map being representative of the disparity between the right image and the left image <NUM>. Each disparity map is advantageously estimated by comparison and pairing of the pixels of the left image <NUM> (respectively right image) to the pixels of the right image (respectively left image <NUM>). The disparity associated with a pixel of the left image <NUM> (respectively right image) advantageously corresponds to the pixel distance between this pixel of the left image <NUM> (respectively right image) and the corresponding (or paired) pixel of right image (respectively left image <NUM>), that is to say, the pixel of the right image (respectively left image <NUM>) having video information (that is to say, color information) identical or similar to that of the pixel of left image <NUM> (respectively right image). The first information representative of the variation of depth is established and associated with the elements of the depth map <NUM> (disparity map) as it will described with more details with regard to <FIG>.

<FIG> shows a part <NUM> of the depth map <NUM> of <FIG>, according to a particular and non-limitative embodiment. This part of the depth map is centred on a first element <NUM>, which is surrounded by <NUM> elements of the depth map <NUM>, among them the elements <NUM>, <NUM> and <NUM>, called second elements. A depth information pz,<NUM> representative of the distance between the point of view <NUM> (not represented on <FIG>) and the fragment PT <NUM> is associated with the centre T <NUM> of the first element <NUM>. The point T <NUM> of the first element corresponds to the projection point of the fragment PT <NUM> along the viewing direction <NUM> which passes through the first element <NUM>. The point T' <NUM>, which belongs to the first element <NUM>, corresponds to the projection point of the fragment PT' <NUM> along the viewing direction <NUM> having as origin the point of view <NUM> associated with the depth map <NUM> and passing through the first element <NUM>. The first fragment <NUM> and the second fragment are advantageously close to each other (for example adjacent) and belongs to the same surface of an object of the scene, i.e. the surface of the object <NUM> which is lit by the light source L (not represented on <FIG>) corresponding to the point of view <NUM> associated with the depth map <NUM>. As to determine whether the fragment PT' <NUM> is lit, the depth of the fragment PT' <NUM> according to the point of view <NUM> is compared with the depth information associated with the point T' <NUM>, which is computed by using the depth information associated with the first element <NUM> (or with the centre T <NUM> of the first element <NUM>) and the one or more first information representative of the variation of depth in the first element <NUM> that is (are) associated with the first element <NUM>. With the depth maps of the prior art, the depth of the fragment PT' <NUM> would have only been compared with the depth information pz,<NUM> associated with the centre T <NUM> of the first element <NUM>, which may have led to visible artefacts. Indeed, by considering that the surface of the object <NUM> to which the two fragments <NUM> and <NUM> belong is nearly tangent to the viewing directions <NUM> and <NUM> of the first and second fragments (or in another words, the angle formed by the surface normal of the object <NUM> and by one viewing direction <NUM> or <NUM> is high, i.e. greater than a predetermined threshold angle, for example greater than <NUM>°), the comparison of the distance ∥LPT'∥ with the depth associated with the centre T <NUM> of the first element <NUM> (and which actually corresponds to the distance pz,<NUM> = ∥LPT∥) could establish that the second fragment PT' <NUM> is in the shadow of the first fragment PT <NUM>, especially when the second fragment PT' <NUM> is behind the first fragment PT <NUM> with regard to the point of view, or when the second fragment PT' <NUM> is located further from the first fragment PT <NUM> with regard to the point of view. This is of course an erroneous consideration due to the imprecision of the depth information stored in depth map of the state of the art that do not comprise the first information established as it will described with regards to <FIG>, <FIG>. Indeed, both fragments PT <NUM> and PT' <NUM> are lit by the light source L (or visible from the point of view <NUM>) as they belong to one and same surface of the object <NUM> and as their projection points T <NUM> and T' <NUM> in the depth map belong to the same first element <NUM>.

<FIG> show a first method to establish the one or more first information representative of the variation of depth in the first element <NUM>, according to a particular and non-limitative embodiment of the present principles. Common elements of <FIG> have the same reference numbers. <FIG> illustrates a first surface element <NUM> which corresponds to the projection of a second surface element belonging to the scene <NUM>, the second surface element corresponding for example to a mesh element (a polygon, for example a triangle) if the scene has been modelled by polygonal modelling or to a subdivision of a surface if the scene <NUM> has been modelled by subdivision of surfaces. According to the example of <FIG>, the first surface element takes the form of a triangle and is defined in the space (x, y) of depth map <NUM>. The first surface element <NUM> comprises at least <NUM> elements <NUM>, <NUM>, <NUM>, <NUM> forming a 2x2 square of elements. The element <NUM> corresponds to the first element <NUM> of <FIG>, the elements <NUM>, <NUM> and <NUM> being called second elements. dx represents the distance between the centre of the first element <NUM> and the centre of the second element <NUM> along the horizontal axis x, in the space (x, y) of the depth map. dy represents the distance between the centre of the first element <NUM> and the centre of the second element <NUM> along the vertical axis y, in the space (x, y) of the depth map. The first information representative of the variation of depth in the first element <NUM> in the space (x, y) of the depth map is advantageously established (i.e. computed or generated) during the rasterization process (which enables to determine which fragments of the scene <NUM> are part of the second surface elements of the scene <NUM> and thus of the corresponding first surface elements of the depth map) of the first surface element <NUM> in the depth map, the rasterization process being performed by the geometry shader of a graphics hardware pipeline associated with a GPU (Graphics Processing Unit) as described hereinafter with respect to <FIG>. The first information representative of the variation of depth in the first element along the x axis is established from the depth information associated with the centre of the first element <NUM> and from the depth information associated with the centre of the second element <NUM>, which is advantageously adjacent to the first element <NUM> along the x axis, by taking into account the distance dx separating the centre of the first element <NUM> and the centre of the second element <NUM> along the x axis. <FIG> illustrates the first element <NUM> and the second element <NUM> and the depth information d1 and d2 associated with their respective centres. d1 and d2 are represented with dashed arrows, the depth difference between the first element <NUM> and the second element <NUM> being represented with a line segment <NUM>, the distance dx <NUM> between the centres of respectively the first element <NUM> and the second element <NUM> being represented by a double arrow. The first information representative of the variation of depth in the first element <NUM> corresponds to the slope of this line segment <NUM>, which is obtained by establishing the partial derivative of the function of the straight line <NUM>.

The first information representative of the variation of depth in the first element in the space of the depth map along the x axis may be established with the following equation: <MAT>.

In a same way, a first information representative of the variation of depth in the first element along the y axis is established from the depth information associated with the centre of the first element <NUM> and from the depth information associated with the centre of the second element <NUM>, which is advantageously adjacent to the first element <NUM> along the y axis, by taking into account the distance dy separating the centre T <NUM> of the first element <NUM> and the centre of the second element <NUM> along the y axis. The first information representative of the variation of depth in the first element <NUM> in the space of the depth map along the y axis may be established with the following equation: <MAT>.

According to a variant, only one of the two first information (either along the x axis or along the y axis) is established as to minimize the computation amount performed in the rendering pipeline of the GPU.

The first information associated with the other elements <NUM>, <NUM> and <NUM> is advantageously established with the same group of elements <NUM> to <NUM>. For example, the first information <MAT> representative of the variation of depth in the element <NUM> along the x axis is established from the depth information p(<NUM>) associated with the centre of the element <NUM> and from the depth information p(<NUM>) associated with the centre T <NUM> of the element <NUM>, by taking into account the distance dx <NUM> separating the centre of the element <NUM> and the centre of the element <NUM>. The first information <MAT> representative of the variation of depth in the element <NUM> along the y axis is established from the depth information p(<NUM>) associated with the centre of the element <NUM> and from the depth information p(<NUM>) associated with the centre of the element <NUM>, by taking into account the distance dy separating the centre of the element <NUM> and the centre of the element <NUM>. Both first information (along the x axis and the y axis respectively) may be established via the following equations: <MAT> <MAT>.

<FIG> show a second method to establish the one or more first information representative of the variation of depth in the first element <NUM> of <FIG>, according to a particular and non-limitative embodiment. <FIG> illustrates a first surface element <NUM> which corresponds to the projection of a second surface element <NUM> belonging to the scene <NUM>, the second surface element corresponding for example to a mesh element (a polygon, for example a triangle) if the scene has been modelled by polygonal modelling or to a subdivision of a surface if the scene <NUM> has been modelled by subdivision of surfaces. The second surface element <NUM> is represented in the world space, i.e. the space of the scene <NUM>. The first surface element <NUM> corresponds to the projection of the second surface element <NUM> in the tridimensional space <NUM> associated with an orthographic camera associated with the depth map <NUM>. Using an orthographic camera enables to project the second surface element, which belong to the world space, into a tridimensional space (x, y, z) of the camera field of view associated with the depth map <NUM>. As to establish the one or more first information representative of the variation of depth in a first element of the depth map in the 2D-space of the depth map <NUM>, the function representative of the plane associated with the first surface element <NUM> in the 3D space of the orthographic camera <NUM> is determined and is as follow: <MAT>.

<FIG> diagrammatically illustrates the structure <NUM> of a memory element (or combination of memory elements) that represents the first element <NUM> of the depth map <NUM>, according to a particular and non limitative embodiment of the present principles. The structure <NUM> comprises <NUM> memory blocks <NUM>, <NUM>, <NUM>. The depth information pz,<NUM> associated with the centre of the first element is stored in the first block <NUM>, the first information dpz,<NUM>/dx representative of the variation of depth along the x axis of the space of the depth map is stored in the second block <NUM> and the first information dpz,<NUM>/dy representative of the variation of depth along the y axis of the space of the depth map is stored in the third block <NUM>. The structure of the depth map is for example a <NUM>-bits RGBA buffer, <NUM> bits being used for the block <NUM>, <NUM> bits for the second block <NUM> and <NUM> bits for the third block <NUM>. According to another example, the structure of the depth map is a <NUM>-bits data array stored in a RGBA buffer with <NUM> bits for the first block <NUM> and <NUM> bits for each block <NUM> and <NUM>. If only one first information is established (either along axis x or axis y), one of the block <NUM> and <NUM> is empty and the freed space memory may be used for storing the only one first information with more precision.

The first information stored in the depth map <NUM> and associated with the first element <NUM> of <FIG> may then be used for computing the depth information associated with the point T' <NUM> of the first element <NUM>. The depth information pz,T' associated with point T' <NUM> is computed with the following equation: <MAT>.

As to determine whether the fragment PT' <NUM> is lit or hidden from the light by the fragment PT <NUM>, the distance ∥LPT'∥ between the point of view <NUM> and the fragment PT' <NUM> is compared with the depth information pz,T' associated with the point T' <NUM> and computed via the equation <NUM>. If ∥LPT'∥ is less than or equal to pz,T' then the fragment PT' <NUM> is lit by the light source L <NUM> (not illustrated on <FIG> but illustrated on <FIG>). If ∥LPT'∥ is greater than pz,T' then the fragment PT' <NUM> is not lit by the light source L <NUM>.

<FIG> illustrate the encoding of the first information representative of the variation of depth in the first element <NUM> of the depth map <NUM>, according to a particular and non-limitative embodiment. Values of the first information representative of the variation of depth in a first element of a depth map may be significantly small in presence of front facing surfaces (i.e. surfaces of object(s) having a normal parallel or roughly parallel with the viewing direction passing through the center of the first element <NUM> of the depth map) and conversely very large in presence of grazing angle surfaces (i.e. surfaces of object(s) of the scene having a normal perpendicular or roughly perpendicular to the normal of the depth map). Such amplitude requires high precision float storage and may significantly increase the depth map storage. On the other hand, as the first information represents slopes of linear function of x (along x axis of the depth map) and y (along y axis of the depth map), it may be represented by slope angles as follow: <MAT> <MAT>.

These angular functions are defined in the boundary range <MAT> and may be stored in an unsigned <NUM> bits integer values for example, with a precision of <MAT>, i. e approximately <NUM> degrees. Figure 4A shows such a angular function <NUM>, the range <MAT> being normalized into [-<NUM>; <NUM>].

It may be observed in <FIG> that for low angles, the first information values (i.e. the derivative or rate) vary slowly and the angular precision is not critical. Conversely it becomes critical near domain bounds, i.e. near <MAT> and <MAT>. A small angular variation represent high derivative variation that may lead to erroneous depth reconstruction. Having higher precision may require an increasing data storage of <NUM> or <NUM> bits size for instance. According to one specific and non-limitative embodiment of the invention, the angular functions representative of the variation of depth in a first element of the depth map in the space of the depth map along axis x and y, i.e. the functions θx(pz,<NUM>) and θy(pz,<NUM>,), are encoded in odd power functions xy (for example x<NUM> or x<NUM>) in order to compress the angular domain around <NUM> and give higher precision for area near the domain bounds, i.e. near <MAT> and <MAT>. Such function <NUM> and effective compression is showed in <FIG> with an odd power function (<FIG> illustrating a normalized angular domain compression using odd power function (i.e. x<NUM> in the example of <FIG>)). Such an embodiment allows to record the first information (i.e. the derivatives) with a good precision with an only <NUM> bits data storage for each first information value. The final encoded first information values Edx(pz,<NUM>) and Edy(pz,<NUM>) (along the x axis and the y axis respectively) in an unsigned <NUM> bits integer are obtained as follow: <MAT> <MAT> with n representing a positive odd integer.

The decoding as to obtain the first information values is advantageously performed in scene rendering pass where depth comparison takes place as to determine whether a fragment is lit (or whether a fragment is visible). Decoded first information values <MAT> and <MAT> are obtained as follow: <MAT> <MAT>.

<FIG> diagrammatically shows a hardware embodiment of a device <NUM> configured for enriching the content associated with a first element of a depth map. The device <NUM> is also configured for the creation of display signals of one or several synthesis images representative of the scene <NUM>. The device <NUM> corresponds for example to a personal computer (PC), a laptop, a tablet, a Smartphone or a games console.

The device <NUM> comprises the following elements, connected to each other by a bus <NUM> of addresses and data that also transports a clock signal:.

The device <NUM> also comprises a display device <NUM> of display screen type directly connected to the graphics card <NUM> to display synthesized images calculated and composed in the graphics card, for example live. The use of a dedicated bus to connect the display device <NUM> to the graphics card <NUM> offers the advantage of having much greater data transmission bitrates and thus reducing the latency time for the displaying of images composed by the graphics card. According to a variant, a display device is external to the device <NUM> and is connected to the device <NUM> by a cable or wirelessly for transmitting the display signals. The device <NUM>, for example the graphics card <NUM>, comprises an interface for transmission or connection (not shown in <FIG>) adapted to transmit a display signal to an external display means such as for example an LCD or plasma screen or a video-projector.

It is noted that the word "register" used in the description of memories <NUM>, <NUM>, and <NUM> designates in each of the memories mentioned, both a memory zone of low capacity (some binary data) as well as a memory zone of large capacity (enabling a whole program to be stored or all or part of the data representative of data calculated or to be displayed).

When switched-on, the microprocessor <NUM> loads and executes the instructions of the program contained in the RAM <NUM>.

The random access memory <NUM> notably comprises:.

The algorithms implementing the steps of the method specific to the invention and described hereafter are stored in the memory GRAM <NUM> of the graphics card <NUM> associated with the device <NUM> implementing these steps. When switched on and once the parameters <NUM> representative of the environment are loaded into the RAM <NUM>, the graphic processors <NUM> of the graphics card <NUM> load these parameters into the GRAM <NUM> and execute the instructions of these algorithms in the form of microprograms of "shader" type using HLSL (High Level Shader Language) language or GLSL (OpenGL Shading Language) for example.

The random access memory GRAM <NUM> notably comprises:.

According to a variant, the first and second identifiers and the distances are stored in the RAM <NUM> and processed by the microprocessor <NUM>.

According to another variant, a part of the RAM <NUM> is assigned by the CPU <NUM> for storage of the identifiers and the distances if the memory storage space available in GRAM <NUM> is insufficient. This variant however causes greater latency time in the composition of an image comprising a representation of the environment composed from microprograms contained in the GPUs as the data must be transmitted from the graphics card to the random access memory <NUM> passing by the bus <NUM> for which the transmission capacities are generally inferior to those available in the graphics card for transmission of data from the GPUs to the GRAM and vice-versa.

According to another variant, the power supply <NUM> is external to the device <NUM>.

<FIG> shows a method for enriching the content associated with a first element of a depth map implemented for example in a device <NUM>, according to a first non-restrictive particularly advantageous embodiment of the invention.

During an initialisation step <NUM>, the different parameters of the device <NUM> are updated. In particular, the parameters representative of the scene <NUM> represented in the image <NUM> as well as the depth information comprised in the depth map <NUM> are initialised in any way.

Then during a step <NUM>, at least a first information representative of the variation of the depth in a first element (for example a texel or a pixel) of a depth map is established in the space of the depth map. The first information is used for enriching the content of depth map of the state of the art which only comprises one depth information (called second information representative of the depth), associated with the centre of the first element, corresponding to the depth of the fragment of the scene that is visible from the point of view associated with the depth map along a viewing direction having as origin the point of view of the depth map and passing through the first element. In an advantageous way, two first information are established, i.e. one first information representative of the variation of the depth along the horizontal axis of the space of the depth map and another first information representative of the variation of the depth along the vertical axis (which is perpendicular to the horizontal axis) of the space of the depth map. According to a variant, only one of the two first information is established, either the first information representative of the variation of the depth along the horizontal axis of the space of the depth map or the first information representative of the variation of the depth along the vertical axis of the space of the depth map. The first information is advantageously used in addition to the depth information associated with the centre of the centre of the first element to establish the depth information associated with any point of the first element, using the coordinates of this point expressed in the space of the depth map, instead of associated the depth information of the centre of the first element with any point of the first element as it is done according to the state of the art.

The first information is for example established from the depth information associated with the centre of the first element and with the depth information associated with the centre of a second element of the depth map. The first and second elements belongs to a same surface element resulting from the projection of a surface element of the scene into the depth map. The first and second elements used to establish the first information representative of the variation of the depth along the horizontal axis of the space of the depth map are advantageously adjacent along the horizontal axis. The first and second elements used to establish the first information representative of the variation of the depth along the vertical axis of the space of the depth map are advantageously adjacent along the vertical axis. The first information advantageously corresponds to the ratio of the difference of depth between the first element and the second element to the distance between the first element and the second element (for example to the distance between the centre of the first element and the centre of the second element).

According to another example, the first information is established by using the equation of a surface element comprising the first element and obtained by projecting a surface element of the scene into the space of a camera field of view associated with the depth map. The first information corresponds to the slope(s) of this surface element, the equation of which being expressed in the space of the camera field of view of the depth map, with regard to the horizontal axis and/or vertical axis of the space of the depth map.

According to the invention the one or two first information is expressed under the form of a slope angle representing the variation of the depth in the first element.

This (these) angular representation(s) of the first information is (are) encoded in odd power function.

The first information is stored in a buffer associated with the depth map, in association with the first element.

An image of the scene is rendered by displaying pixels associated with fragments of the scene which are visible from the point of view (i.e. the camera field of view) associated with the image. The color information (e.g. RGB information) associated with the displayed pixels of the image corresponds to the color information associated with the fragments associated with the pixels of the image. The depth of a fragment associated with a pixel of the image is advantageously compared to the depth associated with the point of projection of the fragment into the depth map associated with the scene along a viewing direction having as origin the point of view of the depth map and as end the considered fragment, the depth associated with the point of projection being retrieved from the first information and the second information associated with the element of the depth map comprising the point of projection. This comparison enables to determine whether the considered fragment is lit (when the light source of the scene corresponds to the point of view of the depth map), which may have an impact on the color information associated with the fragment and thus on the color information associated with the pixel of the image to be rendered.

The step <NUM> is advantageously reiterated for each element of the depth map and for each depth map associated with any image of the scene.

Naturally, the invention is not limited to the embodiments previously described.

In particular, the invention is not limited to a method for enriching the content of a depth map but also extends to any device implementing this method and notably any devices comprising at least one GPU. The implementation of calculations necessary to the generation of elementary geometries, to the determination of index values to be assigned to the elementary geometries is not limited either to an implementation in shader type microprograms but also extends to an implementation in any program type, for example programs that can be executed by a CPU type microprocessor.

The use of the invention is not limited to a live utilisation but also extends to any other utilisation, for example for processing known as postproduction processing in a recording studio for the display of synthesis images for example.

The invention also relates to a method (and a device configured) for encoding depth information representative of the variation of depth in an element of a depth map in the space of the depth map. The invention further relates to a method and device for determining the depth information associated with a point of a first element of depth map based on the depth information associated with the centre of the first element and on the additional depth information representative of the variation of the depth along the horizontal and/or vertical axis of the depth map.

Advantage is taken of the programmable rendering pipeline present in GPU (Graphic Processor Unit) to encode the first information (also called derivatives) in a first pass (shadow map rendering), decode them in a second pass (scene rendering) and use them to perform the depth comparison. The GPU implementation for a whole process combining several embodiments of the invention works as follow:.

The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, Smartphones, tablets, computers, mobile phones, portable/personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and/or depth information. Examples of such equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices. As should be clear, the equipment may be mobile and even installed in a mobile vehicle.

Additionally, the methods may be implemented by instructions being performed by a processor, and such instructions (and/or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette ("CD"), an optical disc (such as, for example, a DVD, often referred to as a digital versatile disc or a digital video disc), a random access memory ("RAM"), or a read-only memory ("ROM"). The instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination. Instructions may be found in, for example, an operating system, a separate application, or a combination of the two. A processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.

As will be evident to one of skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntax-values written by a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Additionally, one of ordinary skill will understand that other structures and processes may be substituted for those disclosed and the resulting implementations will perform at least substantially the same function(s), in at least substantially the same way(s), to achieve at least substantially the same result(s) as the implementations disclosed. Accordingly, these and other implementations are contemplated by this application.

Claim 1:
A computer-implemented
method for generating a depth map (<NUM>) associated with a scene (<NUM>) according to a point of view (<NUM>), depth information (<NUM>) being associated with each first element (<NUM>) of a plurality of first elements of the depth map (<NUM>),
characterized in that the method comprises, in a first pass,
storing at least a first information (<NUM>, <NUM>) in the depth map (<NUM>) in addition to the depth information (<NUM>), said at least a first information (<NUM>, <NUM>) being associated with said each first element (<NUM>) and representative of a variation of depth in said each first element (<NUM>) in the space of the depth map, wherein the at least a first information is expressed under the form of a slope angle (<NUM>) representative of the variation of depth in said each first element, wherein the at least a first information is encoded in odd power function (<NUM>) and
wherein the method further comprises, in a second pass,
decoding the at least a first information stored in the depth map (<NUM>),
determining whether a fragment of the scene associated with a pixel of an image of the scene is lit from the point of view of the depth map by comparing a depth of the considered fragment to a depth associated with a point of projection of the fragment into the depth map along a viewing direction having as origin the point of view of the depth map and as end the considered fragment, the depth associated with the point of projection being retrieved from the at least a first information associated with the first element of the depth map comprising the point of projection, and
displaying pixels associated with the fragments which are lit.