Augmented reality method applied to the integration of a pair of spectacles into an image of a face

Method for creating a final real-time photorealistic image of a virtual object, corresponding to a real object arranged on an original photo of a user, in a realistic orientation related to the user's position, includes: detecting the presence of an area for the object in the photo; determining the position of characteristic points of the area for the object in the photo; determining the 3D orientation of the face, the angles Φ and Ψ of the camera having taken the photo relative to the principal plane of the area; selecting the texture to be used for the virtual object, in accordance with the angle-of-view, and generating the view of the virtual object in 3D; creating a first layered rendering in the correct position consistent with the position of the placement area for the object in the original photo; obtaining the photorealistic rendering by adding overlays to obtain the final image.

This invention relates to the field of image processing and image synthesis. It relates more specifically to the real-time integration of a virtual object into photographs or videos.

BACKGROUND OF THE INVENTION AND PROBLEM STATEMENT

The context of the invention is the real-time virtual trying on of an object in the most realistic way possible; typically these objects are a pair of spectacles to be integrated into a photograph or a video representing the face of a person oriented substantially facing the camera.

The growth in Internet sales, a limited stock, or any other reason preventing or hindering the actual trying on of a real object, generate a need for the virtual trying on of this object. Current solutions, based on a virtual reality or augmented reality, are insufficient in the case of spectacles since they lack realism or interactivity. In addition most of the time they require a lot of data and lots of computing time.

OBJECTIVE OF THE INVENTION

The objective of this invention is to propose a method for modeling virtual spectacles representative of real spectacles and a method of integrating in real time these said virtual spectacles into a photograph or a video representing the face of a person, limiting the number of necessary data.

“Integration” means a positioning and realistic rendering of these virtual spectacles on a photo or a video representing a person without spectacles, thus generating a new photo or video equivalent to the photo or video of the individual that would have been obtained by photographing or filming the same person wearing the real spectacles corresponding to these virtual spectacles.

DESCRIPTION OF THE INVENTION

The invention envisages in the first place a method of creating a real-time photorealistic final image of a virtual object, corresponding to a real object, arranged on an original photo of a person in a realistic orientation linked to the position of said user, characterized in that it comprises the following steps:510: detecting the presence of a placement area for the object in an original photo,530: determining the position of characteristic points of the placement area for the object in the original photo,540: determining the 3D orientation of the face, i.e. the angles φ and ψ of the camera having taken the photo, relative to the principal plane of the placement area for the object,550: selecting the texture to be used for the virtual object, in accordance with the angle-of-view, and generating the view of the virtual object in the 3D (φ, ψ)/2D (Θ, s) position in question,560: creating a first rendering by establishing a layered rendering in the correct position consistent with the position of the placement area for the object in the original photo,570: obtaining the photorealistic rendering by adding overlays, referred to as semantic overlays, so as to obtain the final image.

According to a particular implementation of the method, the object is a pair of spectacles and the placement area is the user's face.

In that case, according to an advantageous implementation, step510uses a first boosting algorithm AD1trained to determine whether the original photo contains a face.

In a particular implementation of the method as described, step530consists of:determining a similarity β, to be applied to an original photo, to obtain a face similar to a reference face in magnification and orientation, anddetermining the position of the precise exterior corner A and the precise interior point B for each eye in the face of the original photo.

More specifically, in this case, step530advantageously uses an iterative algorithm that makes it possible to refine the value of the similarity β and the positions of the characteristic points:defining the first parameters of similarity β0=(tx0, ty0, s0, Θ0),characterizing the eyes in the original photo1of the user, from a predefined set of models of eyes DBmodels—eyesand evaluating the scale,re-evaluating the parameters of similarity β1=(tx1, ty1, s1, Θ1).

According to a particular implementation of the method, step530uses a second boosting algorithm trained with an eyes learning database, comprising a set of positive examples of eyes and a set of negative examples of eyes.

In a particular implementation of the method as described, step550consists of:1/ determining a simplified geometric model of the real pair of spectacles, said model comprising a predefined number N of surfaces and their normals, taking as the orientation of these normals the exterior of the envelop convex to the real pair of spectacles,2/ applying to it, from a predefined set of reference orientations, an orientation closest to angles φ and ψ,3/ calculating a texture of the simplified geometric model, positioned in the 3D orientation of the reference orientation closest to angles φ and ψ, using the texture of this reference orientation; this is equivalent to texturing each of the N surfaces of the simplified geometric model while classifying the surface in the current view into three classifications: interior surface of the frame, exterior frame of the frame, lens.

In this case, according to a more particular implementation, the simplified geometric model of a real pair of spectacles, consisting of a frame and lenses, is obtained in a phase100in which:a set of shots of the real pair of spectacles to be modeled is produced, with different angles-of-view and using different screen backgrounds with and without the real pair of spectacles,the simplified geometric model is constructed, consisting of a number N of surfaces surfacejand their normal {right arrow over (n)}j, beginning with a not very dense surface mesh and using an optimization algorithm that deforms the model's mesh so that the projections of its silhouette in each of the views best match the silhouettes detected in the images.

According to an advantageous embodiment, the number N of surfaces of the simplified geometric model is a value close to twenty.

According to a particular implementation of the method, phase100also comprises a step110consisting of obtaining images of the real pair of spectacles; the lens must match the lens intended for trying on500, and in this step110:the real pair of spectacles is photographed at high resolution according to V different reference orientations Orientationiand in N light configurations showing the transmission and reflection of the spectacle lens,these reference orientations are selected by discretizing a spectrum of orientations corresponding to possible orientations when spectacles are tried on,V*N high-resolution images of the real pair of spectacles, designated Image-spectaclesi,j, are obtained.

In this case, according to a particular implementation, the number V of reference orientations is equal to nine, and if an orthogonal reference space with axes x, y, z is defined, where the y-axis corresponds to the vertical axis, ψ to the angle of rotation around the x-axis, φ to the angle of rotation around the y-axis, the V positions Orientationiselected are such that the angle ψ substantially takes the respective values −16°, 0° or 16°, the angle φ takes the respective values −16°, 0° or 16°.

According to a particular implementation of the method:the first light configuration respects the colors and materials of the real pair of spectacles, using neutral light conditions; the V high-resolution transmission images Transmissionicreated in this light configuration allow the maximum transmission of light through the lenses to be revealed,the second light configuration highlights the geometric characteristics of the real pair of spectacles (4), using conditions of intense reflection; the V high-resolution reflection images Reflectioniobtained in this second light configuration reveal the physical reflection properties of the lens.

According to a particular implementation of the method, phase100comprises a step120of creating a texture overlay of the frame Framei, for each of the V reference orientations.

In this case, more specifically in this step120:for each of the V reference orientations, the high-resolution reflection image Reflectioniis taken,a binary image is generated with the same resolution as the high-resolution reflection image of the reference orientations; said binary image is called the lens silhouette Lensibinary. in this lens silhouette Lensibinary, the value of the pixel is equal to one if the pixel represents the lenses and zero otherwise.

Even more particularly, the shape of the lenses needed to generate the lens silhouette Lensibinaryis extracted using an active contours algorithm based on the assumption that the frame and the lenses have different transparencies.

According to an advantageous implementation, in step120:a lens overlay Lensioverlayis generated for each of the reference orientations by copying, for each pixel with a value equal to one in the binary overlay of the lens Lensibinary, the information contained in the high-resolution reflection image and assigning zero to the other pixels,

this lens overlay Lensioverlayis a high-definition cropped image of the lens using, for cropping the original high-definition image, the lens silhouette Lensibinary.the associated high-resolution reflection image Reflectioniis selected for each of the reference orientations, and a binary background image Backgroundibinaryis generated by automatically extracting the background,a binary image is generated from the binary overlay of the frame Frameibinary, by deducting from a neutral image the outline image of the lenses and the outline image of the background,a texture overlay of the frame behind the lens Frameibehind—lens, with the texture of the frame corresponding to the portion of the frame located behind the lenses, is generated for each of the reference orientations by copying, for each pixel with a value equal to one in the binary lens overlay Lensibinary, the information contained in the high-resolution transmission image Transmissioni, and assigning zero to the other pixels,a texture overlay of the frame outside the lens Frameiexterior—lensis generated by copying, for each pixel with a value equal to one in the binary frame overlay Frameibinarythe information contained in the high-resolution reflection image, and assigning zero to the other pixels,an overlay of the texture of the frame Frameiis defined as the sum of the overlay of the texture of the frame behind the lens Frameibehind—lensand the overlay of the texture of the frame outside the lens Frameiexterior—lens.

According to a particular implementation, in step550, the texture calculation is performed using overlays associated to the reference orientation closest to angles φ and ψ, by the following sub-steps:inversion of the normals {right arrow over (n)}jof each of the surfaces of the pair of spectacles modeled surfacejand projection of the frame overlay Framei, limited to the lens space of the reference orientation closest to angles φ and ψ, to obtain a texture overlay of the internal surface of the frame TextureFrameisurface—interior. that makes it possible to structure the arms of the frame seen through the lens, in a textured reference model, oriented according to the reference orientation closest to angles φ and ψ,projection of the frame overlay Framei, limited to the space outside the lens of the reference orientation closest to angles φ and ψ, to obtain a texture overlay of the external surface of the frame TextureFrameisurface—exteriorthat makes it possible to structure the surfaces of the frame outside the lens, in the textured reference model, oriented according to the reference orientation closest to angles φ and ψ,projection of the lens overlay limited to the lens to obtain a lens texture overlay TextureLensithat makes it possible to structure the lens, in the textured reference model, oriented according to the reference orientation closest to angles φ and ψ.

According to a particular implementation of the method as described, step560consists of generating an oriented textured model, oriented according to angles φ and ψ and according to the scale and orientation of the original photo, from a textured reference model, oriented according to the reference orientation closest to angles φ and ψ, and parameters of similarity β; this step comprises the following sub-steps:using a bilinear affine interpolation to orient an interpolated textured model according to the angles φ and ψ based on the textured reference model oriented according to the reference orientation closest to these angles φ and ψ,using the similarity β to be applied, so as to obtain the same scale, the same image orientation and the same centering as the original photo, thus producing an oriented textured model.

In this case, more specifically, step560also comprises a sub-step of geometrically varying the arms of the virtual spectacles according to the morphology of the face of the original photo, so as to obtain a spectacles overlay Spectadesoverlayof the virtual pair of spectacles and a binary overlay Spectadesoverlay—binary, oriented as the original photo, and which can therefore be superimposed on it.

According to a particular implementation of the method as described, step570consists of taking into account the light interactions due to wearing virtual spectacles, particularly the shadows cast onto the face, the visibility of the skin through the lens of the spectacles, the reflection of the environment on the spectacles.

According to a more particular implementation, step570comprises the following sub-steps:1/ creating a shadow map Visibilityifor each reference orientation, obtained by calculating the light occlusion produced by the real pair of spectacles on each area of the average face when the entire face is lit by a light source, said light source being modeled by a set of point sources emitting in all directions, located at regular intervals in a rectangle,2/ multiplying the shadow map and the photo to obtain a shadowed photo overlay, designated Lskin—Shadowed,3/ blending the shadowed photo overlay Lskin—Shadowedand the spectacles overlay Spectaclesoverlayby linear interpolation, depending on the coefficient of opacity α of the lens in an area limited to the binary overlay of the virtual pair of spectacles Spectaclesoverlay—binary, to obtain a final image; this is an image of the original photo on which an image of the selected model of spectacles is superimposed, oriented as the original picture and given shadow properties

According to a particular implementation, the method as described further comprises a phase200of creating a database of models of eyes DBmodels—eyes, comprising a plurality of photographs of faces referred to as learning photographs Appeyesk

In this case, more specifically, phase200advantageously comprises the following steps:step210, of defining a reference face shape and orientation by setting a reference interpupillary distance di0, by centering the interpupillary segment on the center of the image and orienting this interpupillary segment parallel to the image's horizontal axis,then, for each kthlearning photograph Appeyesknot yet processed:step230, of determining the precise position of characteristic points: exterior point Blk, Brk, and interior point Alk, Arkof each eye and determining the respective geometric center Glk, Grkof these eyes and the interpupillary distance dik,step231, of transforming this kthlearning photograph Appeyeskinto a gray-scale image Appeyes-grayk, and normalizing the gray-scale image by applying a similarity Sk(tx, ty, s, Θ) so as to establish the orientation and scale of the reference face (7) to obtain a kthgray-scale normalized learning photograph Appeyes—gray—normk,step232, of defining a window of fixed dimensions for each of the two eyes, in the kthgray-scale normalized learning photograph Appeyes—gray—normk: left patch Pikand right patch Prk; the position of a patch P is defined by the fixed distance Δ between the exterior point of the eye B and the edge of the patch P closest to this exterior point of the eye Bstep233, for each of the two patches Plk, Prkassociated to the kthgray-scale normalized learning photograph Appeyes—gray—normk, of normalizing the gray-scales,step234, for the first learning photograph Appeyes1, of storing each of the patches Pl1, Pr1, called descriptor patches, in the eyes database DBmodels—eyes,step235, for each of the patches P associated to the kthgray-scale normalized learning photograph Appeyes—gray—normk, of correlating the corresponding normalized texture column-vector T0with each of the normalized texture column-vectors T0iof the corresponding descriptor patches,step236, of comparing, for each of the patches Plk, Prk, this correlation measurement with a previously defined correlation threshold, and, if the correlation is less than the threshold, of storing patch P as a descriptor patch in the eyes database DBmodels—eyes.

According to a particular implementation, in this case, in step232, the fixed distance Δ is chosen so that no texture exterior to the face is included in patch P, and the width w and height h of patches Plk, Prkare constant and predefined, so that patch P contains the eye corresponding to this patch P in full, and contains no texture that is exterior to the face, irrespective of the learning photograph Appeyesk.

The invention also envisages in another aspect a computer program product comprising program code instructions for executing steps of a method as described when said program is run on a computer.

DETAILED DESCRIPTION OF A MODE OF IMPLEMENTATION OF THE INVENTION

The method here comprises five phases:

the first phase100is a method of modeling real pairs of spectacles allowing a spectacles database DBmodels—spectaclesof virtual models of pairs of spectacles to be populated,

the second phase200is a method of creating a database of models of eyes DBmodels—eyes,

the third phase300is a method of searching for criteria for recognizing a face in a photo.

the fourth phase400is a method of searching for criteria for recognizing characteristic points in a face.

the fifth phase500, referred to as trying on virtual spectacles, is a method of generating a final image5, from a virtual model3of a pair of spectacles, and an original photo1of a subject taken, in this example, by a camera and representing the face2of the subject.

The first four phases,100,200,300and400, are performed on a preliminary basis, while phase500of trying on virtual spectacles is utilized many times, on different subjects and different virtual pairs of spectacles, based on the results from the four preliminary phases.

Phase100of Modeling Pairs of Spectacles

To begin with the first phase100, the modeling of pairs of spectacles, is described:

The purpose of this phase of modeling pairs of spectacles is to model a real pair of spectacles4geometrically and texturally. The data calculated by this spectacles modeling algorithm, for each pair of spectacles made available during the trying-on phase500, are stored in a database DBmodels—spectaclesso as to be available during this trying-on phase.

This spectacles modeling phase100is divided into four steps.

Step110: Obtaining Images of the Real Pair of Spectacles4

The procedure for constructing a simplified geometric model6of a real pair of spectacles4, uses a device for taking photographs50.

This device for taking photographs50is, in this example, represented inFIG. 2and consists of:a base51, which allows the real pair of spectacles4to be modeled to be supported. This base51is made of a transparent material such as transparent plexiglass. This base51is formed of two parts,51aand51b, which fit together. Part51bis the portion of the base51that is in contact with the real pair of spectacles4when this is placed on the base51. Part51bcan be separated from part51aand can therefore be chosen from a set of parts with shapes optimized with respect to the shape of the object to be placed (goggles, masks, jewelry). Part51bhas three contact points with the real pair of spectacles4, corresponding to the actual contact points on a face when the real pair of spectacles4is worn, i.e. at the two ears and the nose.a turntable52on which part51aof base51is fixed, said turntable52being placed on a foot53; said turntable52makes it possible to rotate the removable base according to a vertical axis of rotation Z.a vertical rail54allowing digital cameras55to be attached at different heights (the number of digital cameras55is variable, from one to eight in this example). The digital cameras55are respectively fixed on the vertical rail54by a ball joint allowing rotation in pitch and yaw. This said vertical rail54is positioned at a distance from the foot53, which is fixed in this example. The cameras are oriented such that their respective photographic field contains the real pair of spectacles4to be modeled, when it is placed on part51bof base51, part51bbeing fitted onto part51a.a horizontal rail56secured to a vertical mount on which a screen58with a changeable background color59is attached. In this example, screen58is an LCD screen. The background color59is selected in this example from the colors red, blue, green, white or neutral, i.e. a gray containing the three colors red, green, blue in a uniform distribution with a value of two hundred, for example. Said horizontal rail56is positioned such that the real pair of spectacles4to be modeled, placed on part51bfitted onto fixed part51aon the turntable52, is between the screen58and the vertical rail54.possibly a base plate60supporting the vertical rail54, the foot53and the horizontal rail56.

The device for taking photographs50is controlled by a unit associated to a software system61. This control consists of managing the position and orientation of digital cameras55, relative to the object to be photographed, assumed to be fixed, for managing the background color59of the screen58and its position, and managing the rotation of the turntable52.

The device for taking photographs50is calibrated by conventional calibration procedures in order to accurately know the geometric position of each of the cameras55and the position of the vertical axis of rotation Z.

In this example, calibrating the device for taking photographs50consists of:firstly, placing one of the digital cameras55sufficiently precisely at the level of the real pair of spectacles4to be modeled, so that its respective shot is a frontal view,secondly, to remove the real pair of spectacles4and possibly removable part51b, and place a test chart57, not necessarily flat, vertically on the turntable52. In this non-limiting example, this test chart57consists of a checkerboard.thirdly, to determine the precise position of each digital camera55by a conventional method, using images62obtained for each of the digital cameras55with different shots of the test chart57, using different screen backgrounds59.fourthly, to determine the position of the vertical axis of rotation Z of the turntable52using the images62.

The first step110of the spectacles modeling phase consists of obtaining images of the real pair of spectacles4from a number of orientations (preferably keeping a constant distance between the camera and the object to be photographed), and under a number of lighting conditions. In this step110, the lens4bmust match the lens intended for the trying-on phase500.

The real pair of spectacles4is photographed with a camera, at high resolution (typically a higher resolution than 1000×1000) in nine (more generally V) different orientations and in N light configurations showing the transmission and reflection of the spectacle lens4b.

These nine (V) orientations are called reference orientations and in the rest of the description are designated by Orientationi. These V reference orientations Orientationiare selected by discretizing a spectrum of orientations corresponding to possible orientations when spectacles are tried on. V*N high-resolution images of the real pair of spectacles4are thus obtained, designated Image-spectaclesi,j(1≦i≦V, 1≦j≦N).

In the present example, the number V of reference orientations Orientationiis equal to nine, i.e. a relatively small number of orientations from which to derive a 3D geometry of the model. However, it is clear that other numbers of orientations may be envisaged with no substantial change to the method according to the invention.

If an orthogonal reference space with axes x, y, z is defined, where the y-axis corresponds to the vertical axis, 4/ to the angle of rotation around the x-axis, φ to the angle of rotation around the y-axis, the nine positions Orientationiselected here (defined by the pair φ, ψ) are such that the angle ψ takes the respective values −16°, 0° or 16°, the angle φ takes the respective values −16°, 0° or 16°.

FIG. 4represents a real pair of spectacles4and the nine orientations Orientationiof the shots.

In the present implementation example of the method, two light configurations are chosen, i.e. N=2. By choosing nine camera positions (corresponding to the reference orientations Orientationi), i.e. V=9, and two light configurations, N=2, eighteen high-resolution images Image-spectaclesi,jrepresenting a real pair of spectacles4are obtained; these eighteen high-resolution images Image-spectaclesi,jcorrespond to the nine orientations Orientationiin the two light configurations.

The first light configuration respects the colors and materials of the real pair of spectacles4. Neutral conditions of luminosity are used for this first light configuration. The nine (and more generally V) images Image-spectaclesi,lcreated in this light configuration allow the maximum transmission of light through the lenses4bto be revealed (there is no reflection on the lens and the spectacle arms can be seen through the lenses). They are called high-resolution transmission images and in the rest of the description are designated by Transmissioni; the exponent i is used to characterize the ithview, where i varies from 1 to V.

The second light configuration highlights the special geometric features of the real pair of spectacles4, such as, for example, the chamfers. This second light configuration is taken in conditions of intense reflection.

The high-resolution images Image-spectaclesi,2obtained in this second light configuration reveal the physical reflection properties of the lens4b(the arms are not seen behind the lenses, but reflections of the environment on the lens are; transmission is minimal). The nine (or V) high-resolution images of the real pair of spectacles4, created in this second light configuration are called high-resolution reflection images and in the rest of the description are designated by Reflectioni; the exponent i is used to characterize the ithview, where i varies from 1 to V.

According to the method just described, the set of high-resolution images Image-spectaclesi,jof real pairs of spectacles comprises, by definition, both the high-resolution transmission images Transmissioniand the high-resolution reflection images Reflectioni. Obtaining the set of high-resolution images Image-spectaclesi,jby this step110is illustrated inFIG. 5.

Step120: Generating Overlays of Spectacles

The second step120of spectacles modeling phase100consists of generating overlays for each of the nine reference orientations Orientationi. A schematic of this second step120is shown inFIG. 6. It is understood that an overlay is defined here in the sense known to the expert in the field of image processing. An overlay is a raster image with the same dimensions as the image from which it is derived.

For each of the nine (and more generally V) reference orientations Orientationi, the high-resolution reflection image Reflectioniis taken. A binary image is then generated with the same resolution as the high-resolution reflection image of the reference orientations. This binary image actually shows the “outline” shape of the lenses4bof the real pair of spectacles4. This binary image is called a lens silhouette and is designated Lensibinary.

Extraction of the shape of the lenses needed generate the lens silhouette is performed by an active contours algorithm (e.g. of a type known to those skilled in the art under the name “2D snake”) based on the assumption that the frame4aand the lenses4bhave different transparencies. The principle of this algorithm, known per se, is to deform a curve having several deformation constraints. At the end of the deformation, the optimized curve follows the shape of the lens4b.

The curve to be deformed is defined as a set of 2D points placed on a line. The kthpoint of the curve associated with the coordinate xk in the high-resolution reflection image Reflectioniassociated to a current reference orientation, has an energy E(k). This energy E(k) is the sum of an internal energy Einternal(k) and an external energy Eexternal(k). The external energy Eexternal(k) depends on the high-resolution reflection image Reflectioniassociated to a current reference orientation, whereas the internal energy Einternal(k) depends on the shape of the curve. This therefore gives Eexternal(k) ∇(xk), where ∇ is the gradient of the high-resolution reflection image Reflectioniassociated to a current reference orientation. The internal energy Einternal(k) is the sum of an energy referred to as the “balloon” energy Eballoon(k), and a curvature energy Ecurvature(k) This therefore gives Einternal(k)=Eballoon(k)+Ecurvature(k)

The balloon energies Eballoon(k) and the curvature energies Ecurvature(k) are calculated using standard formulas in the field of active contour methods, such as the method known as the Snake method.

In this lens silhouette Lensibinary, the value of the pixel is equal to one if the pixel represents the lenses4b, and zero if not (which, in other words, in effect forms an outline image).

It is understood that it is also possible to use gray scales (values between 0 and 1) instead of binary levels (values equal to 0 or 1) to produce such a lens overlay (for example by creating a gradual transition between the values 0 and 1 either side of the optimized curve obtained by the active contours method described above).

A lens overlay, designated Lensioverlay, is then generated for each of the nine (V) reference orientations by copying, for each pixel with a value equal to one in the lens silhouette Lensibinary, the information contained in the high-resolution reflection image Reflectioniand assigning zero to the other pixels. The exponent i of variables Lensibinaryand Lensioverlayvaries from 1 to V, where V is the number of reference orientations.

This lens overlay Lensioverlayis, to some extent, a high-definition cropped image of the lens using, for cropping the original high-definition image, the lens silhouette Lensibinary(outline shape) created previously.

Designating the term to term matrix product operator by, this gives:
Lensioverlay=LensibinaryReflectioni(Eq 1)
Thus, for a pixel with position x, y
Lensioverlay(x,y)=Lensibinary(x,y)×Reflectioni(x,y)

For each of the reference orientations, the associated high-resolution reflection image Reflectioniis chosen, and then, for each of them, a binary background image Backgroundibinaryis then generated by automatically extracting the background, using a standard image background extraction algorithm. A binary image, called binary frame overlay Frameibinary, is then generated for each of the V reference orientations, by deducting from a neutral image the outline image of the lenses and the outline image of the background, i.e. in more mathematical terms, by applying the formula:
Frameibinary=1−(Lensibinary+Backgroundibinary)  (Eq 2)

An overlay, referred to as the texture overlay of the frame behind the lens Frameibehind—lens, is then generated of the texture of the frame corresponding to the portion of the frame located behind the lenses4b(for example, a portion of the arms may be visible behind the lens4bdepending on the orientation) for each of the nine (V) reference orientations, by copying, for each pixel with a value equal to one in the binary lens overlay Lensibinary, the information contained in the high-resolution transmission image Transmissioni, and assigning zero to the other pixels.
This gives: Frameibehind—lens=Lensibinary(x,y)×Transmissioni(Eq 3)
Thus, for a pixel with position x, y:
Frameibehind—lens(x,y)=Lensibinary(x,y)×Transmissioni(x,y)

Similarly an overlay, referred to as the texture overlay of the frame outside the lens Frameiexterior—lensis generated for each of the nine (V) reference orientations by copying, for each pixel with a value equal to one in the binary frame overlay Frameibinary, the information contained in the high-resolution reflection image Reflectioniand assigning zero to the other pixels.

The exponent i of variables Frameibinary, Backgroundibinary, Frameiexterior—lensand Frameibehind—lensvaries from 1 to V, where V is the number of reference orientations Orientationi.
This gives: Frameiexterior—lens=FrameibinaryReflectioni(Eq 4)

A texture overlay of the frame Frameiis defined as the sum of the texture overlay of the frame behind the lens Frameibehind—lensand the texture overlay of the frame outside the lens Frameiexterior—lens.
This gives: Framei=Frameibehind—lens=Frameiexterior—lens(Eq 5)

Step130: Geometric Model

The third step130, of the spectacles modeling phase100consists of obtaining a simplified geometric model6of a real pair of spectacles4. A real pair of spectacles4comprises a frame4aand lenses4b(the notion of lenses4bcomprises the two lenses mounted in the frame4a). The real pair of spectacles4is represented inFIG. 1a.

This step130does not involve the reflection characteristics of the lenses4bmounted in the frame4a; the real pair of spectacles4may be replaced by a pair of spectacles comprising the same frame4awith any lenses4bhaving the same thickness and curvature.

This simplified geometric model6, can be obtained:either by extracting its definition (radius of curvature of the frame, dimensions of the frame) from a database DBmodels—spectaclesof geometric models associated to pairs of spectacles.or, according to the preferred approach, by constructing the simplified geometric model6using a construction procedure. The new geometric model6, thus created, is then stored in a database of models DBmodels—spectacles.

There are several possible ways to construct a geometric model suitable for the rendering method described in step120. One possible method is to generate a dense3dmesh that faithfully describes the shape of the pair and is extracted either by automatic reconstruction methods [C. Hernández, F. Schmitt and R. Cipolla, Silhouette Coherence for Camera Calibration under Circular Motion, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 2, pp. 343-349, February, 2007] or by exploiting existing 3D models from manual modeling by CAD (Computer Aided Design) systems. A second method consists of modeling the real pair of spectacles4by a 3D active contour linked to a surface mesh. An optimization algorithm deforms the model so that the projections of its silhouette in each of the views best match the silhouettes detected in the images (using a procedure as described).

The real pair of spectacles4is modeled by a surface mesh that is dense or has a low number of facets (traditionally known by the name “low polygon number” or “LowPoly”). This last method is. The initial shape is used to introduce one a priori with a weak shape; it can be generic or chosen from a database of models according to the pair to be reconstructed. In what follows, the case of a simplified geometric model (i.e. a “low polygons” type) will be described.

The mesh comprises N summits, designated Vi. The mesh has the shape of a triangle strip, as shown inFIG. 1b. Furthermore it is assumed that the number of summits on the upper contour of the mesh is equal to the number of summits on the lower contour of the mesh, and that the sampling of these two contours is similar. Thus, an “opposite” summit, Vi+can be defined for each summit Vi.

Regardless of the actual topology of the mesh, the neighborhood Niof the summit Viis described by Ni={Vi+1; Vi−1; Vi+}

The summits Vi+1and Vi−1are the neighbors of Vialong the contour of the mesh. The summit Vi+corresponds to the summit opposite to Vi, as defined earlier. This neighborhood also allows two triangles Ti1and Ti2to be constructed (seeFIG. 1c). Let n1and n2be their respective normals. The normal to the surface in segment Vi+Vi(which is a topological peak or not) is defined by

To develop the active contour to the image data, an energy is associated to the current 3D model: the closer the projected silhouettes of the model are to the contours in the images, the lower this energy is. Each summit is then displaced iteratively so as to minimize this energy until convergence (i.e. until the energy is no longer reduced by a displacement). In addition, one seeks to obtain a smooth model, which leads us to define at each summit an internal energy not dependent on images. The energy associated to the summit Viis given by:
Ei=λd,i+λrEr,i+λcEc,i+λoEo,i(Eq 7)

The term Ed,iis the linking term to the image data, i.e. to the contours calculated in the different views. The three other terms are smoothing terms, which do not depend on images.

The term Er,iis a repulsion term that tends to distribute the summits uniformly.

The term Ec,iis a curvature term that tends to make the surface smooth.

Finally the term Eo,iis an obliquity term aimed at minimizing the gap in the (x; y) plane between Vi and Vi+

The weights λd, λr, λc, λoare common to all the summits and in general λd, λr, λc, λo.

The linking term to data Ed,icharacterizes the proximity of the silhouette of the current active contour with the contours detected in the images (by an active contour procedure as described in step120above). In the acquisition process, an automatic cropping phase, of a type known per se (“difference matting”), provides an opacity map for each view.

The contours are obtained by thresholding the gradient of this opacity map. The contour information is propagated to the entire image by calculating, for each view k, a map of distances to the contours, designated Dk. The projection model of the 3D model in the images is a model of pinhole camera, of a type known per se, defined by the following elements:a matrix Kk(3×3 matrix) containing the camera's internal parameters,a matrix Ek=[Rk|tk](3×4 matrix) describing the switch from the world reference space (as presented inFIG. 1b) to the camera reference space of view k.

ψk⁡(x;y;z)=(uw,vw)T
designates the projection of 3D point (x, y, z)Tin view k. It is obtained by

The linking energy to the data is thus expressed by:

Ed,i=1S⁢∑k∈S⁢(Dk⁡(Ψk⁡(Vi)))2(Eq⁢⁢9)
where S and the set of views in which the summit Viis visible and |S| its cardinal.

The repulsion term Er,itends to minimize the difference in length of two peaks of the contour joining at Vi. It is expressed by:

The curvature term Ec,itends to reduce the curvature perpendicular to segment Vi+Vi

The corresponding energy is expressed by
Ec,i=(1−n1Tn2)2(Eq 11)
where n1and n2are the normals defined above.

The obliquity term Eo,itends to preserve the vertical correspondence between the points of the upper contour and the points of the lower contour. For this, it is assumed that the orientation of the model of the spectacles is as inFIG. 1, namely that the z axis is the axis perpendicular to the natural plane of the pair “placed on the table”.
This thus gives Eo,i=(diTa)2(Eq 12)
where didesignates segment Vi+Vi

The resolution is done by scanning each summit Viof the mesh iteratively and one seeks to minimize the associated energy function Ei. This is a nonlinear function, therefore a Newton type of iterative minimization method is used. The development, limited to the second order of the energy function, for a small displacement δiof the summit, is expressed by:
Ei(Vi+δi)≈(Vi)+∇EiTδi+δiTHEiδi(Eq 13)
where ∇Eiis the gradient of Eiand HEiis its Hessian matrix (both evaluated in Vi).

The initial non-linear minimization problem is replaced by a succession of linear problems.

Let f(δi)=Ei(Vi)+∇EiTδi+δiTHEiδiand one seeks the minimum {circumflex over (δ)}iof f relative to δi.

At each iteration, the summit Vik−1is displaced in the direction {circumflex over (δ)}ik
Vik=Vik−1+δk{circumflex over (δ)}ik(Eq 14)

The length of step λkis either optimized (a standard method referred to as “line-search”), or determined beforehand and left constant throughout the procedure.

The iterative procedure described above is stopped when the step is normally below a threshold, when more than kmaxiterations have been performed, or when the energy Eidoes not reduce sufficiently from one iteration to the next.

In a variant to this construction procedure, 3D modeling software is used to model the geometry of the real pair of spectacles4.

In another variant of this construction procedure, a model of the database of models DBmodels—spectaclesis used and it is adapted manually.

The simplified geometric model6is formed of a number N of polygons and their normals, taking as the orientation of these normals the exterior of the envelop convex to the real pair of spectacles1. In this non-limiting example the number N is a value close to twenty.

FIG. 1drepresents a simplified model for a pair of wraparound sports spectacles. In the remainder of the description, these polygons of the simplified geometric model6, are called the surfaces of the modeled pair of spectacles designated by surfacej. The normal to a surface of the modeled pair of spectacles surfacejis designated by {right arrow over (n)}j; j is a numbering index of the surfaces surfacejwhich varies from 1 to N.

A schematic of step130is shown inFIG. 3.

Step140: Creating a Shadow Map

In this step a shadow map, designated Visibilityi, is created for each of the reference orientations Orientationi. The goal is to calculate the shadow produced by the pair of spectacles on a face, modeled here by an average face20, a 3D model constructed in the form of a mesh of polygons (seeFIG. 7a).

The modeling of the face in question corresponds to an average face20, which makes it possible to calculate a shadow suitable for any person. The method calculates the light occlusion produced by the pair of spectacles on each area of the average face20. The technique envisaged allows very faithful shadows to be calculated while requiring only a simplified geometric model6of the real pair of spectacles4. This procedure is applied to calculate the shadow produced by the pair of spectacles, for each image of said pair of spectacles. The final result obtained are 9 shadow maps Visibilityicorresponding to the 9 reference orientations Orientationiused, in this example, during the creation of the image-based rendering.

For each reference orientation, this shadow map Visibilityiis calculated using the simplified geometric model6of the real pair of spectacles4(“low polygons” surface simplified model, see step130), a textured reference model9(superimposition of texture overlays of the pair of spectacles corresponding to a reference orientation) oriented according to the reference orientation Orientationi, a modeling of an average face20, a modeling of a light source21and a modeling22of a camera.

The shadow map Visibilityiis obtained by calculating the light occlusion produced by each elementary triangle forming the simplified geometric model6of the real pair of spectacles4, on each area of the average face20, when everything is lit by the light source21. The light source21is modeled by a set of point sources emitting in all directions, located at regular intervals in a rectangle, for example as a 3×3 matrix of point sources.

The modeling22of a camera is standard modeling of a type known as pinhole, i.e. modeling without a lens and with a very small and simple opening. The shadow map Visibilityiobtained is an image comprising values between 0 and 1.

The coordinates (X,Y) of the 2D projection of a vertex (x,y,z) of the 3D scene is expressed as follows:

X=u⁢⁢0+f×xz,Y=v⁢⁢0+f×yz(Eq⁢⁢15)
in which the parameters u0, v0, f characterize the camera.

Let K designate the operator that, at a vertex V(x,y,z), associates its projection P(X,Y) in the image. A set of 3D points {V} corresponds to a pixel P with coordinates (X,Z) such that K(V)=P.

The set of these 3D points forms a radius. Subsequently, when reference is made to a 3D radius associated with a pixel, the 3D radius corresponds to the set of 3D points projected on the pixel.

To calculate the shadow, for each pixel P with coordinate (i,j) in the shadow image Visibilityi, the value O(i,j) of the shadow image is calculated. For this, the 3D vertex V of the face that corresponds to the projection P is calculated. This term V is defined as the intersection of the 3D radius defined by the pixel and the 3D model of the face20(seeFIG. 7b).

Then, the light occlusion produced by the pair of spectacles on this vertex is calculated. To do this, the light occlusion produced by each triangle of the low-resolution geometric model6is calculated.

Let A(m), B(m), C(m) designate the three summits of the mth triangle of the low-resolution geometric model6. For each light point source Sn, the intersection tn of the light ray passing through Vis calculated.

Tn is the 2D projection of vertex tn on the texture image (textured reference model9of the pair of spectacles). The transparency of the texture is known from step (120) of cropping on differences, therefore, the pixel Tn has a transparency, designated by a(Tn).

Finally, the value of pixel O(i,j) of the shadow image is expressed as follows:

The Coefficient term allows the opacity of the shadow Visibilityito be adjusted according to the visual rendering wanted.

The data obtained in phase100are stored in a spectacles database DBmodels—spectaclesthat contains, for each pair of spectacles modeled, the simplified geometric model6of this real pair of spectacles4, the lens overlays Lensioverlay, the overlays of the frame behind the lens Frameibehind—lensand the overlays of the frame outside the lens Frameiexterior—lens, for each of the V reference orientations.

In addition, data specific to the lenses4bof the real pair of spectacles4are added to the previously mentioned data in the spectacles database DBmodels—spectacles, such as its coefficient of opacity α, known by the manufacturer, and possibly supplied for each reference orientation.

Phase200of Creating a Database of Models of Eyes DBmodels—Eyes

The second phase200makes it possible to create a database of models of eyes, DBmodels—eyes. To simplify its description, it is subdivided into ten steps (210,220,230to236and240). The database of models of eyes, DBmodels—eyes, thus obtained is used, in the trying-on phase500, to characterize the eyes of a person photographed.

This eyes database DBmodels—eyescan be created, for example, from at least two thousand photographs of faces, referred to as learning photographs Appeyesk(1≦k≦2000). These learning photographs are advantageously, but not obligatorily, the same size as the images of models of spectacles and of the face of the user in the trying-on method.

Step210. When this eyes database DBmodels—eyesis created, first of all a reference face7shape is defined by setting a reference interpupillary distance di0, by centering the interpupillary segment on the center of the image and orienting the interpupillary segment parallel to the horizontal axis of the image (face not tilted). The reference face7is therefore centered on the image, with the face orientation and magnification depending on the reference interpupillary distance di0.

Step220. In a second step a correlation threshold is defined.

Step230—The precise position of characteristic points (corners of the eyes) are determined, manually in this example, i.e. the position of the exterior point Blk, Brkof each eye (left and right respectively with these notations) and the position of the interior point Alk, Ark, as defined inFIG. 8. Each position is determined by its two coordinates within the image.

The respective geometric centers Glk, Grkof these eyes are determined, calculated as the barycenter of the exterior point Bkof the corresponding eye and the interior point Akof this eye, and the interpupillary distance dikis calculated.

Step231—This kthlearning photograph Appeyeskis transformed into a gray-scale image Appeyes-grayk, by an algorithm known per se, and the gray-scale image is normalized by applying a similarity Sk(tx, ty, s, Θ) so as to establish the orientation (front view), scale (reference interpupillary distance di0) of the reference face7.

This similarity Sk(tx, ty, s, Θ) is determined as the mathematical operation to be applied to the pixels of the learning photograph Appeyeskto center the face (center of eyes equal to the center of the photograph), orientation of face and magnification depending on the reference interpupillary distance di0. The terms tx and ty designate the translations to be applied on the two axes of the image so as to establish the centering of the reference face7. Similarly, the term s designates the magnification factor to be applied to this image, and the term Θ designates the rotation to be applied to the image so as to establish the orientation of the reference face7.

A kthgray-scale normalized learning photograph Appeyes—gray—normkis thus obtained. The interpupillary distance is equal to the reference interpupillary distance di0. The interpupillary segment is centered on the center of the kthgray-scale normalized learning photograph Appeyes—gray—normk. The interpupillary segment is parallel to the horizontal axis of the gray-scale normalized learning photograph Appeyes—gray—normk.

Step232—A window, rectangular in this example, with a fixed size (width w and height h) is defined for each of the eyes, in the kthgray-scale normalized learning photograph Appeyes—gray—normk. These two windows are called the left patch Plkand right patch Prkin the remainder of this description, according to a standard usage in this field. For simplicity, the term patch P will be used to denote either one of these patches Plk, Prk. Each patch P is a sub-raster image extracted from an initial raster image of a face. It is clear that, in a variant, a shape other than rectangular may be used for the patch, for example polygonal, elliptical or circular.

The position of the patch P corresponding to an eye (left, right respectively), is defined by the fixed distance Δ between the exterior point of the eye B and the edge of the patch P closest to this exterior point of the eye B (seeFIG. 7).

This fixed distance Δ is chosen so that no texture exterior to the face is included in the patch P. The width w and height h of patches Plk, Prkare constant and predefined, so patch P contains the eye corresponding to this patch P in full, and contains no texture that is external to the face, irrespective of the learning photograph Appeyesk.

Step233—For each of the two patches Plk, Prkassociated to the kthgray-scale normalized learning photograph APPeyes—gray—normk(each corresponding to one eye), the gray-scales are normalized.

To do this, a texture column-vector T, called the original texture column-vector, is defined, comprised of the gray-scales for patch P, in this example stored in row order the size of the texture column-vector T is equal to the number of lines (h) multiplied by the number of columns (I) and a column-vector I with a unit value is defined, the same size as the texture column-vector T.

The mathematical operation therefore consists of calculating the mean of the gray-scales of patch P, mean designated μT, of normalizing the standard deviation of these gray-scales, designated σT, and of applying the formula:

T⁢⁢0=(T-μT⁢I)σT(Eq⁢⁢17)
where T0is the normalized texture column-vector (gray-scale) and T the original texture column-vector.

Step234—This step234is only performed for the first learning photograph Appeyes1. The eyes database DBmodels—eyesis therefore empty.

For the first learning photograph Appeyes1processed, each of the patches Pl1, Pr1is added to the eyes database DBmodels—eyes; with the following data stored:the normalized texture column-vector T0l1, T0r1corresponding to a patch Pl1, Pr1,the precise position of the normalized characteristic points, by applying the similarity S1(tx, ty, s, Θ) to the precise positions of characteristic points identified beforehand in the learning photograph Appeyes1,the similarity S1(tx, ty, s, Θ),and other useful information: morphology, brightness etc.,
then one goes to step230for processing the second learning photograph Appeyes2and the following ones Appeyesk.

Patches Pl1, Pr1stored in the eyes database DBmodels—eyesin this step234and in step236are called descriptor patches.

Step235—For each of the patches P associated to the kthgray-scale normalized learning photograph Appeyes—gray—normk(each corresponding to one eye), the corresponding normalized texture column-vector T0is correlated with each of the normalized texture column-vectors T0of the corresponding descriptor patches.

In this non-limiting example a correlation measurement Znccis used, defined for example by
Zncc(T0,T0i)=tT0*T0i(Eq 18)
(wheretT0designates the transposed vector of the normalized texture column-vector T0). As the sizing of patches Plk, Prkis constant, the normalized texture column-vectors T0, T0iall have the same size.

Step236—For each of the patches Plk, Prk, this correlation measurement Znccis compared against the previously defined correlation threshold. If correlation Znccis below the threshold, i.e. Zncc(T0k, T0i)<threshold, patch P is added to the eyes database DBmodels—eyes, with the following data stored:the normalized texture column-vector T0k,the precise position of the normalized characteristic points, by applying the similarity Sk(tx, ty, s, Θ) to the precise positions of characteristic points identified beforehand in the learning photograph Appeyesk,the similarity Sk(tx, ty, s, Θ)and other useful information: morphology, brightness etc.

A new learning photograph Appeyesk+1can now be processed by returning to step230.

Step240—A statistical operation is performed on all the similarities Sk(tx, ty, s, Θ) stored in the database DBmodels—eyes.

First of all, the mean value of the translation tx and the mean value of the translation ty are calculated; these values will be stored in a two-dimensional vector {right arrow over (μ)}.

Secondly, the standard deviation σ is calculated for position parameters tx, ty relative to their mean, characterized by {right arrow over (μ)}.

In a variant, the precise positions of the characteristic points of the eyes, (these precise positions here are non-normalized) determined beforehand in the kthlearning photograph Appeyesk. The similarity Sk(tx, ty, s, Θ) or the values of all the parameters allowing these precise positions to be re-calculated, are also stored.

Phase300: Method of Searching for Criteria for Recognizing a Face in a Photo.

The purpose of phase300is to detect the possible presence of a face in a photo. A boosting algorithm is used, of a type known per se and, for example, described by P. Viola and L. Jones “Rapid object detection using a boosted cascade of features” and improved by R. Lienhart “a detector tree of boosted classifiers for real-time object detection tracking”.

It is noted that, in the field of automatic learning, the term classifier refers to a family of statistical classification algorithms. In this definition, a classifier groups together in the same class elements presenting similar properties.

Strong classifier refers to a very precise classifier (low error rate), as opposed to weak classifiers, which are not very precise (slightly better than a random classification).

Without going into details, which are outside the framework of this invention, the principle of boosting algorithms is to use a sufficient number of weak classifiers to make a strong classifier, achieving a desired classification success rate, emerge by selection or combination.

Several boosting algorithms are known. In this example the boosting algorithm known under the brand name “AdaBoost” (Freund and Schapire 1995) is used to create several strong classifiers (e.g. twenty) that will be organized in a cascade, in a manner known per se.

In the case of searching for a face in a photo, if a strong classifier thinks it has detected a face at the analysis level at which it operates with its set of weak classifiers, then it passes the image to the next strong classifier, which is more accurate, less robust but freed from some uncertainties due to the previous strong classifier.

In order to obtain a cascade with good classification properties in an uncontrolled environment (variable lighting conditions, variable locations, substantially variable faces to be detected), it is necessary to establish a face learning database DBAfaces.

This face learning database DBAfacesconsists of a set of images referred to as positive examples of faces Facepositive(type of example that one wants to detect) and a set of images referred to as negative examples of faces Facenegative(type of example that one does not want to detect). These images are advantageously, but not obligatorily, the same size as the images of models of spectacles and of the face of the user in the trying-on method.

To generate the set of images referred to as the positive examples of faces Facepositive, first of all reference face images Facereferenceare selected such that:these faces are the same size (e.g. one can require the interpupillary distance in the image to be equal to the reference interpupillary distance di0),the segment between the centers of the two eyes is horizontal and vertically centered on the image, andthe orientation of this face is either a front view or slightly in profile, between −45° and 45°.

The set of these reference face images Facereferencemust comprise several lighting conditions.

Secondly, based on these reference face images Facereference, other modified images Facemodifiedare constructed by applying variations in scale, rotation and translation in bounds determined by a normal trying on of a pair of spectacles (e.g. unnecessary to create an inverted face).

The set of images referred to as the positive examples of faces Facepositiveconsists of reference face images Facereferenceand modified images Facemodifiedbased on these reference face images Facereference. In this example, the number of examples referred to as positive examples of faces Facepositiveis greater than or equal to five thousand.

The set of images of negative examples of faces Facenegativeconsists of images that cannot be included in the images referred to as positive examples of faces Facepositive.

These are, therefore, images that do not represent faces, or images representing parts of faces, or faces that have undergone aberrant variations. In this example, a group of pertinent images is taken for each level of the cascade of strong classifiers. For example, five thousand images of negative examples of faces Facenegativeare selected for each level of cascade. If, as in this example, one chooses to use twenty levels in the cascade, this gives one hundred thousand images of negative examples of faces Facenegativein the face learning database DBAfaces.

Phase300uses this face learning database DBAfacestrain the first boosting to algorithm AD1, designed to be used in step510of phase500.

Phase400: Method of Searching for Criteria for Recognizing Characteristic Points in a Face

The purpose of phase400is to provide a method for detecting the position of the eyes in a face in a photo. In this example, the position of the eyes is detected with a second, Adaboost-type, detection algorithm AD2, trained with an eyes learning database DBAeyesdescribed below.

The eyes learning database DBAeyesconsists of a set of positive examples of eyes Eyespositive(positive examples of eyes are examples of what one wants to detect) and a set of negative examples of eyes Eyesnegative(negative examples of eyes are examples of what one does not want to detect).

To generate the set of images referred to as positive examples of eyes Eyespositive, first of all reference eye images Eyesreference, are selected such that the eyes are of the same size, straight (aligned horizontally) and centered, under different lighting conditions and in different states (closed, open, half-closed, etc.),

Secondly, based on these reference eye images Eyesreference, other modified eye images Eyesmodifiedare constructed by applying variations in scale, rotation and translation in limited bounds.

The set of images referred to as the positive examples of eyes Eyespositivewill therefore consist of reference eye images Eyesreferenceand modified eye images Eyesmodifiedbased on these reference eye images Eyesreference. In this example, the number of examples referred to as positive examples of eyes Eyespositiveis greater than or equal to five thousand.

The set of images of negative examples of eyes Eyesnegativemust be constituted of images of parts of the face that are not eyes (nose, mouth, cheek, forehead, etc.) or of partial eyes (bits of the eye).

To increase the number and pertinence of the negative examples of eyes Eyesnegative, additional negative images are constructed based on reference eye images Eyesreferenceby applying sufficiently great variations in scale, rotation and translation so that these images thus created are not interesting in the context of images of positive examples of eyes Eyespositive.

A group of pertinent images is selected for each level of the cascade of strong classifiers. For example, five thousand images of negative examples of eyes Eyesnegativecan be selected for each level of cascade. If there are twenty levels in the cascade, this gives one hundred thousand images of negative examples of eyes Eyesnegativein the eyes learning database DBAeyes.

Phase400may use this eyes learning database DBAeyesto train a second boosting algorithm AD2, which is used in a variant of the method involving a step520.

Phase500of Trying on Virtual Spectacles

In phase500, trying on virtual spectacles, the method of generating a final image5from the original photo1is divided into seven steps:a step510of detecting the face2of the subject in an original photo1.possibly a step520of the preliminary determination of the position of characteristic points of the subject in the original photo1.a step530of determining the position of characteristic points of the subject in the original photo1.a step540of determining the 3D orientation of the face2.a step550of selecting the texture to be used for the virtual pair of spectacles3and generating the view of the spectacles in the 3D/2D position in question.a step560of creating a first rendering28by establishing a layered rendering in the correct position consistent with the position of the face2in the original photo1.a step570of obtaining the photorealistic rendering by adding overlays, referred to as semantic overlays, so as to obtain the final image5.

Step510: In this example step510uses the first boosting algorithm AD1trained in phase300to determine whether the original photo1contains a face2. If this is the case one goes to step520, otherwise the user is warned that no face has been detected.

Step520: its purpose is to detect the position of the eyes in the face2in the original photo1. Step520here uses the second boosting algorithm AD2trained in phase400.

The position of the eyes, determined in this step520, is expressed by the position of characteristic points. This step520thus provides a first approximation, which is refined in the next step530.

Step530: it consists of determining a similarity β, to be applied to an original photo1, to obtain a face, similar to a reference face7in magnification and orientation, and determining the position of the precise exterior corner A and the precise interior corner B for each eye in the face2in the original photo1.

The position of the eyes, determined in this step530, is expressed by the position of characteristic points. As explained above, these characteristic points comprise two points per eye; the first point is defined by the most innermost possible corner A of the eye (the one nearest the nose), the second point B is the most outermost corner of the eye (the one furthest from the nose). The first point, A, is called the interior point of the eye, and the second point, B, is called the exterior point of the eye.

This step530uses the database of models of eyes DBmodels—eyes. In addition this step530provides information characterizing the offset from center, distance to the camera and 2D orientation of the face2in the original photo1.

This step530uses an iterative algorithm that makes it possible to refine the value of the similarity β and the positions of the characteristic points.

The parameters of similarity β and the positions of the characteristic points are initialized as follows. Step520has provided respectively, for each eye, a first approximate exterior point of the eye A0and a first approximate interior point B0; these points are used for initializing the characteristic points. The initialization values of the similarity β are deduced from them.

The similarity β is defined by a translation tx, ty in two dimensions x, y, a parameter of scale s and a parameter of rotation θ in the image plane.

This therefore gives

β0=(x0y0θ0s0),
the initial value of β.

The different steps of an iteration are as follows:

The characteristic points are used to create the two patches Pl, Prcontaining the two eyes. These patches Pl, Prare created as follows;

The original photo1is transformed into a gray-scale image8, by an algorithm known per se, and the two patches Pl, Prare constructed with the information about the exterior B and interior A points.

The position of a patch Pl, Pris defined by the fixed distance D, used prior to this in step232and following steps, between the exterior edge B of the eye and the edge of the patch closest to this point B. The sizing of the patch Pl, Pr(width and height) was defined in step232and following steps. If the patches Pl, Prare not horizontal (external and interior points of the patch not aligned horizontally), a bilinear interpolation of a type known per se, is used to align them.

The information about the texture of each of the two patches Pl, Pris stored in a vector (Tl), then these two vectors are normalized by subtracting their respective mean and dividing by their standard deviation. This gives two normalized vectors, designated T0rand T0l.

The realization of β is considered in terms of probability. The realizations of the parameters of position tx, ty, orientation Θ, and scale s, are considered to be independent and, in addition, the distributions of Θ, s are considered to follow a uniform distribution.

Finally, the parameters of position tx, ty are considered to follow a Gaussian distribution with mean vector {right arrow over (μ)} (in two dimensions) and standard deviation σ. The probability that β is realized is designated by p(β). Taking the variables {right arrow over (μ)}, σ and

v→=(xy),
stored in the eyes database DBmodels—eyes, and established in step240, an optimization criterion is selected as follows:

where

Drare random variable data representing the right patch Pr, consisting of the texture of the right patch,

Dlare random variable data representing the left patch Pl, consisting of the texture of the left patch,

D=Dr∪Dlare random variable data representing the two patches Pl, Pr. The realizations of Drand Dlare considered to be independent,

p(β/D) is the probability β is realized given D,

K is a constant,

id represents a descriptor patch (patches stored in the eyes database DBmodels—eyes).

The set of descriptor patches in the eyes database DBmodels—eyesare then scanned. The term ρ represents the correlation Zncc(between 0 and 1), formulated in step235and following steps, between patch Prof the right eye (respectively Plof the left eye) and a descriptor patch transformed according to the similarity β.

The maximum of these correlations Zncallows the probabilities P(Dr/β) (respectively p(Dl/β) to be calculated.

The regulation term

-K⁢⁢v→-μ→22⁢σ2
makes it possible to ensure the physical validity of the proposed solution.

The optimization criterion defined above (Equation 19), thus makes it possible to define an optimal similarity β and an optimal patch from the patch descriptors for each of the two patches Pl, Prwhich allows new estimates of the position of the exterior corners A and interior point B of each eye, i.e. characteristic points, to be provided.

It is tested whether this new similarity value β is sufficiently far from the previous value, e.g. by a difference of ε: if ∥βi−1−βi∥>ε an iteration is repeated. In this iteration, βirepresents the value of β found at the end of the current iteration and βi−1is the value of similarity β found at the end of the previous iteration, i.e. also the initial value of similarity β for the current iteration.

The constant K allows the right compromise to be achieved between the correlation measurements Zncc and a mean position from which one does not want to depart too far.

This constant K is calculated, using the method just described, on a set of test images, different from the images used to create the database, and by varying K.

It is understood that the constant K is chosen so as to minimize the distance between the characteristic points of the eyes, manually positioned on the training images, and those found in step530.

Step540: its purpose is to estimate the 3D orientation of the face, i.e. to provide the angle φ and angle ψ of the camera having taken the photo1, relative to the principal plane of the face. These angles are calculated from the precise position38of the characteristic points determined in step530, by a geometric transformation known per se.

Step550: this step consists of:firstly, 1/ finding the simplified geometric model6of the model of a virtual pair of spectacles3, stored in the spectacles database DBmodels—spectacles, and, 2/ applying to it the reference orientation Orientationiclosest to angles φ and ψ(determined in step540),secondly, 3/ assigning a texture to the simplified geometric model6, positioned in the 3D orientation of the reference orientation Orientationiclosest to angles φ and ψ, using the texture of the reference orientation Orientationiclosest to these angles φ and ψ. This is equivalent to texturing each of the N surfaces surfacejof the simplified geometric model6while classifying the surface in the current view into three classifications: interior surface of the frame, exterior frame of the frame, lens.

It is noted that the simplified geometric model6is divided into N surfaces surfacej, each having a normal {right arrow over (n)}j. This texture calculation is performed as follows, using the texture, i.e. the different overlays, of the reference orientation Orientationiclosest to angles φ and ψ:inversion of the normals {right arrow over (n)}jof each of the surfaces surfacejand projection of the frame overlay Framei, limited to the lens space of the reference orientation Orientationiclosest to angles φ and ψ. Designating by proj⊥(image,{right arrow over (n)}) the operator of the orthogonal projection of an image on a 3D surface of normal {right arrow over (n)} in a given position, gives:
Texturesurface(−{right arrow over (n)})=proj⊥(FrameiLensibinary,−{right arrow over (n)})⊥  (Eq 20)
This gives a texture overlay of the internal surface of the frame TextureFrameisurface—interior. This overlay TextureFrameisurface—interiormakes it possible to structure (i.e. determine an image) the arms of the frame4a, seen through the lens4bin the textured reference model9(superimposition of texture overlays of the pair of spectacles corresponding to a reference orientation), oriented according to the reference orientation Orientationiclosest to angles φ and ψ.projection of the frame overlay Framei, limited to the space outside the lens of the reference orientation Orientationiclosest to angles φ and ψ. This is expressed by:
Texturesurface({right arrow over (n)})=proj⊥(Framei(1−Lensibinary),{right arrow over (n)})  (Eq 21)This gives a texture overlay of the exterior surface of the frame TextureFrameisurface—exteriorwhich makes it possible to structure the surfaces outside the lens4bof the frame4a, in the textured reference model9, oriented according to the reference orientation Orientationi, closest to angles φ and ψ.projection of the lens overlay limited to the lens. This is expressed by:
Texturesurface(−{right arrow over (n)})=proj⊥(LensiLensibinary,{right arrow over (n)})  (Eq 22)
This gives a lens texture overlay TextureLensithat makes it possible to structure the lens4b, in the textured reference model9, oriented according to the reference orientation Orientationi, closest to angles φ and ψ.

Step560consist of generating an oriented textured model11, oriented according to the angles φ and ψ and according to the scale and orientation of the original photo1(which can have any value and not necessarily equal to the angles of the reference orientations), from the textured reference model9, oriented according to the reference orientation Orientationi, closest to angles φ and ψ, and parameters ⊖ and s of similarity β (determined in step530).

Firstly, a bilinear affine interpolation is used to orient an interpolated textured model10according to the angles φ and ψ (determined in step540) based on the textured reference model9(determined in step550) oriented according to the reference orientation Orientationiclosest to these angles φ and ψ.

Secondly, the similarity β to be applied is used, so as to obtain the same scale, the same (2D) image orientation and the same centering as the original photo1. This gives an oriented textured model11.

Thirdly, the arms of the virtual spectacles3are varied geometrically according to the morphology of the face of the original photo1

Thus, at the end of step560, a spectacles overlay Spectadesoverlayof the virtual pair of spectacles3is obtained and a binary overlay Spectaclesoverlay—binary(outline shape of this spectacles overlay) is deduced, oriented as the original photo1, and which can therefore be superimposed on it.

Step570consists of taking into account the light interactions due to wearing virtual spectacles, i.e. taking into account, for example, the shadows cast onto the face2, the visibility of the skin through the lens of the spectacles, the reflection of the environment on the spectacles. It is described inFIG. 9. It consists of:

1) multiplying the shadow map Visibilityi(obtained in step140) and the photo1to obtain a shadowed photo overlay, designated Lskin—Shadowed. Designating the original photo1by Photo this gives:
Lskin—Shadowed=VisibilityiPhoto  (Eq 23)

2) “blending” the shadowed photo overlay Lskin—Shadowedand the spectacles overlay Spectaclesoverlayby linear interpolation, depending on the coefficient of opacity α of the lens4bin an area limited to the binary overlay Spectaclesoverlay—binaryof the virtual pair of spectacles3, to obtain the final image5.
Where Cxand Cyare any two overlays, a blendαfunction is defined by: blendα(Cx, Cy)=α*Cx+(1−α)*Cy(Eq 24)
where α is the coefficient of opacity of the lens4bstored in the spectacles database DBmodels—spectacles

This function is therefore applied where

and only in the area of the spectacles determined by the binary overlay Spectaclesoverlay—binary.

The result of this function is an image of the original photo1on which is superimposed an image of the model of spectacles chosen, oriented as the original photo1, and given shadow properties.

Variants of the Invention

In a variant, the construction procedure allowing the simplified geometrical model6of a new shape of a real pair of spectacles4, i.e. of a shape not found in the models database DBmodels—spectaclesto be constructed, is here as follows:this real pair of spectacles4is made non-reflective. For example, to achieve this penetrant powder is used, of a type known per se, used in the mechanical and aeronautical industries to detect faults in parts manufactured. This powder is deposited by known means on the frame4aand the lenses4bto make the whole matte, opaque and therefore not reflective.the geometry of this matte, opaque real pair of spectacles4is established, for example, by means of a scanner using lasers or so-called structured light. The real pair of spectacles4generally has a greater depth than the depth of field accepted by these current types of scanners. Therefore, several scans of parts of this real pair of spectacles4are assembled, by conventional techniques, from images based, for example, on physical reference points. In this example, these physical reference points are created using watercolors on the penetrant powder deposited on the real pair of spectacles4.

In yet another variant, step540, whose purpose is to estimate the 3D orientation of the face, proceeds by detecting, if possible, the two points on the image representing the temples, called temple image points63.

The visual characteristic of a temple point is the visual meeting of the cheek and ear.

The detection of temple image points63may fail in the case where, for example, the face is turned sufficiently (>fifteen degrees), or there is hair in front of the temples etc. The failure to detect a temple image point63can be classified into two causes:first cause: the temple image point63is hidden by the face2itself because the orientation of the latter makes it not visiblesecond cause: the temple image point63is hidden by something other than the morphology, most often the hair.

Step540here uses segmentation tools that also, if detection of a temple image point63fails, allow the class of failure cause to which the image belongs to be determined.

Step540comprises a method for deciding whether or not to use the temples image point or points63, according to a previously stored decision criterion.

If this criterion is not fulfilled, angle φ and angle ψ are considered to be zero. Otherwise, angle φ and angle ψ are calculated from the position of the temple image point or points63detected, and the precise position38of the characteristic points determined in step530.

It is understood that the description just given for images of pairs of spectacles to be placed on an image of a face in real time applies, with modifications in the reach of the expert, to similar problems, for example presenting a model of a hat on the face of a user.