Source: http://patents.com/us-9437035.html
Timestamp: 2018-03-19 08:53:09
Document Index: 149281336

Matched Legal Cases: ['arts 200', 'art 200', 'art 200', 'art 200', 'art 200', 'art 300', 'art 300', 'art 300', 'art 300', 'art 300', 'arts 200']

US Patent # 9,437,035. Light source detection from synthesized objects - Patents.com
United States Patent 9,437,035
Free September 6, 2016
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining a location relative to an object and a type of a light source that illuminated the object when the image was captured, are described. A method performed by a process executing on a computer system includes identifying an object of interest in a digital image. The method further includes projecting at least a portion of the digital image corresponding to the object of interest onto a three dimensional (3D) model that includes a polygon-mesh corresponding to the object's shape. The method further includes determining one or more properties of a light source that illuminated the object in the digital image at an instant that the image was captured based at least in part on a characteristic of one or more polygons in the 3D model onto which the digital image portion was projected.
Free; Robert Mikio (San Jose, CA)
Family ID: 1000002087559
14/482,095
US 20150029192 A1 Jan 29, 2015
13659643 Oct 24, 2012 8855422
12693437 Nov 20, 2012 8315461
Current CPC Class: G06T 15/06 (20130101); G06K 9/00241 (20130101); G06K 9/4661 (20130101); G06T 7/0046 (20130101); G06T 7/40 (20130101); G06T 2207/30201 (20130101)
Current International Class: G06T 15/50 (20110101); G06K 9/00 (20060101); G06T 7/00 (20060101); G06T 7/40 (20060101); G06T 15/06 (20110101); G01B 11/24 (20060101); G06K 9/46 (20060101)
Field of Search: ;382/181 ;356/601 ;345/426
6222937 April 2001 Cohen
6549200 April 2003 Mortlock
7227977 June 2007 Dotsenko
8315461 November 2012 Free
8855422 October 2014 Free
2005/0008199 January 2005 Dong
`Light Source Detection From Synthesized Objects`, in Patent Search Report, Nov. 16, 2009, IP.com Security Innovation, 10 pages. cited by applicant .
Brunelli, R., "Estimation of Pose and Illuminant Direction for Face Processing", 1997, Image and Vision Computing, 15:741-748, 8 pages. cited by applicant .
Han, Heechut, `Face Relighting Based on Virtual Radiance Sphere and Reflection Coefficient`, Feb. 2008, Optical Engineering, 47(2), 8 pages. cited by applicant .
Hjelmas, Erik, et al., `Face Detection: A Survey`, Computer Vision and Image Understanding 83, 2001, pp. 236-274, Academic Press. cited by applicant .
Tumblin et al., `Two Methods for Display of High Contrast Images`, Jan. 1999, ACM Transactions on Graphics, 18(1):56-94. cited by applicant .
Yang, Kriegman, Ahuja: Detecting Faces in Images: A Survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, No. 1, Dated Jan. 2002, 25 pages. cited by applicant .
Zhao, W., "Face Recognition: A Literature Survey", Dec. 2002, 399-458, 35(4), ACM Computing Surveys, US, 61 pages. cited by applicant.
1. A method performed by a process executing on a computer system, the method comprising: identifying an object of interest in a digital image; projecting at least a portion of the digital image corresponding to the object of interest onto a three dimensional (3D) model, the projected portion having a color map; determining a location of a light source relative to the object of interest, wherein the light source illuminated the object of interest in the digital image at a time that the digital image was captured; and determining an orientation of the object of interest relative to the determined location of the light source.
2. The method of claim 1, wherein the light source is a sun, and wherein determining the orientation of the object of interest is based on a time of day that the digital image was captured and a latitude indicating where the digital image was captured.
3. The method of claim 1, wherein determining the location of the light source comprises back-tracing rays that travel from a viewing location relative to the 3D model to portions of a surface of the object of interest, then reflect off the portions of the surface of the object of interest, and travel in a direction of the light source, wherein the viewing location corresponds to a location relative to the object of interest of a camera that captured the image of the object of interest.
4. A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising: projecting onto a provided 3D model a color map corresponding to a light source that illuminated an object of interest in an image at a time that the image was captured; selecting specular regions of the projected color map, wherein the specular regions are indicative of specular reflection of light from the light source; determining a location relative to the 3D model of the light source, based, at least in part, upon back-tracing rays reflecting off the selected specular regions; and determining an orientation of the object of interest relative to the determined location of the light source.
5. The non-transitory computer storage medium of claim 4, wherein the light source is a sun, and wherein determining the orientation of the object of interest is based on a time of day that the image was captured and a latitude indicating where the image was captured.
6. The non-transitory computer storage medium of claim 4, wherein determining a location further comprises: back-tracing rays that travel from a viewing location relative to the 3D model to the selected specular regions, then reflect off the selected specular regions and travel in a direction of the light source, wherein the viewing location corresponds to a location relative to the object of interest of a camera that captured the image of the object.
7. The non-transitory computer storage medium of claim 4, wherein the image is a two-dimensional (2D) digital image, and the identified object of interest in the 2D digital image is a building, structure, geographical feature, or landmark.
8. A method performed by a process executing on a computer system, the method comprising: identifying an object of interest in a digital image; projecting at least a portion of the digital image corresponding to the object of interest onto a three dimensional (3D) model that includes a polygon-mesh corresponding to the object's shape; determining a location of a light source relative to the object of interest, wherein the light source illuminated the object of interest in the digital image at a time that the digital image was captured based, at least in part, on a characteristic of one or more polygons in the 3D model onto which the digital image portion was projected; and determining an orientation of the object of interest relative to the determined location of the light source.
9. The method of claim 8, wherein the light source is a sun, and wherein determining the orientation of the object of interest is based on a time of day that the digital image was captured and a latitude indicating where the digital image was captured.
10. The method of claim 8, further comprising selecting a subset of polygons onto which the digital image was projected, the selection based on a predetermined criterion.
11. The method of claim 10, wherein the predetermined criterion comprises a predetermined hue, and wherein the selecting comprises identifying regions of the selected subset of polygons corresponding to portions of the digital image having the predetermined hue.
12. The method of claim 11, wherein the predetermined criterion comprises a predetermined luminosity spatial frequency, and wherein the selecting comprises identifying regions of the selected subset of polygons corresponding to portions of the digital image having a spatial-change in luminosity less than the predetermined luminosity spatial frequency.
13. The method of claim 8, further comprising: calculating a median luminosity over portions of the image corresponding to the selected subset of polygons; assigning a weight between zero and one to each polygon from the selected subset of polygons onto which the digital image was projected based on a luminosity relative to the calculated median luminosity of an associated portion of the digital image; and responsive to the weight of a polygon being larger than zero, identifying the polygon as a specular polygon.
14. The method of claim 13, wherein assigning a weight comprises: associating a weight of zero to polygons corresponding to respective portions of the digital image having luminosity equal to or smaller than the calculated median luminosity; and associating a weight larger than zero to polygons corresponding to respective portions of the digital image having luminosity larger than calculated median luminosity.
15. The method of claim 13, wherein assigning a weight comprises: associating a weight of zero to polygons corresponding to respective portions of the digital image having luminosity equal to or smaller than a predetermined luminosity threshold, wherein the predetermined luminosity threshold is larger than the calculated median luminosity; and associating a weight of one to polygons corresponding to respective portions of the digital image having luminosity larger than the predetermined luminosity threshold.
16. The method of claim 13, wherein the characteristic of the polygon comprises a direction of a normal to the polygon's face, and wherein determining the location of the light source comprises: back-tracing rays that travel from a viewing location relative to the 3D model to the identified specular polygons, then reflect off respective faces of the identified specular polygons in accordance with respective directions of normals to the respective faces of the identified specular polygons, and travel in a direction of the light source, wherein the viewing location corresponds to a location relative to the object of a camera that captured the image of the object; for the back-traced rays reflecting off a certain group of polygons among the identified specular polygons to travel parallel to a certain direction relative to the 3D model, determining that the light source is of a directional-type and is placed along the certain direction; and for the back-traced rays reflecting off a particular group of polygons among the identified specular polygons to intersect at a particular point relative to the 3D model, determining that the light source is of a point type and is placed at the particular point.
This specification relates to light source detection from synthesized faces, for example, analyzing an image to detect a type and a relative location of a light source that illuminated an object in the image at the instant that the image was captured.
Information gathering agencies can use a variety of techniques for identifying a type and a location of a light source that illuminate a subject of a photograph. One such technique may include taking multiple photographs of the subject from different angles, followed by measuring the length of shadows cast by the subject in the respective photographs for each corresponding angle. Typically, analyzing the shadow-length measurements can be based on an assumption about the type of illuminating source: directional light or point light. Iterative testing of various locations where a point source or a directional source may have been placed may eventually lead to determining a light source location that best reproduces the shadow-length measurements.
This specification describes technologies relating to detection of one or more light sources that illuminated an object in a two-dimensional image when the image was taken. The methods and systems disclosed in this specification enable determining a light source vector of directional light and a position of point light.
One aspect of the subject matter described in this specification can be implemented in methods performed by a process executing on a computer system. A method includes identifying an object of interest in a digital image. The method further includes projecting at least a portion of the digital image corresponding to the object of interest onto a three dimensional (3D) model that includes a polygon-mesh corresponding to the object's shape. The method also includes determining one or more properties of a light source that illuminated the object in the digital image at an instant that the image was captured based at least in part on a characteristic of one or more polygons in the 3D model onto which the digital image portion was projected.
Implementations can include any, all, or none of the following features. The determining of the one or more properties of the light source can include determining a type of the light source, a location of the light source or both. The method can include selecting a subset of polygons onto which the digital image was projected, the selection based on a predetermined criterion. The predetermined criterion can include a predetermined hue. The selecting can include identifying regions of the selected subset of polygons corresponding to portions of the digital image having the predetermined hue. The predetermined criterion can include a predetermined luminosity spatial frequency. The selecting can include identifying regions of the selected subset of polygons corresponding to portions of the digital image having a spatial-change in luminosity less than the predetermined luminosity spatial frequency. The method can include calculating a median luminosity over portions of the image corresponding to the selected subset of polygons; assigning a weight between zero and one to each polygon from the selected subset of polygons onto which the digital image was projected based on a luminosity relative to the calculated median of an associated portion of the digital image; and responsive to the weight of a polygon being larger than zero, identifying the polygon as a specular polygon.
In some implementations, the assigning can include associating a weight of zero to polygons corresponding to respective portions of the digital image having luminosity equal to or smaller than the calculated median luminosity; and associating a weight larger than 0 to polygons corresponding to respective portions of the digital image having luminosity larger than calculated median luminosity. The assigning can include associating a weight of zero to polygons corresponding to respective portions of the digital image having luminosity equal to or smaller than a predetermined luminosity threshold. The predetermined luminosity threshold can be larger than the calculated median luminosity. The assigning further includes associating a weight of one to polygons corresponding to respective portions of the digital image having luminosity larger than the predetermined luminosity threshold.
In some implementations, a characteristic of polygon can include direction of normal to polygon's face. Determining of the light source can include back-tracing rays that travel from a viewing location relative to the 3D model, to the identified specular polygons, then reflect off respective faces of the identified specular polygons in accordance with respective directions of the normals to the identified specular polygons' faces, and travel in a direction of the light source. The viewing location corresponds to a location relative to the object of a camera that captured the image of the object. For the back-traced rays reflecting off a certain group of polygons among the identified specular polygons to travel parallel to a certain direction relative to the 3D model, the method includes determining that the light source can be of directional-type and can be placed along the certain direction. For the back-traced rays reflecting off a particular group of polygons among the identified specular polygons to intersect at a particular point relative to the 3D model, the method includes determining that the light source a point type and can be placed at the particular point. The digital image can be a two-dimensional (2D) digital image, and the identified object of interest in the 2D digital image can be a human face.
In a second aspect, a computer storage medium encoded with a computer program, the program includes instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations includes projecting onto the provided 3D model a color map corresponding to a light source that illuminated an object in an image at an instant that the image was captured. The 3D model includes a polygon-mesh corresponding to the object's shape. The apparatus further includes selecting specular regions of the projected color map. The specular regions are indicative of specular reflection of light from the light source. The apparatus further includes back-tracing rays that travel from a viewing location relative to the 3D model, to the selected specular regions, then reflect off the selected specular regions and travel in a direction of the light source. The viewing location corresponds to a location relative to the object of a camera that captured the image of the object. The apparatus further includes determining a location relative to the 3D model of the light source based on the back-traced rays reflecting off the selected specular regions.
Implementations can include any, all, or none of the following features. The determining can include for the back-traced rays reflecting off a certain group of regions among the selected specular regions to travel parallel to a certain direction relative to the 3D model, the apparatus further includes determining that the light source can be of directional-type and can be placed along the certain direction. For the back-traced rays reflecting off a particular group of regions among the selected specular regions to intersect at a particular point relative to the 3D model, the apparatus further includes determining that the light source a point type and can be placed at the particular point. The selecting of specular regions can include determining regions of interest of the projected color map based on a predetermined criterion; calculating a median of luminosity across the regions of interest. The luminosity represents a dimension in a hue-saturation-luminosity color space (HSL) corresponding to a brightness of a color along a lightness-darkness axis; and assigning a weight between zero and one to each of the regions of interest based on an associated luminosity relative to the calculated median. A weight larger than zero denotes a region selected as a specular region, and a weight of zero denotes a region not selected as specular region.
In some implementations, the assigning can include associating a weight of zero to regions having luminosity equal to or smaller than the calculated median luminosity, and associating a weight larger than 0 to regions having luminosity larger than calculated median luminosity. The computer storage medium can include correlating the back-traced rays reflected off the selected specular regions at least in part based on the respective weights of the selected specular regions. The assigning can include associating a weight of zero to regions having luminosity equal to or smaller than predetermined luminosity threshold. The predetermined luminosity threshold can be larger than the calculated median luminosity, and associating a weight of one to regions having luminosity larger than the predetermined luminosity threshold. The predetermined criterion can include a predetermined hue. The determining of regions of interest can include identifying regions of the projected color map having the predetermined hue. The predetermined criterion can include a predetermined luminosity spatial frequency. The determining of regions of interest can include identifying regions of the projected color map having a spatial-change in luminosity less than the predetermined luminosity spatial frequency. A region of the projected color map can include one of: a contiguous area of a pixels can included in a polygon; all pixels can included in a polygon; a contiguous group of neighboring polygons can included in the polygon-mesh; a planar-surface corresponding to a linear-surface-fit of the contiguous group of neighboring polygons; or a spline-surface corresponding to a non-linear-surface-fit of the contiguous group of neighboring polygons.
In some implementations, the computer storage medium can include finding the viewing location based on information can include one or more of dimensions of the projected color map, dimensions of the object in the image, and the location and photometric parameters of the camera that captured the image of the object. The back-tracing of rays can further include reflecting the back-traced rays in accordance with planar-surface reflection off polygons can included in portions of the polygon-mesh corresponding to the selected specular regions. The back-tracing of rays can further include reflecting the back-traced rays in accordance with curved-surface reflection off spline surfaces generated over neighboring polygons can included in portions of the polygon-mesh corresponding to the selected specular regions.
In a third aspect, a computer system includes a data processor communicatively coupled with a data storage device and with a user device. The data processor is configured to receive, from the user device, a two dimensional (2D) digital image of a face. The system further includes providing, from the data storage device, a three-dimensional (3D) model that includes a polygon-mesh corresponding to the face's shape. The system further includes projecting onto the provided 3D model a color map corresponding to a light source that illuminated the face in the 2D digital image at an instant that the 2D digital image was captured. The system further includes selecting specular regions of the projected color map. The specular regions are indicative of specular reflection of light from the light source. The system further includes back-tracing rays that travel from a viewing location relative to the 3D model, to the selected specular regions, then reflect off the selected specular regions and travel in a direction of the light source. The viewing location corresponds to a location relative to the face of a camera that captured the 2D digital image of the face. The system further includes for the back-traced rays reflecting off a certain group of regions among the selected specular regions to travel parallel to a certain direction relative to the 3D model, determining that the light source is of directional-type and is placed along the certain direction. The system further includes for the back-traced rays reflecting off a particular group of regions among the selected specular regions to intersect at a particular point relative to the 3D model, determining that the light source a point type and is placed at the particular point.
Implementations can include any, all, or none of the following features. The selecting of specular regions can include determining regions of interest of the projected color map based on a predetermined criterion; calculating a median of luminosity across the regions of interest. The luminosity represents a dimension in a hue-saturation-luminosity color space (HSL) corresponding to a brightness of a color along a lightness-darkness axis; and assigning a weight between zero and one to each of the regions of interest based on an associated luminosity relative to the calculated median. A weight larger than zero denotes a region selected as a specular region, and a weight of zero denotes a region not selected as specular region.
In some implementations, the providing can include if the 3D model of the face can be stored on a data storage device that can be communicatively coupled with the computer system, retrieving the stored 3D model of the face from the storage device, otherwise retrieving a 3D model of a generic face from the storage device, and modifying the shape of the 3D model of the generic face to match biometric characteristics of the face in the 2D digital image to obtain the 3D model of the face.
In some implementations, the 3D model can further include a neutral color map that covers each polygon of the polygon-mesh. The neutral color map corresponds to the 3D model illuminated by diffuse light such that each hue of the neutral color map can have a luminosity of 50%, the system can be further configured to synthesize a luminosity map that covers each polygon of the 3D model based on the determined light source. Responsive to overlaying the synthesized luminosity map on the neutral color map, reproducing a full-coverage color map that covers each polygon of the 3D model of the face, the reproduced full-coverage color map corresponding to the illumination conditions of the face when the 2D digital image was captured; and examining dark and bright patterns of the full-coverage color map over portions of the 3D model of the face. The examined portions correspond to areas of the face outside of a field of view in the 2D digital image.
In some implementations, the system is configured to correct the 2D digital image for illumination artifacts based on the determined type and location of the light source, and to perform face recognition for the face in the 2D digital image corrected for illumination artifacts. The correcting can include illuminating the 3D model of the face using the determined light source to obtain a synthetically illuminated color map of the face, determining a luminosity-difference between the synthetically illuminated color map of the face and the projected color map of the face, subtracting the determined luminosity-difference from the projected color map of the face to obtain a correction of the projected color map of the face; and mapping the correction of the color map from the 3D model of the face to the 2D digital image of the face to obtain a 2D digital image corrected for illumination artifacts.
Particular implementations of the subject matter described in this specification can be implemented to realize one or more of the following potential advantages. For example, once a location of an illumination point source has been identified using the technologies disclosed in this specification, an orientation of a subject in an image can be determined based on the identified position of the illumination point source.
Further, a user can correlate a directional light source's (e.g., the Sun's) attitude with a time when an image was captured to determine image location latitude. Conversely, a time of day can be determined knowing the attitude of the Sun and the latitude where the image was captured. In addition, if both time and latitude are known, one can determine the orientation of a subject in the image.
Furthermore, the methods and systems disclosed in this specification can be used to examine dark and bright patterns of a color map over portions of a 3D model of an object. The examined portions can correspond to areas of the object outside of a field of view in a 2D photograph of the object, while the examined patterns correspond to illumination conditions when the 2D photograph was taken.
The disclosed techniques can also be used to remove lighting artifacts (specular reflections, shadows, and the like) from a three-dimensional (3D) synthesized model of an object based on the position and the nature (type) of detected light sources. A color map of the 3D model corrected for illumination artifacts can be mapped back onto a two-dimensional (2D) image of the object to obtain a lighting-neutral texture for improved object correlation, matching and recognition.
Additionally, once the light sources illuminating one or more objects are identified, these can in turn be used as hints for recognizing shadows and highlights on other unrecognized objects, specifically providing hints regarding the unknown objects' shape, size and location.
Further, the identified light sources may also be used to provide hints for tracking the movement and orientation of recognized objects in subsequent frames of sequential images (e.g., videos)--reducing recognition time and improving accuracy. Furthermore, if positions of still objects are known and positions of light sources have been identified, one can determine the orientation and location of the capturing camera from sequential frames of a video.
FIG. 1 shows an input-output diagram of a system for determining location and type of one or more light sources that illuminated an object when a two dimensional (2D) image of the object was captured.
FIGS. 2 and 3 show flow charts of an example process for determining a light source to reproduce a projection of a color map corresponding to a face in a 2D image onto a three dimensional (3D) model of the face.
FIGS. 4A and 4B show aspects of an example process for determining a light source that illuminates a 3D model of a face.
FIGS. 5A and 5B show flow charts of an example process for correcting a specified image of an object for illumination artifacts and for performing object recognition on the corrected image.
The methods and systems disclosed in this specification enable a user to determine a type and a location of one or more light sources that illuminated an object in a two-dimensional image when the image was captured. These methods can be implemented in image processing software.
System 100 can include a data processor in communication with a user device and a storage device. A multitude of hardware and software representations of system 100 are described below in this specification. The data processor is configured to receive from a user an image of an object 10. The user may access system 100 via the user device. The user specified image 10 can be a two dimensional (2D) image of the object. For example, the user specified image can be a digital photograph or a frame of a movie. The object in the specified image can be a human face, an automobile, a china vase, and the like.
The data processor of system 100 is configured to determine a location relative to the object and a type of a given light source that illuminated the object when the specified image was captured. The processes implemented by the data processor are described in detail below in connection with FIGS. 2-3. The data processor is configured to operate on the user specified image 10 and on a three-dimensional (3D) model of the object 20. The 3D model 20 includes a polygon-mesh corresponding to the object's shape, and can be provided by system 100 in various ways, as described below in connection with FIG. 2.
Among other things, the data processor of system 100 is configured to project onto the provided 3D model 20 a color map corresponding to the object in the specified image 10. Further, the processes disclosed in this specification include generating, by the data processor, of one or more light sources for illuminating the 3D model to reproduce the projected color map. The one or more light sources generated by system 100 correspond to the one or more light sources that illuminated the object when the specified image of the object 10 was taken. For example, an output of system 100 can be a determined directional light source 140, characterized by a given direction relative to the object in image 10. Another output of system 100 can be a determined point light source 150, characterized by a given location relative to the object in image 10.
In some implementations, system 100 can be used to implement a process that includes identifying an object of interest in a digital image 10. The process can further include projecting at least a portion of the digital image 10 corresponding to the object of interest onto a 3D model 20 that includes a polygon-mesh corresponding to the object's shape. The process is used for determining one or more properties of light sources that illuminated the object in the digital image 10 at an instant that the image was captured based at least in part on a characteristic of one or more polygons in the 3D model 20 onto which the digital image portion was projected. Determining properties of the light sources includes determining a type of light source or location of light source or both 140, 150. A characteristic of the polygon, used to determine the light sources, may be directions of normals to respective polygons' faces, for instance. The process implemented in system 100 also includes selecting a subset of polygons onto which the digital image was projected, the selected subset corresponding to portions of the image that satisfy certain predetermined criteria. Finally, as shown below in connection with FIGS. 2-3, characteristics of the selected set of polygons are being used to detect the light sources that illuminated the object in the digital image 10 at an instant that the image was captured.
FIGS. 2 and 3 show flowcharts 200, 300 of an example process for determining a light source to reproduce a projection of a color map corresponding to a face in a 2D image onto a 3D model of the face. This process can be implemented, for example, in system 100. Although this example process is implemented to process a face, this technique can be applied to process other objects imaged in a 2D image. The process shown in flowchart 200 starts by obtaining 210 a 3D model of a face that corresponds to a 2D image the face.
To implement the obtaining 210 operation, the system 100 receives from the user the 2D image of the face 10 and provides the 3D model of the face 20. If the 3D model of the face 20 is stored on the data storage device that is part of system 100, the data processor is implemented to retrieve the stored 3D model of the face 20 from the storage device. Otherwise, the data processor can retrieve a 3D model of a generic face from the storage device. Subsequently, the data processor can modify the shape of the 3D model of the generic face to match biometric characteristics of the face in the 2D image to provide a corresponding 3D model.
Also as part of the obtaining 210 operation, system 100 can project onto the provided 3D model of the face 20 a color map corresponding to the face in the 2D image 10. In general, a color map can be represented using hexadecimal and RGB colors. Additionally, the color map can be represented in terms of Hue, Saturation, Lightness (HSL) values. A brief description of the HSL terms is included below for convenience.
Hue corresponds to a degree on the color wheel: 0 (or 360) is red, 120 is green, 240 is blue. Degree values in between the primary colors represent secondary colors and different shades of the primary and secondary colors. Examples of hues are aqua, blue, green, red, orange, etc. Saturation is a percentage value, where 100% is the full color. The saturation component goes from the fully saturated color to the equivalent gray. Luminosity (also called lightness or tone) is a property of a color, or the L-dimension of a color space, that is defined in a way to represent the subjective brightness perception of a color for humans along a lightness-darkness axis. Luminosity is also a percentage: 0% is dark (black), 100% is light (white), and 50% is the average (neutral). Equivalently, the luminosity spans the entire range from black through the chosen hue to white. The HSL representation can provide a very wide spectrum of available colors and tones.
The process shown in flowchart 200 continues by identifying 220 regions of interest of the projected color map. The term region (e.g., as in region of interest, or later as in specular region) is used in this specification to mean one of (i) a contiguous area of a pixels included in a polygon; (ii) all pixels included in a polygon; (iii) a contiguous group of neighboring polygons included in the polygon-mesh; (iv) a planar-surface corresponding to a linear-surface-fit of the contiguous group of neighboring polygons; or (v) a spline-surface corresponding to a non-linear-surface-fit of the contiguous group of neighboring polygons. The determination of the regions of interest of the projected color map can be based on a predetermined criterion. One such criterion may be a predetermined hue, and the regions of interest can be identified 220 as regions of the projected color map having the predetermined hue. The criterion may also include a predetermined luminosity spatial frequency (i.e., the change in luminosity over an area), and the regions of interest can be identified 220 as regions of the projected color map having a spatial-change in luminosity less than the predetermined luminosity spatial frequency.
In the case illustrated in FIG. 2 when the object represents a face, the area of interest can represent skin. There may be different techniques to determine whether a region of interest represents skin. One such technique for detecting skin includes (i) checking a hue of the pixels in each of the polygons of the polygon-mesh 20, and (ii) verifying that the luminosity undergoes a low (spatial) frequency change (i.e., the luminosity stays stable and smooth within a polygon, or across neighboring polygons.) To represent skin, the regions may be characterized by an orange hue and a luminosity within a predetermined luminosity range of the orange hue. To validate the assignment of a region to skin, the data processor of system 100 can verify whether the tone (luminosity) of the detected orange hue involves low spatial-frequency changes. If the verification is positive, the analyzed region is confirmed to represent skin (in contrast, e.g., with cloth texture characterized by high frequency tone changes.) For example, the data processor implementing the process shown in flowchart 200 identified regions 130 of the projected color map to represent skin. Panels 132, 134 and 136 illustrate multiple views of the skin regions 130.
The process shown in flowchart 200 continues by calculating 230 a median of luminosity (Lm) across the region(s) of interest. This operation establishes a baseline luminosity for the identified skin region 130 to account for skin having naturally a darker or a lighter tone of orange depending on ethnicity, for instance. Thus the median luminosity of the skin color (the tone of the skin) can be calculated 230 as a median luminosity of the skin hue over the polygons determined to represent skin 130. Once the median luminosity has been found, system 100 can determine a luminosity difference between each pixel (or each polygon) and the calculated median luminosity of the skin region 130. Highlights of the skin may have luminosity values above the calculated median, and shadows cast on the skin may have values below the calculated median.
The process continues as shown in flowchart 300 of FIG. 3 by selecting 310 specular regions among the identified regions of interest 130 of the projected color map. The specular regions are indicative of specular reflection of light from the light source. The selecting operation 310 includes assigning 320 a weight between zero and one (0.ltoreq.w.ltoreq.1) to each of the regions of interest based on an associated luminosity relative to the calculated median. For example, a weight larger than zero (0<w) corresponds to a specular region, and a weight of zero (w=0) denotes a region that is not a specular region.
In some implementations, a weight of zero (w=0) can be associated to regions having luminosity equal to or smaller than the calculated median luminosity (L.ltoreq.Lm), and a weight larger than zero (0<w) can be associated to regions having luminosity larger than calculated median luminosity (Lm<L). Based on this approach, regions that have values of luminosity that are a large in comparison to the median luminosity Lm are more likely (with higher confidence) part of a specular region. For example, if skin represents a perfect mirror (e.g., "skin"-shell of a robot), light rays from a light source reflect off a small area (one point) of the shell to travel towards the camera (or the eye.) However unlike the robot's shell, human skin reflects light diffusely because the skin scatters the light of a source in a diffuse manner. In view of the above reasons, regions of skin that have high levels of luminosity may be selected as specular regions with a high certainty.
In other implementations, a weight of zero (w=0) can be associated to regions having luminosity equal to or smaller than a predetermined luminosity threshold (L.ltoreq.Lt). The predetermined luminosity threshold can be chosen to be larger than the calculated median luminosity (Lm<Lt). In these other implementations, a weight of one (w=1) can be assigned to regions having luminosity larger than the predetermined luminosity threshold (Lt.ltoreq.L). Therefore, the confidence for identifying specular regions can be expressed in terms of a luminosity predetermined threshold. For example, a first set of regions having respective luminosities above the predetermined threshold (Lt.ltoreq.L) can be identified with high confidence as a set of specular regions, and can be used to perform ray analysis for determining the location and type of illumination sources.
The process shown in flowchart 300 continues by back-tracing 330 rays that travel from a viewing location relative to the 3D model, to the selected specular regions, then reflect off the selected specular regions and travel in a direction of a light source. The viewing location corresponds to a location relative to the object of a camera that captured the specified image of the object (of the face, in this example). The viewing location can be provided to system 100 in the form of metadata along with the received image of the object 10. However, the viewing location may not be available as input to system 100. In the later case, the data processor can be configured to determine the viewing location based on information relating to one or more of (i) dimensions of the projected color map, (ii) dimensions of the object and of the object image, and (iii) the location and photometric parameters of the camera that captured the specified image of the object. Examples of such photometric parameters are a focal length and magnification of a camera lens, and the like.
The operation of back-tracing of rays 330 can be implemented based on a variety of ray-tracing techniques. In some implementations, the back-traced rays can reflect off polygons included in portions of the polygon-mesh corresponding to the selected specular regions in accordance with planar-surface reflection. This first category ray tracing technique may be faster but less accurate than other ray tracing techniques. In other implementations, the back-traced rays can reflect off spline surfaces generated over neighboring polygons included in portions of the polygon-mesh corresponding to the selected specular regions in accordance with curved-surface reflection. This second category ray tracing technique may be more accurate but slower than other ray tracing techniques.
If a weighing technique is being applied in identifying the specular regions in accordance to operations 310, 320, all regions identified to represent skin 130 can be used in the foregoing ray analysis by correlating the back-traced rays reflected off the selected specular regions, at least in part based on the respective weights of the selected specular regions. The weight can be defined in terms of the luminosity delta of a certain region from the median luminosity calculated over all regions of interest. If the luminosity delta is close to zero, the weight of that region may be chosen to be small, while if the luminosity delta is large, the weight of that region may be chosen to be large. Alternatively, ray tracing 330 can be applied to specular regions identified to have delta luminosity above a predetermined threshold.
The process shown in flowchart 300 continues by examining 340 the paths of the rays reflected off the selected specular regions. As a result of the examination 340, the data processor of system 100 can determine that the back-traced rays reflect off a certain group of regions among the selected specular regions to travel parallel to a certain direction relative to the 3D model. In this case, the data processor is configured to determine 350 a directional light source for illuminating the 3D model. The directional light source is placed along the certain direction, and as described above, corresponds to the given source that illuminated the face when the 2D image of the face was captured. Therefore, system 100 has determined a directional-type of the given light source and a location of the given light source along a given direction relative to the object corresponding to the certain direction of the directional light source relative to the 3D model.
FIG. 4A illustrates aspects of the example process shown in flowchart 300. The panel labeled 310'+320' (corresponding to operations 310 and 320) depicts a view 134 of regions identified as skin of the projected color map. The underlying polygon-mesh is not shown (for clarity) except for the regions 412 selected as specular regions. In this example, the selection of the specular regions 412 is based on a threshold: only regions having luminosity above a predetermined threshold (Lt<L) can be included in the specular regions 412. Further, a weight of one (w=1) is associated to each of specular regions 412, which means that each of specular regions 412 contributes equally to the subsequent ray analysis depicted in the panel labeled 330'+340'+350' (corresponding to operations 330, 340 and 350.) Furthermore as described above, each of specular regions 412 can represent either (i) a planar-surface corresponding to a linear-surface-fit of a contiguous group of neighboring polygons, or (ii) a spline-surface corresponding to a non-linear-surface-fit of the contiguous group of neighboring polygons.
The panel labeled 330'+340'+350' depicts a view 136 of the 3D model including regions identified as skin of the projected color map. Further, this panel illustrates the viewing location 30 relative to the 3D model. The image of the face 10 was captured by a camera located at a given location relative the face corresponding to the viewing location 30 relative to the 3D model. Rays 44 that originate at viewing location 30 can have a non-planar wavefront 43 and can be back-traced to the selected specular regions 412. The back-traced rays can reflect off the selected specular regions 412 according to planar-surface reflection if the specular regions represent a planar-surface corresponding to a linear-surface-fit. Alternatively, the back-traced rays can reflect off the selected specular regions 412 according to curved-surface reflection if the specular regions represent a spline-surface corresponding to a non-linear-surface-fit. The reflected-off rays 42 can have a planar wavefront 41 and can travel along a direction 400 relative to the 3D model. Therefore, a directional light source can be located along direction 400 relative to the 3D model.
In the example aspect illustrated in FIG. 4A, all of rays 42 reflected off the selected specular regions 412 travel along direction 400, which causes the data processor to determine one directional light source for illuminating the 3D model to reproduce the projected color map that corresponds to the face in the 2D image 10. In another example, however, light rays may reflect off a first group of specular regions to travel parallel to a first direction, and other light rays may reflect off a second group of specular regions to travel parallel to a second direction. In this later example, the first and second directions can define two directional light sources illuminating the 3D model.
Returning to FIG. 3, as a result of the examination 340, the data processor of system 100 can determine that the back-traced rays reflect off a particular group of regions among the selected specular regions to intersect at a particular point relative to the 3D model. In this case, the data processor is configured to determine 360 a point light source for illuminating the 3D model. The point light source is placed at the particular point, and as described above, corresponds to the given source that illuminated the face when the 2D image of the face was captured. Therefore, system 100 has determined a point-type of the given light source and a location of the given light source at a given point relative to the object corresponding to the particular point of the point light source relative to the 3D model.
FIG. 4B illustrates aspects of the example process shown in flowchart 300. The panel labeled 310''+320'' (corresponding to operations 310 and 320) depicts a view 134 of regions identified as skin of the projected color map. The underlying polygon-mesh is not shown (for clarity) except for the regions 416 selected as specular regions. In this example, the selection of the specular regions 416 is based on a threshold: regions having luminosity above a predetermined threshold (Lt<L) can be included in the specular regions 416. Further, a weight of one (w=1) is associated to each of specular regions 416, which means that each of specular regions 416 contributes equally to the subsequent ray analysis depicted in the panel labeled 330''+340''+360'' (corresponding to operations 330, 340 and 360.) Furthermore as described above, each of specular regions 416 can represent either (i) a planar-surface corresponding to a linear-surface-fit of a contiguous group of neighboring polygons, or (ii) a spline-surface corresponding to a non-linear-surface-fit of the contiguous group of neighboring polygons.
The panel labeled 330''+340''+360'' depicts a view 136 of the 3D model including regions identified as skin of the projected color map. Further, this panel illustrates the viewing location 30 relative to the 3D model. The image of the face 10 was captured by a camera located at a given location relative the face corresponding to the viewing location 30 relative to the 3D model. Rays 44 that originate at viewing location 30 can have a non-planar wavefront 43 and can be back-traced to the selected specular regions 416. The back-traced rays can reflect off the selected specular regions 416 according to planar-surface reflection if the specular regions represent a planar-surface corresponding to a linear-surface-fit. Alternatively, the back-traced rays can reflect off the selected specular regions 416 according to curved-surface reflection if the specular regions represent a spline-surface corresponding to a non-linear-surface-fit. The reflected-off rays 47 can have a non-planar wavefront 46 and can intersect at a particular point 450 relative to the 3D model. Therefore, a point light source can be located at the particular point 450 relative to the 3D model.
In the example aspect illustrated in FIG. 4B, all of rays 47 reflected off the selected specular regions 416 intersect at the particular point 450, which causes the data processor to determine one point light source for illuminating the 3D model to reproduce the projected color map that corresponds to the face in the 2D image 10. In another example, however, light rays may reflect off a first group of specular regions to converge to a first point, and other light rays may reflect off a second group of specular regions to converge to a second point. In this later example, the first and second points can define locations of two point light sources illuminating the 3D model.
System 100 can be used to implement the process shown in flowcharts 200, 300 for examining dark and bright patterns over portions of a 3D model of an object. The examined portions may correspond to areas of the object outside of a field of view in a 2D photograph, and the dark and bright patterns correspond to illumination conditions when the 2D photograph was taken. For example, the 3D model 20 can further include a neutral color map that covers each polygon of the polygon-mesh (in 3D-space). The neutral color map corresponds to the object being illuminated by diffuse light, such that each hue of the neutral color map has a luminosity of 50%. Once the processes described previously in this specification are applied to determine the one or more light sources illuminating the 3D model, system 100 can synthesize a luminosity map that covers each polygon of the 3D model based on the one or more light sources. The synthetic luminosity map can be overlaid by system 100 on the neutral color map. Subsequently, the data processor of system 100 can reproduce a full-coverage color map that covers each polygon of the 3D model of the object. The reproduced full-coverage color map corresponds to the illumination conditions of the object when the 2D photograph was captured. In this fashion, a user of system 100 can examine dark and bright patterns of the full-coverage color map over portions on the 3D model of the object. The examined portions correspond to areas of the object outside of a field of view in the 2D photograph.
FIGS. 5A and 5B show flow charts of an example process for correcting a specified image of an object for illumination artifacts and for performing object recognition on the corrected image. For example, process 500 can be implemented in system 100. Process 500 begins with the data processor receiving 510 a user specified image of an object. System 100 can provide 520 a 3D model of the object that contains at least a polygon-mesh corresponding to the object's shape. The data processor can further project onto the provided 3D model a color map corresponding to the object in the specified image. Process 500 continues with the data processor determining 530 a given light source that illuminated the object when the specified image was taken. System 100 can determine a location relative to the object and a type of the given light source.
Process 500 continues with the data processor correcting 540 the specified image for illumination artifacts based on the determined type and location of the given light source. Finally, the data processor is configured to perform object recognition 550 for the object in the specified image corrected for illumination artifacts.
To perform the correcting operation 540, the data processor is configured to illuminate 542 the 3D model of the object based on the determined given light source to obtain a synthetically illuminated color map of the object. Operation 540 continues with the data processor determining 544 a luminosity-difference between the synthetically illuminated color map of the object and the projected color map of the object. The determined luminosity-difference can be subtracted 546 from the projected color map of the object to obtain a correction of the projected color map of the object. Finally, the data processor is configured to map 548 the correction of the color map from the 3D model of the object to the 2D image of the object to obtain a 2D image corrected for illumination artifacts.
Previous Patent US 9,437,034 | Next Patent US 9,437,036