Patent Description:
Advancements in the field of computer graphics have led to development of various techniques of estimation of 3D shape and texture of human faces for photorealistic 3D face modeling. High fidelity 3D models may be in demand in a number of industries, such as entertainment industry, gaming industry, design industry, and healthcare industry. For example, in entertainment and gaming industry, 3D face modeling may be utilized to create a photorealistic face animation or to develop a 3D face of a game character. Conventional imaging setup used for 3D modeling may have inconsistent lighting conditions. As a result, a 3D model constructed based on images acquired by the conventional imaging setup may have inaccuracies associated with the 3D shape of the 3D model. Additionally, the 3D model may include poor surface-level details in term of surface geometry and surface reflection.

<NPL>] disclose a method to acquire dynamic properties of facial skin appearance, including dynamic diffuse albedo encoding blood flow, dynamic specular intensity, and per-frame high resolution normal maps for a facial performance sequence. In particular, the paper presents a capture setup consisting in <NUM> cameras: <NUM> monochrome for multi-view stereo geometry reconstruction and <NUM> colour cameras for appearance capture. <CIT> (DEBEVEC PAUL E [US] ET AL) discloses a method for acquiring detailed facial geometry with high resolution diffuse and specular photometric information from multiple viewpoints. This solution uses a lighting system to illuminate a face from multiple directions with polarized light. The light may be polarized and parallel and perpendicular to a reference axis by using a camera system including one or more cameras, where each camera is configured to capture an image of the face along a materially different optical axis. Finally, this method uses a controller to control the lighting system and/or the camera system so as to cause each of the cameras to capture an image of the face while the lighting system is in the parallel polarization mode or in the perpendicular polarization mode of operation.

Limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of described systems with some aspects of the present disclosure, as set forth in the remainder of the present application and with reference to the drawings.

A system and a method of three-dimensional (3D) microgeometry and reflectance modeling is provided substantially as shown in, and/or described in connection with, at least one of the figures, as set forth more completely in the claims.

These and other features and advantages of the present disclosure may be appreciated from a review of the following detailed description of the present disclosure, along with the accompanying figures in which like reference numerals refer to like parts throughout.

The following described implementations may be found in the disclosed system and method for three-dimensional (3D) microgeometry and reflectance modeling. Exemplary aspects of the disclosure provide a system and method that may be configured to receive a plurality of images. The plurality of images may include a first set of images of a face and a second set of images of the face. In accordance with an embodiment, the system may control a plurality of imaging devices to capture the plurality of images of the face of the human subject from a plurality of viewpoints. In some embodiments, the system may activate a set of flash units concurrently while the plurality of imaging devices captures the first set of images. The face in the first set of images may be exposed to omni-directional lighting. The system may further activate the set of flash units in a sequential pattern while the plurality of imaging devices captures the second set of images. The face in the second set of images may be exposed to directional lighting.

Based on the received plurality of images, the system may be configured to generate a 3D face mesh. The system may execute, by using the generated 3D face mesh and the second set of images, a set of skin-reflectance modeling operations to estimate a set of texture maps for the face. In accordance with an embodiment, the set of skin-reflectance modeling operations may include a diffused reflection modeling operation, a specular separation operation, and a specular reflection modeling operation. In some embodiments, the system may execute the diffused reflection modeling operation (for example, based on Lambertian light model) to generate a diffuse normal map of the face and a diffuse albedo map of the face of the human subject. The diffuse albedo map may be a first texture map of the estimated set of texture maps. In one or more embodiments, the system may execute the specular separation operation to separate specular reflection information from the second set of images, based on the generated diffuse normal map and the generated diffuse albedo map. In some embodiments, the system may execute the specular reflection modeling operation (for example, based on Blinn-Phong light model) to generate a specular albedo map of the face, a specular normal map of the face, and a roughness map of the face. The specular albedo map, the specular normal map, and the roughness map may be referred to as second texture maps of the estimated set of texture maps. The system may texturize the generated 3D face mesh based on the estimated set of texture maps (such as the first texture map and the second texture maps). The texturization may include a mapping of texture information, including microgeometry skin details and skin reflectance details in the estimated set of texture maps, onto the generated 3D face mesh.

In some conventional methods, images of the face may be acquired only in the omni-directional lighting. Therefore, such images may lack adequate information required to generate accurate texture maps for generation of high fidelity 3D models. However, the system of the present disclosure may control the plurality of imaging devices and may activate the set of flash units to capture the images under both omni-directional lighting and directional lighting. While images under omni-directional lighting may be used to estimate accurate 3D shape of the face, images under directional lighting may be used to estimate texture maps which include both microgeometry skin details (such as pore-level details, ridges, and furrows) and the skin reflectance details (such as specular albedo and roughness). The system may make use of both the accurate 3D shape and texture maps to generate a high fidelity and photorealistic 3D model of the face.

<FIG> is a block diagram that illustrates an exemplary network environment for three-dimensional (3D) microgeometry and reflectance modeling, in accordance with an embodiment of the disclosure. With reference to <FIG>, there is shown a network environment <NUM>. The network environment <NUM> may include a system <NUM>, a plurality of imaging devices <NUM>, a set of flash units <NUM>, and a communication network <NUM>. The system <NUM>, the plurality of imaging devices <NUM>, and the set of flash units <NUM> may be configured to communicate with each other via the communication network <NUM>. In the network environment <NUM>, there is shown a face <NUM> of a person, for example. The plurality of imaging devices <NUM> may acquire a plurality of images <NUM>, which includes a first set of images <NUM> and a second set of images <NUM> of the face <NUM>.

The system <NUM> may include suitable logic, circuitry, and interfaces that may be configured to receive the plurality of images <NUM> (such as the first set of images <NUM> and the second set of images <NUM>) associated with the face <NUM> of a person. The system <NUM> may be further configured to generate a 3D face mesh <NUM> based on the received plurality of images <NUM> and execute a set of skin-reflectance modeling operations to generate texture maps for texturization of the generated 3D face mesh <NUM>. The texturization may generate microgeometry skin details and skin reflectance details on the 3D face mesh <NUM>. Examples of the system <NUM> may include, but are not limited to, a mainframe machine, a server, a computer work-station, a gaming device (such as a game console), a head-mounted display (such as an eXtended Reality (XR) headset), a wearable display device, a consumer electronic (CE) device, or a mobile computer.

The plurality of imaging devices <NUM> may include suitable logic, circuitry, and interfaces that may be configured to capture the plurality of images <NUM> of the face <NUM> of a person from a corresponding plurality of viewpoints. The plurality of imaging devices <NUM> may be further configured to transmit the captured plurality of images <NUM> to the system <NUM>. Examples of an imaging device may include, but are not limited to, an image sensor, a wide-angle camera, an action camera, a camcorder, a digital camera (such as a digital single reflex camera (DSLR) or a digital single lens mirrorless (DSLM)), a camera phone, and/or any image capture device with capability to capture images in multiple formats and at different framerates.

The set of flash units <NUM> may be configured to produce a flash of light based on trigger signals produced by the system <NUM>. The flash of light may be produced to illuminate the face <NUM> of the person. Examples of the set of flash units <NUM> may include, but are not limited to, a built-in and pop up camera flash unit, a dedicated camera flash unit, a macro ring light camera flash unit, and a hammerhead camera flash unit.

It should be noted that the present disclosure is not be limited to the implementation of the plurality of imaging devices <NUM> and the set of flash units as devices separate from the system <NUM>. Accordingly, in some embodiments, the plurality of imaging devices <NUM> and the set of flash units may be included in the system <NUM>, without departing from the scope of the present disclosure.

The communication network <NUM> may include a communication medium through which the system <NUM>, the plurality of imaging devices <NUM>, and the set of flash units <NUM> may communicate with each other. The communication network <NUM> may be one of a wired connection or a wireless connection. Examples of the communication network <NUM> may include, but are not limited to, the Internet, a cloud network, a Cellular or Wireless Mobile Network (such as Long-Term Evolution and <NUM> New Radio), a Wireless Fidelity (Wi-Fi) network, a Personal Area Network (PAN), a Local Area Network (LAN), or a Metropolitan Area Network (MAN). Various devices in the network environment <NUM> may be configured to connect to the communication network <NUM> in accordance with various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of a Transmission Control Protocol and Internet Protocol (TCP/IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Zig Bee, EDGE, IEEE <NUM>, light fidelity (Li-Fi), <NUM>, IEEE <NUM>, IEEE <NUM>, multi-hop communication, wireless access point (AP), device to device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

In operation, the system <NUM> may be configured to control the plurality of imaging devices <NUM> to capture the plurality of images <NUM> from a corresponding plurality of viewpoints. The plurality of imaging devices <NUM> may be arranged at a corresponding first plurality of locations on a 3D structure. For example, the 3D structure may be a dome shaped cage structure that may offer enough space to accommodate at least one person. The set of flash units <NUM> may be arranged at a corresponding set plurality of locations on the 3D structure. Each imaging device and each flash unit may be arranged on the 3D structure so as to surround the person inside the space within the 3D structure from a plurality of viewpoints. An example of such arrangement is provided in <FIG>.

The set of flash units <NUM> may be activated within a duration in which the plurality of imaging devices <NUM> captures the plurality of images <NUM> of the face <NUM>. By way of example, and not limitation, the set of flash units <NUM> may be activated for two camera shots within a defined duration (~<NUM> seconds). In the first shot, while the plurality of imaging devices <NUM> may capture the first set of images <NUM>, the set of flash units <NUM> may be activated concurrently to expose the face <NUM> to omni-directional lighting. In the second shot, while the plurality of imaging devices <NUM> may capture the second set of images <NUM>, the set of flash units <NUM> may be activated in a sequential pattern to expose the face <NUM> to directional lighting. Details of the capture of the plurality of images <NUM> are further provided for example, in <FIG>.

At any time-instant, the system <NUM> may be configured to receive the plurality of images <NUM> (that may include the first set of images <NUM> and the second set of images <NUM>) from the plurality of imaging devices <NUM>. In an embodiment, the system <NUM> may receive the plurality of images <NUM> from a server which maintains a repository of images from various sources. The face <NUM> in the first set of images <NUM> may be exposed to omni-directional lighting and the face <NUM> in the second set of images <NUM> may be exposed to directional lighting.

In accordance with an embodiment, the system <NUM> may be further configured to generate a first 3D face mesh based on the received first set of images <NUM>. The first 3D face mesh may be a raw 3D scan of the face <NUM> and may include artifacts, such as spikes or pointy edges, large and small holes, and other shape irregularities. The system <NUM> may be configured to apply a set of model clean-up operations on the generated first 3D face mesh to obtain a refined first 3D face mesh. By way of example, and not limitation, such model clean-up operations may be applied to remove the artifacts from the first 3D face mesh. The system <NUM> may further generate a second 3D face mesh based on the received second set of images <NUM>. For 3D reconstruction of a 3D face mesh from 2D images, there are many techniques which may be known to one ordinarily skilled in the art. For example, both the first 3D face mesh and the second 3D face mesh may be generated using a photogrammetry-based method (such as structure from motion (SfM)), a method which requires stereoscopic images, or a method which requires monocular cues (such as shape from shading (SfS), photometric stereo, or shape from texture (SfT)). Details of such techniques have been omitted from the disclosure for the sake of brevity.

The system <NUM> may estimate an affine transformation between the refined first 3D face mesh and the generated second 3D face mesh. Thereafter, the system <NUM> may apply the estimated affine transformation on the refined first 3D face mesh to generate the 3D face mesh <NUM>. The generated 3D face mesh may be rigid aligned with the generated second 3D face mesh and may be un-textured. Details of the generation of the 3D face mesh <NUM> are further provided for example, in <FIG> and <FIG>.

By using the generated 3D face mesh <NUM> and the second set of images <NUM>, the system <NUM> may execute a set of skin-reflectance modeling operations to estimate a set of texture maps for the face <NUM>. In accordance with an embodiment, the set of skin-reflectance modeling operations may include a diffused reflection modeling operation, a specular separation operation, and a specular reflection modeling operation. Details of the execution of the set of skin-reflectance modeling operations are provided for example, in <FIG> and <FIG>.

The system <NUM> may texturize the generated 3D face mesh <NUM> based on the estimated set of texture maps. The texturization may include an operation in which texture information, including microgeometry skin details and skin reflectance details, of the estimated set of texture maps is mapped onto the generated 3D face mesh <NUM>. Details of the texturization of the generated 3D face mesh <NUM> are further provided for example, in <FIG>.

<FIG> is a block diagram that illustrates an exemplary system for 3D microgeometry and reflectance modeling, in accordance with an embodiment of the disclosure. <FIG> is explained in conjunction with elements from <FIG>. With reference to <FIG>, there is shown a block diagram <NUM> of the system <NUM>. The system <NUM> may include circuitry <NUM>, a memory <NUM>, an Input/output (I/O) device <NUM>, and a network interface <NUM>. The circuitry <NUM> may be communicatively coupled to the memory <NUM>, the I/O device <NUM>, and the network interface <NUM>.

The circuitry <NUM> may include suitable logic, circuitry, and interfaces that may be configured to execute program instructions associated with different operations to be executed by the system <NUM>. The circuitry <NUM> may include one or more specialized processing units, each of which may be implemented as a separate processor. In an embodiment, the one or more specialized processing units may be implemented as an integrated processor or a cluster of processors that perform the functions of the one or more specialized processing units, collectively. The circuitry <NUM> may be implemented based on a number of processor technologies known in the art. Example implementations of the circuitry <NUM> may include, but are not limited to, an x86-based processor, x64-based processor, a Graphics Processing Unit (GPU), a Reduced Instruction Set Computing (RISC) processor, an Application-Specific Integrated Circuit (ASIC) processor, a coprocessor (such as a Vision Processing Unit (VPU)), a Complex Instruction Set Computing (CISC) processor, a microcontroller, a central processing unit (CPU), and/or a combination thereof.

The memory <NUM> may include suitable logic, circuitry, and interfaces that may be configured to store the program instructions to be executed by the circuitry <NUM>. The memory <NUM> may be configured to store the plurality of images <NUM> (which includes the first set of images <NUM> and the second set of images <NUM>). The memory <NUM> may be also configured to store the generated 3D face mesh <NUM> and the estimated set of texture maps. Example implementations of the memory <NUM> may include, but are not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Hard Disk Drive (HDD), a Solid-State Drive (SSD), a CPU cache, and/or a Secure Digital (SD) card.

The I/O device <NUM> may include suitable logic, circuitry, and interfaces that may be configured to receive an input from a user and provide an output based on the received input. The I/O device <NUM> which may include various input and output devices, may be configured to communicate with the circuitry <NUM>. For example, the system <NUM> may receive a user input, via the I/O device <NUM>, to control the plurality of imaging devices <NUM> to capture the plurality of images <NUM>. The I/O device <NUM>, such as a display may render inputs and/or outputs, such as the generated 3D face mesh <NUM>, the estimated set of texture maps, or the texturized 3D face mesh. Examples of the I/O device <NUM> may include, but are not limited to, a touch screen, a display device, a keyboard, a mouse, a joystick, a microphone, and a speaker.

The network interface <NUM> may include suitable logic, circuitry, and interfaces that may be configured to facilitate communication among the circuitry <NUM>, the plurality of imaging devices <NUM>, and the set of flash units <NUM>, via the communication network <NUM>. The network interface <NUM> may be implemented by use of various known technologies to support wired or wireless communication of the system <NUM> with the communication network <NUM>. The network interface <NUM> may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuitry.

The network interface <NUM> may be configured to enable wired or wireless communication with networks, such as the Internet, an Intranet or a wireless network, such as a cellular telephone network, a wireless local area network (LAN), and a metropolitan area network (MAN). The wireless communication may be configured to use one or more of a plurality of communication standards, protocols and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), wideband code division multiple access (W-CDMA), Long Term Evolution (LTE), <NUM> NR, code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE <NUM>. 11a, IEEE <NUM>. 11b, IEEE <NUM> or IEEE <NUM>. 11n), voice over Internet Protocol (VoIP), light fidelity (Li-Fi), Worldwide Interoperability for Microwave Access (Wi-MAX), a protocol for email, instant messaging, and a Short Message Service (SMS).

The functions or operations executed by the system <NUM>, as described in <FIG>, may be performed by the circuitry <NUM>. Operations executed by the circuitry <NUM> are described in detail, for example, in <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>.

<FIG> is a diagram that illustrates an exemplary photogrammetry setup for 3D microgeometry and reflectance modeling, in accordance with an embodiment of the disclosure. <FIG> are explained in conjunction with elements from <FIG> and <FIG>. With reference to <FIG>, there is shown a diagram <NUM> which includes a 3D structure <NUM>, a plurality of imaging devices <NUM>, a set of flash units <NUM>, and a set of diffusers <NUM>. The diagram <NUM> shows a person <NUM> in a seated position inside the 3D structure <NUM>.

The plurality of imaging devices <NUM> may be arranged (or mounted) at a corresponding first plurality of locations on the 3D structure <NUM>. As shown, for example, the 3D structure <NUM> may be a dome-shaped lighting rig with enough space to include the person <NUM> or at least the face of the person <NUM>. The plurality of flash units <NUM> and the set of diffusers <NUM> may be arranged or mounted at a corresponding second plurality of locations and a corresponding third plurality of locations, respectively, on the 3D structure <NUM>. The arrangement of the plurality of imaging devices <NUM>, the set of flash units <NUM>, and the set of diffusers <NUM> on the 3D structure <NUM> may be such that each imaging device, flash unit and diffuser may be facing towards face <NUM> of the person <NUM> from a particular viewpoint. While an imaging device may acquire an image of the face <NUM> from a first viewpoint, the flash unit may illuminate the face <NUM> from a second viewpoint (which may be same as or different from the first viewpoint).

In the diagram <NUM>, there is shown a set of coded targets <NUM> which are placed on the face <NUM> (such as on the forehead) of the person <NUM>. In some instances, such coded targets <NUM> may include unique codes or identifiers, each of which may help to uniquely identify a location of a portion of the face <NUM> in each of the captured plurality of images <NUM>. For example, when placed on the face <NUM>, the set of coded targets <NUM> may appear at different angles to a number of imaging devices. If code value of a particular coded target is identified in multiple images from different viewpoints, then the location of the coded target in each image may be referenced to a common portion of the face <NUM>.

The circuitry <NUM> may be configured to control the plurality of imaging devices <NUM> to capture the plurality of images <NUM> from a corresponding plurality of viewpoints. By way of example, and not limitation, the circuitry <NUM> may control the plurality of imaging devices <NUM> at a first time-instant to capture the first set of images <NUM> and The circuitry <NUM> may further control the plurality of imaging devices <NUM> at a second time-instant to capture the second set of images <NUM>. Between the first time-instant and the second time-instant, there may be a time difference of about <NUM> seconds.

In an embodiment, the circuitry <NUM> may be configured to activate the set of flash units <NUM> concurrently while the plurality of imaging devices <NUM> captures the first set of images <NUM> at the first instance of time. The number of flash units in the set of flash units <NUM> may be less than or equal to the plurality of imaging devices <NUM>. The concurrent activation of the set of flash units <NUM> may allow the lighting in the 3D structure <NUM> to be omni-directional. The omni-directional lighting may allow the face <NUM> of the person <NUM> to be illuminated evenly. The first set of images <NUM> may include the images of the face <NUM> of the person <NUM> exposed to the omni-directional lighting. The first set of images <NUM> may be utilized to generate an accurate 3D face mesh (such as the 3D face mesh <NUM>) of the face <NUM> of the person <NUM>.

In an embodiment, the circuitry <NUM> may further activate the set of flash units <NUM> in sequential pattern while the plurality of imaging devices <NUM> captures the second set of images <NUM> at the second time-instant. The sequential activation of the set of flash units <NUM> may allow the lighting in the 3D structure <NUM> to be directional. The directional lighting may partially illuminate the face <NUM> of the person <NUM> in each image. In accordance with an embodiment, a light intensity of the directional lighting may be greater than a light intensity of the omni-directional lighting. The light intensity of the omni-directional lighting may be reduced to decrease an amount of illumination on the face <NUM> of the person <NUM>. The set of diffusers <NUM> may be utilized to soften the effect of lighting on the face <NUM> while the plurality of imaging devices <NUM> captures images of the face <NUM>.

In some scenarios, each imaging device may have a delay associated therewith due to a difference between a time of the control (such as activation of a shutter) of the imaging device and a time of capturing of an image (of the plurality of images <NUM>) by the imaging device. Such delay in time may be due to hardware limitation of the plurality of imaging devices <NUM>. The circuitry <NUM> may activate each flash unit in the sequential pattern at a set interval to match the delay encountered by each imaging device. In an embodiment, the circuitry <NUM> may activate a first subset of flash units of the set of flash units <NUM> while a first group of imaging devices (of the plurality of imaging devices <NUM>) captures one or more first images of the second set of images <NUM>. The circuitry <NUM> may activate a second subset of flash units of the set of flash units <NUM> while a second group of imaging devices (of the plurality of imaging devices <NUM>) captures one or more second images of second set of images <NUM>. The second set of images <NUM> may be utilized to capture the microgeometry skin details and the skin reflectance details of the face <NUM> of the person <NUM>.

<FIG> illustrates example images captured under omni-directional lighting conditions, in accordance with an embodiment of the disclosure. <FIG> is explained in conjunction with elements from <FIG>, <FIG>, and <FIG>. With reference to <FIG>, there is shown a diagram 400A which includes a first set of images <NUM>. The first set of images <NUM> may be generated by the plurality of imaging devices <NUM> at a first time-instant. The first set of images <NUM> may include the face <NUM> of the person <NUM> from a corresponding plurality of viewpoints. The face <NUM> in the first set of images <NUM> may be exposed to the omni-directional lighting. For example, a first image may include a left side view of the face <NUM>, a second image may include a right side view of the face <NUM>, and a third image may include a front view of the face <NUM>. The number of images in the first set of images <NUM> may depend on the number imaging devices that may be controlled at the first time-instant to capture the first set of images <NUM>. By way of example, and not limitation, the number of imaging devices may be <NUM> and the number of images in the first set of images <NUM> may be <NUM>.

<FIG> illustrates example images captured under directional lighting conditions, in accordance with an embodiment of the disclosure. <FIG> is explained in conjunction with elements from <FIG>, <FIG>, <FIG>, and <FIG>. With reference to <FIG>, there is shown a diagram 400B which includes a second set of images <NUM>. The second set of images <NUM> may be output by the plurality of imaging devices <NUM> at the second time-instant. The second set of images <NUM> may include the face <NUM> of the person <NUM> from a corresponding plurality of viewpoints. The face <NUM> in the second set of images <NUM> may be exposed to the directional lighting. For example, a first image of may include a left side view of the face <NUM>, a second image may include a right side view of the face <NUM>, and a third image may include a front view of the face <NUM>. The number of images in the second set of images <NUM> may be dependent on the number of imaging devices. For example, the number of imaging devices may be <NUM> and the number of images in the second set of images <NUM> may be <NUM>.

<FIG>, <FIG>, <FIG>, and <FIG> collectively illustrate exemplary operations for 3D microgeometry and reflectance modeling, in accordance with an embodiment of the disclosure. With reference to <FIG>, <FIG>, <FIG>, and <FIG>, there is shown a block diagram <NUM> that illustrates exemplary operations from <NUM> to <NUM>, as described herein. The exemplary operations illustrated in block diagram <NUM> may start at <NUM> and may be performed by any computing system, apparatus, or device, such as by system <NUM> of <FIG> or the circuitry <NUM> of <FIG>. Although illustrated with discrete blocks, the exemplary operations associated with one or more blocks of the block diagram <NUM> may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on implementation of the exemplary operations.

At <NUM>, the plurality of images <NUM> may be received. In accordance with an embodiment, the circuitry <NUM> may be configured to receive the plurality of images <NUM> from the plurality of imaging devices <NUM>. The plurality of images <NUM> may include the first set of images <NUM> and the second set of images <NUM>. As shown, for example, the first set of images <NUM> may include a first image 402A, a second image 402B, a third image 402C, and an Nth image 402N. The second set of images <NUM> may include a first image 404A, a second image 404B, a third image 404C, and an Nth image 404N. The first set of images <NUM> and the second set of images <NUM> may include the face <NUM> of the person <NUM>. While the face <NUM> in the first set of images <NUM> may be exposed to omni-directional lighting, the face <NUM> in the second set of images <NUM> may be exposed to the directional lighting.

At <NUM>, a first 3D face mesh 504A may be generated, based on the received first set of images <NUM>. In accordance with an embodiment, the circuitry <NUM> may be configured to generate the first 3D face mesh 504A, based on the received first set of images <NUM>. For 3D reconstruction of a 3D face mesh from 2D images, there are many techniques which may be known to one ordinarily skilled in the art. For example, the first 3D face mesh 504A may be generated using a photogrammetry-based method (such as structure from motion (SfM)), a method which requires stereoscopic images, or a method which requires monocular cues (such as shape from shading (SfS), photometric stereo, or shape from texture (SfT)). Details of such techniques have been omitted from the disclosure for the sake of brevity.

In an embodiment, the first 3D face mesh 504A may be a raw 3D scan of the face <NUM> of the person <NUM> and may include artifacts, such as pointy edges (i.e. edges with large dihedral angles), spikes, or holes (large and small size). To refine the first 3D face mesh 504A, a set of model clean-up operations may be performed, as described herein.

At <NUM>, a set of model clean-up operations may be applied on the generated first 3D face mesh 504A to obtain a refined first 3D face mesh 506A. In accordance with an embodiment, the circuitry <NUM> may be configured to apply the set of model clean-up operations on the generated first 3D face mesh 504A to obtain the refined first 3D face mesh 506A. In an embodiment, the set of model clean-up operations may include removal of unwanted regions from the first 3D face mesh 504A, filling small holes (such as vacant spaces) in the first 3D face mesh 504A, and removal of spikes from the first 3D face mesh 504A.

The unwanted regions, such as incorrect estimated polygons on the first 3D face mesh 504A (which may not be a part of the face <NUM> of the person <NUM>) may be removed. In the first 3D face mesh 504A, there may be a few vacant spaces or holes (i.e. a sufficiently large space with no polygons) in the first 3D face mesh 504A. Such holes may affect the fidelity of the first 3D face mesh 504A. Therefore, such unwanted holes may be removed, for example, using a suitable prediction method that may rely on arrangement or geometry of nodes in proximity of the holes in the first 3D face mesh 504A. In some instances, incorrect estimation of depth at some locations may lead to generation of unwanted spikes on the first 3D face mesh 504A. The circuitry <NUM> may remove or smoothen such unwanted spikes on the first 3D face mesh 504A to obtain the refined first 3D face mesh 506A.

At <NUM>, a second 3D face mesh 508A may be generated based on the received second set of images <NUM>. In accordance with an embodiment, the circuitry <NUM> may be configured to generate the second 3D face mesh 508A based on the received second set of images <NUM>. Similar to the first 3D face mesh 504A, the second 3D face mesh 508A may be generating using a photogrammetry-based method (such as structure from motion (SfM)), a method which requires stereoscopic images, or a method which requires monocular cues (such as shape from shading (SfS), photometric stereo, or shape from texture (SfT)). Details of such techniques have been omitted from the disclosure for the sake of brevity.

The second 3D face mesh 508A may be a raw 3D scan of the face <NUM> of the person <NUM> and may include one or more of the artifacts. In accordance with an embodiment, the second 3D face mesh 508A may further be refined. The circuitry <NUM> may refine the generated second 3D face mesh 508A based on application of the set of model clean-up operations (as described at <NUM>) on the second 3D face mesh 508A.

At <NUM>, an affine transformation may be estimated between the refined first 3D face mesh 506A and the generated second 3D face mesh 508A. The circuitry <NUM> may be configured to estimate the affine transformation between the refined first 3D face mesh 506A and the generated second 3D face mesh 508A. In accordance with an embodiment, the affine transformation may be estimated based on the set of coded targets <NUM>. By way of example, and not limitation, the circuitry <NUM> may be configured to determine first locations of the set of coded targets <NUM> in the received first set of images <NUM>. The circuitry <NUM> may further determine second locations of the set of coded targets <NUM> on the face <NUM> in the received second set of images <NUM>. The affine transformation may be estimated based on a comparison of the determined first locations and the determined second locations.

As both the refined first 3D face mesh 506A and the second 3D face mesh may not be rigid-aligned initially, the difference between corresponding nodes of the refined first 3D face mesh 506A and the second 3D face mesh 508A may be non-zero. The difference may be calculated using L1 or L2 norm, for example. As shown, for example, a heatmap 510A represents a node-wise difference between the refined first 3D face mesh 506A and the generated second 3D face mesh 508A. Points on the left half of the heatmap 510A represent a higher difference between corresponding nodes of the two meshes as compared to points on the right half of the heatmap 510A.

At <NUM>, the estimated affine transformation may be applied on the refined first 3D face mesh 506A. In accordance with an embodiment, the circuitry <NUM> may be configured to apply the estimated affine transformation on the refined first 3D face mesh 506A to generate the 3D face mesh 512A. The generated 3D face mesh 512A may be rigid aligned with the generated second 3D face mesh 508A.

The affine transformation may include a matrix (or matrices) of rotation and translation values. The relative position and orientation of the refined first 3D face mesh 506A may be updated based on the matrix to match with that of the second 3D face mesh 508A. As shown, for example, a heatmap 510B represents a node-wise difference between the 3D face mesh 512A and the generated second 3D face mesh 508A. All the points on the face region of the heatmap 510B represent a near-zero difference between corresponding nodes of the 3D face mesh 512A and the generated second 3D face mesh 508A. Thus, the heatmap 510B indicates that both the 3D face mesh 512A and the generated second 3D face mesh 508A may be rigid aligned.

At <NUM>, a white balancing operation may be applied on the second set of images <NUM> to generate a set of white-balanced images. Due to lighting variations, the color of the face <NUM> may vary slightly in different images of the second set of images <NUM>. The white balancing operation may be applied to correct the skin color of the face <NUM> in all or some images of the second set of images <NUM>. In accordance with an embodiment, the circuitry <NUM> may be configured to apply the white balancing operation on the second set of images <NUM> to generate the set of white-balanced images. Thereafter, the circuitry <NUM> may obtain a set of specular-less images (such as a first specular-less image 514A and a second specular-less image 514B) by removal of specular information from the set of white-balanced images. The removal of specular information may lead to removal of highlights from the face <NUM> of the person <NUM> in the set of white-balanced images. The specular information may be removed based on conversion of color information, within each image of the set of white-balanced images, from a red-green-blue (RGB) space to an SUV color space. The color information from the RGB space may be converted to the SUV color space by a rotation of RGB coordinate vectors of the RGB space.

At <NUM>, a UV coordinate map of the face <NUM> may be determined based on the generated 3D face mesh 512A. In accordance with an embodiment, the circuitry <NUM> may be configured to determine a UV coordinate map of the face <NUM> based on the generated 3D face mesh 512A. The UV coordinate map of the face <NUM> may be a representation of the 3D face mesh 512A on a 2D UV coordinate space. Thereafter, the circuitry <NUM> may generate an initial texture map of the face <NUM> by texture-mapping the set of specular-less images (such as the first specular-less image 514A and the second specular-less image 514B) onto the determined UV coordinate map. The initial texture map of the face <NUM> may include texture information and color information from skin and/or other visible portions of the face <NUM> in the set of specular-less images. The initial texture map may be represented in UV coordinate space, where "U" and "V" may be 2D coordinates of texture values.

At <NUM>, a set of skin-reflectance modeling operations may be executed. By using the generated 3D face mesh 512A and the second set of images <NUM>, the circuitry <NUM> may execute the set of skin-reflectance modeling operations, as described herein. Such operations may be executed to estimate a set of texture maps for the face <NUM> of the person <NUM>. In accordance with an embodiment, the set of skin-reflectance modeling operations may include a diffused reflection modeling operation, a specular separation operation, and a specular reflection modeling operation. The diffused reflection modeling operation may be executed to generate a diffuse normal map and a diffuse albedo map. The specular separation operation may be executed to generate a specular reflection information separation map. The specular reflection modeling operation may be executed to generate a specular albedo map, a specular normal map, and a roughness map of the face <NUM> of the person <NUM>.

At <NUM>, the diffused reflection modeling operation may be executed. In an embodiment, the circuitry <NUM> may be configured to execute the diffused reflection modeling operation on the second set of images <NUM>. The diffused reflection modeling operation may be executed to generate a diffuse normal map 520A of the face <NUM>, based on the initial texture map (obtained at <NUM>) of the face <NUM>. The diffused reflection modeling operation may be further executed to generate a diffuse albedo map 520B of the face <NUM> based on the initial texture map and the generated diffuse normal map 520A. The diffuse albedo map 520B may be referred to as a first texture map of the estimated set of texture maps (at <NUM>, for example).

In an embodiment, the generation of the diffuse normal map 520A and the diffused albedo map may be based on a Lambertian light model. The Lambertian light model may be represented by equation (<NUM>), as follows: <MAT> where n is a diffuse normal, ρ is a diffuse albedo, and L<NUM> is a direction of light. The direction of light may be determined from a pre-defined position and orientation of each of the plurality of imaging devices <NUM>.

At <NUM>, the specular separation operation may be executed. In accordance with an embodiment, the circuitry <NUM> may be configured to execute the specular separation operation to separate specular reflection information from the second set of images <NUM>. The specular reflection information may be separated based on the generated diffuse normal map 520A and the generated diffuse albedo map 520B. A map 522A which includes the separated specular reflection information is shown as an example.

At <NUM>, the specular reflection modeling operation may be executed. In accordance with an embodiment, the circuitry <NUM> may be configured to execute the specular reflection modeling operation on the second set of images <NUM>. The specular reflection modeling operation may be executed to generate a specular normal map 524A of the face <NUM>, a specular albedo map 524B of the face <NUM>, and a roughness map 524C of the face <NUM>. The specular normal map 524A, the specular albedo map 524B, and the roughness map 524C may be generated based on the separated specular reflection information (and the second set of images <NUM>). The specular normal map 524A may include the shine and highlight information of the face <NUM> in the second set of images <NUM>. The specular albedo map 524B may include the color information of the face <NUM> and may exclude the highlight information and shadow information of the face <NUM> of the person <NUM>. The roughness map 524C may represent a roughness of the skin of the face <NUM> of the person <NUM>. The roughness map 524C may be represented as a black and white color texture image. The specular normal map 524A, the specular albedo map 524B, and the roughness map 524C may be referred to as second texture maps of the set of texture maps (estimated at <NUM>). The first texture map and the second texture maps may include the microgeometry skin details and the skin reflectance details of the face <NUM> of the person <NUM>.

In an embodiment, the generation of the specular normal map 524A, the specular albedo map 524B, and the roughness map 524C may be based on the Blinn-Phong light model. The Blinn-Phong light model may be represented by equation (<NUM>), as follows: <MAT> where n is a specular normal, ρ is a specular albedo, and α relates to a surface roughness.

At <NUM>, the generated 3D face mesh 512A may be texturized. In accordance with an embodiment, the circuitry <NUM> may be configured to texturize the generated 3D face mesh 512A based on the estimated set of texture maps. The texturization may include an operation in which texture information, including microgeometry skin details and skin reflectance details, of the estimated set of texture maps is mapped onto the generated 3D face mesh 512A. In accordance with an embodiment, the estimated set of texture maps may include the diffuse albedo map 520B of the face <NUM>, the specular normal map 524A of the face <NUM>, the specular albedo map 524B of the face <NUM>, and the roughness map 524C of the face <NUM>. The microgeometry skin details may include texture information for various skin components, such as pores, ridges, freckles, and furrows. Similarly, the skin reflectance details may include information for a diffused reflection component, a specular reflection component, an albedo component, and a roughness component. The texturized 3D face model 526A may include both the microgeometry skin details and the skin reflectance details. Therefore, the texturized 3D face model 526A may be treated as a high fidelity 3D model of the face <NUM> of the person <NUM>.

Although the block diagram <NUM> is illustrated as discrete operations, such as <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, the disclosure is not so limited. Accordingly, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation without detracting from the essence of the disclosed embodiments.

<FIG> is a flowchart that illustrates an exemplary method 3D microgeometry and reflectance modeling, in accordance with an embodiment of the disclosure. <FIG> is explained in conjunction with elements from <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>. With reference to <FIG>, there is shown a flowchart <NUM>. The method illustrated in the flowchart <NUM> may be executed by any computing system, such as by the system <NUM> or the circuitry <NUM>. The method may start at <NUM> and proceed to <NUM>.

At <NUM>, the plurality of images <NUM> that may include the first set of images <NUM> of the face <NUM> and the second set of images <NUM> of the face <NUM> may be received. In accordance with an embodiment, the circuitry <NUM> may be configured to receive the plurality of images <NUM> that may include the first set of images <NUM> of the face <NUM> and the second set of images <NUM> of the face <NUM> of the person <NUM>. The face <NUM> in the first set of images <NUM> may be exposed to the omni-directional lighting and the face <NUM> in the second set of images <NUM> may be exposed to the directional lighting. Details of the reception of the plurality of images <NUM> are further provided for example, in <FIG>.

At <NUM>, the 3D face mesh 512A may be generated based on the received plurality of images <NUM>. In accordance with an embodiment, the circuitry <NUM> may be configured to generate the 3D face mesh 512A based on the received plurality of images <NUM>. Details of the generation of the 3D face mesh 512A are further provided for example, in <FIG>.

At <NUM>, a set of skin-reflectance modeling operations may be executed, by using the generated 3D face mesh 512A and the second set of images <NUM>, to estimate the set of texture maps for the face <NUM>. In accordance with an embodiment, the circuitry <NUM> may be configured to execute the set of skin-reflectance modeling operations, by using the generated 3D face mesh 512A and the second set of images <NUM>, to estimate the set of texture maps for the face <NUM>. Details of the execution of the set of skin-reflectance modeling operations are provided, for example, in <FIG> and <FIG>.

At <NUM>, the generated 3D face mesh 512A may be texturized based on the estimated set of texture maps. In accordance with an embodiment, the circuitry <NUM> may be configured to texturize the generated 3D face mesh 512A based on the estimated set of texture maps. The texturization may include an operation in which texture information, including microgeometry skin details and skin reflectance details, of the estimated set of texture maps is mapped onto the generated 3D face mesh 512A. Details of the texturization of the 3D face mesh 512A are further provided for example, in <FIG>. Control may pass to end.

Although the flowchart <NUM> is illustrated as discrete operations, such as <NUM>, <NUM>, <NUM>, <NUM> and <NUM>, the disclosure is not so limited. Accordingly, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation without detracting from the essence of the disclosed embodiments.

Various embodiments of the disclosure may provide a non-transitory computer-readable medium and/or storage medium having stored thereon, instructions executable by a machine and/or a computer to operate a system (such as the system <NUM>). The instructions may cause the machine and/or computer to perform operations that may include receiving a plurality of images (such as the plurality of images <NUM>) which may include a first set of images (such as the first set of images <NUM>) of a face (such as the face <NUM>) and a second set of images (such as the second set of images <NUM>) of the face <NUM>. The face <NUM> in the first set of images114 may be exposed to omni-directional lighting and the face <NUM> in the second set of images <NUM> may be exposed to directional lighting. The operations may further include generating a three-dimensional (3D) face mesh (such as the 3D face mesh <NUM>) based on the received plurality of images <NUM>. The operations may further include executing, by using the generated 3D face mesh <NUM> and the second set of images <NUM>, a set of skin-reflectance modeling operations to estimate a set of texture maps for the face <NUM>. The operations may further include texturizing the generated 3D face mesh <NUM> based on the estimated set of texture maps. The texturization may include an operation in which texture information, including microgeometry skin details and skin reflectance details, of the estimated set of texture maps is mapped onto the generated 3D face mesh <NUM>.

Exemplary aspects of the disclosure may provide a system (such as the system <NUM> of <FIG>) that includes circuitry (such as the circuitry <NUM>). The circuitry <NUM> may be configured to receive a plurality of images (such as the plurality of images <NUM>) which may include a first set of images (such as the first set of images <NUM>) of a face (such as the face <NUM>) and a second set of images (such as the second set of images <NUM>) of the face <NUM>. The face <NUM> in the first set of images114 may be exposed to omni-directional lighting and the face <NUM> in the second set of images <NUM> may be exposed to directional lighting. The circuitry <NUM> may be further configured to generate a three-dimensional (3D) face mesh (such as the 3D face mesh <NUM>) based on the received plurality of images <NUM>. The circuitry <NUM> may be further configured to execute, by using the generated 3D face mesh <NUM> and the second set of images <NUM>, a set of skin-reflectance modeling operations to estimate a set of texture maps for the face <NUM>. The circuitry <NUM> may be further configured to texturize the generated 3D face mesh <NUM> based on the estimated set of texture maps. The texturization may include an operation in which texture information, including microgeometry skin details and skin reflectance details, of the estimated set of texture maps is mapped onto the generated 3D face mesh <NUM>.

In accordance with an embodiment, the system <NUM> may further include a plurality of imaging devices (such as the plurality of imaging devices <NUM>) arranged at a corresponding first plurality of locations on a 3D structure (such as the 3D structure <NUM>). The circuitry <NUM> may be further configured to control the plurality of imaging devices <NUM> to capture the plurality of images <NUM> from a corresponding plurality of viewpoints.

In accordance with an embodiment, the system <NUM> may further include a set of flash units (such as the set of flash units <NUM>) arranged at a corresponding second plurality of locations on the 3D structure <NUM>. The circuitry <NUM> may be further configured to activate the set of flash units <NUM> concurrently while the plurality of imaging devices <NUM> captures the first set of images <NUM>. The circuitry <NUM> may activate the set of flash units <NUM> in sequential pattern while the plurality of imaging devices <NUM> captures the second set of images <NUM>.

In accordance with an embodiment, the light intensity of the directional lighting may be greater than a light intensity of the omni-directional lighting.

In accordance with an embodiment, the circuitry <NUM> may be further configured to generate a first 3D face mesh (such as the first 3D face mesh 504A), based on the received first set of images <NUM>. The circuitry <NUM> may apply a set of model clean-up operations on the generated first 3D face mesh 504A to obtain a refined first 3D face mesh (such as the refined first 3D face mesh 506A). The circuitry <NUM> may further generate a second 3D face mesh (such as the second 3D face mesh 508A) based on the received second set of images <NUM>. The circuitry <NUM> may estimate an affine transformation between the refined first 3D face mesh 506A and the generated second 3D face mesh 508A. The circuitry <NUM> may further apply the estimated affine transformation on the refined first 3D face mesh 506A to generate the 3D face mesh 512A. The generated 3D face mesh 512A may be rigid aligned with the generated second 3D face mesh 508A.

In accordance with an embodiment, the circuitry <NUM> may be further configured to determine first locations of a set of coded targets (such as the set of coded targets <NUM>) on the face <NUM> in the received first set of images <NUM>. The circuitry <NUM> may determine second locations of the set of coded targets <NUM> on the face <NUM> e in the received second set of images <NUM>. The circuitry <NUM> may further estimate the affine transformation based on comparison of the determined first locations and the determined second locations.

In accordance with an embodiment, the circuitry <NUM> may be further configured to apply a white balancing operation on the second set of images <NUM> to generate a set of white-balanced images. The circuitry <NUM> may obtain a set of specular-less images (such as the first specular-less image 514A and the second specular-less image 514B) by removal of specular information from the set of white-balanced images. The specular information may be removed based on conversion of color information, within each image in the received second set of images <NUM>, from a red-green-blue (RGB) space to an SUV color space.

In accordance with an embodiment, the circuitry <NUM> may be further configured to determine a UV coordinate map of the face <NUM> based on the generated 3D face mesh 512A. The circuitry <NUM> may further generate an initial texture map of the face <NUM> by texture-mapping the set of specular-less images onto the determined UV coordinate map.

In accordance with an embodiment, the set of skin-reflectance modeling operations may include a diffused reflection modeling operation, a specular separation operation, and a specular reflection modeling operation.

In accordance with an embodiment, the circuitry <NUM> may be further configured to execute the diffused reflection modeling operation to generate a diffuse normal map (such as the diffuse normal map 520A) of the face <NUM> based on the initial texture map. The circuitry <NUM> may further generate a diffuse albedo map (such as the diffuse albedo map 520B) of the face <NUM> based on the initial texture map and the generated diffuse normal map 520A. The diffuse albedo map 520B may be a first texture map of the estimated set of texture maps.

In accordance with an embodiment, the circuitry <NUM> may be further configured to execute the specular separation operation to separate specular reflection information from the second set of images <NUM>, based on the generated diffuse normal map 520A and the generated diffuse albedo map 520B.

In accordance with an embodiment, the circuitry <NUM> may be further configured to execute the specular reflection modeling operation to generate, based on the separated specular reflection information, a specular albedo map (such as the specular albedo map 524B) of the face <NUM>, a specular normal map (such as the specular normal map 524A) of the face <NUM>, and a roughness map (such as the roughness map 524C) of the face <NUM>. The specular albedo map 524B, the specular normal map 524A, and the roughness map 524C may be second texture maps of the estimated set of texture maps.

In accordance with an embodiment, the estimated set of texture maps may include the diffuse albedo map 520B of the face <NUM>, the specular albedo map 524B of the face <NUM>, the specular normal map 524A of the face <NUM>, and the roughness map 524C of the face <NUM>.

The present disclosure may be realized in hardware, or a combination of hardware and software. The present disclosure may be realized in a centralized fashion, in at least one computer system, or in a distributed fashion, where different elements may be spread across several interconnected computer systems. A computer system or other apparatus adapted to carry out the methods described herein may be suited. A combination of hardware and software may be a general-purpose computer system with a computer program that, when loaded and executed, may control the computer system such that it carries out the methods described herein. The present disclosure may be realized in hardware that comprises a portion of an integrated circuit that also performs other functions.

The present disclosure may also be embedded in a computer program product, which comprises all the features that enable the implementation of the methods described herein, and which when loaded in a computer system is able to carry out these methods. Computer program, in the present context, means any expression, in any language, code or notation, of a set of instructions intended to cause a system with information processing capability to perform a particular function either directly, or after either or both of the following: a) conversion to another language, code or notation; b) reproduction in a different material form.

Claim 1:
A system (<NUM>), characterised in comprising circuitry configured to:
receive a plurality of images comprising a first set of images of a face and a second set of images of the face,
wherein the face in the first set of images is exposed to omni-directional lighting and the face in the second set of images is exposed to directional lighting;
generate a three-dimensional (3D) face mesh based on the received plurality of images;
execute, by using the generated 3D face mesh and the second set of images, a set of skin-reflectance modeling operations to estimate a set of texture maps for the face; and
texturize the generated 3D face mesh based on the estimated set of texture maps,
wherein the texturization comprises an operation in which texture information, including microgeometry skin details and skin reflectance details, of the estimated set of texture maps is mapped onto the generated 3D face mesh.