Patent Document ID: 8928734
Application ID: 13113178
Patent Flag: 1

Claim One:
1. A method for free-view relighting of a dynamic scene based on photometric stereo, comprising the following steps of: (1) performing multi-view dynamic videos capturing of an object using a multi-view camera array under a predetermined controllable varying illumination, and setting a controllable lighting sequence by a controllable light source array comprising a plurality of LED light sources to provide the desired predetermined controllable varying illuminations, wherein step (1) further includes the steps of: placing a crystal ball at a center of the acquisiting scene surrounded by the multi-view camera array and the controllable light source array, and capturing the crystal ball by the multi-view camera array using the predetermined controllable lighting sequence provided by the controllable light source array; and forming a panorama of the dynamic scene based on frames of the crystal ball in the multi-view dynamic video to obtain an environment map corresponding to the time; (2) obtaining a three-dimensional shape model and surface reflectance peculiarities of the object based on the captured multi-view dynamic videos, and obtaining surface normal information and surface reflectance peculiarities of a visible area in each single view using the obtained dynamic videos of the object for each single view, wherein the step of obtaining the normal information and surface reflectance peculiarities further includes the steps of: aligning any two image frames captured under uniform lighting of the controllable lighting sequence in the obtained dynamic videos for each single view and obtaining a two-dimensional trajectory of the object; interpolating the obtained two-dimensional trajectory of the object and aligning each captured video under variable lighting of the controllable lighting sequence with the image frame captured under the uniform lighting; performing a confidence verification to pixels in the image frames captured under the variable lighting and the uniform lighting using Robust Principal Component Analysis method; and recovering the surface normal information and the surface reflectance peculiarities in the visible areas of the object in the each single view using the image frames acquired under the predetermined controllable varying illumination after the confidence verification based on a polynomial texture model wherein the polynomial texture model is I s =  L  ⁢ ∑ k = 1 K ⁢ ρ k s ⁡ ( N * H ) k , ⁢ and ⁢ 
 H= ½( V+L ), where K represents an order of the polynomial texture model; L represents an incident light direction; V represents a viewing direction; and H represents a halfway vector based on L and V respectively; and the step of recovering the surface normal information and the surface reflectance peculiarities further includes the following steps: recovering the surface normal information N and a diffuse albedo coefficient ρ d in the visible areas of the object in the single view based on a Lambertian lighting model I d =ρ d (N*L), where L is calculated based on the environment maps; recovering the specularity (ρ 1 s ,. .. , ρ K s ) of the surfaces in the corresponding visible areas based on I s =I−I d and the polynomial texture model, where ρ 1 s , ρ K s are specularity albedo coefficients corresponding to the first to k th order, and I represents a pixel value in the captured image frame; and obtaining reflectance peculiarities ρ=(ρ d , ρ 1 s ,. .. , ρ K s ) of the surfaces in the visible areas by combining the diffuse albedo coefficient ρ d and the specularity (ρ 1 s ,. .. , ρ K s ); and the step (2) further includes integrating the surface normal information and the surface reflectance peculiarities of the visible area in each view to obtain the three-dimensional shape model of the object and model surface reflectance peculiarities corresponding to the three-dimensional shape model, (3) obtaining a static relighted three-dimensional model of the object based on the obtained three-dimensional shape model and surface reflectance peculiarities thereof, and obtaining a three-dimensional trajectory of the object based on the three-dimensional shape model, the surface reflectance peculiarities and the multi-view dynamic videos; (4) obtaining a dynamic relighted three-dimensional model based on the static relighted three-dimensional model and the three-dimensional trajectory of the object; and (5) performing a free-view dependent rendering to the dynamic relighted three-dimensional model of the object, whereby a free-view relighting of the object is achieved.