A Model-Based Approach to Image Relighting with a Potential for Real-Time Implementation

AbstractImage relighting is a very unique special visual effect which promises to have many important practical applica-tions. Image relighting is essentially the process of, given one or more images of some scene, computing what thatscene would look like under some other (arbitrary) lighting conditions, e.g., changing positions and colors of lightsources. Image relighting can for example be used for interior light design. This paper describes an approach toimage relighting which can be implemented to run in real-time by utilizing graphicshardware, as opposed to otherstate-of-the-art approaches which at best run at a few frames per second.Categories and Subject Descriptors (according to ACM CCS) : I.3. 4 [Computer Graphics]: Three-DimensionalGraphics and Realism1. IntroductionThis paper addresses the subject of developing relightingtechniques, i.e. techniques allowing a user to completely al-ter the lighting conditions in an image of a real scene. Specif-ically, the paper focuses on techniques providing real-timerelighting functionalities, thus enabling the user to interac-tively change lighting conditions and get "instant" visualfeedback. Figure 1 provides an example of the kind of re-lighting this paper addresses. It should be noted that the workpresented here presumes the availability of three things: 1)an original image of the scene, 2) a 3D model of the scene,and 3) a model of the lighting conditions in the scene at thetime the original image is acquired. We will return to waysin which the two last pieces of knowledge can be obtained.Conceptually image relighting in this manner is a two stepprocess. In the ﬁrst step all effects of the original lightingconditions are removed, e.g., highlights, shadows, and dif-ferences in shading across surfaces due to varying light in-cidence angles. In the second step the scene is subjected tosome arbitrary new lighting conditions and the appearanceof the scene in these conditions is computed. The secondstep thus "adds" new highlights, shadows and shading etc.Of these two steps the former is the tricky one, while thelatter can be performed using any preferred rendering tech-nique, e.g., ray tracing, radiosity or standard hardware accel-erated approaches. Which rendering technique is employeddepends on the preferred balance between rendering speedand accuracy in handling various lighting phenomena. In or-der to achieve true real-time performance we have chosen touse a hardware accelerated approach for step 2, thus sacri-ﬁcing certain global illumination phenomena.In our approach step 1 is achieved by a computational ap-proach which requires, as stated above, a 3D model of thescene and a model of the original lighting conditions. Alter-natively, one could in principle acquire fronto-parallel digitalimages of the surfaces in the scene under perfectly diffuse,white-balanced lighting conditions and use these images astextures on the 3D model, which is subsequently renderedunder novel lighting conditions (step 2). This would be amechanical or image acquisition approach to step 1, but inreality acquiring such ’clean’ textures devoid of lighting ef-fects is not practical for general scenes.The contributions in this work lie in the speciﬁc manner inwhich the operations performed in steps 1 and 2 are carriedout. With the approach described here the two steps can becombined such that the image relighting becomes a matter ofmodulating the original image on a pixel by pixel basis witha "relighting map". The relighting map can be computed in