Perception of Lighting Errors in Image Compositing

Image compositing is a standard computer graphics technique used to merge independently created visual elements into a single image. However, errors in the source material or the compositing process can destroy the realism of the result. In this paper we describe a series of experiments that investigate the role that lighting errors in the source material have on the realism of composite images. We study two classes of errors: pixel errors differences in illumination intensity or color temperature; and cue errors – differences in illumination or shadow direction. The results show that the sensitivity to errors varies with the type of error. While observers are reasonably sensitive to pixel errors that change the statistics of the foreground and background image regions, they are less sensitive to cue errors that leave the image statistics unchanged but introduce conflicting information about scene lighting conditions. These trends are modulated by the scene content. These studies represent some first steps towards developing perceptual metrics for error tolerance in image compositing that can be used to improve the fidelity and efficiency of compositing process. Introduction Image compositing is a technique in computer graphics that is used to merge independently created visual elements into a single image. The origins of compositing date back to 1857, when Oscar G. Rejlander created a single image by combining different regions of 32 photographs [5]. Digital image compositing has become ubiquitous in filmmaking [1,8], television [22], virtual sets [11], and augmented reality [7]. When used successfully, compositing creates the illusion that the composited elements are part of a single, cohesive scene. However, errors in either the compositing process or the source material can destroy the illusion. For example, a standard compositing technique is to photograph an object against a uniformly colored screen, and then to superimpose this object on a new background. Figure 1 (left) shows that errors in the compositing process, such a colored silhouette around the object caused by poor segmentation, clearly indicate that the image is not real. Recent advances in compositing technology [22,16] have eliminated the majority of these process errors. Unfortunately, even scenes that are “process perfect” may not appear realistic because of lighting errors in the source material. An example of a common source error is shown in Figure 1 (right), where the composited object’s lighting properties do not match the target environment’s lighting. Although in the compositing industry there are rules of thumb for creating realistic results, there have been no systematic studies of the tolerance for such errors. An understanding of how these source errors affect the realism of composite images would greatly facilitate the compositing process. In this paper, we study the visual effects of source errors in image compositing. In a series of psychophysical experiments we measure visual sensitivity to four common source lighting errors: brightness errors, color errors, illumination direction errors and shadow direction errors. We use two scenes that are representative of common compositing situations: a tabletop still life, and a television “talking heads” scene with two subjects sitting behind a desk. Our results show that different lighting errors are not equally detectable, and also that the detectability of errors depends in part on the scene content. Background In 1984 Porter and Duff [17] introduced a digital matting algebra that is now widely used in compositing applications, that allowed two images to be merged by employing an extra image “alpha” channel that contains transparency information. This operation allowed objects photographed against a uniform “blue screen” to be segmented and superimposed over selected backgrounds. Brinkmann [5] and Rickitt [20] offer excellent surveys of the history and development of digital image compositing. Process errors in compositing Recent advances in blue screen technology [22,16] have solved many of the problems traditionally associated with the compositing process. For example, one early problem was how to correct for the blue (or green) light that would reflect from the background and would produce halos around composited objects. This problem is now solved automatically by most systems (known as spill suppression). Camera errors have been another source of problems in image compositing. In standard cameras, the finite depth of field causes image focus differences. Film grain or CCD noise is imparted to the images. Motion blur is recorded due to finite exposure times. Perspective distortion is added because of the nature of lenses. Color imbalances may be present in the media. Finally, the CCD or film stock will have a limited dynamic range that will impart some transfer function onto the recorded images. To effectively Figure 1. Errors in image compositing. Left: Composite process error (improper segmentation), Right: Image source error (differences in lighting color and direction). composite images created through different processes, one must understand the limitations inherent in each process and compensate for them. Fortunately, for the most part in modern compositing systems, these “process” problems have been solved [5]. Source errors in image compositing Unfortunately, the focus on the technical “process” of compositing, ignores the role that “source” factors have on the realism of the result. However, manufacturers of compositing systems are beginning to realize that creating better source material is a critical issue. For example, in a technical bulletin [21], Ultimatte makes the following plea to users: “Proper lighting is the key to realism in image compositing. Not only will it be difficult to make the Ultimatte function properly if the lighting is not right, but even a technically perfect composite will look phony with bad lighting”. Books and papers on compositing techniques [21,5] have begun to note rules of thumb to guide users towards better results: • Composite scenes that are matching exterior shots should be filmed outside. • Discrepancies in skin tones should be avoided. • Matching the location of light sources is important for scenes with strongly cast shadows. • Color filters (gels) should be used to recreate the lighting color for scenes. While these rules of thumb are widely known, guidelines for applying them to achieve visually realistic results are lacking. Studies that link composting practice to knowledge of the requirements and limitations of human vision would be of great value. Source lighting errors Lighting errors in image compositing can be placed into two categories: pixel errors and cue errors. Pixel errors include brightness and/or color differences between the composite layers. These are typically produced by differences in the intensity or color temperature of the illumination in the scenes that are being composited. Cue errors on the other hand, are inconsistencies in the spatial properties of illumination in the composited scenes. Sources of cue errors include discrepancies in the direction of illumination (surface shading) and differences in the directions of cast shadows. With pixel errors, the image statistics of the composited element and the scene context are different, according to a uniform, histogram-modifying rule. In contrast, cue errors preserve the image statistics of the composited region and the scene context, but the visual information about direction and qualities of illumination are contradictory. In the experiments that follow we study four lighting errors that are considered most likely to affect the realism of image composites. With respect to pixel errors we study the effects of source brightness and color temperature differences. With respect to cue errors we study the effects of differences in shading and shadows caused variation in the spatial properties of lighting. The errors introduced by these manipulations are representative of many real-world compositing situations and therefore the results of the studies should be directly applicable to practice. Contributions from vision and graphics In the vision literature, there are many studies on the perception of objects and scene illumination [3,2,10,12,15], but little of this work is directly applicable to the analysis of lighting errors in compositing. One exception is the work of Ostrovsky et al. [14], which directly addresses the issue of lighting direction inconsistencies in images. They found that observers were relatively insensitive to even large lighting inconsistencies, however, they did not study the errors parametrically, which limits the applicability of the results. In the computer vision literature there have recently been efforts [26,27,28] to develop algorithms for automatically detecting image forgeries and composites on the basis of errors in shading, shadows, and perspective. These algorithms have focused on taking advantage of photometric and projective inconsistencies and have not focused on human abilities for detecting errors. One exception is the work of Farid and Bravo [29] who have recently started to investigate the perception of shading, shadow, and perspective errors. In the computer graphics literature, studies by a number of researchers [4,13,19,18,23] have focused on the promise of perceptually-based rendering, in which models of human vision are used to improve the efficiency and fidelity of the image synthesis process. One recent project that is particularly relevant is Ramanarayan et al.’s [25] work on the concept of “visual equivalence” that quantifies how changes in the light field in a scene affects the appearance of objects, and develops a metric that can predict when two different images are equally realistic as representations of a scene. Figure 2. Test scenes and lighting errors. Rows show the library (top) and kitchen (bottom)