The Role of Bright Pixels in Illumination Estimation

The White-Patch method, one of the very first colour constancy methods, estimates the illuminant colour from the ma ximum response of three colour channels. However, this simpl e method has been superseded by advanced physical, statistic al and learning based colour constancy methods. Recently, a few re search works have suggested that the simple idea of using max imum pixel values is not as limited an idea as it seems on first glance. These works show that in several situations some man ipulations can indeed made it perform very well. Here, we exten d the White-Patch assumption to include any of: white patch, h ighlights or light source; let us refer to these pixels in an imag e as the “bright” pixels areas. We propose that bright pixels are surprisingly helpful in the illumination estimation process. In this paper, we investigate the effects of bright pixels on several current colour constancy algorithms. Moreover, we describe a simple framework for an illumination estimation me thod based on bright pixels and compare its accuracy to well-know n colour constancy algorithms applied to four standard datas e s. We also investigate failure and success cases, using bright pixels, and propose desiderata on input images with regard to th e proposed method. Introduction Illumination estimation, which is the main step in colour constancy processing, is an important prerequisite for man y computer vision applications. One of the first colour constancy methods, the so-called White-Patch or Max-RGB method estimates the illuminant colour from the maximum response of three col our channels [24]. With the advent of newer and more precise colo ur constancy methods such as Grey-World [3], Gamut Mapping [12 ], Grey-Edge [31] and many other complex methods (refer to [21] for an overview), few researchers or commercial cameras use the White-Patch method. On the other hand, recent research such as that on perceptual color contrast enhancement by Choudhu ry and Medioni [4] or on the “rehabilitation” of MaxRGB by Funt and Shi [15] propose that a local mean calculation such as local blurring as a preprocessing step can significantly impro ve the performance of this simple method, consisting of simply find ing the maximum in each colour channel. Simply put, these works propose it is advantageous to calculate the max of a local mea n image. Recently, Drew et al. [6] found analytically that the geomet ric mean of bright (generally, specular) pixels is the optim al estimate for the illuminant, based on a standard dichromatic m odel for image formation (which accounts for the matte and highli ght appearance of objects). This work proposes that in the prese nc of specular highlights the “mean of the max” is the best illuminant estimate, in contradistinction to previous works [1 5, 4] which say it is the “max of the mean. ” The analytical approach [6] claims performance comparable with very complex colour con stancy methods despite its simplicity. The bright areas of images can be white surfaces or light sources as well as highlights and specularity, and all are he lpful in the illumination estimation process. Highlights and whi te surfaces both tend to have the colour of light in ideal condition s for dielectric materials such as plastic. In this paper, we investigate the effects of bright pixels on different colour constancy algorithms. We describe a simpl e framework for an illumination estimation method based on br ight pixels and compare its accuracy to well-known colour consta ncy algorithms applied to four standard datasets. We also inves tigate failure and success cases, using bright pixels, and draw con clusions on input images with regard to the proposed method. Illumination Estimation by Specular reflection In specular reflection, light from a single incoming directi on is reflected into a small cone of outgoing directions. This co ntrasts with diffuse reflection, where light is partially absorbed a nd partially scattered within the surface material. Areas of imag es that are specular tend to be bright. Moreover, the spectral power distribution (SPD) of specular reflections is the same as the ill um nation’s SPD, within a Neutral Interface Reflection (NIR) [25] condition, which mostly obtains for the surfaces of optically i nhomogeneous objects (such as ceramics, plastics, paints, etc.) ; however it does not always hold for the surfaces of optically homogen eous objects (such as gold, bronze, copper, etc.) [20]. These pro perties make specular reflection, which is usually in bright are as of image, an appropriate tool for estimating illumination. Ma ny illumination estimation methods derive from the dichromatic mo del for specular reflection proposed by Shafer [28]. Klinker et al. [23] showed that when the diffuse colour is constant over a surface, the colour histogram of its image fo rms a skewed-T shaped distribution, with the diffuse and specula r pixels forming linear clusters. They used this information to esti mate a single diffuse colour. Therefore in order to use this princ iple, their approach needed to segment an image into several regio ns of homogeneous diffuse colour. Lee [26] proposed a method which uses specularity to compute illumination by using the fact that in the CIE chromatic ity diagram [33] the coordinates of the colours from different p oints from the same surface will fall on a straight line connected t o the specular point. This is the case when the light reflected from a uniform surface is an additive mixture of the specular compo nent and the diffuse component. This seminal work initiated a sub stantial body of work on identifying specular pixels and using th ese to attempt to discover the illuminant [27, 30]. Another appr oach extending these algorithms is to define a constraint on the po ssible colours of illumination, making estimation more robust [8, 9]. Extending the White Patch Hypothesis The White-Patch hypothesis is essentially that there is always a white surface in the image. Let us extend this assumpti on to include any of: white patch, specularities, or light sour ce (or an effective white, e.g. a bright yellow and red pixel which c ombined have the same max R, G and B as a white patch). We also use the termgamutof bright pixels, in contradistinction to maximum channel response of the White-Patch method, which typi cally deals only with the brightest pixel in the image. Obvio usly, using a single pixel or very small area is noisy and not robust . Since are we dealing with bright pixels we need to be very careful about clipped pixels, i.e. pixels where the light re flection exceeds the dynamic range of the camera. Here for each colour channel we remove pixels which exceed 90% of the dynamic range. Then we simply define bright pixels as the top T% of luminance given byR+G+B. To investigate the utility of this assumption, we carry out a simple experiment. We check whether or not the actual illuminant colour falls inside the 2D gamut of bright pixels. We find that the actual illuminant colour falls in the 2D gamut of the top 5% brightness pixels of each image for the SFU Laboratory Dataset [2] for 88 . 16% of images, in 74 . 7% of images for the ColorChecker dataset [29], and in 66 . 02% of images for the GreyBall Dataset [5]. Fig. 1 shows the 2D gamut in chromatici ty space{r,g} = {R,G}/(R+G+B), with the top-5% brightness pixels in green. The actual measured illuminant is shown as a red star. Clearly, as Fig. 1(c) shows, with no supporting evi d nce it may happen that the illuminant does not fall within the bri ght region. 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8