Removing Outliers in Illumination Estimation

A method of outlier detection is proposed as a way of improving illumination-estimation performance in general, and for scenes with multiple sources of illumination in particular. Based on random sample consensus (RANSAC), the proposed method (i) makes estimates of the illumination chromaticity from multiple, randomly sampled sub-images of the input image; (ii) fits a model to the estimates; (iii) makes further estimates, which are classified as useful or not on the basis of the initial model; (iv) and produces a final estimate based on the ones classified as being useful. Tests on the Gehler colorchecker set of 568 images demonstrate that the proposed method works well, improves upon the performance of the base algorithm it uses for obtaining the sub-image estimates, and can roughly identify the image areas corresponding to different scene illuminants. Introduction There are two types of outliers that create problems for illumination-estimation algorithms. In the illumination-estimation context, an outlier is an observation that does not fit the illumination model well. One type of outlier arises from noise in the image data created, for example, by a speck of dust on the imaging sensor, or by clipping of high digital counts. A second, more interesting, type of outlier arises from scenes that do not fit the expected model of the scene illumination—for example, a predominantly indoor scene with some light also coming through a window. In this paper, we apply the RANSAC (random sample consensus) [5] technique to randomly sampled subwindows of the input image as a means of handling outliers of both these types. The proposed algorithm was evaluated on Shi’s reprocessed version of Gehler’s original ‘colorchecker’ set of 568 images [7, 14] and found to reduce the high-percentile angular errors by roughly 30%. Illumination estimation is the crucial step in standard automatic white balancing or ‘color constancy’. The goal in illumination estimation is to determine the chromaticity of the overall scene illumination. The accuracy of the estimate is often measured in terms of the angular difference in degrees between the estimated chromaticity and the actual chromaticity when treated as vectors in 3-space. Many illumination-estimation algorithms have been proposed and are surveyed by Barnard et al. [1], Hordley et al. [10] and Gijsenij et al. [8]. Noting that outliers of the two types mentioned above will mislead most illumination-estimation algorithms, we propose a novel technique to improve upon any given algorithm or set of algorithms by explicitly taking outliers into account. The proposed method divides the input image into smaller sub-images, runs the algorithm(s) on each of the parts independently and then combines the resulting estimates. Outliers are identified and eliminated as part of the process of combining the estimates. The rational behind the proposed method is that many illumination-estimation algorithms rely on information that can be significantly influenced by a small part of an image. For example, MaxRGB [7] and retinex [11] both can be influenced by a single pixel having a spuriously high R, G or B value. Gamut mapping algorithms such as Forsyth’s [6] can be influenced by a single erroneous pixel that happens to stretch the convex hull of the gamut significantly in the wrong direction. For a single-illuminant scene, the estimates from multiple sub-images should be consistent. Those that are inconsistent can be identified as outliers and eliminated. Sub-images, of course, do not carry as much information as the whole image. Therefore, the performance of an algorithm on each of the parts is likely to be worse than its performance on the full image; that is, assuming the full image contains no outliers. However, it is often the case that even a fairly small sub-image contains enough information for the underlying illuminationestimation algorithm to work reasonably well. As an example, consider the images in Figure 1. It is likely that quite a few of the vertical or horizontal slices will cover a sufficient proportion of the complete set of image colors to make gamut mapping algorithms work. Similarly, there is a good chance that they contain the necessary high R, G or B digital counts that MaxRGB requires, or are sufficiently textured for Edge-based Color Constancy [15] to succeed. Illumination-estimation algorithms generally assume there is a single illuminant lighting the imaged scene, or at least that even if there is more than one illuminant then there is only one dominant illuminant. It is expected that white balancing the image relative to the dominant illuminant will suffice. In terms of the second type of outlier—those related to multiple illuminants—the information from a sub-image is likely to be more reliable, not less, than that from the image as a whole, because by being smaller the sub-image is more likely to involve only a single illuminant. Proposed Algorithm The proposed algorithm combines illumination estimates obtained from sub-images using RANSAC as a method of eliminating outliers. The core idea of RANSAC is to determine which observations are inliers and which are outliers and to base the final result only on the inliers. The data is assumed to fit some underlying model defined by some parameters (e.g., a model could be a straight line with the slope and intercept being the parameters). The process works by: (1) randomly selecting some observations; (2) determining the model parameters that best fit those observations; (3) testing all the remaining observations and classifying them as inliers or outliers based on how well they conform to the model; (4) checking to see that a sufficient number of inliers remain, and if not, discarding the model; (5) recomputing the model parameters based on the complete set of inliers. The algorithm repeats the steps (1) to (5) to generate many 20th Color and Imaging Conference Final Program and Proceedings 105 possible models. The model that fits the observations the best is returned as the result. RANSAC is used to deal with the two types of outliers mentioned above. The underlying model of the image data that RANSAC fits is different for each of them. To handle multiple sources of illumination, the model is that the scene contains 3 distinct illuminants. Any additional illumination chromaticities that are observed (i.e., calculated from a sub-image by an illumination-estimation algorithm) will be classed as outliers. Applying the RANSAC steps in this case means obtaining illumination estimates from 3 sub-images of random size and location (the ‘observations’), sorting the remaining estimates as either inlier or outlier, checking that a sufficient number of inliers has been found, and re-computing the estimates for each of the 3 clusters obtained this way based on the final set of inliers. For this last step, the inlier estimates within each cluster are simply averaged. In terms of the outliers of the pixel-noise variety, the scene model is that there is only a single illuminant, and hence all the sub-image estimates should conform to this. Those that do not will be considered to be outliers. Note that this definition of outlier will also deal with secondary illuminants to a certain extent. If a secondary illuminant only lights a small portion of the scene then the estimates from the corresponding sub-windows will be excluded as outliers and therefore not influence the estimate of the dominant illuminant. Related Work Combining estimates from multiple illumination-estimation algorithms applied to whole images has been explored before, but generally not to sub-regions of images. One simple method of combining estimates is to use the arithmetic mean [3, 2]. Other strategies are to use a weighted average with weights determined by a training stage using least squares, or to use a trained neural network or support vector regression [12] to combine the estimates [3]. Shades of Grey [4] and Edge-based Color Constancy [15] combine MaxRGB-type and Greyworld-type clues. In both cases, the results are proportional to the Minkowski or p-norm of the form