# Colorimetric Analysis Using Scene-Adaptive Color Conversion Matrix of Calibrated CIS

Authors: Sung-Hak Lee
Journal: Journal of Sensors
 (2016)

---

## Abstract

TheRGB signals of different CISs (color image sensors) do not register the same values for the same viewing scene owing to their different spectral sensitivity and white balance mechanisms. Thus, CISs must be characterized based on CIE standard observation for colorimetric purposes. One general method for characterizing CISs is least square polynomial modeling to derive the colorimetric transfer matrix betweenRGB outputs and CIE XYZ tristimulus inputs. However, the transfer matrix that is obtained under the standard CIE illumination is unable to estimate various conditions of a CIS that is operated under various illuminations with varying chromaticity and luminance. Therefore, repeated experiments are necessary to obtain accurate colorimetric analysis results. This paper presents a scene-adaptive colorimetric analysis method using images captured by a general consumer camera under various environments.


---

## Body

## 1. Introduction

Generally, theRGB outputs generated by CISs (color image sensors) are device-dependent. This means that theRGB signals do not correspond to device-independent tristimulus based on CIE CMFs (color matching functions) [1]. This is because the spectral sensitivity of CISs is not matched to the standard CMFs, and the characteristics of color sensors that are utilized in consumer cameras usually depend on the surrounding environment when capturing a scene [2]. Therefore, it is necessary to determine the transfer characteristics defining the relationship betweenRGB signals and standard color stimuli for post-color matching and enhancement processes such as color reproduction, color correction, and HDR (high dynamic range) rendering to facilitate accurate scene rendering in general display devices [3–8].The derivation of a transform relationship betweenRGB signals and CIE XYZ stimulus is known as CIS characterization [9], and it can be performed by spectral-sensitivity-based and color-target-based methods. The color-target-based method uses reference colors obtained from their XYZ values on a color chart and thus is relatively simple and practical when compared to the spectral-sensitivity-based method that requires spectral analysis of the camera. The polynomial regression method based on least squares fitting has been widely adopted by many color researchers for calculation of the transfer matrix from the capturedRGB values and their XYZ values [9–13]. However, camera characterization under the specific standard illuminant does not provide good estimations for the different ambient light conditions in which an image is captured. Accordingly, for more accurate color analysis, flexible and rigorous experiments in various white balance conditions should be performed.Generally, most color images from CISs should be calibrated chromatically to remove specific color cast. The changed transfer matrix for any other white balance condition can be obtained based on the surrounding illumination and the phosphor primaries of a camera [12, 13]. However, color measurement performance can be enhanced using a recalculated transfer matrix based on illuminant estimation techniques. Commonly, the illuminant estimation is referred to as white point estimation. It is assumed that the CIS is then calibrated to produce similarRGB values for white under any illuminant. However, it is uncertain which illuminant will be used and how similar theRGB values will be to each other. In theory, color correction with unknown white balance is not problematic, because the calibrating coefficients used to scale the sensor parameters are absorbed into the color transformation required for the color correction [14].In this paper, a novel colorimetric analysis method is proposed using images captured by a consumer digital camera. In general cases, it is hard to estimate the internal calibration state of the camera sensor and surrounding illumination. For this reason, without the illumination estimation process, the color conversion function for variousRGB levels is derived using only two images captured on the AWB (auto white balance) mode and the preset mode. The difference of the two images is used for the internal calibration process that corrects the color balance. It is possible to estimate the color conversion of the image sensor through the color transfer function. In simulation results, performance comparisons are made with the characterized cameras based on the single matrix and the multimatrices. This demonstrates that the proposed camera characterization method performs effectively relative to previous methods.


## 2. Image Calibration

In handling captured images, an important issue concerns the color balance status of the image sensor. In most consumer digital cameras, the sensor color channels are calibrated according to sensor response to the intensity and wavelength of the scene. The AWB techniques find the optimal color balanced condition for varying surroundings, by using various illumination estimation methods such as grey world, white patch based, neural network based, and bootstrapping methods [14]. Moreover, conventional digital cameras adopt a knee curve for dynamic range compression and the nonsaturated highlight range [15].Figure1 shows the difference between the sensor responses of Nikon and Canon cameras. Both cameras are balanced for identical illumination of A and D65 illuminants for the same scene. This allows a comparison between different digital cameras as regards color balance transformation. Sensor responses are plotted in the relativeRGB space and show how theRGB values are calibrated between white balanced and unbalanced images. For some color patches, theRGB values are synthesized with the different sensor characteristics of two cameras. It can be seen that theRGB channel gains are controlled differently for different signal levels and that theRGB responses vary significantly even though light conditions are consistent. These are affected by the camera-specific color constancy algorithms. In other words, the color sensors of two cameras perform differently even for the case of color balanced for the same illumination. The internal calibrating parameters are absorbed into the color transformation and cause nonlinear transformation owing to the gain, gamma, and knee control properties.Figure 1
Comparisons of relativeRGB transform relationships obtained from color balanced images by two different cameras for the same illuminant: (a) under illuminant A, (b) under illuminant D65.
(a) (b)


## 3. Colorimetric Analysis

Color imaging systems using CISs can be characterized by the colorimetric method. The resulting camera characterization generates a transfer matrix for the colorimetric analysis of captured images. The color-target-based method uses reference colors and their correspondingXYZ values to determine an approximated linear transformation between CIE XYZ values and cameraRGB signals. The transfer characterization matrix is derived through polynomial regression based on measurements of known color samples. The color target-based characterization using polynomial regression is generally adopted owing to its simplicity and effectiveness.The transformation processing betweenXYZ tristimulus for N color test targets and corresponding cameraRGB signals is shown in the following:(1)R1R2⋯RNG1G2⋯GNB1B2⋯BN=McX1X2⋯XNY1Y2⋯YNZ1Z2⋯ZN,Mc=R1R2⋯RNG1G2⋯GNB1B2⋯BNX1X2⋯XNY1Y2⋯YNZ1Z2⋯ZNTX1X2⋯XNY1Y2⋯YNZ1Z2⋯ZNX1X2⋯XNY1Y2⋯YNZ1Z2⋯ZNT-1.The camera characterization matrix Mc is determined to minimize the color differences over all test targets. Because 3×NRGB values should correspond to 3×N  XYZ tristimulus through a 3×3 matrix, the 3×3Mc matrix is derived by using the least squares method [16].Nevertheless, this method is accurate only under certain standard illuminants because the characteristics of CISs are dependent on the surrounding illumination. In addition, the internal AWB process controls the variations of chromaticity and intensity for correct imaging. The cameras have different colorimetric characteristics according to white balance conditions. Accordingly, the transfer matrix should be recalculated for these inconsistent cases and thus illumination-adaptive measuring methods have been proposed.A recently proposed method uses the estimated phosphor primaries and reference white points to calculate the tristimulus constant matrix for certain illumination conditions [12]. However, this requires phosphor chromaticity information for theRGB and white reference to be previously measured. An alternative method is color measurement based on multicharacterization and illuminant estimation [13]. First, the representative camera transfer matrix is measured for A, D50, and D65 illuminants. Then, the CCT (correlated color temperature) of an illuminant is estimated to select the correct representative transfer matrix for the illuminant. Finally, the tristimulus weighting factors from ALC (auto luminance control) functions are modified to control signal gains for sustained luminance levels. This enables more precise estimation results through the selective Mc.


## 4. Colorimetric Analysis Based on a Calibrated Color Conversion Matrix

The characteristics and calibration of CISs are dependent on the surrounding illuminants. To obtain a precise camera characteristic that changes under different ambient conditions, it is necessary to know calibrating parameters such as the status of the ALC registers and balanced white points. However, the characteristics of balanced images and the sensor mechanism are unknown for consumer cameras; thus, it is very difficult to find out the condition of the calibrated sensors in an arbitrary ambient situation. The previous analytical methods are applicable only under restricted measuring circumstances, in which it is possible to understand the sensor’s signal processing by interfacing with the internal registers. Thus, in this section, an ambient-independent method is proposed for enhanced colorimetric measurements. It approaches the resulting color transformation based on the characterizations for the uncalibrated and calibrated images in each preset and AWB mode.First,RGB input signals are converted to normalized values that are compensated linearly with the inverse of the camera gamma:(2)RN=R2n-11/γR,GN=G2n-11/γG,BN=B2n-11/γB,where RNGNBN are the gamma compensated and normalized outputs from theRGB inputs, n is the bit-depth (n=8), and γRγGγB are gamma values for eachRGB channel.Then, the camera characterization for the transfer function should be performed under the reference illuminant. The color transfer matrix is calculated for a camera set by the preset mode. Herein, the 5000K illuminant is adopted for the reference, andRGB channel gains are set internally such thatRGB outputs are in a ratio of 1 : 1 : 1 for white and grey samples under that illuminant. Thus, the preset mode means the reference mode for the D50 illuminant. The transfer characterization of a CIS can be conducted using polynomial regression with the least squares method. For the reference illuminant, XYZ tristimulus are estimated from the derived 3×3 matrix (McD50) andRGB signals in the preset mode as per the following:(3)XYZ=McD50-1RNGNBN.However, for the arbitrary illuminant condition, the single characterization does not perform an adequate estimation of the reflected color signal. The transfer matrix should be modified to correspond to the changed surroundings.As shown in Figure1, the calibrated images represent the converted color balances and it is possible to find the weight of primaries and channel gains by using the difference between the uncalibrated and calibrated images. The difference between the uncalibrated image in the preset (or reference) mode and the calibrated image in the specific AWB condition can be used to determine a color conversion for different signal levels. The transfer gains of each R, G, and B for the lower part (Clg) and upper part (Cug) are calculated approximately as per the following:(4)Clgx=R^low_AWBR^low_refforx∈RN∣x<xcG^low_AWBG^low_refforx∈GN∣x<xcB^low_AWBB^low_refforx∈BN∣x<xc,Cugx=R^up_AWBR^up_refforx∈RN∣x≥xcG^up_AWBG^up_refforx∈GN∣x≥xcB^up_AWBB^up_refforx∈BN∣x≥xc,where “^” denotes the average value, subscript “low_AWB” and “low_ref” refer to the lower part in the AWB mode and the reference mode, respectively, xc is the average value of total reference mode inputs, and “up_AWB” and “up_ref” refer to the upper part in each mode. The calculated data are sampled at a rate of 10 : 1 to reduce computing burden.According to the separated signals, the approximation function for transfer gains consists of two parts. The bi-segmental channel transfer function is derived to reflect the nonlinear transformation property as per the following:(5)ftx=Clgxifx<xlcfAWBxxifx≥xlc,fAWBx=y0+ax+bx2,y0ab=1xmaxxmax21xlcxlc21xucxuc2-1ymaxClgxxlcCugxxuc,where fAWB is the estimated AWB output, xlc is the average value of reference values less than xc, xuc is the average value of reference values more than xc, and xmax and ymax are the maximum values in the reference and AWB modes, respectively.The value offAWB has been estimated nonlinearly using a quadratic function for the estimation of calibrated outputs corresponding to the region of reference inputs more than xlc, while a transfer function has been derived linearly for the relatively narrow region of lower values less than xlc. The construction of the channel transfer function using bi-segmental color gains, Clg and Cug, is illustrated in Figure 2.Figure 2
Illustration of the estimation of the channel transfer function using lower and upper color gains,Clg and Cug.Then, the new color transfer matrix of the CIS is reconstructed using a color conversion matrix (MCC) between the uncalibrated and calibrated images as per (6). Finally, the tristimulus of color objects in a scene can be estimated using the generated transfer matrix.(6)X′Y′Z′=MCC×McD50-1RNAGNABNA,MCC=ftRNR000ftGNR000ftBNR,where RNRGNABNA are the calibrated inputs in the AWB mode and RNRGNRBNR are the uncalibrated inputs in the reference mode. X′Y′Z′ are values for general nonstandard illuminant environments.When the illuminant changes, the color conversion matrix controls 3 channel gains according to the image calibration by AWB processing. The proposed colorimetric analysis flow is shown in Figure3(b) in comparison with the multi-Mc method of Figure 3(a). The multi-Mc method requires finding characteristic transfer matrices under various standard illuminations and estimating the illuminant. On the other hand, the adaptive-Mc gives estimated results for illuminants and color information by obtaining the color conversion matrix using images captured in two different modes without the illuminant estimation and the AWB information of the image sensor.Figure 3
Block diagrams of color analysis processes: (a) the multi-Mc method and (b) the scene-adaptive-Mc method.
(a) (b)


## 5. Simulation Results

The proposed algorithm has been tested on images containing Macbeth 24 color samples and various objects under the representative D50, D65, and A illuminants using a Macbeth lighting booth. A schematic diagram of the camera characterization and color estimation is shown in Figure4. The viewing conditions considered for the verification of the method are restricted within the range of indoor environments and three illuminants. A CMOS CIS (TOSHIBA TCM8230MD) module including a control interface was used for the experiments. To evaluate the multi-Mc method compared with the proposed method, first the illuminant was estimated using a single characterization matrix (D50) for higher luminance pixels corresponding to the upper 10% of Y signals. Then, the most close matching of the three illuminants was selected and the corresponding characterization matrix was used for color estimations. For the consideration of the preset mode, the internal parameters of the camera were fixed so that theRGB outputs maintained the white balance state under the 5000 K illuminant.Figure 4
Schematic of the camera characterization and color estimation for the evaluation.Tables1, 2, and 3 show the relative performance of the color and illuminant estimations in color differences between the measured and predicted values and a comparison of the processing steps between multi-Mc and adaptive-Mc. The average errors were computed in CIE Lu′v′ color space. Comparisons of average color differences for color samples and illuminants for the three methods are shown in Tables 1 and 2, respectively. The results show that the adaptive-Mc method performs better than the multi-Mc method by 16% and 65% for the estimation of color samples and illuminants, respectively. The meaningful results are shown in color estimations under illuminant D65 and white estimations for D50, D65, and A illuminants. In comparison with the conventional multicharacterization for typical illuminants through the illuminant estimation process, the proposed method requires only a single characterization, no ALC information, and no Mc selection based on illuminant estimation; however, it does require acquiring two-mode images.Table 1
Comparisons of estimation performance among single-Mc, multi-Mc, and adaptive-Mc in color differences.
Test patches Color difference,Δu′v′ Single-Mc (D50) Multi-Mc Adaptive-Mc D50 D65 A D50 D65 A D50 D65 A Red 0.0183 0.0086 0.0354 0.0183 0.0111 0.0126 0.0063 0.0124 0.0229 Green 0.0061 0.0046 0.0208 0.0061 0.0027 0.0019 0.0031 0.0073 0.0024 Blue 0.0078 0.0570 0.0450 0.0078 0.0376 0.0086 0.0268 0.0110 0.0174 Cyan 0.0105 0.0182 0.0174 0.0105 0.0113 0.0212 0.0179 0.0105 0.0379 Magenta 0.0092 0.0226 0.0456 0.0092 0.0151 0.0321 0.0142 0.0044 0.0277 Yellow 0.0101 0.0160 0.0184 0.0101 0.0202 0.0198 0.0041 0.0090 0.0045 White 0.0117 0.0174 0.0273 0.0117 0.0148 0.0210 0.0098 0.0067 0.0003  Average 0.0105 0.0206 0.0300 0.0105 0.0161 0.0167 0.0117 0.0088 0.0162 Model average 0.0204 0.0145 0.0122Table 2
Comparisons of illuminant estimation among single-Mc, multi-Mc, and adaptive-Mc in color differences.
CCT of illuminant Color difference,Δu′v′ Single-Mc (D50) Multi-Mc Adaptive-Mc D50 0.0117 0.0117 0.0098 D65 0.0174 0.0148 0.0067 A (2800 K) 0.0273 0.0210 0.0003  Average 0.0188 0.0158 0.0056Table 3
Comparisons between multi-Mc and adaptive-Mc in terms of processing steps and results.
Multi-Mc based on illuminant estimation Adaptive-Mc based on color conversion estimation Processing steps (1) Set the CIS : AWB off, manual RGB gain (1 : 1 : 1), gamma correction off.(2) Characterizing a CIS for 3 standard illuminants (A, D50, D65).(3) Illuminant estimation (white point).(4) Selecting the transfer matrix.(5) Reading ALC registers.(6) Color estimation. (1) Characterizing CIS for only a standard illuminant (D50).(2) Capturing with AWB on.(3) Capturing in preset mode (D50).(4) Calculation of color conversion matrix.(5) Color estimation. Δ u ′ v ′ Color samples 0.0142 0.0133 Illuminants (white samples) 0.0158 0.0056In Table3, the multi-Mc method requires finding the characteristic transfer matrices for various standard illuminations. Additionally, the illuminant estimation for Mc selection using pixel samples of higher levels is inaccurate. However, the adaptive-Mc demonstrates satisfactory results for illuminants and color information by obtaining the color conversion matrix using images captured in two different modes without any other interfacing with the CIS.


## 6. Conclusions

In color imaging systems, colorimetric measuring is necessary to correctly reproduce color images. This is generally performed by techniques such as chromatic adaptation transformation that requires absolute luminance and chrominance information.It is also indispensable to estimate illuminants and object colors for the correction of degraded effects by tonal mapping and color distortion in rendering high dynamic range scenes on low dynamic range displays. However, it is difficult to obtain accurate colorimetric measurement because the AE and AWB of cameras operate according to the exposed environment. The status of sensor calibration is unknown because it differs in many imaging systems. In general, color conversion techniques have been developed to convert between different illuminants based strictly on the physical quantities, and thus the measured results depend on the chosen illuminant.In this paper, to solve this problem, the transfer matrix according to white-balance conditions is estimated but does not use the internal CIS register values. The difference between the uncalibrated and calibrated images is solely used to derive the color conversion matrix. This method of color analysis can be consistent with the assumption of known color-processing and illuminants. Experimental results show the proposed method is valid in terms of measuring performance. The prediction of luminance and chromaticity of scenes is applicable to CISs in automatic systems responding to various environments.



---

*Source: 6731572-2016-10-11.xml*