数理解析研究所講究録989 巻 1997 年 122-131

122

Computer Simulation for Improvement of Color Reproduction
in Electronic Endoscopes
Yoichi Miyake and Hideaki Haneishi

Abstract
Image quality of the electronic endoscopes is poor compared with that from conventional
optical endoscopes that use color film. Our goal is to improve the image quality, particularlycolor reproduction of electronic endoscopes. To achieve our goal, we have developed theendoscopic spectrophotometer to measure the spectral reflectance of gastric mucous membrane.
Three hundred and ten spectra have been analyzed by principal component analysis. The results
can be adequately described using only three principal
indicate that the reflectancecomponents. Based on the above experimental result, it is shown that the spectral reflectance ofgastric mucous membranes can be calculated from thesignals of the conventional
electronic endoscopes. Therefore, spectral reflectance of all pixels in gastric mucous membranetaken by electronic endoscopes can be estimated within less than the average color differencein thecolor space. Computer simulations of the color reproduction of$\mathrm{s}\mathrm{p}\overline{\mathrm{e}}\mathrm{C}\mathrm{t}\mathrm{r}\mathrm{a}$$\mathrm{R},$$\Delta E_{\iota l\mathcal{V}}^{*}=2.66$$\mathrm{G},$$\mathrm{B}$$\mathrm{L}^{*}\mathrm{u}^{*}\mathrm{V}^{*}$

electronic endoscopes were performed, and the resulting spectral characteristics under various
illuminants are described and analyzed.

1. Introduction
In recent years, electronic endoscopes with CCD area sensor have been developed andwidely used instead of the conventional optical endoscopes that use color film. Electronicendoscopes can be easily applied to telemedicine, image filing, image processing and Picture
Archiving and Communication Systems(PACS).
Color information is important for the early stage of diagnosis in various kinds of stomach
diseases. However, color reproduction- of the images taken
electronic endoscopes is poor
compared with the optical endoscope images because the present electronic endoscopes use the
NTSC system.
To improve the color reproduction of electronic endoscopes, we have developed an$.\mathrm{b}\mathrm{y}$123endoscopic spectrophotometer to measure the spectral reflectance of gastric mucous membrane.1
We have measured about 1000 spectral reflectance using this spectrophotometer and reported
the capability of automatic diagnosis using spectral data and chromaticity.23 We have also
reported a color correction method in the electronic endoscope’s input system, using a simple
matrix to reproduce a colorimetrically correct color based on the analysis of the spectral$3\mathrm{x}3$reflectance.4
These color correction methods are based on the analysis of pixel to pixel, because theendoscopic spectrophotometer can only measure the spectral reflectance of gastric mucous
membrane atpoints and cannot measure the spectra in entire viewing fields. To
sampled points, normally, a multi-band method has
measure the spectral reflectance ofbeen used, but the method is difficult to apply to the electronic endoscope.
In this study, we analyzed 310 spectral reflectance of gastric mucous membranes byprincipal component analysis, and it became clear that those measured reflectance spectra canbe adequately described using only thuee principal components. Therefore, the spectralreflectance of all pixels in gastric mucous membranes can be estimated bysignals taken
by electronic endoscopes. On the basis of the experimental results, the computer simulation of
the color reproduction of electronic endoscopes was performed.
$\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\dot{\mathrm{c}}$$\mathrm{M}\mathrm{x}\mathrm{N}$$\mathrm{R},$$\mathrm{G},$$\mathrm{B}$2. Principal Component Analysis of the Spectral Reflectance of Gastric Mucous
Membrane
Figure 1 shows the block diagram of developed endoscopic spectrophotometer. The instrument
consist ofsource with xenon lamp; a conventional spectrophotometer; an opticalmultichannel analyzer with 1024 silicon photodiodes; and a personal computer. We measured
about thousand reflectance spectra of gastric mucous membrane. Figure 2 (a) and (b) are$1\mathrm{i}_{\mathrm{o}}\sigma \mathrm{h}\mathrm{t}$examples of spectral reflectance of gastric mucous membrane. Thuee hundred and ten spectralreflectance of gastric mucous membranes excluding blurred reflectance spectra, have been
analyzed by the principal component analysis. Figure 3 shows the cumulative contribution ratioof the pnincipal components. The result shows that the contribution ratio from first to third
components is about 99%; in other words, the reflectance spectra of gastric mucous membranes
can be represented approximately 99% of the time using a linear combination of three principal
and . Figure 4 shows these three principal components or eigenvectors
components$\mathrm{u}_{1},$$\mathrm{u}_{2}$$\mathrm{u}_{3}$obtained from the covariance matrix of the reflectance spectra.
sequencing,
Figure 5 shows the plan of an electronic endoscopes that utilizes
filters used in the simulation. The
incorporating a rotating color wheel comprising of
are
value
is the spectral radiance distnibution of light source,
filters,
is spectral sensitivity of CCD
respectively, spectral transmittance of the$\mathrm{R},$$\mathrm{R},$$\mathrm{G},$$f_{R}(\lambda),$$\mathrm{G},$$\mathrm{B}$$\mathrm{B}$$\mathrm{B}$$E(\lambda)$$\mathrm{R},$$\mathrm{G},$$S(\lambda)$$f_{G}(\lambda),$$f_{B}(\lambda)$

124$..\vee\wedge\backslash _{\mathrm{o}_{\underline{\mathrm{O}}}}^{\mathrm{o}}\vee\varpi$$.\cdot\vee 0\underline{\mathrm{o}\subset}\supset$$\frac{=}{\mathrm{o}\mathrm{c},0}$$. \frac{\sim\varpi\geq\omega}{0=\in\supset}$$\mathrm{P}\mathrm{r}\mathrm{l}\mathrm{n}\mathrm{C}\mathfrak{l}\mathrm{P}\mathrm{a}\mathrm{I}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{p}\circ \mathrm{r}\iota 6l\iota 1$

Fi.gure 3. Cumulative contribution ratio of theprincipal components.

Figure 1. Diagram of the endoscopic spectrophotometer.

$\angle_{\mathrm{k}\cup\cup}$

OUU$3\mathrm{U}\mathrm{U}$wavelength(a).

$(\mathrm{n}\mathrm{m})$$/\mathrm{U}\mathrm{U}$wavelcngm$(\mathrm{n}\mathrm{m})$(b)Figure 2. Examples of reflectance spectra of gastoric mucous membranes.

125
area sensor,
lens, andand$\mathrm{B}$is combined spectral transmittance of the light guide fiber and the imaging
is the spectral reflectance of gastric mucous membranes. Then the output$L(\lambda)$$o(\lambda)$$\mathrm{R},$$\mathrm{G}$values of the electronic endoscopes are calculated by Eq. (1).
$v_{R}= \int f_{R}(\lambda)E(\lambda)L(\lambda)s(\lambda)_{o()d\lambda}\lambda$(1)$v_{G}= \int f_{G}(\lambda)E(\lambda)L(\lambda)S(\lambda)_{o()d\lambda}\lambda$$v_{B}= \int f_{B}(\lambda)E(\lambda)L(\lambda)s(\lambda)_{o()d\lambda}\lambda$

Although the spectral characteristics of the systems are inherently continuous, for theconvenience the discrete vector notation is introduced as,,,$E(\lambda):\mathrm{E}$$f_{B}(\lambda):\mathrm{f}\mathrm{i}(\mathrm{i}=\mathrm{R},\mathrm{G},\mathrm{B}),$

Then the$\mathrm{R},$$\mathrm{G}$and$\mathrm{B}$$L(\lambda):\mathrm{L},$$S(\lambda):\mathrm{s},$$o(\lambda):\mathrm{O}$$f_{R}(\lambda)$$f_{G}(\lambda)$.

values of an object that are obtained by an endoscope are expressed as$v_{i}=\mathrm{f}_{i}f\mathrm{E}\mathrm{L}\mathrm{S}\mathrm{o}$(2)$=\mathrm{F}_{i}^{t}\mathrm{o}$$(i=R,G,B)$

where$\mathrm{F}_{i}^{t}(i=R,G,B)$

is an overall spectral sensitivities of the electronic endoscopes, and$[\bullet]^{f}$represents transposition.

Because the orthonormal vectors with -dimensional space were obtained by the principal
component analysis, the reflectance spectra of gastric mucous membranes can be calculatedby$\mathrm{n}$

$0$

(3)

$0= \sum_{i=1}\alpha_{i}n\mathrm{u}_{i^{+}}\mathrm{m}$where$\alpha_{i}$is an expansion coefficient and$\mathrm{m}$is the mean vector of 310 reflectance spectra. From

the result of principal component analysis, can be approximated by the linear
the three eigenvectors
andwith the mean vector .
$0$$\mathrm{u}_{1},$$\mathrm{u}_{2}$$\mathrm{u}_{3}$$\mathrm{m}$$0 \equiv\sum_{i=1}^{3}-\alpha_{i}\mathrm{u}i+\mathrm{m}=[\mathrm{u}_{1} \mathrm{u}_{2} \mathrm{u}_{3}]+\mathrm{m}$From Eqs.(2) and (4)$\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{a}\dot{\mathfrak{a}}_{\mathrm{O}}\mathrm{n}$

(4)

of

126

$3\mathrm{U}\mathrm{U}$$4\mathrm{U}\mathrm{U}$wavelength

OUU$/\mathrm{U}\mathrm{U}$$(\mathrm{n}\mathrm{m})$Figure 5. Simulation model of an electronic endoscope.

(a)

(b)

Figure 6. Examples of estimated reflectance spectra.

127$[_{\mathcal{V}_{B}}^{\mathcal{V}}\mathcal{V}_{G}]R=[_{\mathrm{F}_{B}^{t}}^{\mathrm{F}_{R}^{t}}\mathrm{F}_{c]\{[}^{t}\mathrm{u}1$

Then, the constant value$\alpha_{1},$$\alpha_{2}$and$\alpha_{3}$is given by$=$

where,(5)$\mathrm{u}_{3}][_{\alpha}\alpha 2]_{-}\alpha_{3}1+\mathrm{m}$$\mathrm{u}_{2}$$.\cdot-\}$ ,(6)$=\mathrm{m}$ .

Becauseand are known and$v_{i}(i=R,G,B)$ and $v_{im}(i=R,G,B)$
are obtained from the practical electronic endoscopes, the coefficientsandcan be$\mathrm{u}_{i}(i=1,2,3)$$\mathrm{F}_{i}^{t}(i=R,G,B),$$\mathrm{m}$$\alpha_{1},$$\alpha_{2}$$\alpha_{3}$by.
The reflectance spectra of gastnic mucous membrane can be estimated by Eq. (4); therefore it
becomes possible to calculate the reflectance spectra of all pixels
from theoutput signals obtained by a general endoscope system.
We estimated 310 reflectance spectra of gastric mucous membranes by the above method.
Examples ofreflectance spectra are shown in Fig. 6 (a) and (b). Estimated spectralreflectance are approximately equal to the measured spectral reflectance. On the other hand,color differences of(under illuminant D65) between measured and estimatedspectra are shown in Fig. 7. The average color differenceof three hundred and ten
samples was 2.66, the minimum color difference
was 0.64 and maximum color difference
was 9.14. As shown in the next section, we simulated color reproduction of endoscope
images under 12 different illuminants. The color reproduction of 310 measured and estimatedreflectance spectra under those illuminants was also evaluated inspace. The
resultantcolor difference
between measured and estimated ranged from 2.20 to$\mathrm{c}\mathrm{a}1_{\mathrm{C}\mathrm{u}}\mathrm{I}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{d}$$\mathrm{F}\mathrm{q}.(6)$$O(x,y,\lambda)$$\mathrm{R},$$\mathrm{G},$$\mathrm{B}$$\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{e}\dot{\grave{\mathrm{d}}}$$CIEL^{*}\mathcal{U}^{**}\mathcal{V}$$\Delta E_{uv}^{*}$$\Delta E_{u\nu}^{*}$$\Delta E_{u\nu}^{*}$$CIEL^{\mathrm{s}}uv\wedge*\mathrm{C}\mathrm{o}\mathrm{l}\mathrm{o}\mathrm{r}$$\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}_{\mathrm{o}}\sigma \mathrm{e}$$\Delta E_{uv}^{*}$

3.29. These results indicate that the proposed method has sufficient accuracy to estimate thereflectance spectra of gastric mucous membrane and that the slight estimation error does notaffect the simulation of color reproduction of endoscope images.

3. Simulation of Color Reproduction in Electronic Endoscopes
The method was applied to improve color reproduction of practical eight kinds of endoscopic

128
pixels. Namely gastnic membranes with (a) cancer, (b) early stage ofimages withadvanced stage of chronic gastritis, (e) gastricchronic gastritis, (c) chronic gastritis
ulcer, ( normal membrane, (g) polyp and (h) xanthoma were used for computer simulation. In
the simulation, first the spectral reflectance of gastric mucous membrane of all pixels was
signals, then the color reproduction of electronic endoscopes undercalculated fromtwelve illuminants was estinated by computer simulation. The illuminants used in thesimulation are CIE standard illuminant ; D65; CIE fluorescent lamps Fl, F2, and F3, xenon
lamp, metal halide lamp; fluorescent lamps for home use Ml, M2, M3, M4, M5. The density
balance for each iuninant was chosen so that the integral of the spectral radiance of each
illuminant weighted by the spectral sensitivity of human visual system was constant.
The color temperature of the illuminants is shown in Table 1. Figure 8 shows thechromaticities for the images under illuminant CIE standard A and xenon
distribution oflamp. The distribution shifts depending on the illuminants. To estimate the optimum illuminant,those obtained images were displayed on a CRT and evaluated by two physicians who are
specialist of endoscopic diagnosis. Images by xenon lamp was selected as most preferred.
Because the xenon lamp has been used as light source in conventional electronic endoscopes,
we consider that the color memory of doctors may be fixed to the color reproduction by xenon
lamp illumination. However, the doctors said that in the diagnosis of the membrane,
particularly in the vascular pattem of polyp, image by fluorescent M2 is more significant than
the image with other illuminants.
As the physical measure of the illuminant, we assumed that the illuminant with wide color
gamut in reproduced images may be better for electronic endoscopes. Then, we defined theas Eq. (7).
evaluation value$270\mathrm{x}270$$\mathrm{I}\mathrm{I}\mathrm{b},$$(\mathrm{d})$$\mathrm{t}\gamma$$\mathrm{G},$$\mathrm{R},$$\mathrm{B}$$\mathrm{A}$$\mathrm{u}’ \mathrm{v}$’$E_{V}$$E_{V}=\underline{\sum_{i=1j=}^{m-}\sum_{1}1mi+\{(L_{j}^{*}-L_{j})2(^{\wedge*}+\mathcal{U}-u)^{2}+(*\mathcal{V}iji*-\mathcal{V})^{2}j\mathrm{I}*/_{\sim}1_{\circ}}$

(7)
$m2C$

where

$\mathrm{m}$is the number of measured spectra. The equation means that as$E_{V}$

becomes larger,

the color differences between pixels becomes large.
Figure 9 shows the results of the calculated evaluation values. The evaluation values of theiuuminant M3, M4, M2, A and Fl are larger than that of the xenon lamp. Simulation images
with large
enhanced the color difference between pixels. Thus, for example, the vascular$E_{V}$

pattems of the polyp and chronic gastritis became more clear although the color reproduction is

poor.

129

$\iota’.|$$\iota’..r$$\mathrm{u}.\supset$’$\mathrm{u}$(a) C1E A

sample no.

$().()$

Figure 7. Color differences ofbetween measured and estimated reflectancespectra.

$\mathrm{C}\mathrm{I}\mathrm{E}\mathrm{L}^{*}\mathrm{u}*_{\mathrm{V}}*$$\mathrm{v}^{\iota}\mathrm{t}$) $.4$.

$\{)_{\vee}^{\gamma}${ $).|$

0.5$()..1$‘$\mathrm{t}1$(b) xenon lamp

chromaticity of a polyp image
Figure 8. The
under two kinds of illuminants:(a)CIE
xenon
lamp.
$\mathrm{u}’ \mathrm{v}$’$\mathrm{A},$.

Table 1. The color temperatures of the 12
illuminants used{llumlnant

Color temperature

C1E standard A

2856
6500
about 3000
about 4000
about 6500
about 5100
4274
6245
7763
2766
5150
6758

D65
C1E fluorescent Fl
C1E
C1E fluorescent
Xenon
Metal Halogen
FIuorescent lamp Mllamp M2
FIuorescent lamp M3
lamp M4$\mathrm{C}\mathrm{l}\mathrm{E}$Figure 9. Results of the calculated$\mathrm{e}\mathrm{V}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}.\mathrm{n}$valu.es.

standa$r\mathrm{d}$$\mathrm{t}\downarrow \mathrm{u}\mathrm{o}\prime \mathrm{e}S\mathrm{o}\mathrm{e}\mathrm{n}\downarrow \mathrm{F}.2$$\mathrm{F}3|$$-$$\mathrm{F}\mathrm{l}\cup 0\Gamma \mathrm{e}\mathrm{S}\mathrm{C}\mathrm{e}\mathrm{n}\mathrm{t}$$\mathrm{F}\mathrm{l}\mathrm{u}0r\mathrm{e}\epsilon \mathrm{C}\mathrm{e}\mathrm{n}\dagger$$\mathrm{F}\mathrm{l}\mathrm{u}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{c}\mathrm{e}\mathrm{n}\iota$

lamp M5$\mathrm{K}$$\mathrm{K}$$\mathrm{K}$$\mathrm{K}$$\mathrm{K}$$\mathrm{K}$$\mathrm{K}$$\mathrm{K}$$\mathrm{K}$$\mathrm{K}$$\mathrm{K}$$\mathrm{K}$$(\mathrm{b})$130
4. Conclusion and Discussion
In this paper, we showed that the spectral reflectance of gastric mucous membranes can be
cipalcomponents-. On the basis of theaccuracy using only threeestimated
pixels were estimated from theanalysis, the spectral reflectance of 270
signals ofgastric mucous membrane taken byelectronic endoscopes. The color reproduction
characteristics of eight kinds of membranes with different disorders under the 12 different
illuminants were estimated by computer simulation.
images were rated by physicians,
and the color reproduction of the image taken by xenon lamp was most preferred, however, inthe diagnosis of vascular pattem, another illuminant was better than the xenon.
In this paper, we only considered the improVe..ment of color reproduction of electronicendoscopes based on colorimetric color reproduction under various illuminants. The perceived
colors, however, are different in practice from the colorimetric color reproduction, that arecaused by the chromatic adaptation of human vision. The color appearance can be predictedusing chromatic adaptationsuch as the von Kries, Fairchild, RLAB , Hunt, etc..
Therefore, furthermore research considering color appearance models for the optimum color
reproduction should be performed for each of the various diagnosis and viewing condition.
$\mathrm{w}\mathrm{i}\mathrm{t}\acute{\mathrm{h}}\dot{\mathrm{h}}\mathrm{i}\mathrm{g}\dot{\mathrm{h}}$$\vee \mathrm{p}\dot{\mathrm{m}}$$\mathrm{x}270$$\mathrm{R},$$\mathrm{G},$$\mathrm{B}$$\mathrm{p}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{i}\dot{\mathrm{C}}\mathrm{a}\mathrm{l}$se-$\mathrm{T}\mathrm{h}\mathrm{o}$$\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}\mathrm{S}6,78$

Acknowledgment
Author wishes to thank Dr. Kunihiko Nagahara and Dr. Takama Maekawa, Kyoto National
Hospital for their cooperation for image evaluation and their useful comments.
References(*)Presented in part at the J. Imaging Science and Technology, 40(6), 494(1996)
1. Y. Miyake, T. Sekiya, T. Yano, S. Kubo and T. Hara; A New
for
Measuning the Spectral Reflectance of Gastric Mucous Membrane “, J. Pho-togr. Sci, ,37, 13“$\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\iota \mathrm{r}\mathrm{Q}..\mathrm{p}\mathrm{h}_{0}\mathrm{t}\mathrm{o}\mathrm{m}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{r}$(1989).

2. T. Sekiya, T. Hara, H. Yaguchi and Y. Miyake ; Evaluation and Analysis ofReflectance of Gastric Mucous Membranes (I) “, J. SPSTJ, 54 (2), 125 (199i) in Japanese.
3. T. Sekiya, T. Hara, H. Haneishi, H. Yaguchi and Y. Miyake; Evaluation and Analysis ofSpectral Reflectance of Gastric Mucous Membranes (II) “, J. SPSTJ, 54 (3), 346 (1991) in“$\mathrm{s}_{\mathrm{P}^{\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{r}\mathrm{a}}}!$“

Japanese.
4. H. Haneishi, T. Shiobara and Y. Miyake; Color correction for colorimetric color
reproduction in an electronic endoscope “, Optics Communications, 114 (1), 57 (1995).
5. N. Ojima, H. Haneishi and Y. Miyake; The Appearance of Skin with Make-up (III)(Estimation for spectral reflectance of skin with the HDTV color image) “, J. SPSTJ, 57 (2),78 (1994) in Japanese.
“

“

131
6.

.Bems: Image Color Appearance Specification Through Extension of
CIELAB, Color Res. and Appl., 18, 178(1993)
7. Francisco H. Imai, N.Tsumura, H.Haneishi andof
Reproduction for Skin Color Under Different Illuminants Based on the Color Appearance
Models, J.Imaging Science and Technology (in print)
8.
Model of Colour Vision for Predicting Colour Appearance in Various
Viewing Conditions, Color Res. Appl., 12, 297(1987)$\mathrm{M}.\mathrm{D}$.Fairchild,$\mathrm{R}.\mathrm{S}$$\mathrm{Y}.\mathrm{M}\mathrm{i}\mathrm{y}\mathrm{a}\mathrm{k}\mathrm{e}:\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{C}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}$$\mathrm{C}\mathrm{o}1.\mathrm{o}\mathrm{r}$$\mathrm{R}.\mathrm{W}.\mathrm{G}.\mathrm{H}\mathrm{u}\mathrm{n}\mathrm{t}:\mathrm{A}$Y. Miyake
Department ofInformation and Computer Sciences, Chiba University
1-33 Yayoi-cho, Inage-ku, Chiba 263, Japan-rnail: miyake@ics. . chiba-u. $ac$.jp$E$$tj$