Abstract:
An image coloring/recoloring apparatus can provide natural-looking color images of an object illuminated by a source color. An input color image or a shaded image (grey scale image) which corresponds to the object is input to the coloring/recoloring apparatus. The coloring/recoloring apparatus logarithmically transforms a pixel sub-region corresponding to all or a portion of the object into intermediate color density data, which lies on a straight line in a color density space. This straight line starts at a point in the color density space corresponding to the source color and has a direction corresponding to the object&#39;s absorptive properties. By inputting a desired color and/or a new source color, it is possible to create a new image, which shows the object with the sub-region painted the desired color while being illuminated by the new source color.

Description:
This application is a continuation of application Ser. No. 07/432,841, filed Nov. 7, 1989, now abandoned. 
    
    
     BACKGROUND OF THE INVENTION 
     The present invention relates to a method and an apparatus for coloring or recoloring a desired domain in a visual image without impairing the naturalness of shades. 
     Computer-aided design (CAD) has been introduced into the field of color design, and is now desirable to apply it to the coloring and recoloring of visual images. More specifically, a visual image including the object of design (an automobile, for instance) is inputted into a computer system through a scanner or a color camera. After reproducing it on a color display, the object domain is colored if it was previously uncolored or, is recolored, if it was already colored (for example, to recolor the red body of the automobile into blue). It is not simple to change the color in a natural-looking way because the color of the object, even though it is evenly colored, is shaded by the reflection and/or transmission of light. However, the following technique has already been proposed in the U.S. patent application Ser. No. 07/173,164 filed Mar. 25, 1988 by the same inventor as that of the present invention, as will now be described. Light beams coming from a source and reflected by an object to enter into a camera can be broadly classified into a mirror reflection beam and a diffuse reflection beam. The former, reflected off the surface of the object, changes only in intensity, while its spectral composition remains the same as the source. The latter results from the absorption of the light from the source into the object and its isotropic external emission. The spectral composition of the diffuse reflection beam usually differs from that of the source beam. Accordingly, the three components (R (Red), G (Green), B (Blue)) of each point on the object face can be represented by: ##EQU1## where (R o , G o , B o ) are the three components of the diffuse reflection beam, which vary from object to object and, therefore, will be called the &#34;object color&#34;; (R s , G s , B s ) are those of the mirror reflection beam, which are respectively equal to those of the light at its source and, therefore, will be here after called the &#34;source color&#34;; α and β are matrices, which represent the contributions of the diffuse reflection beam and the mirror reflection beam, respectively, to the luminous energy emitted, and are affected by the material and orientation of the reflecting surface. Since they differ from pixel to pixel, the three components of a given pixel whose position in the image is (x, y), can be represented by: ##EQU2## If the source colors and the object colors are known, α(x, y) and β(x, y) can be calculated from (R.sub.(x, y), G.sub.(x, y), B.sub.(x, y)). 
     To figure out how an object of the same material in a different color will look, the pixel values are calculated by Equation (3) in which only the color of the object is altered to (R o  &#39;, G o  &#39;, B o  &#39;). ##EQU3## 
     This model, however, presupposes that the object color and the source color are independent of each other in the RGB space. As a consequence, if an originally uncolored (i.e., achromatic) region is to be colored, α(x, y) and β(x, y) cannot be figured out because the directions of the object color and the source color are parallel in the chromatic space. Hence, uncolored object cannot be colored by this conventional manner. 
     Moreover, since this is a physical model constructed for use with an object made of plastic or the like, its application to a color transmitted by a transparent object or one reflected by cloth or the like, even if the image is similarly shaded, may not necessarily provide a natural-looking image. 
     SUMMARY OF THE INVENTION 
     An object of the present invention is, therefore, to provide a method and an apparatus to enable uncolored regions to be colored in a natural-looking manner. 
     Another object of the invention is to provide a recoloring method and apparatus, which enables the object color to be altered in a natural-looking manner even where Equation (1) does not hold. 
     A visual image coloring or recoloring method according to the invention is based on the assumption that the three primary color values of pixels in a given colored region on an object contained in a color image, logarithmically transformed, lie on a first straight line in the direction of a first absorption vector passing a first source color in a three-dimensional chromatic space. The method includes the steps of: transforming the density of each pixel in an uncolored shaded region of the original visual image into intermediate coordinate values on the first straight line; giving either a desired color for the object or a second source color together with the desired color for the object; setting a second straight line in the direction of a second absorption vector passing the first source color or the second source color and calculating the altered three primary color values of the pixels from said intermediate coordinate values to thereby alter the color of said uncolored region to the desired color. 
     A visual image coloring or recoloring apparatus using this method according to the invention comprises: color image memory means for storing a color image having an uncolored region, source color memory means for storing the source color of said color image; intermediate image calculating means for logarithmically transforming the densities of the pixels in the uncolored region in said color image and transforming the color of each pixel into an intermediate coordinate value; intermediate image memory means for storing said intermediate coordinate value of each pixel; absorption vector calculating means for calculating an absorption vector from a newly given color desired for the object; and altered image calculating means for calculating the altered three primary color values for each pixel either from said absorption vector and said source color or from the newly given source color and said intermediate coordinate values, whereby the apparatus gives the desired color to the object uncolored region in the image. 
     Another visual image recoloring method according to the invention, is also based on the assumption that the three primary color values of pixels in a given colored region, when logarithmically transformed, lie on a first straight line in the direction of a first absorption vector passing a first source color in a three-dimensional chromatic space. The method includes the steps of: transforming the values of said three primary colors into intermediate coordinate values on the first straight line; giving either a desired color for the object or a second source color and the desired color for the object; setting a second straight line in the direction of a second absorption vector passing the first source color or the second source color; and calculating the altered three primary color values of the pixels from the intermediate coordinate values to thereby alter the color of said colored region to the desired color. 
     A visual image recoloring apparatus using this method comprises: color image memory means for storing a color image; source color memory means for storing the source color of said color image; color distribution calculating means for figuring out the distributing direction in a three-dimensional chromatic space by logarithmically transforming the primary color values of pixels in said color image; absorption vector calculating means for calculating an absorption vector from said distributing direction and the color of said color image; intermediate image calculating means for transforming the color of each pixel into an intermediate coordinate value by using said source color and said absorption vector; intermediate image memory means for storing said intermediate coordinate value of each pixel; and altered image calculating means for calculating the altered three primary color values for each pixel either from a second absorption vector set on the basis of a newly given color for the color image and said source color or from the newly given source color and said intermediate coordinate values, whereby the apparatus alters the color of the object colored region in the color image to the desired color. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIG. 1 is an illustrative diagram for use in describing the effect of the present invention with respect to a transparent body; 
     FIG. 2, an illustrative diagram for use in describing the effect of the present invention with respect to a reflective body;. 
     FIG. 3, a diagram illustrating a sectionally constant spectral characteristic which the effect of the invention presupposes; 
     FIG. 4, a diagram illustrating a color distribution which the invention presupposes; 
     FIG. 5, a block diagram illustrating a first embodiment of the invention; 
     FIG. 6a and 6b illustrate its corresponding color image memory and a masked image memory, respectively as used in the first embodiment of the invention; and 
     FIG. 7, a block diagram of a second embodiment of the invention. 
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     The effect of the present invention will be described below with reference to FIGS. 1 and 2. For instance, the hatches of light passing a lampshade made of colored glass can be considered to be generated, with reference to FIG. 1, by the repeated reflection of the light from a bulb 30 within a body of colored glass 31. The shades of the color of deep-pile cloth, such as a carpet, presumably result, as in FIG. 2, from the repeated reflection of light from a source 32 by pile pieces 33 before it reaches the eyes. In these two cases, the reflected color can be approximated by modeling the extent of the absorption of energy from a light source by a medium having a certain absorptive characteristic. 
     Representing the spectral energy distribution of light from a source by S(λ) and the transmittance through a unit length of the object matter, or the reflectance after a unit number of repetitions by ρ(λ), the spectral energy distribution S&#39;(λ) of the light having been subjected to the absorptive effect of transmission or reflection is represented by Equation (4) below: 
     
         S&#39;(λ)=S(λ)ρl (λ)                  (4) 
    
     where l is the length of the section of the optical path passing the medium in the case of transmission, or the relative number of reflections in the case of reflection. 
     The source color represented by (R s , G s , B s ) before transmission or reflection has the relationship of: ##EQU4## where r(λ), g(λ) and b(λ) are the sensitivities of sensors respectively corresponding to red, green and blue. The color after transmission or reflection, represented by (R, G, B), has the relationship of: ##EQU5## In the forms of Equations (5) and (6), (R, G, B) cannot be figured out from the relationship between the source color and the optical path length or the number of repeated reflections unless the spectral characteristic ρ(λ) of the medium is known. 
     If it is supposed, however, that every spectral characteristic smoothly varies with respect to the wavelength, the spectral characteristic to be identified can be approximated by one that is sectionally constant, such as shown in FIG. 3. Thus, the intensity of the light light is at the source is approximated as R s  at λ&gt;λ 2 , G s  at λ 1  &lt;λ 2  or B s  at λ&lt;λ 1 , and the transmittance or reflectance of the object is approximated as ρ r  at λ&gt;λ 2 , ρ g  at λ 1  &lt;λ&lt;λ 2  or ρ b  at λ&lt;λ 1 . In this case the color of the object, relative to l, can be figured out from: 
     
         R=ρ.sub.r.sup.l ·R.sub.s, G=.sub.g.sup.l ·G.sub.s, B=ρ.sub.b.sup.l ·B.sub.s                     (7) 
    
     Taking the logarithms of these equations for representation in the form of density will give: ##EQU6## If, for instance, the values of R, G and B are between 0 and 255, these equations will become: ##EQU7## Here, 
     
         P.sub.i .tbd.-logρ.sub.i, D.sub.is .tbd.-log(I.sub.s /255) (i=r,g,b; I=R,G,B) 
    
     so that variations in color with changes in l will take place on a straight line 34 passing a source color density vector C s  .tbd.(D rs , D gs , D bs ) in a (D r , D g , D b ) density space as shown in FIG. 4. The color density vector C.sub.(x,y) .tbd.(D r  (x,y), D g  (x,y), D b  (x,y)) of each point in a given chromatic region can be represented by: 
     
         C(x,y)=l(x,y)P+C.sub.s                                     (9) 
    
     where the absorption vector P is defined to be P.tbd.(Pr, Pg, Pb). 
     Now, in an uncolored region, the following relationships hold with respect to the source color density vector C s  and the absorption vector P: 
     
         D.sub.rs =D.sub.gs =D.sub.bs P.sub.r =P.sub.g =P.sub.b     (10) 
    
     Therefore, the density can be considered with respect to any one of the three primary color components for each pixel value. 
     On the basis of the foregoing theory, an uncolored region can be colored in the following manner. 
     (1) Determine the source color. Represent the density value D i  (i=r, g, b) of the brightest part in the region by D is , for instance. 
     (2) Designate an appropriate point (here l is assumed to be 1), and regard the density D i  of its brightness as the reference density D io . 
     (3) Give a color so as to alter the part of the reference density to a new reference level (R o  &#39;, G o  &#39;, B o  &#39;). The density vector of this color is represented by C o  &#39;.tbd.(D&#39;ro, D&#39;go, D&#39;bo). Then, as the reference density D io  is first obtained from l=1, the absorption vector: 
     
         P.sub.i =D.sub.io -D.sub.is (i=r, g, b)                    (11) 
    
     can be obtained from Equation (9). Similarly, with respect to each point in the uncolored region, the absorption quantity l(x, y) can be calculated from its density D i  (x, y) by the following equation, ##EQU8## 
     Further, since the point of l=1 is altered to C o  &#39;, the absorption vector P&#39;.tbd.(P r  &#39;, P g  &#39;, P b  &#39;) for the new color can be obtained from 
     
         P&#39;=C.sub.o &#39;-C.sub.s                                       (13) 
    
     As the l(x, y) for each pixel is already determined, the color of the colored pixels can be calculated by Equation (14) in the same way as Equation (9), 
     
         C&#39;(x,y)=l(x,y) P&#39;+C.sub.s                                  (14) 
    
     In, the case of recoloring an already colored region the color distribution spreads from a linear line because of the presence of noise contents in the visual image. However, the direction of the absorption vector P can be calculated from the pattern of color distribution in the density space of a given chromatic region, according to the directional vector v of a first main axis. The directional vector v is obtained as follows, 
     The covariance matrix Σ of the density distribution of points in a density space in a given chromatic region takes the following symmetric form. ##EQU9## wherein the components A ij  of the matrix A are: 
     
         A.sub.11 =Σ(D.sub.r (x, y)-D.sub.r).sup.2 
    
     
         A.sub.12 =Σ(D.sub.g (x, y)-D.sub.g) (D.sub.r (x, y)-D.sub.r) 
    
     
         A.sub.13 =Σ(D.sub.r (x, y)-D.sub.r)(D.sub.b (x, y)-D.sub.b) 
    
     
         A.sub.21 =A.sub.12 
    
     
         A.sub.22 =Σ(D.sub.g (x, y)-D.sub.g).sup.2 
    
     
         A.sub.23 =Σ(D.sub.g (x, y)-D.sub.g) (D.sub.b (x, y)-D.sub.b) 
    
     
         A.sub.31 =A.sub.13 
    
     
         A.sub.32 =A.sub.23 
    
     
         A.sub.33 =Σ(D.sub.b (x, y)-D.sub.b).sup.2 
    
     These can be diagonalized by using an orthogonal matrix O and can be rewritten into: ##EQU10## The proper vectors (v 1 , v 2 , v 3 ) constitute O. The proper v 1 , which corresponds to the greatest proper value δ 1   2 , is considered the directional vector v of the first main axis. 
     On the basis of the foregoing theory, a given chromatic region is recolored in the following manner. 
     (1) Determine the source color (R s , G s , B s ), from which the density vector C s  of the source color is obtained. 
     (2) Designate a point (l is assumed to be 1 here), represent its color by (R o , G o , B o ) (hereinafter called the reference color) and calculate the reference color density (D ro , D go , D bo ) in the same way as Equation (8). 
     (3) Give a color so as to alter the reference color to a desired new reference color (R o  &#39;, G o  &#39;, B o  &#39;). 
     Then, representing the density vector of the reference color D&#39; ro , D&#39; go , D&#39; bo ) at l=1 by C o2  and D&#39; projecting onto the nearest point on the straight line of Equation (9), the absorption vector P is obtained i.e., 
     
         P={(C.sub.o -C.sub.s)·v}v,                        (17) 
    
     where {.} represents the inner product of the vector. Similarly, l(x, y) for every point in a given chromatic region can be obtained from its color density vector C(x, y). ##EQU11## 
     Further as the point of l=1 is altered to (R o  &#39;, G o  &#39;, Bo&#39;) with the new reference color being represented by C o  &#39;.tbd.(D ro  &#39;, D go  &#39;, D bo  &#39;) the absorption vector P&#39; .tbd.(P r  &#39;, P g  &#39;, P b  &#39;) for the new color can be calculated by Equation (13), and the color of the pixels in the new color, by Equation (14). 
     Now will be described a first preferred embodiment of the present invention with reference to FIG. 5. A color image memory 1 stores a full color image, including an uncolored object to be colored, in a tone scale of eight bits each for R, G and B. This content is displayed as a color image on a CRT 3 via a digital-to-analog (D/A) converter 2. A masked image memory 4 stores the mask of the region to be colored in the image at a rate of one bit per pixel. For instance, referring to FIG. 6(b), the color image memory 1 stores a scene inside a room including a lamp 1L with an uncolored transparent glass shade. The masked image memory 1 has a masked image 4L in which only the glass shade part, which is to be colored, has the value of &#34;1&#34; and all the rest has the value of &#34;0&#34;, as shown in FIG. 6(a). The masked image 4L is supposed to have already been prepared by a technique well known to those skilled in the art. For instance, by drawing the contour with a pointing device 5 and shading the part surrounded by the contour while observing the color image on the CRT 3. 
     To color the object region, the user picks out the parts of a source color and of a reference color from said color image with the pointing device 5, for instance, and color designating section 6 transforms these pieces of information into a source density D is  and a reference density D io , which are stored in source color memory 7 and reference color memory 8, respectively. 
     Intermediate image calculating section 9 scans the pixels in the masked image memory 4, and figures out the density D i  (x, y) of each pixel having the value of &#34;1&#34; from the pixel values stored in the color image memory 1. Then, the calculating section 9 calculates l(x, y) in accordance with Equation (12) and writes the result into an intermediate image memory 10 as image data. Upon completion of an intermediate image, the preparations are all set for coloring. 
     In an actual coloring process, first the user picks out a color in the image with the pointing device 5 or gives the three primary color values (R o  &#39;, G o  &#39;, B o  &#39;) of the reference color to the color designating section 6 through a keyboard 11. Then, the reference color density vector C o  &#39; is stored into reference color memory 12. Further, absorption vector calculating section 13 calculates the absorption vector P&#39; from this density vector by Equation (13), and stores it into absorption vector memory 14. Altered image calculating section 15 scans the pixels in the masked image memory 4 so as to figure out, for pixels whose values are &#34;1&#34;, C&#39; (x y) by Equation (14) from l(x, y) available from the intermediate image memory 10, P&#39; available from the absorption vector memory 14 and C s  available from the source color memory 7. Then, the calculating section 15 retransforms the density values thereby obtained into three primary color values, and writes the pixel value (R&#39;(x, y), G&#39;(x, y), B&#39;(x, y)) into the color image memory 1 again. The equations of transformation from density values into three primary color values are Equations (19) below, which are reverse to Equations (8). 
     
         R=255 10.sup.-D r, G=255 10.sup.-D g, B=255 10.sup.-D b    (19) 
    
     Upon completion of the foregoing for every pixel, the color of the pixels having the value of &#34;1&#34; in the mask image within the image stored in the color image memory 1 will have been replaced with the desired color with natural shades. 
     The foregoing referred to a case in which the uncolored part of the object was to be colored without altering the source color. However, if the source is to be altered in color or given a new color, recoloring can also be achieved by designating a new source color C s  with the color designating section 6 and the pointing device 5 or the keyboard 11, then rewriting the content of the source color memory 7 and finally executing Equation (14) with the altered image calculating section 15 in the same manner as described above. 
     By using the method and apparatus described above, an uncolored region, which should be essentially uniform in color but have shades differing with the degree of transmission or reflection, can be given a designated color without looking unnatural. The present invention is most suitable for objects whose shades of color are varied by continual absorption. 
     Next will be described a second embodiment of the invention. Referring to FIG. 7, color distribution calculating section 105 scans the pixels in the masked image memory 4 and the color image memory 1. The color distribution calculating section 105 first calculates, densities (D r , D g , D b ) from (R, G, B) stored in the color image memory 1 for pixels having the value of &#34;1&#34; in the masked image memory 4, and then prepares a covariance matrix of color distribution in accordance with Equation (15). Then, the proper vectors are obtained by diagonalizing the matrix, the directional vector of the first main axis being represented by v. The color distribution calculating section 105 achieves the foregoing calculations with a programmed microcomputer or the like. The vector v thereby obtained is stored into main axis vector memory 104. 
     Color designating section 106, as instructed by the user, picks out a source color and a reference color from an image by using the CRT 3 and the pointing device 5. The color designating section 106 then transforms the source color and reference color into densities, and stores the source color density vector C s  into source color memory 107 and the reference color density vector C o  into reference color memory 108. Absorption vector calculating section 113, with reference to the main axis vector v, transforms C o  into the absorption vector P in accordance with Equation (17) and stores it into first absorption vector memory 114a. The absorption vector calculating section 113 can be realized with a microcomputer or the like. 
     Intermediate image calculating section 109 scans the pixels in the masked image memory 4. The image calculating section 109 first figures out, for pixels having the value of &#34;1&#34; in it, the density vector C(x, y) from pixel values (R(x, y), G(x, y), B(x, y)) stored in the color image memory 1, calculates l(x, y) in accordance with Equation (18), and writes the result into the intermediate image memory 10 as image data. Thus, upon completion of an intermediate image, the preparations are all set for recoloring. 
     In an actual coloring process, first the user gives the RGB values (R o  &#39;, G o  &#39;, B o  &#39;) of a new reference color to replace the current reference color of (R o , G o , B o ) by, for instance, using the pointing device 5 or through the keyboard 11 to the color designating section 106. The density vector C o  &#39; of the new reference color is stored into reference color memory 108. Further, the absorption vector calculating section 113 calculates a second absorption vector P&#39; from this density vector by Equation (13), and stores it into a second absorption vector memory 114b. Altered image calculating section 115 scans the pixels in the masked image memory 4. The calculating means 115 figures out, again for pixels whose values are &#34;1&#34;, C&#39; (x, y) by Equation (14) from l(x, y) available from the intermediate image memory 10, P&#39; available from the second absorption vector memory 114b and C s  available from the source color memory 107. The calculating section 115 writes the pixel value (R&#39;(x, y), G(x, y), B&#39;(x, y)), resulting from the retransformation of the density values thereby obtained into three primary color values, into the color image memory 1 again. Upon completion of the foregoing for every pixel, the color of the pixels having the value of &#34;1&#34; in the mask image within the image stored in the color image memory 1 is replaced with the desired color with no unnatural shades. 
     The foregoing referred to a case in which the color of the object was to be altered without changing the source color. However, if the source color is to be altered, too, this can be achieved as readily by designating a new source color C s  with the color designating section 106 and the pointing device 5 or the keyboard 11, rewriting the content of the source color memory 107 and then executing Equation (14) with the altered image calculating section 115 in the same manner as described above. 
     By using the method and apparatus described above, the color of a region, which should be essentially uniform in color but have shades differing with the degree of transmission or reflection, contained in a color image can be altered to a designated color without looking unnatural. As the present invention enables the color of a design to be changed and displayed in a natural looking manner without altering the surrounding situation in the image, it affords the significant industrial and economic advantage of making it possible to check color coordination without having to actually make a product in the contemplated color.