Abstract:
The invention concerns a device and a method for colour correction of an input image. According to the invention, the device comprises: means for analysing the colours of an input image in order to get a colours parametric representation of said input image, means for selecting a set of reference images, means for analysing the colours of at least one of said reference images or one region of said reference images in order to get a colours parametric representation of said reference image, means for calculating a set of possible transformations on the colour of said input image such that the colours parametric representation of said input image is close to the colours parametric representation of said reference image according to a distance criterion, means for selecting among the said possible transformations, a subset of transformations which applied to said input image, gives a physically possible colour corrected input image, means for interactively selecting visually one transformation out of said subset. Application to the post-production of films.

Description:
[0001]     The invention concerns a method and a device for correcting colour of images using reference images.  
       BACKGROUND OF THE INVENTION  
       [0002]     In the age of introduction of digital film processing and digital cinema, multiple digital processing steps appear in the—classically analog-cinematographic workflow. While special effect generation and post-production are often already digital, now digital cameras and new digital display devices and digital media appear. This digital equipment changes completely the habits during capture and post-processing. Digital technologies bring new features but represent also a danger for artistic experience and heritage. It is thus necessary to transfer known artistic effects from conventional cinematographic post processing into digital cinema post processing. Since the introduction of digital cinema is long term, also technologies for co-existing analog and digital processes have to be studied. This may concern a film captured by an analog film camera and then display by a digital projector. This concerns also integration of digital special effects in an analog film production chain.  
         [0003]     One of artistic workflows is colour correction. Colour correction is applied to raw film material after production to compensate for illumination colour artifacts. Colour correction is also applied before film distribution to fine-tune the colour tones in order to realize artistic intents for certain scenes. Colour correction is further applied to photographs, paintings or graphics before printing.  
         [0004]     Colour correction can be applied for a sequence of video frames, for a single video frame, for still images or even for parts of an image, e.g. an object.  
         [0005]     Usually colour correction is carried out in cooperation of artistic director and highly skilled operators. The artistic director describes the intent of colour correction while the operator has to transform the intent into a colour transform applied to the visual content. Results of colour correction thus depend on the experts who do it.  
         [0006]     An efficiency problem of colour correction is that colour experts have to re-start work on each new image or video scene.  
         [0007]     An evident solution for this problem is to choose for a first image an appropriate colour correction transform. This transform is stored and reused for other images.  
         [0008]     This solution is not sufficient as the following example shows. Let us assume that the experts have already defined a colour transform for an image of a cloudy day scene such that an output image results that gives impression of a sunny late afternoon scene. For this purpose, the operator manually defined a transform adding notably more red tones. Lets assume further that now an image of a very sunny high noon scene should be transformed into a sunny late afternoon scene. Applying the same transform as for the cloudy day scene gives a result image with certainly too much red in it.  
         [0009]     Another existing solution to automate and simplify colour correction of images is the following. An input image is to be transformed into an output image such that the colours of the output image are close to the colours of a reference image. This could be a daylight image to be transformed into a night image using another night reference image. Therefore R. C. Gonzales et al. propose in 1987 in their book entitled “Digital image processing” to calculate the colour histogram H I ;H R  for the input image I and for the reference image R, respectively. From the histograms H I ;H R  two colour transforms T I   E ;T R   E  are calculated such that the images T I   E I;T R   E R have equalized histograms, i.e. all histogram values are, in ideal case, constant, where T E  indicates a colour transform that equalizes histograms. A final colour transform T 1 =T 1   fixed =T R   E−1 T I   E  is calculated and applied to the input image, where T E−1  indicates the inverse of a colour transform that equalizes histograms. By this method, input images can be automatically colour corrected.  
         [0010]     The presented solution has two disadvantages. First, it is based on the assumption that two images having the same or a close histogram show two scenes with the same or close illumination colour. Second, no fine-tuning method is proposed to correct eventual imperfections of the calculated transform.  
         [0011]     A further existing solution to automize and simplify colour correction of images is the following. In 1990, a publication entitled “A novel algorithm for colour constancy” by D. A. Forsyth in the International Journal of Computer Vision No. 1 Volume 5, proposes to calculate a colour transform T 1  such that an input image I having a convex colour gamut M I  transformed into an output image O=T 1 I with the convex colour gamut M O  such that M O  is included in the convex colour gamut M R  of a reference image or a number of reference images. For this, the method uses diagonal transforms, for example  
         T   1     =     (           c   R         0       0           0         c   G         0           0       0         c   B           )         
 
 for a three-dimensional colour space R,G,B. It is shown how to determine a specific gamut M T  of transforms in the space of transforms having the coordinates C R ,C G ,C B . The gamut M T  is determined such that it includes the totality of mathematical possible transforms for T 1  fulfilling the above explained constraint. The method then chooses either a transform T 1 =T 1   medium  being in the centre of the gamut M T  or a transform T 1 =T 1   max  giving a gamut M O  of maximal volume. A gamut M O  of maximum volume is the gamut that contains as many colours as possible. 
 
         [0012]     This solution has two disadvantages. First, it uses a heuristic method to chose a transform among the possible transforms and second, no fine tuning method is proposed to correct eventual imperfections of the calculated transform.  
       BRIEF SUMMARY OF THE INVENTION  
       [0013]     The present invention proposes to solve at least one of the above mentioned drawbacks.  
         [0014]     According to the present invention, the colour correction applied to the input images is not fixed and independent from the input image and it lets the choice between several possible colour corrections.  
         [0015]     To this end, the present invention concerns a device for colour correction of an input image comprising: 
        means for analysing the colours of an input image in order to get a colour parametric representation of said input image,     means for selecting a set of reference images,     means for analysing the colours of at least one of said reference images or one region of said reference images in order to get a colour parametric representation of said reference image,     means for calculating a set of possible transformations on the colour of said input image such that the colours parametric representation of said input image is close to the colours parametric representation of said reference image according to a distance criterion,     means for selecting among the said possible transformations, a subset of transformations which applied to said input image, gives a physically possible colour corrected input image,     means for interactively selecting visually one transformation out of said subset.        
 
         [0022]     Thus, the invention can enable colour correction according to the input image and to the desired output aspect.  
         [0023]     The selection of a subset can enable to get images showing physically possible scenes or also images which effect is desired by the user.  
         [0024]     According to a preferred embodiment, the means for selecting the reference images select the images having a colour parametric representation containing a wide range of colours representing the possible colours for the desired aspect after colour correction.  
         [0025]     According to a preferred embodiment, the means for selecting the reference images are controlled by a user in order to perform a manual selection of the reference images.  
         [0026]     Indeed, the manual selection is interesting in order to get a more precise set of expected colours. The user can then input the desired colours he would like to get after transformation by selecting reference images containing such colours.  
         [0027]     According to a preferred embodiment, the colour parametric representation is a colour gamut.  
         [0028]     According to a preferred embodiment, the colour parametric representation is a colour histogram.  
         [0029]     According to a preferred embodiment, a criterion of possible transformation is that the gamut of the input image is inside the gamut of the reference images after applying the transformation to the input image.  
         [0030]     Such an embodiment makes sure that the colours of the transformed image are colours contained in the reference images and that the transformed image has the expected aspect as far as colours are concerned.  
         [0031]     According to a preferred embodiment, the means for calculating a set of possible transforms calculate a single transform and calculate a set of possible transforms by calculating a transform gamut around this single transformation.  
         [0032]     According to a preferred embodiment, the means for selecting a subset of possible transformations select said subset as being the transformations on a curve in the space of transformations between the neutral transform and the transform giving the largest gamut of the input image.  
         [0033]     According to a preferred embodiment, the means for interactively selecting visually one transformation out of the subset comprise a graphical user interface.  
         [0034]     The invention is also related to a method for colour correction of an input image comprising the steps of: 
        analysing the colours of an input image in order to get a colour parametric representation of said input image,     selecting a set of reference images,     analysing the colours of at least one of said reference images or one region of said reference images in order to get a colour parametric representation of said reference image,     calculating a set of possible transformations on the colour of said input image such that the colours parametric representation of said input image is close to the colours parametric representation of said reference image according to a distance criterion,     selecting among the said possible transformations, a subset of transformations which applied to said input image, gives a physically possible colour corrected input image,     interactively selecting visually one transformation out of said subset.       
 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0041]     Other characteristics and advantages of the invention will appear through the description of a non-limiting embodiment of the invention, which will be illustrated, with the help of the enclosed drawings:  
         [0042]      FIG. 1  represents a bloc diagram of a preferred embodiment of the invention,  
         [0043]      FIG. 2  represents an enlarged colour gamut,  
         [0044]      FIG. 3  represents a gamut in the transformation space showing the subset of transforms selection,  
         [0045]      FIG. 4  represents another embodiment of the subset of transforms. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0046]     The represented modules on  FIG. 1  are functional units, which can correspond to units physically separated or which can be included in a same physical unit. For instance, these modules or at least some of them can be grouped in a unique component or can constitute functionalities of a same software. In opposition, some modules can eventually be divided into separate units.  
         [0047]     A set of references images, or regions in reference images, is selected by a user. In the following, this is called selected reference images. These reference images are for instance selected by skilled experts such as directors of photography. These reference images are images which colour looks like the desired colour. The set of selected reference images contain a wide range of colours representing the possible colours for the desired expected effect, for instance the colours that could represent a late evening atmosphere, skin colour or indoor colour. The input image to be colour-corrected will have after colour correction only those colours that are closely related to the selected colours.  
         [0048]     The selection of the reference images is mainly done manually by the operator as colour correction is close to artistic creation. In other embodiments, it can be done automatically, the user giving for instance input parameters related to colour, illumination, time span of a film.  
         [0049]     For instance, if the colour correction aims to obtain images having a night atmosphere, the reference images will also have a night atmosphere.  
         [0050]     A colour analysis means  1  receives as input the reference images  10  from the user. It calculates and outputs a colour parametric representation  7  of all reference images  10 . As colour parametric representation, a colour gamut M R  can be chosen. The colour parametric representation describes all colours contained in the reference images  10 .  
         [0051]     In other embodiments of the invention, a colour histogram H R  can also be chosen as colour parametric representation, or a selection of dominant colours or a vector quantization.  
         [0052]     A colour analysis is also carried out on an input image  11  which has to be colour corrected. This is the function of module  2  which receives as input the input image  11 .  
         [0053]     The same analysis is done on the input image  11  as on the reference images  10 . If a colour gamut-based analysis is carried out on reference images  10 , a colour gamut-based analysis is also carried out on the input image  11 .  
         [0054]     The colour parametric representation represents the colours of all the reference images in a space such as XYZ as defined by the International Commission of Illumination (CIE) or CIE xy (chromaticity) spaces. The colour parametric representation can be based on a convex volume in colour space, called gamut, or on other volume representations such as colour histograms for instance. In other embodiments, other colour representations can be used. The term volume is used in a general sense and it corresponds to a three-dimensional volume in the case of a three-dimensional colour space.  
         [0055]     The colour analysis module  2  outputs also a colour gamut  8  corresponding to the input image  11 .  
         [0056]     Based on these gamuts, a means  4  calculates a set  13  of possible colour transformations for the input image  11 .  
         [0057]     The possible transformations  13  are a set of transformations that each make the colour of the input image  11  close to the colour of the reference images  10  according to a distance criterion.  
         [0058]     Each transformation consists of a mathematical function mapping an input colour to a transformed colour. The input colour and the transformed colour can be expressed for example in XYZ or xy coordinates. The transformation can consist for example of a linear mathematical function, a non-linear mathematical function or a look up table.  
         [0059]     An example for a linear transform is the transform T according to  
               (           X   ′               Y   ′               Z   ′           )     =       ⁢     T   ⁡     (         X           Y           Z         )                     with   ⁢           ⁢   T     =       ⁢     (           c   R         0       0           0         c   G         0           0       0         c   B           )               
 
         [0060]     The parameters C R ,C G ,C B  of all possible transforms  13  lie inside a gamut of transforms defined by a polygon P which is generated iteratively by the following steps: 
        1. Initialize P to a large RGB cube such that it encloses all colors,     2. Take the first point (R,G,B) of the polygon of the input image gamut,     3. For the N points (R n ,G n ,B n ) of the polygon of the gamut of the reference images, calculate N transformations: 
 
( C   R,n   =R   n   ′/Rc   G,n   =G   n   ′/Gc   B,n   ′=B   n   ′/B );0≦ n&lt;N.  
    These N transformations build a temporal gamut P tmp .     4. Calculate the intersection between P and P tmp  and define P to be the result of this intersection.     5. Take the next point (R,G,B) of the polygon of the input image gamut and continue with step  2 . If all points of the polygon of the input image gamut have already been processed, calculation of P is terminated.        
 
         [0067]     The possible transformations  13  can be described by a gamut M T  of transformations in the space of transforms. The space of transforms is the space of parameters defining a transform. For example, two transforms that are close in transform space have usually similar transformation characteristics.  
         [0068]     When the colour gamut is used for characterizing the images, a criterion of possible transformation is that the gamut of the input image after applying the transformation is inside the gamut of the reference images up to a set of outliers. Outliers are transformed colours outside the gamut of the reference images. Outliers can be handled in different ways. For example their number can be limited to a certain percentage of the number of all transformed colours. Another possibility is to allow outliers up to a maximum distance from the gamut of the reference images. In other embodiments, other criterions can be used.  
         [0069]     For reference images  10 , showing not balanced scenes (such as close-ups, scenes with objects having all very close colours, scenes with low saturation), it may occur that no colour transform exists that fulfills the constraint which consists in having the gamut of the transformed input image in the gamut of the reference images. In this case, the gamut M R  is enlarged resulting in an enlarged gamut M′ R , as shown on  FIG. 2 .  
         [0070]      FIG. 2  shows an example for the case of a two-dimensional space where gamuts are represented by polygons. A gamut can be represented by other means such as geometric primitives (cubes, spheres, . . . ) or curves (splines, mathematical functions, . . . ) The colour space can be for example defined by the xy chromaticity coordinates of the CIE. Since the xy coordinates are constrained to values between zero and one, the amplitude of enlargement can be calculated such that the volume of enlarged gamut M′ R  is in between the volume of MR and the maximum volume possible without passing the limits of the colour space. Such an operation can be parameterised by a parameter k, where k=0 represents no enlargement and k=1 represents the maximum enlargement. For enlargement, the centre P of gravity of M R  is calculated. Then by putting straight lines c from the centre to points on the hull of the gamut, the points of the hull are moved outside of the gamut forming the hull of the enlarged gamut M′ R .  
         [0071]     In another embodiment, there can be points that move stronger than other points. For example points far away from the limit of the colour space move stronger than other points.  
         [0072]     In a further embodiment, when the colour analysis modules  1  and  2  produce colour histograms, a criterion for a possible transformation is related to the colour histograms.  
         [0073]     In this embodiment 
        a colour transform T I   E  is applied to the input image  11  such that T I   E I has an equalized histogram, i.e. histogram values are either zero or constant,     a colour transform T R   E  is applied to the reference image  10  such that T R   E R has an equalized histogram,     a single transform T 1 =T 1   fixed =(T R   E ) −1 T 1   E  is calculated.        
 
         [0077]     E means a transformation that gives an equalized histogram.  
         [0078]     Once this single transform T 1  is obtained, in order to obtain a set of possible transforms  13 , a cubic, spheric or ellipsoidal transform gamut is created around T 1  in a space of transforms. The size of this gamut M T  of transformations is set to relate to the distance between T 1  and the neutral transform, which is the transform without effect to images. For example, when choosing a circular gamut, the radius is set to the distance between T 1  and the neutral transform. The gamut may be clipped when exceeding certain limits such as negative coefficients. Clipping means that concerned parts of the gamut are eliminated from the gamut. Other embodiments can be forecast.  
         [0079]     The module  4  transmits to a module  5 , the set of possible transforms  13  calculated as mentioned above.  
         [0080]     The module  5  is in charge of selecting a subset of transforms  14  among the set of transforms  13 .  
         [0081]     The set of transforms  13  is described by a gamut M T  of transforms in the preferred embodiment. All transforms included in the gamut M T  transform the input image into an output image that fulfils the mathematic constraints.  
         [0082]     One purpose of selecting a subset of transforms  14  among the possible transforms  13  is to get transforms which produce images showing physically possible scenes.  
         [0083]     Indeed, some of the mathematically transforms in the set of transforms  13  may give too reddish, too greenish, too bluish or too dark output images that show non-realistic scenes.  
         [0084]     This can occur when the scenes in the input image  11  and in the reference images  10  do not cover all colours, contain reflective objects or have fluorescent parts. In this case, the gamut M T  is too large to give a plausible choice for a final transform.  
         [0085]     The module  5  produces a subset of transforms by using the following physical and heuristic criteria: 
        the module  5  of subset selection produces as output a curve C T  included in the gamut M T  as shown on  FIG. 3 .        
 
         [0087]     The curve is a straight line. Such a line can be defined for example between two transformations T start , T end  in the space of transformations inside the gamut M T .  
         [0088]     T start  is chosen to be the neutral transform T o .  
         [0089]     If T o  is not inside the gamut M T , M T  can be enlarged resulting in M′ T  and M T  is replaced by M′ T  as shown on  FIG. 3 .  
         [0090]     T end  is chosen as being the transform giving the largest gamut M o  of the output image transformed according to O=T max I.  
         [0091]     The curve C T  can also be a closed curve in other embodiments and T start  and T end  can be chosen differently. For example, the curve can be defined to be a curve lying on the hull of the gamut M T  starting at T start  being the transform giving the smallest gamut M o  and running to T end  being the transform giving the largest gamut M o .  
         [0092]     In another embodiment of the invention, the subset of transforms  14  produced by subset selection module  5  is a gamut of transforms M″ T  which is smaller than M T  as shown on  FIG. 4 .  
         [0093]     This gamut of transforms M″ T  is generated by calculating the centre of gravity T c  of M T , setting lines through T c  and points on the hull of M T , then moving the points of the hull on the lines in inside direction, finally giving the hull of the smaller gamut M″ T .  
         [0094]     The centre point T c  can be set to the transform T o  that is the neutral transform, defined by T 0 I=I. The neutral transform is an important physical point in the gamut M T  since it corresponds to the case where the input image keeps as it is. If T o  is not included in the gamut, T c  can be set to the centre of gravity.  
         [0095]     The module  5  of subset selection produces the subset of transforms to a module  6  which chooses a single transform  9  out of the set of transforms.  
         [0096]     This step is controlled by an operator The subset selection means  5  produce a reduced number of transforms, thus the selection by the user does not require a long time and does not require special skills from the operator.  
         [0097]     The subset being represented in this embodiment by a curve CT in the space of transforms, the operator can choose a transform by sliding on the curve between its start and its ending point. An instantaneous calculation and display of the transformed input image allows the operators to choose the final transform T∈C T  considering artistic aspects. For this interactive selection, the module  5  is linked to a module  3  which enables the instantaneous display of the transformed image. The module  6  can be a remote control having a graphical user interface (GUI) which enables the user to slide a cursor along the curve and to obtain as output the results of the transform applied to the input image on the module  3 .  
         [0098]     An output image  12  is output by module  3 . This output image  12  corresponds to the input image to which the selected transformation  9  is applied.