Abstract:
Image data is processed by identifying a region of substantially constant color within a first image. Color variations, in terms of standard deviation about an average color, are analyzed and a similar level of variation is applied to a second image. The first image may be derived from cinematographic film, with color variations due to film grain. The second image may be a video image or a computer generated image and the application of a similar level of variation may reproduce noise similar to that present within the film due to grain. In this way, it is possible to add grain to non-filmed images so as to match said images to images derived from film.

Description:
The present invention relates to processing image data in order to facilitate composting and editing. 
     INTRODUCTION 
     The processing of broadcast quality video signals in the digital domain has become increasingly popular and has created a significant number of new production techniques. By using fast digital frame stores and parallel disk drives it is possible to produce full bandwidth digital video signals in real-time, thereby facilitating on-line video editing and compositing. This has led to a demand for such techniques to be used in cinematographic film editing, although the editing of cinematographic film is significantly more difficult given the relatively high information content of film compared to broadcast quality video. 
     In order for images conveyed by film media to be processed using digital techniques, it is necessary to scan the image frames to produce manipulatable digital image signals. After manipulation, these digital signals are supplied to an exposing device, arranged to expose destination film stock, thereby returning the manipulated image back onto a film medium. From the film viewer&#39;s point of view, a manipulated film image should be indistinguishable from an unmanipulated film image. 
     One area in which there is a problem of being able to distinguish between a manipulated film image and an unmanipulated film image, is where a film clip comprises a plurality of film image frames, each made up of a film image and a video or computer generated image. For example, where a video image or computer generated image of a foreground feature is composited with a filmed image of a background feature, there may be a mismatch in the textural qualities between the foreground image and the background image due to the “graininess” of the film image. The video or computer generated portion of the composite image can lack graininess or noise, or have a different noise characteristic to the portion of the composite clip which originated as a filmed image. 
     The graininess of a film is a visual sensation experienced by a viewer of the film, in which the viewer experiences a subjective impression of a random dot-like textural pattern in an image. When an image is projected onto a large area screen, the graininess of a filmed image may be readily apparent. 
     Graininess in a viewed film clip results from the physical composition of the film itself. Motion picture films consist of silver halide crystals dispersed in a gelatin emulsion. The exposure and development of the crystals form the photographic image, which is made up of the discrete particles of silver. In color processes, the silver is removed after development of the film, and dyes form dye clouds centered on sites of developed silver crystals. The crystals can vary in size, shape and sensitivity to light, and are generally randomly distributed within the emulsion. 
     Individual silver particles can range from about 0.002 mm, down to about 0.0002 mm. In a motion picture, the eye cannot distinguish individual grain particles, however the particles resolve into random groupings of dense and less dense areas which, when viewed, result in the visual sensation of graininess. 
     FIG. 1 of the accompanying drawings illustrates clouds of dye formed at sites occupied by exposed silver halide. FIG. 2 of the accompanying drawings shows discrete grains at a magnification of around 400×. Grains nearer the surface of the film are in focus, whilst grains deeper in the emulsion are out of focus. FIG. 3 of the accompanying drawings shows the make up of individual grains of filamentary silver enlarged by an electron microscope. 
     It is known for suppliers of particular film types to produce sets of curves characterizing the graininess of a batch or type of film in terms of the “granularity” of the film. A statistical measure of the granularity of a sample may be supplied by the film manufacturers in the form of an RMS granularity characteristic measured by an electro-optical microdensitometer. 
     However, an RMS granularity characteristic of a batch of film, or film type, may become inaccurate, due to factors such as temperature, the aging of film, exposure levels and other factors. A knowledge of the RMS granularity of the film on which a particular film clip is recorded will often not give an accurate enough basis for matching the granularity of an individual film clip. 
     SUMMARY OF THE INVENTION 
     According to the first aspect of the present invention, there is provided a method of processing image data, comprising steps of identifying a region of substantially constant color within a first image; analyzing color variation within said region; and applying a similar level of variation to a second image. 
     Preferably, a plurality of regions are selected, wherein each of said regions has a different color. In a preferred embodiment, mean or average values are calculated for the intensities. 
     In a preferred embodiment, standard deviation values are calculated for the intensities and these values may be considered as a function with respect to color. Preferably, the step of applying a similar level of variation to a second image involves applying a similar standard deviation to areas within said second image. 
     According to a second aspect of the present invention, there is provided an image data processing apparatus, comprising identification means configured to identify a region of substantially constant color within a first image; analyzing means configured to analyze color variation within said region; and applying means configured to apply a similar level of variation to a second image. 
     In a preferred embodiment the identification means is configured to identify a plurality of regions of differing colors and said analyzing means may be arranged to calculate mean or average values for said color variations. Preferably, the analyzing means is configured to calculate standard deviation values and said deviation values may be calculated with respect to each color. 
     In a preferred embodiment, the analyzing means analyzes standard deviation values of said first image and said applying means is configured to apply similar degrees of standard deviation to said second image. 
     Means may be provided for deriving the first image from cinematographic film. Means may be provided for deriving said second image from video data or, alternatively, a computer may be configured to generate said second image. 
     In a preferred embodiment, the analyzing means is configured to analyze a plurality of regions by traversing linearly across the first image. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIGS. 1 to  3  show examples of granularity in cinematographic film; 
     FIG. 4 shows a general overview of a compositing process; 
     FIG. 5 shows an image processing apparatus adapted for compositing a video image or computer generated image with a film image and applying an appropriate granularity texture to the composite image; 
     FIG. 6 shows a graphics work station comprising the image processing apparatus; 
     FIG. 7 shows an overview of a process for applying a grain texture in a composite image; 
     FIG. 8 shows a background film image; 
     FIG. 9 shows a foreground video image or computer generated image; 
     FIG. 10 shows a composite image of the background and foreground images of FIGS. 8 and 9; 
     FIG. 11 shows the composite image of FIG. 10, having applied thereto a grain texture; 
     FIG. 12 shows a method of scanning the film background image of FIG. 8; 
     FIG. 13 shows a method of sampling and characterizing a granularity of a portion of the filmed image of FIG. 12; 
     FIG. 14 shows schematically a process for characterizing a granularity of a region of a filmed image; 
     FIG. 15 shows a spatial intensity distribution of a region of the filmed image; 
     FIG. 16 shows a gradient of the spatial intensity distribution function of FIG. 15; 
     FIG. 17 shows schematically characterization of granularity of first and second regions of the filmed image of FIG. 8; 
     FIG. 18 shows schematically a process for production of granularity characteristics from a plurality of spatial intensity distributions sampled at various regions on the filmed image of FIG. 8; 
     FIG. 19 shows a granularity characteristic determined in accordance with a specific method according to the present invention; 
     FIG. 20 shows a general process for synthesization of a granularity template signal for a video image or computer generated image; and 
     FIG. 21 shows schematically an apparatus for synthesization of the granularity template signal for a video image or computer generated image. 
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     Referring to FIG. 4 of the accompanying drawings, there is shown an overview of a general preferred process for compositing film image data with video image data or computer generated image data. The filmed image is converted to a first digital image signal corresponding to the film image data, and the first digital image signal is entered into a digital editing suite  400 . A second digital signal corresponding to a video image or computer generated image, generated by a video source or computer source  401 , is input to the editing suite  400 , and the first digital image signal is composited with the second digital image signal to produce a composite digital image signal. The composite video image signal is converted to a film image on a reel of film by exposing the film. 
     Referring to FIG. 5 of the accompanying drawings, there is shown an apparatus for performing the process as described above. The first digital image signal may be produced by a film scanner  501  and passed along a wideband data bus  502  for storage in a high capacity storage device  504 . The second digital image signal, corresponding to a video image or a computer generated image may be input from a video signal source device or computer signal source device attached to the data bus  502 . The first digital image signal and second digital image signal may be composited in a graphics processor  505 , linked to a graphics workstation  506  comprising a screen  507 , a control keypad  508 , and a tablet and pen  509 . A resulting composite digital image signal may be downloaded and converted back to a film storage medium by a film recorder  510  receiving the composite digital image signal from the data bus  502 . 
     In the present specification it will be understood that the signals carry data corresponding to images and the data may be converted to signal form, for storage purposes and data processing operations. Selection or identification of a portion of a displayed image may correspond to selection or identification of the corresponding data for that portion. 
     FIG. 6 shows schematically an editing artist in overall control of the compositing process at the graphics work-station  506 . 
     Referring to FIG. 7 of the accompanying drawings there is shown a background image frame on film, the background image being subject to a spatial intensity texture or granularity, which as seen by a viewer would be described as a “graininess” of the image. 
     Referring to FIG. 8 of the accompanying drawings, there is shown a video image frame or computer generated image frame, having no graininess as seen by the observer. 
     Shown in FIG. 9 of the accompanying drawings, is a composite image comprising the background image of FIG.  7  and the foreground image of FIG.  8 . In the composite image, the observer perceives a difference in the graininess of the foreground object and the background. The difference in graininess between foreground and background objects is undesirable in the composite image since the human observer perceives the composite image to be unrealistic. 
     Referring to FIG. 10 herein, there is described an overview of a process for producing a composite image data derived from filmed image data or signal and a video or computer generated image or signal, in which the video or computer generated portion of the composite image data or signal has applied thereto a textural pattern data or signal synthetically generated to match the grain texture of the filmed background image. 
     FIG. 11 herein illustrates the resultant composite image having applied thereto the synthetically generated grain texture. 
     There will now be described a method of characterizing a grain texture of a filmed image, with reference to FIGS. 12 to  19  of the accompanying drawings. 
     A frame of filmed image is converted into a first digital image signal as described above. The first digital image signal may be a video signal. The first digital image signal is displayed upon a display device, in this case the display unit  507  of the graphics work-station. The first digital image as displayed, is illustrated schematically with reference to FIG. 12 herein. 
     An operator of the graphics work-station, e.g. a film editing artist, using an electronic cursor and the tablet and stylus  509 , identifies a region of the displayed image in which there is a relatively uniform region, for example the region  1  in FIG.  12 . By relatively uniform it is meant relatively uniform with respect to overall color. 
     Referring to the upper portion of FIG. 13 herein, there is shown the region  1  of FIG. 12 in magnified form. The region  1  of the image displayed on the display unit corresponds to a corresponding region of the original film image contained on the film. The granularity of the original film of the film clip causes fluctuations in the intensity of the portion of the first digital image signal corresponding to the displayed region of relatively uniform color  1 . On selecting a region of relatively uniform color, the assumption is made that any variations in the intensity across region  1  are caused by the granularity of the original film frame corresponding to region  1 . 
     Once the operator has identified relatively uniform region  1 , the graphics processor  505  operates to scan the region of relative uniformity, for example along a straight line between spatially disparate points A and B as shown in FIG.  13 . As the graphics processor  505  scans the corresponding region of the first digital image signal, corresponding to the spatially disparate positions A and B on the displayed image, the processor records values of intensity against distance. Where the first digital image signal is a video signal, the processor may record variations in intensity on a pixel by pixel basis. 
     The portion of the first digital image signal corresponding to the line AB in the region  1  may be filtered into separate color components corresponding to red, green and blue components. For each of the red, green and blue components, a characteristic of intensity versus distance may be recorded and stored. The resultant plot of intensity versus distance, for the purposes of convenience in this specification shall be referred to as a spatial intensity distribution. 
     The spatial intensity distribution is then further processed as follows. 
     It has been found that the variation of intensity with distance over a region of a filmed image fluctuates in accordance with the granularity of the film. To characterize the granularity of the film the spatial intensity distribution from the line A-B of the region  1  of the first digital image signal is firstly averaged by computer program applying an algorithm to the data of the spatial intensity distribution, to produce a mean or average value mu. The standard deviation of intensity as it varies above and below the mean or average intensity mu is determined. For example the standard deviation sigma may be determined in accordance with the formula shown in FIG. 13, where: 
     sigma=standard deviation of intensity. 
     mu=mean or average intensity over the region AB 
     K=a constant, and 
     lambda=frequency of intensity fluctuations. 
     Secondly, by determining the gradient of intensity fluctuations with distance over the line A-B, a periodicity P of intensity fluctuations around the mean or average value mu may be determined from the gradient characteristic. The periodicity P is dependent upon the grain size in the original filmed image. 
     Thus, for the region  1 , sampled along the line A-B, information about the granularity of the original film may be determined and characterized in terms of the parameter&#39;s standard deviation, mean or average intensity (mu), and periodicity P of the gradient of intensity change. This data gives information about the granularity of the film used in the film clip from a small relatively uniform sample region  1  of one frame of the film clip at one overall color. 
     However, the film clip comprises a large number of image frames, and the color varies over each image frame. A general objective in characterizing the grain texture of the film used for the film clip is to select specific regions of relative uniformity. Since, in the absence of grain structure in the film, individual regions of relatively constant overall color ought not to show any variations in color intensity in either of the three red, green or blue components, any variations in intensity determined from the first digital image signal portions corresponding to the regions of relative uniformity ought to be due solely to the granularity of the originating film. 
     To obtain a better characterization of the grain texture of the originating film, the steps shown in FIG. 13 are repeated for a number of different regions of relative uniformity. The different regions of relative uniformity should be of different colors. Each region of relative uniformity of different color will have its own particular value of mean or average intensity mu for the red, green and blue components. 
     The steps are summarized in schematic form in FIG.  14 . 
     Referring to FIG. 15 of the accompanying drawings, there are shown first and second regions of relative uniformity  1 ,  2  which are respectively sampled along lines A-B and C-D. Each sampling results in corresponding spatial intensity distributions along lines A-B and C-D in the red, green and blue components. There may be determined by a plurality of standard deviations sigma r1, sigma g1, sigma b1, sigma r2, sigma g2, sigma b2 corresponding to the spatial intensity distributions of the first and second regions. Similarly, for each spatial intensity distribution there may be determined a particular value of P relating to the grain size. 
     Referring to FIGS. 12 to  17  of the accompanying drawings, where a number N of individual regions of relative uniformity are sampled, there may result in a number N of values mu, mu 1 . . . mu N. For each value of mu, there will result a corresponding value of sigma. Values of mu and sigma may be plotted as shown in FIG. 17 with mu on the horizontal axis and sigma on the vertical axis, to produce a color plane granularity characterization map which characterizes the granularity of the originating film by finding discrete points of standard deviation at discrete values of mean or average color intensity mu. 
     By sampling N regions, a continuous curve for each of the red, green and blue components may be approximated by fitting a polynomial expression to the determined values of sigma for the respective red, green and blue components. 
     Referring to FIGS. 18 and 19 of the accompanying drawings, there is shown graphically, a preferred method for determining the grain size P from the spatial intensity distributions. 
     FIG. 18 shows a spatial intensity distribution of, for example, a red component. FIG. 19 shows the gradient of the spatial intensity distribution of FIG. 18. A periodicity P is determined as a most prominently present fundamental frequency of the gradient of the intensity, over the sample region A-D. 
     A preferred method of synthetically generating a granularity template signal or data for applying to the second digital image signal corresponding to the video image or the computer generated image, will now be described. 
     Referring to FIG. 20 of the accompanying drawings, a noise generator  600  generates signal intensity noise to mimic the intensity fluctuations in the first digital image signal produced by the granularity of the film. A low pass filter  601  filters the noise signal generated by the noise generator  600 . 
     The noise generator  600  produces noise having a standard deviation which can be specified by inputting a standard deviation value sigma. The low pass filter can filter the random noise signal generated by the noise generator  600  with a cut off frequency determined by the periodicity P, which characterizes the grain size of the originating film clip. 
     In the composite signal, comprising the first digital image signal and the second digital image signal, the first digital image signal contains information relating to the granularity of the originating film. By applying the granularity template signal produced by the random noise generator  600  and low pass filter  601 , the granularity template signal is produced having characteristics which relate to the granularity of the film. This is then used to produce an effect on the second digital image signal (the signal corresponding to the video image or to the computer generated image) which mimics the granularity of the originating film when an image corresponding to the composite digital image signal is displayed. 
     When the composite digital image signal is transferred back to film medium via the film recorder  510 , the resultant film holds an image comprising the image on the original originating film clip plus the keyed-in video image or computer generated image, in which the keyed-in portion of the composite image contains a granularity effect comparable to that on the originating film.