Abstract:
A first visual image or other pattern (A) is represented by a first ordered set of elements (x i ) each having a value and a second pattern (B) is a represented by a second ordered set of element (u i ) each having a value. Iteratively, a comparison is made by: (i) selecting a plurality of elements from the first ordered set;(ii) for each selected element of the first ordered set, selecting an element from the second ordered set, such that the selected elements of the second ordered set have, within the second ordered set, a set of positional relationships relative to each other that is the same as, or a transformation of, the set of positional relationships that the selected plurality of elements of the first ordered set have relative to each other; (iii) selecting a random shift value (γ); and (iv) determining whether the selected elements meet the match criterion that the value of each of the selected elements of the first ordered set differs from the value of the correspondingly positioned selected element of the second ordered set by an amount that is equal to, or within a set margin of, and in the same sense as, said shift value. In the event of a match, a similarity score (S AB ) is updated. Alternatively, or in addition, the shift amount in the event of a match is recorded and, at the conclusion of the iteration, a single representative shift value is estimated from said recorded shift value.

Description:
This application is the U.S. national phase of International Application No. PCT/GB2006/002001 filed 1 Jun. 2006 which designated the U.S. and claims priority to European Patent Application No. 05253589.5 filed 10 Jun. 2005, the entire contents of each of which are hereby incorporated by reference. 
   BACKGROUND 
   1. Technical Field 
   The present invention is concerned with comparing one pattern with another, and is of particular interest in the comparison of two-dimensional patterns such as visual images, although applicable also to one-dimensional patterns and patterns having three or more dimensions. 
   2. Related Art 
   Some techniques for analysis of images or other patterns where the pattern is compared with other parts of the same pattern are described in our earlier patent applications as follows.
     European patent application 00301262.2 (publication No. 1126411);   International patent application PCT/GB01/00504 (publication No. WO 01/61648);   International patent application PCT/GB01/03802 (publication No. WO02/21446));   U.S. patent application Ser. No. 977,263/09 filed 16 Oct. 2001 (publication No. 20020081033);
       as well as the following papers published by the inventor:   
       Stentiford F W M, “An estimator for visual attention through competitive novelty with application to image compression”, Proc. Picture Coding Symposium 2001, Seoul, 25-27 April, pp 101-104, 2001.   Stentiford F W M, “An evolutionary programming approach to the simulation of visual attention”, Proc. Congress on Evolutionary Computation 2001, Seoul, pp 851-858, 27-30 May 2001.   

   Methods and apparatus for comparing patterns are described in our earlier international patent application WO03/081532 and our co-pending international patent application WO2006/030173. 
   BRIEF SUMMARY 
   Aspects of the present invention are set out in the claims. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
     Some embodiments of the present invention will now be described, with reference to the accompanying drawings, wherein: 
       FIG. 1  depicts computer apparatus suitable for practice of this invention; 
       FIGS. 2   a  and  2   b  depict image arrays stored in the memory of  FIG. 1 ; 
       FIGS. 3-5  depict flow charts of processes involved in some aspect of the invention; and 
       FIG. 6  depicts exempla comparison values for an embodiment. 
   

   DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS 
     FIG. 1  shows an apparatus consisting of a general purpose computer programmed to perform image analysis according to a first embodiment of the invention. It has a bus  1 , to which are connected a central processing unit  2 , a visual display  3 , a keyboard  4 , a scanner  5  (or other device, not shown) for input of images, and a memory  6 . 
   In the memory  6  are stored an operating system  601 , a program  602  for performing the image analysis, and storage areas  603 ,  604  for storing two images, referred to as image A and image B. Each image is stored as a two-dimensional array of values, each value representing the brightness and/or colour components of a picture element within the array. It will be understood, however that the apparatus might more generally be arranged to analyse data sets other than images, in which case the storage areas  603 ,  604  would each contain a data set A, B, each being an ordered set of values, ordered in any number of dimensions (the same number of dimensions in each set). 
   The image arrays are shown schematically in  FIGS. 2   a  and  2   b . Image A consists of an array of picture elements x i =(x i , y i ) where x i  and y i  are the horizontal and vertical positions of the elements within the image. A 20×20 array is shown, for the purposes of illustration. In the general case, x i =(X i1 , x i2 , x i3 , . . . x in ) where x ij  are the n coordinates of x i  in n dimensions. Each element has a respective value a=a(x i ). This may be a scalar value a or vector (multidimensional) value a. In the case of an image these would be the brightness a, or alternatively a set of colour components such as (in r,g,b representation) a=(a r , a g , a b ) Similarly, the image B consists of an array of picture elements u i =(u i , v i ) having brightness values b(u i ) or colour vectors b(u i ). In the general case, u i =(u i1 , u i2 , u i3 , . . . u in ). 
     FIG. 3  is a flowchart explaining the application of the method as applied to two-dimensional images A, B. In Step  100 , a score C AB  is set to zero, and a counter t set to zero. 
   The first task of the method is to make a random selection of elements in image A. This may be visualised as stabbing the image with an irregular fork having m tines. The actual number m of elements may be fixed, or if preferred could be selected at random within a range m min ≦m≦m max . The fork size may be limited by a diameter D, which could be fixed or selected as a random number D min ≦D≦D max . Other approaches to varying the distribution of fork size are discussed in our above-mentioned U.K. patent application 0420727.0. Note that, throughout this specification, references to a random selection also envisage the possibility of selection by means of a pseudo-random process. Also, a reference to random selection envisages the possibility not only of a random selection for which the selection of any value (within a range specified) is equally probable, but also of a selection in accordance with a non-uniform probability distribution. 
   At Step  101  a set N X  of m elements at random positions x i  is selected in image A, subject to the constraint that all elements lie within the image. If desired, the elements may be constrained to lie within a certain distance of one another, though this is not essential. One possible method of doing this is as follows:
         a) Choose a random element x 0  in A.   b) Choose a set of random displacements d i =(d xi , d yi ), where 0≦d xi ≦D/2 and 0≦d yi ≦D/2.   c) If the Euclidean distance |d i |≦D/2 for all i, then continue; otherwise repeat the previous step for any i needing it until this condition is satisfied. Where |d i |=√{d xi   2 +d yi   2 }.   d) Compute positions x i =x 0 +d i      e) If any x i  lies outside the image area, repeat steps (b) to (e) in respect of the offending d i .       

   Note that this process permits the occurrence of one or more pairs x i =x j  i≠j i.e. the x i  are not constrained to be distinct. This is not objectionable in practice but if desired may be eliminated by testing for it and reselecting one member of the matching pair in the same way as for elements falling outside the image area. 
   Step  102  is optional and will be described later. At  103 , a counter n is initialised to zero. The next stage (Step  104 ) is to make a selection N u  of m elements u i  within image B, at a random overall position, but having the same positions relative to one another as the selected elements of image A. That is, x i −x j =u i −u j  for all i,j. This may be visualised as stabbing image B with the same fork as used on image A. 
   Assuming that a set of displacements d i  has been calculated as above, this can be done by: 
   Choose a random element u 0  in B. 
   Compute positions u i =u 0 +d i    
   If any ui lies outside the image area, repeat these steps in respect of the offending d i . 
   It is to be determined whether each of the selected elements in image A matches the element having the same relative position in image B. First, however, a random colour shift γ is chosen. In the case of monochrome image this would simply be a brightness shift but in the colour case it requires selection of a random shift value for each colour axis, e.g., in an r, g, b system,
 
γ=(γ r , γ g , γ b ,)
 
   where each shift γ r , γ g , γ b  is selected to lie in the range −γ MAX  to +γ MAX  (different ranges could be defined for the different colour components if desired—for example in y, u, v systems one range might be chosen for the luminance and a different range for the chrominance values). Supposing that the colour components take values in the range 0 to 255, then a typical value for γ MAX  might be 64, or perhaps 128. 
   The test for a match occurs at Step  105 . In the r, g, b case a match occurs if
 
Dist[ a ( x   i )− b ( u   i )+γ]&lt;δ for all  i= 1  . . . m  
 
   Where Dist is some distance operator (e.g. Euclidean or city-block), or the vector components may be thresholded separately, e.g. for colour components the match condition would be
 
| a   r ( x   i )− b   r ( u   i )+γ r |&lt;δ r  and
 
| a   g ( x   i )− b   g ( u   i )+γ g |&lt;δ g  and
 
| a   b ( x   i )− b   b ( u   i )+γ b |&lt;δ b  for all  i= 1  . . . m  
 
   where δ r , δ g  and δ b  are some small threshold values which will commonly be equal. 
   If a match occurs, then at Step  106  the Score C AB  is incremented. It could simply be incremented by 1 (or other fixed value). 
   Once the score has been updated, then the iteration counter t is incremented at  109  and tested at  110  to see if it has reached a maximum iteration count T, and if so, the process terminates. Otherwise the process returns to Step  101  for a further iteration. If no match occurs, then the fork is retained and the process repeats from  104 . However this repetition with the same x i  is subject to a limit of N iterations: the counter n is incremented at  107  and if at  108  it reaches N then control passes to  109  for a fresh fork. 
   The result of this process is the final value of C AB , which is a score indicating the degree of similarity between image A and image B. As compared with the method described in our international patent application WO03/081523, this method has the advantage that a high score can be obtained despite the fact that there may be differences in colour between the two images, due for example to differences in lighting. 
   In the case of images A, B both containing large plain areas of similar brightness (or colour), a large number of matches and hence a high score might be obtained even though there may be little similarity between the parts of the image that contain detail. In this case, the process need not necessarily be carried out for the whole image. For example, if regions of images A and B have been identified as being of special interest—perhaps using the method described in one of our earlier patent applications referred to above—then the picture elements x, u dealt with may be just those lying in the vicinity of the identified region. This could be implemented by constraining the selection of x and u such that at least one element x i  is an element of a special interest region of image A and at least one element u i  is an element of a special interest region of image B. This means that each match establishes a similar relationship in each image between the foreground and background, or foreground and foreground (if all tines lie on high interest pixels). Small diameter forks will tend to obtain similarities between localised regions in A and B, and larger forks will determine global relationships between foreground and background. 
   Alternatively, we find that improved results can be obtained if the process is biased towards making comparisons involving the more detailed parts of the image, and with this in mind, the operation represented in Step  102  serves to test each new fork to determine whether, when it is in its initial position in image A, each of its tines points at an element that differs by at least a threshold amount from those pointed at by at least h of its other tines. This is shown in detail in  FIG. 4 . Counters q, p are initialised at  200 ,  201  so each distinct combination of p and q represents a pair of tines of the fork. The process then enters a loop where at  202  the picture elements at the corresponding two points in image A are compared to see whether they differ by more than a threshold amount. If not, then p is incremented ( 205 ) until it exceeds m ( 206 ) and the comparison repeated. If the values match, then a match counter Z is incremented at  203 . As long as ( 204 ) the counter Z has not reached h, then the comparison is again repeated for a new value of p. Once Z has reached h, it has been established that picture element x q  differs significantly from at least h others in the fork and at  208 ,  209  Z is reset and q advanced for a further iteration. If all values of q result in a value of z that reaches h, then the test is passed ( 211 ). But is any value of q fails to result in Z reaching h them the tests fails  207 . 
   The above process is effective but rejects many candidate forks and is, therefore, slow. A faster but cruder alternative is to select elements at random until m-h have been found that match each other, and then looking for h random pixels that mismatch all of the first group of h pixels. 
   In this context a mismatch between two pixels means that they differ by ε or more in at least one colour component. 
     FIG. 5  is a flowchart showing the operation of a second embodiment of the invention. Steps that are identical to those of  FIG. 3  are given the same reference numerals. Here, the object is not to generate a similarity score, but rather to estimate the colour difference between the two images, though the score C AB  is still calculated, but for a different purpose. At the start, a set of counters C r (i), C g (i), C b (i) is initialised. The number of counters corresponds to the number of possible values of γ r , γ g , γ b , so there are 3(γ MAX +1)counters, corresponding to i=0 . . . γ MAX . 
   Operation is then as before up to the point where a match is found. Then the value of γ that resulted in the match is recorded by incrementing the corresponding counters. Thus, at step  200  the counters C r (γ r ), C g (γ g ), C b (γ b ) are incremented. The remained of the iteration process is as before, until the limit T is reached at  110 . At this point the counters C r (i), C g (i), C b (i) contain, in effect, three histograms indicating how successful each of the values of γ r , γ g , γ b  have been in enabling matches to be obtained.  FIG. 6  shows actual histograms plotted to show the results of a test in which image A was a photograph of a bowl of fruit under normal illumination and image B was the same scene but with a green cast. These results were obtained with T=500, m=6, N=500. 
   The next step is  201 , where the aim is to determine a single, representative colour shift for each colour component. This is achieved by taking the average of the colour shifts that resulted in matches, that is to say the shift μ AB =(μ r , μ g , μ b ) given by 
   
     
       
         
           
             
               
                 
                   μ 
                   r 
                 
                 = 
                 
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         0 
                       
                       
                         γ 
                         MAX 
                       
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       
                         iC 
                         r 
                       
                       ⁡ 
                       
                         ( 
                         i 
                         ) 
                       
                     
                   
                   
                     C 
                     AB 
                   
                 
               
             
           
           
             
               
                 
                   μ 
                   g 
                 
                 = 
                 
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         0 
                       
                       
                         γ 
                         MAX 
                       
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       
                         iC 
                         g 
                       
                       ⁡ 
                       
                         ( 
                         i 
                         ) 
                       
                     
                   
                   
                     C 
                     AB 
                   
                 
               
             
           
           
             
               
                 
                   μ 
                   b 
                 
                 = 
                 
                   
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           0 
                         
                         
                           γ 
                           MAX 
                         
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           iC 
                           b 
                         
                         ⁡ 
                         
                           ( 
                           i 
                           ) 
                         
                       
                     
                     
                       C 
                       AB 
                     
                   
                   . 
                 
               
             
           
         
       
     
   
   Note that C AB  was counted separately, but could alternatively be calculated from 
   
     
       
         
           
             C 
             AB 
           
           = 
           
             
               ∑ 
               
                 i 
                 = 
                 0 
               
               
                 γ 
                 MAX 
               
             
             ⁢ 
             
               
                 
                   C 
                   b 
                 
                 ⁡ 
                 
                   ( 
                   i 
                   ) 
                 
               
               . 
             
           
         
       
     
   
   Of course, rather than counting the number of times each value occurs, the same result could be obtained by storing in a memory each value that resulted in a match, and then taking the average of the stored values. 
   For the example shown in  FIG. 6 , a value of μ AB =(−28, 42, 2) was obtained. If desired, this result can be subtracted from image B by way of a correction, that is to say, the original pixel values b(u i ) are replaced by b(u i )−μ AB . Or if image B is considered to be correct the correction could be added to image A. 
   As an alternative to taking the average, the values of μ r , μ g , μ b  could be obtained by estimating the position of the peak of the histogram. This may be done by looking for the maximum value, the median, or the position of a window of width w in which the sum of w values was maximised. 
   In practice it is found that good colour correction can be obtained in this manner. In tests, it has become clear that it is not necessary for colour correction purposes that the two images should be identical in all respects except brightness/colour. For example, good results have been observed in a case where image A covered only part of the scene shown in image B. More significantly, the images do not have to be of the same scene: it is only necessary to have a range of matching colour adjacencies that allow forks to match in both images in order to obtain a good estimate, though similar structures present in both images will allow a greater proportion of colour shifted forks to match and therefore produce a more statistically reliable colour shift estimate. This offers the opportunity for automatically tailoring images to individual preferences by simply supplying examples of acceptable images and allowing the system to adjust the colour balance accordingly. If desired, the system could be incorporated in a camera for setting the exposure or aperture, and/or (in a digital camera) for adjusting the colour balance at the time a photograph is taken, or immediately afterwards. 
   The introduction of random colour variations into the scoring process, as described above with reference to  FIG. 3  can also be combined with the system described in our aforementioned international patent application WO2006/030173 which suggested the introduction of random variations in the number of tines m, and/or in the maximum fork diameter D and/or the fork shape to be applied to the second image (reflection, rotation, scaling). However, the more variables that are perturbed, the greater is the number of iterations need to produce a good result, so for fast performance one prefers to vary only two or three parameters at once. 
   Thus, in such an implementation, Step  105  of  FIG. 3  would be preceded by one or more of the following steps for transformation of the displacements d i : 
   A. a random reflection is applied to the fork: 
   
       
       
         
           Choose a random integer R as 0 or 1; 
           If R=1, set d xi =−d xi  for all i=1 . . . m. d yi  are unchanged. 
         
       
     
  
   This is a reflection about a single vertical axis. Reflections about other axes are encompassed by this reflection plus the rotation step that comes next. 
   B. choose a random rotation angle β in the range β max  to β min . Then the fork represented by d i  is rotated by this angle by performing, for each i:
 
 d   xi   ′=d   xi . cos β− d   yi . sin β
 
 d   yi   ′=d   xi . sin β+ d   yi . cos β.
 
C. a random scale factor α is chosen in the range α min  to α max . The fork is then scaled by this factor:
 
 d   xi   ″=α.d   xi ′
 
 d   yi   ″=α.d   yi ′
 
   The scaling and rotation are relative to (0,0), though scaling or rotation about any other centre would do just as well. 
   These processes has been described in terms of values representing the colour or brightness of elements of a two dimensional visual image. This would include a pattern that is viewable as a visual image even if it was not originally generated by detection of visible light, such as an x-ray image or tomography. It may be applied also to three-dimensional visual images (as in tomography), or to other patterns, especially where the values represent measurements where it may be of interest to identify a global shift which is hard to spot using conventional techniques; examples include measurements of magnetic fields, or of temperature.