Patent Publication Number: US-8538075-B2

Title: Classifying pixels for target tracking, apparatus and method

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to an image processing apparatus and an image processing method. More specifically, the present invention relates to a method for tracking an object by using an image. 
     2. Description of the Related Art 
     A conventional method for tracking an arbitrary object uses a pattern matching method, which searches for an area having a high degree of matching with a previously stored template image. In addition, a conventional object tracking method uses a relative difference method, in which template images are serially updated and the location of an object is identified based on a difference between a current frame and a previous flame. Furthermore, a conventional object tracking method uses a histogram matching method, in which the location of an object is identified based on a result of searching for an area having a high degree of matching with a color or luminance histogram of the object. 
     If the above-described pattern matching method is used, a still object can be searched at a high accuracy. However, the performance of tracking a moving object when the pattern matching method is used is not sufficiently high. More specifically, if the distance between an object and a tracking apparatus varies, if an object rotates, or if the orientation of the tracking apparatus itself varies, the pattern matching method cannot track the object at a high accuracy. 
     If the above-described relative difference method is used, if another object comes in front of a tracking target object or if the tracking target object has gone out of the scope of imaging, then the tracking target object may not be appropriately tracked. 
     If the histogram matching method is used, a moving tracking target object can be tracked but the performance of identifying the tracking target object is low if the identification is executed based on a color or luminance histogram only. Furthermore, the tracking performance of the histogram matching method is not sufficiently high if similar colors or areas having the approximately the same level of luminance is largely distributed in the background of an object. 
     In order to solve the above-described situation, Japanese Patent Application Laid-Open No. 09-181953 discusses a method for determining a characteristic color, which is uniquely used to a tracking target area and which can be securely extracted, considering the state of both a tracking target object and another object that is not a tracking target. 
     However, in the conventional method discussed in Japanese Patent Application Laid-Open No. 09-181953, a method for determining one characteristic color useful in using during tracking is discussed but the tracking may not always be executed at a high accuracy. More specifically, unless the state of the current tracking target object is useful in using during tracking, the probability of a wrong search may not be lowered if one optimum characteristic color is selected and used. 
     SUMMARY OF THE INVENTION 
     According to an aspect of the present invention, an apparatus includes a storage unit configured to classify pixels existing inside a tracking target area set on an image and pixels existing outside the tracking target area according to an attribute and to store a result of classification of the pixels on a storage medium, a first derivation unit configured to derive a first ratio of the pixels existing inside the tracking target area and having the attribute to the pixels existing outside the tracking target area and having the attribute, a second derivation unit configured to derive a second ratio of of pixels, of the pixels existing inside the tracking target area, whose the first ratio is higher than a first predetermined value, to all pixels existing inside the tracking target area, and a determination unit configured, if the second ratio is higher than a second predetermined value, to determine that the tracking target area can be tracked. 
     Further features and aspects of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the invention and, together with the description, serve to explain the principles of the present invention. 
         FIG. 1  illustrates an example of a basic configuration of an imaging apparatus according to a first exemplary embodiment of the present invention. 
         FIG. 2A  illustrates a state in which an animal, whose image is located approximately in the center of the imaging scope, is set as a tracking target object.  FIG. 2B  illustrates a state in which the animal set as the tracking target object has moved. 
         FIG. 3  is a flow chart illustrating an example of processing for tracking a tracking target object, which is executed by the imaging apparatus. 
         FIG. 4  is a flow chart illustrating an example of processing for recognizing a tracking target object. 
         FIG. 5  illustrates an example of an image and pixels included in a tracking target area of the image. 
         FIG. 6A  illustrates an example of a hue histogram.  FIG. 6B  illustrates an example of an average hue histogram. 
         FIG. 7A  illustrates an example of a luminance histogram.  FIG. 7B  illustrates an example of an average luminance histogram. 
         FIG. 8A  illustrates an example of pixels existing adjacent to two pink pixels.  FIG. 8B  illustrates an example of pixels existing adjacent to three yellow pixels.  FIG. 8C  illustrates an example of pixels existing adjacent to two green pixels.  FIG. 8D  illustrates an example of pixels existing adjacent to one blue2 pixels. 
         FIG. 9A  illustrates an example of pixels existing adjacent to one black0 pixel.  FIG. 9B  illustrates an example of pixels existing adjacent to twelve black1 pixels.  FIG. 9C  illustrates an example of pixels existing adjacent to two gray2 pixels.  FIG. 9D  illustrates an example of pixels existing adjacent to three white pixels. 
         FIG. 10  illustrates an example of an adjacent histogram, which corresponds to a hue histogram of a main histogram. 
         FIG. 11  illustrates an example of an adjacent histogram, which corresponds to a luminance histogram of a main histogram. 
         FIG. 12 , composed of  FIGS. 12A and 12B , is a flow chart illustrating an example of trackability determination processing in step S 103  in the flow chart illustrated in  FIG. 3 . 
         FIG. 13  is a flow chart illustrating an example of continued tracking availability determination processing in step S 105  in the flow chart illustrated in  FIG. 3 . 
         FIG. 14  is a flow chart illustrating an example of tracking target object detection processing in step S 107  in the flow chart illustrated in  FIG. 3 . 
         FIG. 15  schematically illustrates an example of a method for tracking a tracking target area. 
         FIG. 16A  illustrates an example of a state in which a pixel to be processed matches a chromatic color of the main histogram.  FIG. 16B  illustrates an example of a state in which a pixel to be processed matches an achromatic color of the main histogram.  FIG. 16C  illustrates an example of a state in which a pixel existing adjacent to the pixel to be processed that matches the chromatic color of the main histogram matches a chromatic color or an achromatic color included in the adjacent histogram.  FIG. 16D  illustrates an example of a state in which a pixel existing adjacent to the pixel to be processed that matches the achromatic color of the main histogram matches a chromatic color or an achromatic color included in the adjacent histogram. 
         FIG. 17  illustrates an example of a result of calculation of an evaluation value. 
         FIG. 18  is a flow chart illustrating an example of processing for tracking a tracking target object, which is executed by the imaging apparatus. 
     
    
    
     DESCRIPTION OF THE EMBODIMENTS 
     Various exemplary embodiments, features, and aspects of the invention will be described in detail below with reference to the drawings. 
       FIG. 1  illustrates an example of a basic configuration of an imaging apparatus  100  according to a first exemplary embodiment of the present invention. The imaging apparatus  100  is capable of capturing both a still image and a moving image. Referring to  FIG. 1 , light reflected from an object goes through an imaging optical system  101 , which includes a focus lens, a variable magnification lens, and a fixed lens. An image of the reflection light from the object is formed by an image sensor  102 . 
     The image sensor  102  is a photoelectric conversion element for converting a signal of the light from an image formed on a light-receiving plane into an electric signal for each light for each photodetection pixel existing at a corresponding location. The light whose image has been formed on the image sensor  102  is converted by the image sensor  102  into an electric signal. The electric signal is input to an analog-to-digital (A/D) converter  103 . The A/D converter  103  converts the input electric signal into a digital signal. Then, the digital signal is input to an image processing circuit  104 . The image processing circuit  104  executes image processing on the input digital signal (image data). 
     A display device  105  is a display unit, which is provided on a back surface of the imaging apparatus  100 . The display device  105  functions as an electronic viewfinder. In addition, the display device  105  displays information about a setting status on the imaging apparatus  100 . Furthermore, the display device  105  displays a thumbnail image generated by the imaging apparatus  100 . 
     An imaging optical system control circuit  106  generates lens information. The lens information includes an object distance, a focal length, which is acquired based on information about the location of a variable power lens group, and an F number, which is acquired based on the aperture diameter of a diaphragm. 
     A system controller  107  includes a microcomputer. The system controller  107  controls an operation of the entire imaging apparatus  100  by executing a control program. An image recording medium  108  records image data. In the present exemplary embodiment, an example of a method for tracking an image of an arbitrary object, which is executed on the imaging apparatus  100  having the configuration described above, will be described in detail. 
       FIGS. 2A and 2B  illustrate an example of a scene which includes an image that a conventional object tracking method cannot appropriately track. 
     Referring to  FIG. 2A , an animal existing around the center of a screen  200  is set as a tracking target object. For a method for designating a tracking target object, it is useful if an area satisfying a predetermined color condition or luminance condition is automatically identified and set as a tracking target object. In addition, it is also useful if a user touches the display device  105  on a touch panel included therein at an arbitrary location and an image corresponding to the location touched by the user is set as a tracking target object. 
     In the example illustrated in each of  FIGS. 2A and 2B , it is supposed that an area  202 , which is an area corresponding to the belly and around of an animal, is set as a tracking target area, that the animal has moved to another location after that, and that thus the orientation of the animal has been changed. 
     If the above-described change in the tracking target object has been made, the pattern matching method, in which an image included in the tracking target area  202  illustrated in  FIG. 2A  and an image included in an area  210  corresponding to the belly and around of the animal after it moved to the different place are compared to each other, cannot track the animal. 
     As illustrated in  FIGS. 2A and 2B , the histogram matching method, which compares a histogram  220  of the tracking target area  202  with another histogram  222  of the image of the area  210  corresponding to the belly and around of the animal after it moved to the different place, cannot appropriately track the animal because the histograms do not match each other. 
     Now, an example of a method according to the present exemplary embodiment for tracking an arbitrary object will be described in detail below with reference to the following flow charts. 
     In the present exemplary embodiment, the system controller  107  executes processing according to the following flow chart. However, it is also useful if a part of or the whole of the processing is executed by a component other than the system controller  107  (e.g., the image processing circuit  104 ). 
       FIG. 3  is a main flow chart illustrating an example of processing executed by the imaging apparatus  100  for tracking a tracking target object. Referring to  FIG. 3 , in step S 101 , the system controller  107  determines a location and a size of the tracking target object. 
     As described above, a tracking target object can be automatically identified. Alternatively, a user can designate an arbitrary area by viewing through an electronic viewfinder to determine a tracking target object. After the location and the size of the tracking target object have been determined, then the processing advances to step S 102 . In step S 102 , the system controller  107  executes processing for recognizing the tracking target object based on color information and luminance information about the tracking target area. 
       FIG. 4  is a flow chart illustrating an example of processing for recognizing a tracking target object in step S 102  illustrated in  FIG. 3 . Referring to  FIG. 4 , in step S 201 , the system controller  107  generates a histogram of a tracking target area (main histogram). 
       FIG. 5  illustrates an example of an image and pixels included in a tracking target area of the image. Referring to  FIG. 5 , an image  300  includes a scene of a penguin captured by a camera. The user can designate a portion of the image  300  by touching the touch panel on a tracking target area  302  to determine the tracking target area. 
     The tracking target area  302  corresponds to an area  304 , which is illustrated in  FIG. 5  in its right portion, surrounded by a thin-line rectangle. The system controller  107  generates histograms of pixels inside and outside the area  304 . 
     The histograms include a hue histogram and a luminance histogram. The distribution is taken on each of the hue histogram and the luminance histogram according to whether each pixel is a chromatic color pixel or an achromatic color pixel. Determination as to whether each pixel is a chromatic color pixel or an achromatic color pixel is executed based on a result of comparison of saturation of each pixel with its threshold value. In the present exemplary embodiment, the system controller  107  generates a histogram of each area included in the tracking target area  302  and each area existing outside the tracking target area  302 . 
       FIG. 6A  illustrates an example of a hue histogram (i.e., a histogram of chromatic color pixels).  FIG. 6B  illustrates an example of an average hue histogram. The average hue histogram indicates an average luminance, an average saturation, and an average hue of a pixel classified by the hue histogram.  FIG. 7A  illustrates an example of a luminance histogram (i.e., a histogram of achromatic color pixels).  FIG. 7B  illustrates an example of an average luminance histogram. The average luminance histogram indicates an average luminance of a pixel classified by the luminance histogram. A chromatic color pixel is classified in (registered in) a hue histogram  400 . 
     The system controller  107  counts the number of pixels existing inside the tracking target area (“In” in  FIG. 6A ) and those existing outside the tracking target area (“Out” in  FIG. 6A ) for each hue (“Hue” in  FIG. 6A ) existing inside the tracking target area  302 . 
     In the example illustrated in  FIGS. 6A and 6B , the system controller  107  generates a hue histogram  400  in the unit of division (the unit of classification) of an angle of 40° each. However, it is also useful if the unit of division is changed according to an accuracy of tracking. An achromatic color pixel is classified in (registered in) a luminance histogram  440 . 
     The system controller  107  counts the number of pixels existing inside the tracking target area (“In”) and the number of pixels existing outside the tracking target area (“Out”) for each luminance Y existing inside the tracking target area  302 . In the examples illustrated in  FIGS. 7A and 7B , the system controller  107  generates a luminance histogram  440  in the division unit of 40 LSB. However, the division unit can be changed according to an accuracy of tracking. 
     After generating the hue histogram  400  and the luminance histogram  440 , the system controller  107  calculates a ratio of existence of each hue inside and outside the tracking target area. The ratio of existence of each hue inside and outside the tracking target area can be calculated by the following expression:
 
Ratio of Existence of Each Hue inside and outside Tracking Target Area [%]=Number of Pixels Existing inside Tracking Target Area/Number of Pixels Existing outside Tracking Target Area.
 
     For example, in the example illustrated in  FIG. 6A , fifteen pixels having the pink hue, i.e., two pixels having the pink hue inside the tracking target area  302  and thirteen pixels having the pink hue outside the tracking target area  302 , exist. Accordingly, the existence ratio thereof is 15% (=(2/13)×100). 
     However, if one or more pixels exist within the tracking target area and if the number of pixels existing outside the tracking target area is zero, then the existence ratio is 100%. The existence ratio is calculated in the above-described manner. However, the present invention is not limited to this. More specifically, it is also useful if the existence ratio is calculated by the following expression:
 
Ratio of Existence [%]=[Number of Pixels Existing inside Tracking Target Area/(Number of Pixels Existing inside Tracking Target Area+Number of Pixels Existing outside Tracking Target Area)]×100.
 
     In addition to calculating the distribution of the hue, the system controller  107  calculates an average luminance AveY, an average saturation AveChroma, and an average hue AveHue of a pixel classified in a unit of division, for the unit of division in the histogram of pixels existing inside the tracking target area  302 . 
     More specifically, in the hue histogram  400  illustrated in  FIG. 6A , three pixels having the yellow hue exist inside the tracking target area  302 . Accordingly, in an average hue histogram  420  illustrated in  FIG. 6B , the three pixels have the average luminance AveY of 153 LSB, the average saturation AveChroma of 55 LSB, and the average hue AveHue of 113°. 
     Similarly, in a histogram  440  illustrated in  FIG. 7A , three pixels having achromatic color hue (white hue) exist inside the tracking target area  302 . Accordingly, in an average luminance histogram  480  illustrated in  FIG. 7B , the three pixels have the average luminance of 210 LSB. 
     As described above, as information about the chromatic color pixel existing inside the tracking target area  302 , the hue histogram  400  and the average luminance AveY, the average saturation AveChroma, and the average hue AveHue classified in the hue histogram  400  (the average hue histogram  420 ) are acquired. In addition, as information about the achromatic color pixel existing inside the tracking target area  302 , the luminance histogram  440  and the average luminance AveY classified in the luminance histogram  440  (the average luminance histogram  480 ) are acquired. In the following description, the hue histogram  400 , the average hue histogram  420 , the luminance histogram  440 , and the average luminance histogram  480  are collectively referred to as a “main histogram” where necessary. 
     Returning to  FIG. 4 , in step S 202 , the system controller  107  generates histograms  500  and  520  of an aggregate of pixels existing adjacent to each pixel classified in the division unit in each of the hue histogram  400  and the luminance histogram  440  of the main histogram. In the following description, the histograms  500  and  520  are collectively referred to as an “adjacent histogram” where necessary. 
       FIGS. 8A through 8D  and  FIGS. 9A through 9D  schematically illustrate a first and a second example of a pixel whose adjacent histogram is to be generated. As illustrated in  FIGS. 6A and 6B  and  FIGS. 7A and 7B , it is supposed that eight distributions including four distributions of pink, yellow, green, and blue2, i.e., information about the chromatic color pixel inside the tracking target area  302 , and four other distributions of black0, black1, gray2, and white, i.e., information about the achromatic color pixel inside the tracking target area  302 , have been acquired.  FIGS. 8A through 8D  and  FIGS. 9A through 9D  illustrate an example of adjacent pixels of each pixel having the eight distributions. 
     Referring to  FIG. 8A , pixels existing adjacent to two pink pixels  550  and  552  are indicated by circles surrounding the same. A total of eight pixels including one blue2 pixel, one gray2 pixel, one black1 pixel, two pink pixels, one white pixel, and two green pixels are distributed around the pink pixel  550 . In addition, a total of eight pixels including one gray2 pixel, one black1 pixel, one black0 pixel, one white pixel, one green pixel, and two pink pixels are distributed around the pink pixel  552 . 
     The system controller  107  generates the histograms based on the sixteen pixels surrounding the pink pixels  550  and  552 , which constitute the aggregate of pixels existing adjacent to the pink pixels. 
     Similarly, a total of twenty-four pixels exist adjacent to three yellow pixels  554 ,  556 , and  558  in the example illustrated in  FIG. 8B . A total of sixteen pixels exist adjacent to two green pixels  560  and  562  in the example illustrated in  FIG. 8C . In the example illustrated in  FIG. 8D , a total of eight pixels exist adjacent to one blue2 pixel  564 . 
       FIGS. 9A through 9D  illustrate an example of pixels existing adjacent to an achromatic color pixel. More specifically, adjacent pixels, i.e., a black0 pixel, a black1 pixel, a gray2 pixel, and a white pixel are illustrated in  FIGS. 9A through 9D . 
     In the present exemplary embodiment, for the main histogram for the tracking target area  304  illustrated in  FIG. 5 , the system controller  107  generates a histogram of pixels existing inside the tracking target area  304 . On the other hand, for the adjacent pixels, the system controller  107  executes recognition thereof existing in an area wider than the tracking target area  304 . 
     With the above-described configuration, the system controller  107  according to the present exemplary embodiment can acquire information about adjacent pixels including information about the relationship between the tracking target object and the background thereof. Accordingly, the present exemplary embodiment can acquire an appropriate result of tracking. However, it is also useful if the same area is referred to for both. 
     The system controller  107  associates the histogram of the aggregate of the pixels including pixels existing adjacent to each pixel classified in each division unit in the main histogram illustrated in  FIGS. 8A through 8D  and  FIGS. 9A through 9A  (the adjacent histogram) with the main histogram. Furthermore, the system controller  107  stores the adjacent histogram and the main histogram in a format illustrated in  FIGS. 10 and 11 . 
       FIG. 10  illustrates an example of the adjacent histogram, which corresponds to a hue histogram of the main histogram.  FIG. 11  illustrates an example of the adjacent histogram, which corresponds to a luminance histogram of the main histogram. 
     In the adjacent histogram, as information about the distribution of the adjacent pixel, the system controller  107  can extract a luminance (“Y”), a saturation (“Chroma”), and a hue (“Hue”) from the chromatic color (“Color”) pixel and can acquire the distribution of each of them. Furthermore, the system controller  107  can acquire the distribution of the luminance Y based on an achromatic color (“Colorless”) pixel. 
     After the tracking target object recognition processing including acquisition of the main histogram and the adjacent histogram illustrated in the flow chart of  FIG. 4  ends, then in step S 103  ( FIG. 3 ), the system controller  107  executes processing for determining whether the tracking target object can be tracked. 
       FIG. 12 , composed of  FIGS. 12A and 12B , is a flowchart illustrating an example of trackability determination processing in step S 103  in the flow chart illustrated in  FIG. 3 . 
     Referring to  FIG. 12 , in steps S 301  and S 302 , the system controller  107  generates a main histogram including the hue histogram and the luminance histogram for the pixels existing inside or outside the tracking target area in the manner as described above with reference to  FIG. 4 ,  FIGS. 6A and 6B , and  FIGS. 7A and 7B . 
     In the exemplary flow chart illustrated in  FIG. 12 , “aHistColorTarget[m]” denotes the number of pixels existing inside the tracking target area (“In”) for each hue (“Hue”) in the hue histogram  400  illustrated in  FIG. 6A . In addition, “aHistYTarget[n]” denotes the number of pixels existing inside the tracking target area (“In”) for each luminance Y in the luminance histogram  440  illustrated in  FIG. 7A . Furthermore, “aHistColorTarget[m]” denotes the number of pixels existing outside the tracking target area (“Out”) for each hue (“Hue”) in the hue histogram  400  illustrated in  FIG. 6A . Moreover, “aHistYOther[n]” denotes the number of pixels existing outside the tracking target area (“Out”) for each luminance Y in the luminance histogram  440  illustrated in  FIG. 7A . In addition, the system controller  107  generates the average hue histogram illustrated in  FIG. 6B  and the average luminance histogram illustrated in  FIG. 7B  based on the values described above. 
     In step S 303 , the system controller  107  determines whether pixels are distributed inside the tracking target area in the division unit included in the hue histogram. In the present exemplary embodiment, “aHistColorTarget[i]” used in step S 303  denotes a value of the number of pixels existing inside the tracking target area (“In”) corresponding to a specific hue (“Hue”) (i) in the hue histogram  400 . More specifically, in step S 303 , the system controller  107  determines whether aHistColorTarget[i]&gt;0. 
     If one or more pixels having the hue (i) exist inside the tracking target area (YES in step S 303 ), then the processing advances to step S 304 . In step S 304 , the system controller  107  calculates the existence ratio for the inside or the outside of the tracking target area for the hue (“Hue”) (i). 
     The existence ratio of the hue inside or outside the tracking target area is calculated as described above with reference to  FIGS. 6A and 7A . More specifically, the system controller  107  calculates the existence ratio by multiplying a result of dividing the number of pixels existing inside the tracking target area (“In”) for the hue (“Hue”) (i) (aHistColorTarget[i]) by the number of pixels existing outside the tracking target area (“Out”) (aHistColorOther[i]) by 100. However, if aHistColorTarget[i]≧1 and if aHistColorOther=0, then the existence ratio is 100%. 
     In step S 305 , the system controller  107  determines whether the existence ratio is higher than a ratio, which is a predetermined hue employment condition. If it is determined that the existence ratio is higher than the ratio, which is the predetermined hue employment condition (NO in step S 305 ), then it is determined that the same color as the color of the tracking target object largely exists in the background and that if the tracking using the color is executed, the accuracy of the tracking may be low. Accordingly, in step S 306 , the system controller  107  determines not to employ the hue (i) to be processed in order to achieve a high tracking accuracy. 
     On the other hand, if it is determined that the existence ratio is higher than the ratio that is the predetermined hue employment condition, then the processing advances to step S 307 . In step S 307 , the system controller  107  determines that the hue (i) to be processed is to be employed. 
     In step S 308 , the system controller  107  stores “SumOKTarget” and “SumOKOther” of the processing target hue (i). In the present exemplary embodiment, “SumOKTarget” denotes the total number of the number of pixels existing inside the tracking target area (“aHistColorTarget[i]”). Furthermore, in the present exemplary embodiment, “SumOKOther” denotes the total number of the number of pixels existing outside the tracking target area (“aHistColorOther[i]”). In step S 309 , the system controller  107  repeats the processing in steps S 303  through S 308  for all the hues (i). 
     As described above, in the present exemplary embodiment, the system controller  107  determines whether to employ the hue (i) in all the division units in the hue histogram. In addition, the system controller  107  stores the total sum of the number of pixels employed. 
     Furthermore, the system controller  107  determines whether the existence ratio of pixels inside and outside the tracking target area is higher than the ratio that is the predetermined luminance employment condition, with respect to the luminance histogram. In steps S 310  through S 316 , the system controller  107  determines whether to employ the luminance Y (i) in the division unit in the luminance histogram. 
     “SumOKTarget”, which denotes the total number of pixels existing inside the tracking target area, and “SumOKOther”, which denotes the total number of pixels existing outside the tracking target area, are values calculated by adding the number of pixels having the employed luminance Y (i) to the total sum of the number of pixels having the employed hue (i), respectively. 
     Accordingly, the resulting “SumOKTarget”, which denotes the total number of pixels existing inside the tracking target area, is the total sum of the number of pixels having the employed hue (i). Furthermore, the resulting “SumOKOther”, which denotes the total number of pixels existing outside the tracking target area, is the total sum of the number of pixels having the employed luminance Y (i). 
     In the present exemplary embodiment, the “hue employment condition” and the “luminance employment condition” can be set independently from each other. In particular, it is less easy to use an achromatic color, than a chromatic color, as a characteristic of a tracking target object. Accordingly, for the achromatic color, an employment condition stricter than the hue employment condition can be set for the luminance employment condition. 
     In addition, although not illustrated in the drawing, it is also useful if the employment condition is changed for only a specific color or luminance instead of setting the employment conditions of chromatic color or achromatic color. In addition, it is also useful if either the processing in steps S 303  through S 309  or the processing in step S 310  through S 316  is not to be executed. 
     After executing the above-described employment determination processing for in the division unit in the hue histogram and the luminance histogram (the main histogram), the processing advances to step S 317 . In step S 317 , the system controller  107  determines the ratio of the employed chromatic color pixel or achromatic color pixel existing inside the tracking target area. 
     More specifically, the system controller  107  calculates the ratio by multiplying a value calculated by dividing “SumOKTarget”, which is the employed total number of pixels existing inside the tracking target area, by the number of pixels existing within the frame by 100. 
     In step S 318 , the system controller  107  determines whether the ratio is higher than a predetermined within-frame condition. If it is determined that the ratio is not higher than the predetermined within-frame condition (NO in step S 318 ), then the processing advances to step S 319 . In step S 319 , the system controller  107  determines that no appropriate characteristic of the tracking target object has been acquired and that the tracking target object cannot be tracked. 
     On the other hand, if it is determined that the ratio is higher than the within-frame condition (YES in step S 318 ), then the system controller  107  determines that an appropriate characteristic of the tracking target object has been acquired. Then the processing advances to step S 320 . In step S 320 , the system controller  107  calculates the ratio of existence of the employed chromatic color pixel or achromatic color pixel within the screen. 
     More specifically, the system controller  107  calculates the ratio in the following manner. At first, the system controller  107  adds SumOKTarget, which is the total number of pixels existing inside the tracking target area to SumOKOther, which is the total number of pixels existing outside the tracking target area. Then, the system controller  107  divides the result of the addition by the number of pixels existing within the screen. Then, the system controller  107  multiplies the result of the division by 100. 
     In step S 321 , the system controller  107  determines whether the ratio calculated in the above-described manner is lower than a predetermined entire-screen condition B. If it is determined that the ratio is not lower than the predetermined entire-screen condition B (NO in step S 321 ), then the processing advances to step S 319 . In step S 319 , the system controller  107  determines that that tracking target object cannot be tracked because the tracking performance may degrade due to a large number of chromatic color pixels or achromatic color pixels to be tracked existing in the background area, which is outside the tracking target area. 
     On the other hand, if it is determined that the ratio is lower than the predetermined entire-screen condition B (YES in step S 321 ), then the processing advances to step S 322 . In step S 322 , the system controller  107  determines that the tracking target object can be tracked. 
     As described above, the system controller  107  determines whether the tracking target object can be tracked in a plurality of stages including single-attribute evaluation processing in relation to each color and luminance, the evaluation processing as to the pixels existing inside the tracking target area, and the evaluation processing as to the pixels included in the entire screen. Accordingly, the system controller  107  according to the present exemplary embodiment can securely and appropriately determine whether a tracking target object can be tracked. However, it is not always necessary to execute the entire-screen evaluation in steps S 320  and S 321 . 
     Returning to  FIG. 3 , in step S 104 , the system controller  107  determines whether the tracking target area can be tracked based on the result of the trackability determination processing described above. If it is determined that the tracking target area cannot be tracked (NO in step S 104 ), then the processing according to the flow chart of  FIG. 3  ends. On the other hand, if it is determined that the tracking target area can be tracked (YES in step S 104 ), then tracking loop processing starts. 
     In the tracking loop processing, in step S 105 , the system controller  107  determines whether continued tracking is available.  FIG. 13  is a flow chart illustrating an example of continued tracking availability determination processing in step S 105  in the flow chart illustrated in  FIG. 3 . 
     Referring to  FIG. 13 , in step S 401 , the system controller  107  calculates a ratio of chromatic color pixels and achromatic color pixels employed in the histograms illustrated in  FIGS. 6A and 7A  that exist within the screen. More specifically, the system controller  107  calculates the ratio by multiplying a value calculated by dividing “SumOKALL”, which is the total number of chromatic color pixels and achromatic color pixels employed in the histograms illustrated in  FIGS. 6A and 7A , by the number of pixels included in the entire screen, by 100. In the present exemplary embodiment, the total number of pixels “SumOKALL” is a value calculated by adding “SumOKTarget”, which is the total number of pixels existing inside the tracking target area, to “SumOKOther”, which is the total number of pixels existing outside the tracking target area. 
     In step S 402 , the system controller  107  determines whether the ratio is lower than a predetermined entire screen condition C. If it is determined that the ratio is not lower than the predetermined entire screen condition C (NO in step S 402 ), then the processing advances to step S 403 . In step S 403 , the system controller  107  determines that the tracking target object cannot be tracked because the tracking performance may degrade due to a large number of chromatic color pixels or achromatic color pixels to be tracked existing in the background area, which is outside the tracking target area. 
     On the other hand, if it is determined that the ratio is lower than the predetermined entire-screen condition C (YES in step S 402 ), then the processing advances to step S 404 . In step S 404 , the system controller  107  determines that the tracking target object can be tracked. In the present exemplary embodiment, the “entire screen condition C” can be set as a condition different from the “entire screen condition B”, which is a final determination condition for determining whether the tracking target object can be tracked, which is processing in step S 321  described above with reference to  FIG. 12 . 
     By using the entire screen condition C, which is stricter than the entire screen condition B, the system controller  107  according to the present exemplary embodiment can determine whether to immediately suspend the tracking when the status of the background has changed after the start of the tracking and thus the number of pixels having the same hue or luminance as that of the tracking target object has increased. 
     In the example illustrated in  FIG. 13 , the condition is fixed at the entire screen condition C. However, it is also useful if the condition is gradually changed (narrowed down) in accordance with the time elapsed after the start of tracking. 
     In addition, it is also useful if the “within-frame condition” used in step S 318  ( FIG. 12 ), the “entire screen condition B” used in step S 321  ( FIG. 12 ), and the “entire screen condition C” used in step S 402  ( FIG. 13 ) are narrowed down for each case where the object is tracked based on the hue only, where the object is tracked based on the luminance only, and where the object is tracked based on both the hue and the luminance. In addition, it is also useful if the condition for a specific color only and the condition for the specific luminance only are further changed. 
     As described above, in the present exemplary embodiment, by executing the processing in steps S 301  and S 302 , an example of a storage unit is implemented. In addition, in the present exemplary embodiment, by executing the processing in step S 301 , an example of a first histogram storage unit is implemented. Furthermore, in the present exemplary embodiment, by executing the processing in step S 302 , an example of a second histogram storage unit is implemented. 
     In the present exemplary embodiment, for a chromatic color pixel, the hue thereof implements an example of the predetermined attribute while for an achromatic color pixel, the luminance implements an example of the predetermined attribute. In addition, in the present exemplary embodiment, by executing the processing in steps S 304  and S 312 , an example of a first derivation unit is implemented. In the present exemplary embodiment, for a chromatic color pixel, the hue employment condition implements an example of a first predetermined value while for an achromatic color pixel, the luminance employment condition implements an example of the first predetermined value. 
     In addition, in the present exemplary embodiment, by executing the processing in step S 317 , an example of a second derivation unit is implemented while the within-frame condition implements an exemplary second predetermined value. Furthermore, by executing the processing in at least either one of the processing in step S 320  and the processing in step S 401 , an example of a third derivation unit is implemented. Moreover, the entire screen conditions B and C implement an example of a third predetermined value. 
     Returning to  FIG. 3 , in step S 106 , the system controller  107  determines whether the tracking of the tracking target area can be continued based on the result of the continued tracking availability determination processing. If it is determined that the tracking of the tracking target area cannot be continued (NO in step S 106 ), then the processing illustrated in the flow chart of  FIG. 3  ends. On the other hand, if it is determined that the tracking of the tracking target area can be continued (YES in step S 106 ), then the processing advances to step S 107 . In step S 107 , the system controller  107  executes tracking target object detection processing. 
       FIG. 14  is a flow chart illustrating an example of tracking target object detection processing in step S 107  in the flow chart illustrated in  FIG. 3 . The tracking target object detection processing can be executed either on the entire screen or only on an arbitrary area within the screen. However, in the present exemplary embodiment, it is supposed that the tracking target object detection processing is executed on the entire screen. 
       FIG. 15  schematically illustrates an example of a method for tracking a tracking target area by executing tracking target object detection processing. Various starting points, directions, or orders can be set in tracking a tracking target object within the screen. However, in the present exemplary embodiment, it is supposed that the tracking of the tracking target object is executed horizontally starting with the upper-left portion of the screen as illustrated in  FIG. 15 . Then, the tracking is serially executed on lines below. 
     Referring to  FIG. 14 , in step S 501 , the system controller  107  determines whether the pixel to be processed (pixel to be evaluated) matches the pixel distributed in the histogram. More specifically, in step S 501 , the system controller  107  determines whether the pixel to be processed is a chromatic color pixel or an achromatic color pixel stored in the main histogram illustrated in the flow chart of  FIGS. 6A and 7A  as a pixel existing inside the tracking target area. In addition, in step S 501 , the system controller  107  determines whether the pixel to be processed has the average luminance, average saturation, or the average hue, which have been stored in the average histogram illustrated in  FIGS. 6A and 7A , that fall within a predetermined range thereof. 
     If it is determined that the pixel to be processed does not match the pixel distributed in the main histogram (NO in step S 501 ), then the processing advances to step S 506 . On the other hand, if it is determined that the pixel to be processed matches the pixel distributed in the main histogram (YES in step S 501 ), then the processing advances to step S 502 . In step S 502 , the system controller  107  allocates rating points to the location of the pixel. 
       FIGS. 16A through 16D  schematically illustrate an example of a method of allocating a rating point to a pixel to be searched for to detect the tracking target object. Referring to  FIG. 16A , if a pixel to be processed (processing target pixel)  650  matches the chromatic color in the main histogram, then the system controller  107  allocates 1 rating point to the processing target pixel  650 . 
     On the other hand, as illustrated in  FIG. 16B , if a processing target pixel  660  matches the achromatic color in the main histogram, then the system controller  107  allocates zero point to the processing target pixel  660  (does not allocate a rating point to the processing target pixel  660 ). 
     Alternatively, it is also useful to employ the following configuration. More specifically, in this case, if the processing target pixel  650  matches the chromatic color in the main histogram, the system controller  107  allocates 2 rating points to the processing target pixel  650 . On the other hand, in this case, if the processing target pixel  650  matches the achromatic color in the main histogram, the system controller  107  allocates 1 point, which is a rating point smaller than that allocated in the case of the chromatic color. 
     In step S 503 , the system controller  107  determines whether the pixel existing adjacent to the processing target pixel matches the pixel distributed in the adjacent histogram. More specifically, with respect to the pixel matching the main histogram, the system controller  107  determines whether eight pixels existing around a processing target pixel  610  match the adjacent histograms illustrated in  FIGS. 10 and 11 . 
     If it is determined that the pixel existing adjacent to the processing target pixel does not match the pixel distributed in the adjacent histogram (NO in step S 503 ), then the processing advances to step S 505 . On the other hand, if it is determined that the pixel existing adjacent to the processing target pixel matches the pixel distributed in the adjacent histogram (YES in step S 503 ), then the processing advances to step S 504 . In step S 504 , the system controller  107  allocates rating points to the location of the pixel. 
     More specifically, if, as illustrated in  FIG. 16C , a chromatic color pixel  652 , which is distributed in the adjacent histogram and having a hue different from the hue of a chromatic color pixel matching the chromatic color in the main histogram exists adjacent to the chromatic color pixel, then the system controller  107  allocates 5 points. 
     On the other hand, if, as illustrated in  FIG. 16C , a chromatic color pixel  654 , which is distributed in the adjacent histogram and having the same hue as the hue of a chromatic color pixel matching the chromatic color in the main histogram exists adjacent to the chromatic color pixel, then the system controller  107  allocates 4 points. 
     Furthermore, if, as illustrated in  FIG. 16C , an achromatic color pixel  656 , which is distributed in the adjacent histogram exists adjacent to the chromatic color pixel matching the chromatic color in the main histogram, then the system controller  107  allocates 3 points. 
     On the other hand, if, as illustrated in  FIG. 16D , a chromatic color pixel  662 , which is distributed in the adjacent histogram exists adjacent to the chromatic color pixel matching the chromatic color in the main histogram, then the system controller  107  allocates 3 points. 
     Furthermore, if, as illustrated in  FIG. 16D , an achromatic color pixel  664  having a luminance different from the luminance of an achromatic color pixel matching the achromatic color in the main histogram exists adjacent thereto, then the system controller  107  allocates 2 points. On the other hand, if, as illustrated in  FIG. 16D , an achromatic color pixel  666  having the same luminance as the luminance of an achromatic color pixel matching the achromatic color in the main histogram exists adjacent thereto, then the system controller  107  allocates 2 points. 
     In determining whether a pixel existing adjacent to a processing target pixel matches any of the pixels distributed in the adjacent histograms illustrated in  FIGS. 10 and 11 , it is also useful if the determination is executed based on the hue only if the adjacent pixel is a chromatic color pixel. Alternatively, in this case, it is also useful if the determination is executed based on all of the hue, the saturation (“Chroma”), and the luminance Y. 
     In the examples illustrated in  FIGS. 16A through 16D , the rating points to be allocated are changed according to the combination of the processing target pixel and the pixel existing adjacent thereto. However, it is also useful if the same rating points are allocated in all of the above-described cases. Furthermore, it is also useful if the rating points different from those described above are allocated. 
     As described above with reference to  FIG. 15  and  FIGS. 16A through 16D , in the present exemplary embodiment, the system controller  107  evaluates the relationship among the pixel to be processed that exists within the screen and the pixels existing adjacent to the pixel. In addition, the system controller  107  determines the matching status between the processing target pixel and the main histogram and the matching status between the adjacent pixel and the adjacent histogram. If it is determined that they match each other, the system controller  107  allocates rating points to the location of the evaluated pixel. In step S 505 , the system controller  107  determines whether the above-described processing has been completely executed on all the adjacent pixels. 
     If it is determined that the above-described processing has been completely executed on all the adjacent pixels (YES in step S 505 ), then the processing advances to step S 506 . In step S 506 , the system controller  107  calculates an evaluation value for each location of the evaluated pixel by using the allocated rating points and the average luminance, average saturation, and average hue in each division unit in the main histogram, which have been stored in the manner described above with reference to  FIGS. 6B and 7B . 
     More specifically, if the evaluated pixel is a chromatic color pixel, the system controller  107  uses the following expression (1) to calculate the evaluation value while if the evaluated pixel is an achromatic color pixel, the system controller  107  uses the following expression (2) to calculate the evaluation value: 
                   EvaColor   =     ∑       Point   ×     [     K   ⁢           ⁢   0     ]                 Δ   ⁢           ⁢     Hue   ⁡     (     x   ,   y     )       ×     [     K   ⁢           ⁢   1     ]       +     Δ   ⁢           ⁢     Chroma   ⁡     (     x   ,   y     )       ×     [     K   ⁢           ⁢   2     ]       +               Δ   ⁢           ⁢     Distance   ⁡     (     x   ,   y     )       ×     [     K   ⁢           ⁢   3     ]                         (   1   )                 EvaY   =     ∑       Point   ×     [     K   ⁢           ⁢   0     ]           Δ   ⁢           ⁢     Y   ⁡     (     x   ,   y     )       ×     [     K   ⁢           ⁢   4     ]       +     Δ   ⁢           ⁢     Distance   ⁡     (     x   ,   y     )       ×     [     K   ⁢           ⁢   3     ]               ,           (   2   )               
where “ΔHue” denotes a difference between the hue of the evaluated pixel and the average hue, which has already been stored in the average histogram illustrated in  FIGS. 6B and 7B , “Δchroma” denotes a difference between the saturation of the evaluated pixel and the average saturation previously stored in the average histogram illustrated in  FIGS. 6B and 7B , “ΔY” denotes a difference between the luminance of the evaluated pixel and the average luminance previously stored in the average histogram illustrated in  FIGS. 6B and 7B , and “Δdistance” denotes a distance between the location of the frame that indicates the location of the tracking target object displayed by the display device  105  and the location of the evaluated pixel. If the accuracy of detecting the tracking target object is extremely high, the distance “Δdistance” is either unnecessary or can be very small.
 
     However, if a large number of pixels having the similar hue or luminance exist around the evaluated pixel, the phenomenon of hunting of the display of the frame, which may otherwise occur due to low accuracy of detecting a tracking target object, can be suppressed by adding a specific distance coefficient to the terms of the expressions (1) and (2). 
     In the expressions (1) and (2), terms of different unitary systems are added to each other. Accordingly, in order to keep the balance between them, coefficients “K0” through “K4” are used. Alternatively, in order to simplify the processing, it is also useful if a constant is used as the denominator of each of the expressions (1) and (2). This is because if the criterion of the determination in step S 501  is set strict, the range of variation of the values of “ΔHue” and “ΔChroma” in the denominator of each of the expressions (1) and (2) becomes small. 
     In step S 507 , the system controller  107  determines whether the calculation of the evaluation value for all the pixels existing within the screen has been completed. If it is determined that the calculation of the evaluation value for all the pixels existing within the screen has not been completed (NO in step S 507 ), then the processing returns to step S 501 . On the other hand, if it is determined that the calculation of the evaluation value for all the pixels existing within the screen has been completed (YES in step S 507 ), then the processing advances to step S 508 . 
     In step S 508 , the system controller  107  uses the calculated evaluation value to identify the location of the tracking target object. More specifically, it is useful if the system controller  107  searches for a location at which the calculated evaluation value comes to its peak in the unit of a pixel and sets the location as the location of the tracking target object. In addition, it is useful if the system controller  107  searches for the location of an aggregate of pixels having an evaluation value higher than a predetermined value and sets the location of the aggregation of the pixels as the location of the tracking target object. 
     However, if the location at which the evaluation value reaches a specific value cannot be determined, i.e., if the location of the tracking target object cannot be identified by using the calculated evaluation value, then the system controller  107  does not identify the location of the tracking target object. 
     If the location of the tracking target object has been identified, then the system controller  107  determines the location for displaying the frame according to the identified location of the tracking target object. If the location of the tracking target object has not been identified, the system controller  107  does not determine the location for displaying the frame. 
       FIG. 17  illustrates an example of a result of calculation of the evaluation value. In the example illustrated in  FIG. 17 , an evaluation value of the image illustrated in  FIG. 2B  in the case where the image has changed from the image illustrated in  FIG. 2A  to that illustrated in  FIG. 2B  is illustrated. 
     In the example illustrated in  FIG. 17 , coordinates of the image illustrated in  FIG. 2B  in the vertical direction and coordinates thereof in the horizontal direction are taken on the vertical axis and the horizontal axis, respectively. The evaluation value at each coordinate is taken on another axis perpendicular to a plane parallel to these axes. 
     In the example illustrated in  FIG. 2A , the image at the time the tracking target object is determined is illustrated. The system controller  107  recognizes the portion inside the area  202  as the tracking target area. In addition, the system controller  107  generates the main histogram and the adjacent histogram thereof. 
     In the example illustrated in  FIG. 2B , an image after the tracking target area has moved is illustrated. The evaluation value calculated in this state has the distribution illustrated in  FIG. 17 . As a result, the system controller  107  can display a frame that indicates the tracking target area in the area having a high evaluation value  612 . 
     In the present exemplary embodiment, in the example illustrated in  FIG. 17 , a plurality of areas having high evaluation values  620 ,  622 , and  624 , which are different from the evaluation value  612 , exists. 
     Suppose that a frame indicating the currently displayed tracking target area exists at the location having the evaluation value  620 . The evaluation value  620  is lower than the evaluation value  612 . Furthermore, the location of the pixel having the evaluation value  620  is closer to the pixel having the evaluation value  612 . Accordingly, the frame may have already been displayed on the tracking target object. 
     Accordingly, in this case, it is also useful if the system controller  107  does not change the location of displaying the frame. More specifically, it is also useful if the frame is displayed at the location of the evaluation value  620  instead of changing the location to the location of the evaluation value  612 . 
     Returning to  FIG. 3 , in step S 108 , the system controller  107  determines whether the tracking target area has been detected as a result of the processing in step S 508  illustrated in  FIG. 14 . If it is determined that the tracking target area has been detected (YES in step S 108 ), then the processing advances to step S 109 . In step S 109 , the system controller  107  displays the frame at the frame display location determined in step S 509  in  FIG. 14 . Then, the processing returns to step S 105 . 
     On the other hand, if the tracking target area has not been detected (NO in step S 108 ), then the processing advances to step S 110 . In step S 110 , the system controller  107  determines whether timeout has occurred (i.e., predetermined time has elapsed since a timing at which it is determined that the tracking target area has not been detected). 
     If it is determined that timeout has occurred (YES in step S 110 ), then the processing illustrated in the flow chart of  FIG. 3  ends. On the other hand, if it is determined that timeout has not occurred (NO in step S 110 ), then the processing returns to step S 105 . 
     As described above, for a chromatic color pixel, the system controller  107  according to the present exemplary embodiment calculates the ratio of existence of the pixels existing inside the tracking target area to the number of pixels existing outside the tracking target area in the division unit in the hue histogram as described above with reference to  FIG. 6A . 
     Similarly, for an achromatic color pixel, the system controller  107  according to the present exemplary embodiment calculates the ratio of existence of the pixels existing inside the tracking target area to the number of pixels existing outside the tracking target area in the division unit in the luminance histogram as described above with reference to  FIG. 7A . 
     If the existence ratio is higher than a threshold value, the system controller  107  employs the pixel belonging to the division unit in the hue histogram and the luminance histogram. If the employed existence ratio of the total number of pixels existing inside the tracking target area to the number of pixels existing within the frame corresponding to the tracking target area is smaller than the threshold value, then the system controller  107  according to the present exemplary embodiment determines that the tracking target area cannot be tracked. 
     On the other hand, if the employed existence ratio of the total number of pixels existing inside the tracking target area to the number of pixels existing within the frame corresponding to the tracking target area is larger than the threshold value, then the system controller  107  according to the present exemplary embodiment determines whether the employed existence ratio of the total number of pixels to the number of pixels included in the entire screen is smaller than a threshold value. 
     If it is determined that the employed existence ratio of the total number of pixels to the number of pixels included in the entire screen is smaller than the threshold value, then the system controller  107  determines that the tracking target area can be tracked. On the other hand, if it is determined that the employed existence ratio of the total number of pixels to the number of pixels included in the entire screen is not smaller than the threshold value, then the system controller  107  determines that the tracking target area cannot be tracked. 
     Accordingly, the present exemplary embodiment can execute tracking of the tracking target area based on a result of the appropriate determination as to whether the tracking target area can be tracked. With the above-described configuration, the present exemplary embodiment can reduce the probability of wrong tracking of the tracking target area. 
     In addition, the present exemplary embodiment having the configuration described above is capable of appropriately determining whether the tracking target area can be tracked in order to improve the tracking performance even when a pixel having the attribute similar to that of a pixel included in the tracking target area exists in the background of the image. 
     Furthermore, with the above-described configuration, the present exemplary embodiment is capable of determining whether the tracking target area can be tracked within a very short time period because the memory capacity used for determining whether the tracking target area can be tracked is small. 
     Now, a second exemplary embodiment of the present invention will be described in detail below. In the present exemplary embodiment, the tracking target object is tracked by an interframe difference method if the method is available. On the other hand, in the present exemplary embodiment, if tracking by the interframe difference method is not available, the tracking target object is tracked in the manner similar to the manner described above in the first exemplary embodiment. 
     In other words, the present exemplary embodiment is different from the first exemplary embodiment described above in a point that the present exemplary embodiment uses the interframe difference method. The components of the present exemplary embodiment that are the same as those of the first exemplary embodiment are provided with the same reference numerals or symbols as those of the first exemplary embodiment described above with reference to  FIGS. 1 through 17 . Accordingly, the description thereof will not be repeated here. 
       FIG. 18  is a main flow chart illustrating an example of processing for tracking a tracking target object, which is executed by the imaging apparatus according to the present exemplary embodiment. 
     Referring to  FIG. 18 , in step S 1001 , the system controller  107  determines the location and the size of the tracking target object. In step S 1002 , the system controller  107  executes the tracking target object recognition processing. In step S 1003 , the system controller  107  executes trackability determination processing. In steps S 1001  through S 1003 , the system controller  107  executes the processing similar to that in steps S 101  through S 103  illustrated in the flow chart of  FIG. 3 . 
     In step S 1004 , the system controller  107  tracks the tracking target area by using the interframe relative difference method. During the interframe relative difference method, the system controller  107  uses image data existing at a designated tracking location as a template image and searches for an area to be subjected to pattern matching from a subsequent image frame. 
     In addition, the system controller  107  updates the image data existing at the location detected as the area to be subjected to pattern matching as a template image. Furthermore, the system controller  107  continues the search for the area to be subjected to pattern matching from the subsequent frame image. 
     As described above, in the present exemplary embodiment, the system controller  107  continues the tracking of the tracking target object from a subsequent frame image while serially updating the template images. When the interframe relative difference method is used, the pattern matching cannot be executed and it is determined that the tracking target object cannot be tracked in some cases. 
     Accordingly, in step S 1005 , the system controller  107  determines whether tracking by the interframe relative difference method is available. More specifically, if the orientation of the tracking target object has suddenly changed due to rotation thereof or if the attitude of the tracking apparatus itself in the vertical or horizontal direction has varied, and if pattern matching cannot be executed due to these causes, the tracking by the interframe relative difference method becomes unavailable. 
     If it is determined that tracking by the interframe relative difference method is available (YES in step S 1005 ), then the processing returns to step S 1004  and the system controller  107  continues the tracking of the tracking target object by the interframe relative difference method. On the other hand, if it is determined that the tracking by the interframe relative difference method is not available (NO in step S 1005 ), then the processing advances to step S 1006 . 
     In this case, in step S 1006 , the system controller  107  executes continued tracking availability determination processing. In step S 1007 , the system controller  107  determines whether the tracking of the tracking target object can be continued. In step S 1008 , the system controller  107  executes tracking target object detection processing. More specifically, in step S 1009 , the system controller  107  determines whether the tracking target object has been detected. 
     If it is determined that the tracking target object has been detected (YES in step S 1009 ), then the processing advances to step S 1010 . In step S 1010 , the system controller  107  displays the frame at the frame display location. Then, the processing returns to step S 1004 . 
     In step S 1004 , the system controller  107  resumes the tracking of the tracking target object by the interframe relative difference method from the tracking target area detected in step S 1008 . On the other hand, if the tracking target area has not been detected, then the processing advances to step S 1011 . In step S 1011 , the system controller  107  determines whether timeout has occurred. If it is determined that timeout has occurred (YES in step S 1011 ), then the processing according to the flow chart of  FIG. 18  ends. On the other hand, if it is determined that timeout has not occurred (NO in step S 1011 ), then the processing returns to step S 1006 . 
     The processing in steps S 1006  through S 1011  is similar to the processing in steps S 105  through S 110  illustrated in the flow chart of  FIG. 3 . 
     As described above, in the processing according to the flow chart illustrated in  FIG. 18 , the tracking by the interframe relative difference method and the tracking described above in the first exemplary embodiment are not simultaneously executed. However, the present invention is not limited to this. More specifically, it is also useful if both processing is simultaneously executed in parallel to each other. In this case, the system controller  107  determines which of the processing has a high accuracy and uses the result of the processing whose accuracy is higher to execute the tracking of the tracking target object. In addition, it is also useful if any other tracking method, such as those described above in Description of the Related Art, is used instead of the interframe relative difference method. 
     Furthermore, the present invention can also be achieved by providing a system or a device with a storage medium (or a recording medium) which stores program code of software implementing the functions of the embodiments and by reading and executing the program code stored in the storage medium with a computer of the system or the device (a CPU or an MPU). In this case, the program code itself, which is read from the storage medium, implements the functions of the embodiments mentioned above. In addition, the program that implements an exemplary embodiment of the present invention can be stored on a computer-readable storage medium. 
     While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all modifications, equivalent structures, and functions. 
     This application claims priority from Japanese Patent Application No. 2009-183769 filed Aug. 6, 2009, which is hereby incorporated by reference herein in its entirety.