An image processing apparatus comprising: setting means configured to set a partial area of a first image as a tracking target area; data generation means configured to generate classification data based on the first image, wherein the data generation means generates, a first pixel being any of the pixels included in the tracking target area and a second pixel being any of the pixels surrounding the first pixel, combinations of luminance or hue of the first pixel and luminance or hue of each of the second pixels, for each of the pixels included in the tracking target area, as the classification data; allocation means configured to allocate an evaluation value to each pixel included in a second image by using the classification data, wherein, a third pixel being any of the pixels included in the second image and a fourth pixel being any of the pixels surrounding the third pixel, the allocation means allocates the evaluation value to the third pixel in the case that a combination of luminance or hue of the third pixel and luminance or hue of any of the fourth pixels is included in the classification data; and wherein the evaluation value allocated to the third pixel in the case that a combination of hue of the third pixel and hue of any of the fourth pixels is included in the classification data is larger than the evaluation value allocated to the third pixel in the case that a combination of luminance of the third pixel and luminance of any of the fourth pixels is included in the classification data; detection means configured to detect an area of the second image in which the same object as the tracking target area exists, based on each of the evaluation values allocated by the allocation means, by searching for an area at which the evaluation value comes to its peak in the unit of a pixel; changing means configured to change position data of the tracking target area set by the setting means to position data of the area detected by the detection means; and display means configured to display a frame that indicates the position of the tracking target area based on the position data changed by the changing means. setting means configured to set a partial area of a first image as a tracking target area; data generation means configured to generate classification data based on the first image, wherein the data generation means generates, a first pixel being any of the pixels included in the tracking target area and a second pixel being any of the pixels surrounding the first pixel, combinations of luminance or hue of the first pixel and luminance or hue of each of the second pixels, for each of the pixels included in the tracking target area, as the classification data; allocation means configured to allocate an evaluation value to each pixel included in a second image by using the classification data, wherein, a third pixel being any of the pixels included in the second image and a fourth pixel being any of the pixels surrounding the third pixel, the allocation means allocates the evaluation value to the third pixel in the case that a combination of luminance or hue of the third pixel and luminance or hue of any of the fourth pixels is included in the classification data; and wherein the evaluation value allocated to the third pixel in the case that a combination of hue of the third pixel and hue of any of the fourth pixels is included in the classification data is larger than the evaluation value allocated to the third pixel in the case that a combination of luminance of the third pixel and luminance of any of the fourth pixels is included in the classification data; detection means configured to detect an area of the second image in which the same object as the tracking target area exists, based on each of the evaluation values allocated by the allocation means, by searching for an area at which the evaluation value comes to its peak in the unit of a pixel; changing means configured to change position data of the tracking target area set by the setting means to position data of the area detected by the detection means; and display means configured to display a frame that indicates the position of the tracking target area based on the position data changed by the changing means.