An auto focus system having an AF frame auto-tracking function, comprising: an auto focus unit that controls the focus of an imaging optical system which focuses an imaging unit on an object such that an object in a portion corresponding to an AF frame in the range of the object captured by the imaging unit is in focus; a face detecting unit that detects the image of a person's face from the image captured by the imaging unit; a pattern matching unit that detects the image of an object, which is a tracking target, from the image captured by the imaging unit, using a pattern matching process, on the basis of a reference pattern image that is stored as the image of the object, which is the tracking target; a first AF frame determining unit that, when the image of the person's face is include in the AF frame, determines the position of the AF frame in the image captured by the imaging unit on the basis of the position of the image of the face detected by the face detecting unit; a second AF frame determining unit that determines the position of the AF frame in the image captured by the imaging unit on the basis of the position of the image of the object, which is the tracking target, detected by the pattern matching unit, wherein one of said AF frame determining units is set as a preferential AF frame determining unit; and an AF frame changing unit that changes the position of the AF frame to a position determined by the preferential AF frame determining unit, and when the determination of the position of the AF frame by the preferential AF frame determining unit is unavailable, changes the position of the AF frame to a position determined by the other AF frame determining unit. an auto focus unit that controls the focus of an imaging optical system which focuses an imaging unit on an object such that an object in a portion corresponding to an AF frame in the range of the object captured by the imaging unit is in focus; a face detecting unit that detects the image of a person's face from the image captured by the imaging unit; a pattern matching unit that detects the image of an object, which is a tracking target, from the image captured by the imaging unit, using a pattern matching process, on the basis of a reference pattern image that is stored as the image of the object, which is the tracking target; a first AF frame determining unit that, when the image of the person's face is include in the AF frame, determines the position of the AF frame in the image captured by the imaging unit on the basis of the position of the image of the face detected by the face detecting unit; a second AF frame determining unit that determines the position of the AF frame in the image captured by the imaging unit on the basis of the position of the image of the object, which is the tracking target, detected by the pattern matching unit, wherein one of said AF frame determining units is set as a preferential AF frame determining unit; and an AF frame changing unit that changes the position of the AF frame to a position determined by the preferential AF frame determining unit, and when the determination of the position of the AF frame by the preferential AF frame determining unit is unavailable, changes the position of the AF frame to a position determined by the other AF frame determining unit.