A control apparatus to control focal point adjustment operation of an imaging optical system, the control apparatus comprising: a calculation means configured to calculate a focal point evaluation value for each of a plurality of images captured successively by the imaging optical system; a determination means configured to determine whether an amount of change in the focal point evaluation value calculated by the calculation means with respect to a reference value is greater than or equal to a threshold; a count value calculation means for determining, as a count value, the number of times the amount of change is determined to be greater than or equal to the threshold; a means for determining, as a number of consecutive counts, the number of consecutive times the amount of change is determined to be greater than or equal to the threshold; a storage means configured to store the count value and the number of consecutive counts; and a control means (108, 111) configured to control the focal point adjustment operation of the imaging optical system to be executed in a case where the count value is greater than or equal to a first value, wherein: when determining that a predetermined number of frames has passed and the number of consecutive counts is less than a predetermined number, the control means is configured to clear the count value and the number of consecutive counts, when determining that the predetermined number of frames has passed while the number of consecutive counts is greater than or equal to the predetermined number, the control means is not configured to clear the count value and the number of consecutive counts, and when determining that the count value is greater than or equal to the first value, the control means is configured to clear the count value and the number of consecutive counts. a calculation means configured to calculate a focal point evaluation value for each of a plurality of images captured successively by the imaging optical system; a determination means configured to determine whether an amount of change in the focal point evaluation value calculated by the calculation means with respect to a reference value is greater than or equal to a threshold; a count value calculation means for determining, as a count value, the number of times the amount of change is determined to be greater than or equal to the threshold; a means for determining, as a number of consecutive counts, the number of consecutive times the amount of change is determined to be greater than or equal to the threshold; a storage means configured to store the count value and the number of consecutive counts; and a control means (108, 111) configured to control the focal point adjustment operation of the imaging optical system to be executed in a case where the count value is greater than or equal to a first value, wherein: when determining that a predetermined number of frames has passed and the number of consecutive counts is less than a predetermined number, the control means is configured to clear the count value and the number of consecutive counts, when determining that the predetermined number of frames has passed while the number of consecutive counts is greater than or equal to the predetermined number, the control means is not configured to clear the count value and the number of consecutive counts, and when determining that the count value is greater than or equal to the first value, the control means is configured to clear the count value and the number of consecutive counts. when determining that a predetermined number of frames has passed and the number of consecutive counts is less than a predetermined number, the control means is configured to clear the count value and the number of consecutive counts, when determining that the predetermined number of frames has passed while the number of consecutive counts is greater than or equal to the predetermined number, the control means is not configured to clear the count value and the number of consecutive counts, and when determining that the count value is greater than or equal to the first value, the control means is configured to clear the count value and the number of consecutive counts.