Patent Publication Number: US-2021195119-A1

Title: Image processing apparatus, image capturing apparatus and image processing method

Description:
BACKGROUND OF THE INVENTION 
     Field of the Invention 
     The present invention relates to an image processing technique for correcting a tilt of a captured image. 
     Description of the Related Art 
     Japanese Patent Laid-Open No. 2017-58660 discloses a technique that detects a tilt angle of an image capturing apparatus such as a digital camera with a shake sensor such as an acceleration sensor, and electronically rotates a part (clipping area) of a captured image depending on the detected tilt angle with enlargement thereof to acquire a tilt-corrected image. 
     However, in such tilt correction of the captured image, a large tilt correction angle makes the clipping area small, which increases an enlargement amount of the clipping area. Thereby, a resolution of the tilt-corrected image is lowered. 
     On the other hand, the degree of necessity of the tilt correction differs depending on objects to be captured. For example, when the object is a building or a landscape, the degree of necessity of the tilt correction is high, but when the object is a plant or a face, the degree of necessity of the tilt correction is low. 
     SUMMARY OF THE INVENTION 
     The present invention provides an image processing apparatus, an image capturing apparatus and an image processing method that are capable of appropriately performing tilt correction on a captured image depending on objects. 
     The present invention provides as an aspect thereof an image processing apparatus including at least one processor or circuit configured to execute a plurality of tasks. The tasks includes a correction task configured to perform, using information on a tilt of an image capturing apparatus around an optical axis, tilt correction on image data generated by the image capturing apparatus, and an acquiring task configured to acquire information on an object included in the image data. The correction task is configured to set a tilt correction amount in the tilt correction depending on the information on the object. 
     The present invention provides as another aspect thereof an image capturing apparatus including the above-described image processing apparatus, and an image sensor configured to capture an object image. 
     The present invention provides as still another aspect thereof an image processing method including the step of performing, using information on a tilt of an image capturing apparatus around an optical axis, tilt correction on image data generated by the image capturing apparatus, and the step of acquiring information on an object included in the image data. The method sets a tilt correction amount in the tilt correction depending on the information on the object. 
     The present invention provides as yet another aspect thereof a non-transitory computer-readable storage medium for storing a computer program to cause a computer to execute a process according to the above-described image processing method. 
     Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a camera shake in Embodiment 1 of the present invention. 
         FIG. 2  is a block diagram illustrating a configuration of a camera in Embodiment 1. 
         FIG. 3  is a flowchart of a tilt correction amount switching process in Embodiment 1. 
         FIG. 4  is a flowchart of a tilt correction amount switching process in Embodiment 2 of the present invention. 
         FIG. 5  is a flowchart of an automatic image capturing process in Embodiment 3 of the present invention. 
         FIG. 6  illustrates tilt correction in Embodiment 1. 
     
    
    
     DESCRIPTION OF THE EMBODIMENTS 
     Exemplary embodiments of the present invention will hereinafter be described with reference to the accompanying drawings. 
     Embodiment 1 
       FIG. 1  illustrates an external view of a digital still camera (hereinafter referred to as “a camera”)  101  as an image capturing apparatus in a first embodiment (Embodiment 1) of the present invention.  FIG. 1  further illustrates directions of tilt and shake of the camera  101  (the shake is hereinafter referred to as “camera shake”). An optical axis  102  of an image capturing optical system of the camera  101  is defined as a Z axis, and a direction in which the Z axis extends is defined as a Z direction. A direction  103   r  around the Z axis is defined as a roll direction. Of the two axes orthogonal to the Z axis and orthogonal to each other, one is defined as an X axis and the other is defined as a Y axis. A direction in which the X axis extends is defined as an X direction, a direction in which the Y axis extends is defined as a Y direction, a direction around the X axis  103   p  is defined as a pitch direction, and a direction  103   y  around the Y axis is defined as a yaw direction. As will be described in detail later, the camera  101  of this embodiment has a function of correcting, by image processing, a tilt of a captured image caused by the tilt of the camera  101  in the roll direction and a function of correcting blur of the captured image caused by the camera shake in the roll, pitch and yaw directions. 
     Although this embodiment describes the digital still camera as the image capturing apparatus, various image capturing apparatuses such as a digital video camera, a mobile phone, a tablet, a surveillance camera and a Web camera are also included in embodiments of the present invention. 
       FIG. 2  illustrates an internal configuration of the camera  101 . The image capturing optical system includes a magnification-varying lens  201 , an aperture stop/shutter unit  202 , and a focus lens  203  arranged in order from an object side (right side in the drawing). The magnification-varying lens  201  and the focus lens  203  respectively move in the Z direction (optical axis direction) to perform variation of magnification and focusing. The aperture stop/shutter unit  202  has a stop function for adjusting a light amount and a shutter function for controlling an exposure amount of an image sensor  204  described later. The image capturing optical system causes light from an object (not illustrated) to form an optical image (object image). 
     The image sensor  204  constituted by a CCD sensor or a CMOS sensor captures (photoelectrically converts) the object image formed by the image capturing optical system to output an analog image capturing signal to an image capturing signal processor  205 . 
     The image capturing signal processor  205  converts the analog image capturing signal from the image sensor  204  into image data as a digital signal to output it to an image signal processor  206  and a controller  215 . 
     The image signal processor  206  performs image processes such as a distortion correction process, a white balance adjustment process and a color interpolation process on the image data from the image capturing signal processor  205  to generate image data as a captured image. Further, the image signal processor  206  performs a tilt correction process that corrects (reduces) the tilt of the image data caused by the tilt of the camera  101  with respect to a horizontal direction or a gravity direction in the roll direction. Specifically, the image signal processor  206  performs a rotation process and an enlargement process on the image data as illustrated in  FIG. 6  depending on a tilt correction amount set by a correction amount switcher  219  described later. The processed image data is output to a format converter  207 . The tilt correction process is hereinafter simply referred to as “tilt correction”. 
     In  FIG. 6 , image data  301   a  is original image data generated by the image signal processor  206  before the tilt correction. Image data  302   a  is tilt-corrected image data generated by the tilt correction performed on the image data  301   a  by the image signal processor  206 . The image signal processor  206  clips, from the tilt-corrected image data  302   a , a partial area (clipping area) having the same aspect ratio as that of the original image data  301   a  to generate output image data  303   a . The image signal processor  206  further enlarges the output image data  303   a  to the same size as that of the original image data  301   a.    
     Furthermore, the image signal processor  206  performs a roll blur correction process for correcting (reducing) the blur of the image data (hereinafter referred to as “roll blur”) caused by the camera shake in the roll direction. Specifically, the image signal processor  206  performs the same process as that in the tilt correction described above depending on a blur correction amount calculated by a blur correction amount calculator  217  described later, and outputs the processed image data to the format converter  207 . The roll blur correction process is hereinafter simply referred to as “roll blur correction”. 
     The image signal processor  206  can also correct blur of the image data (shift blur) caused by the camera shake in the pitch and yaw directions detectable by a three-axis angular velocity meter  214  by clipping a part (clipping area) of the image data and enlarging the clipping area. However, such blur correction performed by the image signal processor  206  for correcting the blur of the image data caused by the camera shake in the roll, pitch and yaw directions can reduce blur between continuous images, but cannot reduce blur during exposure. 
     The format converter  207  converts the image data output from the image signal processor  206  into image data in a recording format such as a JPEG format to output the image data to an image recorder  208 . 
     The image recorder  208  records the image data in the recording format to a recording medium such as a non-volatile memory. A display controller  209  displays the image data generated by the image signal processor  206  on a display device such as a liquid crystal display element (LCD). 
     The camera  101  is provided with a three-axis accelerometer  213  as a tilt detector and the three-axis angular velocity meter  214  mentioned above as a shake detector. The three-axis accelerometer  213  detects an acceleration in each of the X, Y and Z directions to output an acceleration signal to the controller  215 . The three-axis angular velocity meter  214  detects an angular velocity in each of the pitch, yaw and roll directions to output an angular velocity signal to the controller  215 . The acceleration detected by the three-axis accelerometer  213  corresponds to information on the tilt of the camera  101 , and the angular velocity detected by the three-axis angular velocity meter  214  corresponds to information on the shake of the camera  101  (camera shake). 
     The controller  215  as a control unit is a computer constituted by a CPU, a memory and the like, and controls the entire camera  101 . A power unit  210  supplies power necessary for operations of the camera  101 . A communication unit  211  has a terminal for inputting/outputting a communication signal or a video signal to and from an external apparatus (not illustrated), and has a wireless module for performing wireless communication with the external apparatus. An operation unit  212  includes operation members that are operated by a user for making various settings of the camera  100  and inputting various instructions to the camera  101 . A shutter button  105  and a mode dial  106  illustrated in  FIG. 1  are also included in the operation unit  212 . 
     A user&#39;s half-press operation of the shutter button  105  inputs an image capturing preparation instruction to the controller  215 , and thereby the controller  215  performs an image capturing preparation process such as autofocus (AF) control and automatic exposure (AE) control. Further, a user&#39;s full-press operation of the shutter button  105  inputs an image capturing instruction to the controller  215 , and thereby the controller  215  controls the aperture stop/shutter unit  202  to perform an image capturing process for generating a recording captured image. A user&#39;s rotation operation of the mode dial  106  causes the controller  215  to set (change) an image capturing mode. 
     The controller  215  includes a tilt correction amount calculator  216 , a blur correction amount calculator  217 , an object information detector  218 , and the correction amount switcher  219  mentioned above. The tilt correction amount calculator  216  calculates, by using the acceleration signal from the three-axis accelerometer  213 , the tilt correction amount (angle) to be used in the tilt correction for the image data. The blur correction amount calculator  217  calculates, by using the angular velocity signal from the three-axis angular velocity meter  214 , the blur correction amount to be used in the blur correction for the image data. 
     The object information detector  218  as an object information acquiring unit detects an object included in the image data from the image capturing signal processor  205  to acquire information on the object (hereinafter referred to as “object information”). Examples of the object information will be described later. The object information detector  218  may perform learning of objects to be detected in advance using Convolutional Neural Networks (CNN) or the like, and acquire the object information with a method of applying CNN to face recognition or object recognition. Further, the object information detector  218  may acquire the object information on an object set by the user via the operation unit  212  or via an external apparatus capable of communicating with the camera  101 . 
     The tilt correction amount calculator  216  calculates, from the acceleration detected by the three-axis accelerometer  213 , the tilt correction amount (angle) for correcting the tilt generated in the image data due to the tilt of the camera  101  in the roll direction. The tilt correction amount corresponds to an angle difference from a reference angle. The blur correction amount calculator  217  calculates, from the angular velocity detected by the three-axis angular velocity meter  214  and a position of the magnification-varying lens  201  detected by a zoom position sensor (not illustrated) (that is, a focal length of the image capturing optical system), the blur correction amount for correcting the blur generated in the image data due to the camera shake in the roll, pitch and yaw directions. 
     The correction amount switcher  219  calculates, depending on the object information, a tilt correction permissible amount that is a permissible value (upper limit) of the tilt correction amount. Further, the correction amount switcher  219  switches the tilt correction amount to be output to the image signal processor  206  depending on the tilt correction permissible amount, the tilt correction amount from the tilt correction amount calculator  216  and the like. The correction amount switcher  219  may switch the tilt correction amount to be output to the image signal processor  206  in response to a user&#39;s operation of the operation unit  212 . 
     The tilt correction amount calculator  216 , the blur correction amount calculator  217 , the correction amount switcher  219  and the image signal processor  206  constitute a correction unit, and they and the object information detector  218  constitute an image processing apparatus. 
     Although this embodiment describes the case where the image processing apparatus is provided in the camera  101 , a personal computer may be an image processing apparatus by having functions of the tilt correction amount calculator  216 , the blur correction amount calculator  217 , the correction amount switcher  219 , the image signal processor  206  and the object information detector  218 . 
     A non-volatile memory (EEPROM)  221  is a memory that is electrically erasable and recordable, and stores constants, computer programs and the like to be used for operations of the controller  215 . 
       FIG. 3  is a flowchart of a correction amount switching process executed by the controller  215  (specifically, the tilt correction amount calculator  216  and the correction amount switcher  219 ) according to a computer program. In this embodiment, the tilt correction amount to be used by the image signal processor  206  for the tilt correction (hereinafter referred to as “a use tilt correction amount”) is set as follows. In the following description, “S” means a step. 
     At S 301 , the correction amount switcher  219  calculates (sets) the tilt correction permissible amount depending on the object information from the object information detector  218 . The tilt correction permissible amount is a permissible amount (upper limit) of the tilt correction amount as a rotation amount when rotating the clipping area clipped from the image data in the tilt correction, and is calculated depending on characteristics of the object indicated by the object information. Specifically, the correction amount switcher  219  calculates a larger tilt correction permissible amount when the object information indicates an object whose degree of necessity of the tilt correction is high than when the object information indicates an object whose degree of necessity of the tilt correction is low. The object whose degree of necessity of the tilt correction indicated by the object information is high is, for example, a) an architectural structure such as a building and a tower, b) an object whose vertical extension should be emphasized such as a forest, c) an object whose horizontal extension should be emphasized such as a horizontal line, a horizon and a staircase seen from front, and d) a group of persons. 
     On the other hand, the correction amount switcher  219  calculates a smaller tilt correction permissible amount when the object information indicates an object whose degree of necessity of the tilt correction is low (in other words, the tilt of the object does not become a matter) than when the object information indicates the above-described object whose degree of necessity of the tilt correction is high. The object whose degree of necessity of the tilt correction indicated by the object information is low is, for example, a close-up face, a close-up flower, and an aerial photograph. 
     As described above, in this embodiment, the correction amount switcher  219  sets a first tilt correction permissible amount (larger tilt correction permissible amount) when the object information indicates an object whose degree of necessity of the tilt correction is high, and sets a second tilt correction permissible amount smaller than the first tilt correction permissible amount when the object information indicates an object whose degree of necessity of the tilt correction is low. 
     Next, at S 302 , the tilt correction amount calculator  216  calculates a tilt angle of the camera  101  from the acceleration detected by the three-axis accelerometer  213 , and calculates the tilt correction amount with respect to the image data (captured image) generated by the camera  101  at the detected tilt angle. 
     Next, at S 303 , the correction amount switcher  219  compares the tilt correction amount calculated by the tilt correction amount calculator  216  with the tilt correction permissible amount calculated at S 301 . As a result of this comparison, the correction amount switcher  219  proceeds to S 304  when the tilt correction amount is smaller than the tilt correction permissible amount, and proceeds to S 305  when the tilt correction amount is equal to or more than the tilt correction permissible amount. 
     At S 304 , the correction amount switcher  219  sets the tilt correction amount calculated by the tilt correction amount calculator  216  as the use tilt correction amount to output it to the image signal processor  206 . On the other hand, at S 305 , the correction amount switcher  219  sets the tilt correction permissible amount as the use tilt correction amount to output it to the image signal processor  206 . 
     Description will herein be made of the calculation of the use tilt correction amount when a person&#39;s face is indicated by the object information with reference to the flowchart of  FIG. 3 . When calculating the use tilt correction amount only from the person&#39;s face indicated by the object information, the correction amount switcher  219  calculates at S 301  the tilt correction permissible amount from information on a size of the person&#39;s face (face size) in an image capturing area detected by the object information detector  218 . When the face size is smaller than a threshold (predetermined value), the proportion of the person&#39;s face in the image capturing area is small, so that the possibility is high that a landscape or a building whose degree of necessity of the tilt correction is high is captured in the background. Therefore, the correction amount switcher  219  determines that the degree of necessity of the tilt correction is high to calculate a large tilt correction permissible amount (first permissible value). 
     On the other hand, when the face size is as large as the threshold or more, the proportion of the person&#39;s face in the image capturing area is large, so that the degree of necessity (importance) of the tilt correction is low even if the background is a landscape or a building. Therefore, the correction amount switcher  219  calculates a small tilt correction permissible amount (second permissible value smaller than the first permissible value). 
     Next, at S 302 , as described above, the tilt correction amount calculator  216  calculates the tilt angle of the camera  101  from the acceleration detected by the three-axis accelerometer  213 , and calculates the tilt correction amount with respect to the image data (captured image) generated by the camera  101  at the detected tilt angle. 
     Next, at S 303 , as described above, the correction amount switcher  219  compares the tilt correction permissible amount calculated at S 301  with the tilt correction amount calculated by the tilt correction amount calculator  216  to set the use tilt correction amount. Description will herein be made of a case where multiple person&#39;s faces are detected at S 301 . In this case, at S 301 , the object information detector  218  calculates an average of the detected sizes of the multiple person&#39;s faces. The correction amount switcher  219  calculates the tilt correction permissible amount depending on the average of the face sizes detected by the object information detector  218 . 
     The correction amount switcher  219  may calculate a total face area that is a sum of the face sizes, calculate the ratio of the total face area in the image capturing area, and calculate the tilt correction permissible amount depending on the ratio. Alternatively, the correction amount switcher  219  may calculate the tilt correction permissible amount depending on an average of the sizes of only the faces determined as main objects among the multiple person&#39;s faces, or may calculate the tilt correction permissible amount depending on the maximum face size among the multiple person&#39;s faces. 
     Next, at S 304 , the correction amount switcher  219  compares the tilt correction permissible amount calculated at S 301  depending on the multiple person&#39;s face sizes with the tilt correction amount calculated by the tilt correction amount calculator  216  to set the use tilt correction amount. 
     At S 301 , the object information detector  218  may detect a position of the person&#39;s face (face position) in the image capturing area in addition to the face size. In this case, the correction amount switcher  219  may compare a face position tilt correction permissible amount calculated from the face position such that the face is not out of the clipping area when the tilt correction is performed, with a face size tilt correction permissible amount calculated from the face size, and set a smaller tilt correction permissible amount as the tilt correction permissible amount to be used (use tilt correction permissible amount). 
     As described above, this embodiment sets, in the tilt correction, the use tilt correction amount equal to or smaller than the tilt correction permissible amount calculated depending on the degree of necessity of the tilt correction for the object. As a result, this embodiment can provide, from a captured image of an object whose degree of necessity of the tilt correction is high, an image whose tilt is well corrected. Further, this embodiment can provide, from a captured image of an object whose degree of necessity of the tilt correction is low, an image whose tilt may not be well corrected but whose image quality is good. 
     Although this embodiment described an example of switching the tilt correction permissible amount between the object whose degree of necessity of the tilt correction is high and the object whose degree of necessity of the tilt correction is low, more tilt correction permissible amounts may be set. For example, the tilt correction permissible amounts may be set depending on types of objects (a building, a tree, a person, etc.) whose degree of necessity of the tilt correction is high. Similarly, the tilt correction permissible amounts may be set depending on types of objects (a face, a flower, an aerial photograph, etc.) whose degree of necessity of the tilt correction is low. 
     In addition, the above-described setting of the use tilt correction permissible amount depending on the size, number and position of the person&#39;s face can be applied to a case where the object is a plant or an animal. In the case where the object is a plant, the use tilt correction permissible amount may be set by replacing the person&#39;s face with a petal area of the plant. In the case where the object is an animal, the use tilt correction permissible amount may be set by replacing the person&#39;s face with an animal&#39;s face or the entire animal&#39;s body. 
     Further, although this embodiment detects the camera shake by the shake detector provided in the camera  101 , the camera shake may be detected by analyzing the image data using an external apparatus provided outside the camera  101 . In this case, the detected camera shake may be input from the external apparatus to the controller  215  (the tilt correction amount calculator  216  and the blur correction amount calculator  217 ) via the communication unit  211 . Further, the tilt detection and the shake detection may use the three-axis accelerometer  213  and the three-axis angular velocity meter  214  in combination or use another sensor (such as a geomagnetic sensor). 
     Further, this embodiment calculates the use tilt correction amount by using tilt angles of 0, 90, 180 and 270 degree of the camera  101  calculated from the acceleration detected by the three-axis accelerometer  213  as reference tilt angles. The reference tilt angles are as follows. When the tilt angle is −45 (315) degree or more and less than 45 degree, the reference tilt angle is 0 deg. When the tilt angle is 45 degree or more and less than 135 degree, the reference tilt angle is 90 degree. When the tilt angle is 135 degree or more and less than 225 (−135) degree, the reference tilt angle is 180 degree. When the tilt angle is 225 degree or more and less than 315 (−45) degree, the reference tilt angle is 270 degree. 
     For example, when the tilt angle is 10 degree, the use tilt correction amount is calculated as −10 degree by using the reference tilt angle of 0 degree. When the tilt angle is −10 (350) degree, the use tilt correction amount is calculated as 10 degree by using the reference tilt angle of 0 degree. When the tilt angle is 100 degree, the use tilt correction amount is calculated as −10 degree by using the reference tilt angle of 90 degree. When the tilt angle is 80 degree, the use tilt correction amount is calculated as 10 degree by using the reference tilt angle of 90 degree. When the tilt angle is 190 (−170) degree, the use tilt correction amount is calculated as −10 degree by using the reference tilt angle of 180 degree. When the tilt angle is 170 degree, the use tilt correction amount is calculated as 10 degree by using the reference tilt angle of 180 degree. When the tilt angle is 280 (−80) degree, the use tilt correction amount is calculated as −10 degree by using the reference tilt angle of 270 degree. When the tilt angle is 260 (−100) degree, the use tilt correction amount is calculated as 10 degree by using the reference tilt angle of 270 degree. 
     As described above, this embodiment changes the reference tilt angle (reference value) when correcting the tilt with respect to the reference value in the tilt correction depending on the posture of the camera  101 . Thereby, this embodiment can appropriately perform the tilt correction on the image data while reflecting a user&#39;s intended image capturing posture such as a vertical image capturing posture and a horizontal image capturing posture. 
     Embodiment 2 
     In a second embodiment (Embodiment 2) of the present invention, the use tilt correction amount is switched depending not only on the comparison result between the tilt correction amount and the tilt correction permissible amount described in Embodiment 1, but also on an enlargement permissible amount of the image data. In the switching, a magnitude of the camera shake is also taken into consideration. The camera of this embodiment has the same configuration as that of the camera  101  of Embodiment 1.  FIG. 4  is a flowchart of a correction amount switching process executed by the controller  215  (the tilt correction amount calculator  216 , blur correction amount calculator  217  and correction amount switcher  219 ) according to a computer program. 
     The processes at S 401  to S 405  are the same as those at S 301  to S 305  in Embodiment 1 ( FIG. 3 ). However, at S 404  and S 405 , the used tilt correction amount set at S 304  and S 305  is referred to as “a tilt correction amount 1” in this embodiment. 
     After the process at S 404  or S 405  is completed, the correction amount switcher  219  calculates at S 406  an enlargement amount of the image data (hereinafter referred to as “a tilt correction enlargement amount”) for the tilt correction amount 1 set at S 404  or S 405 . 
     Next, at S 407 , the correction amount switcher  219  calculates a blur enlargement permissible amount depending on information on the camera shake from the blur correction amount calculator  217 . The blur enlargement permissible amount is a permissible vale (upper limit) of an enlargement amount for enlarging the clipping area of the image data in the tilt correction, which is calculated in consideration of a degree of visibility of blur generated in the image data. 
     Specifically, when the camera shake is large, greatly enlarging the image (clipping area) in the tilt correction makes the blur generated in the image data highly visible. Therefore, when the camera shake is large, the correction amount switcher  219  sets a smaller blur enlargement permissible amount than when the camera shake is small, thereby limiting image enlargement in the tilt correction. When setting the blur enlargement permissible amount, the correction amount switcher  219  may take the object information in consideration or may use the object information instead of the information on the camera shake. For example, when the object information indicates an object whose uplifting feeling is to be emphasized, a moving object tracked by the camera, or an object whose brightness is dark, the blur of the object in the image data tends to be large. Therefore, when the object information indicates such objects, the correction amount switcher  219  may set a small blur enlargement permissible amount. 
     On the other hand, when the camera shake is small, even if the image is greatly enlarged in the tilt correction, the blur generated in the image data is not so visible. Therefore, when the camera shake is small, the correction amount switcher  219  sets a larger blur enlargement permissible amount than when the camera shake is large, thereby allowing image enlargement in the tilt correction. 
     When the object information indicates the object whose degree of necessity of the tilt correction is low described in Embodiment 1 such as a stationary object, a landscape or a bright object, blur of the object in the image data tends to be small. Therefore, when the object information indicates these objects, the correction amount switcher  219  may set a large blur enlargement permissible amount. 
     Next, at S 408 , the correction amount switcher  219  compares the tilt correction enlargement amount calculated at S 406  with the blur enlargement permissible amount calculated at S 407 . As a result of this comparison, when the tilt correction enlargement amount is smaller than the blur enlargement permissible amount, the correction amount switcher  219  proceeds to S 409 , and when the tilt correction enlargement amount is equal to or more than the blur enlargement permissible amount, the correction amount switcher  219  proceeds to S 410 . 
     At S 409 , the correction amount switcher  219  sets the tilt correction amount 1 as the use tilt correction amount to output it to the image signal processor  206 . On the other hand, at S 410 , the correction amount switcher  219  sets the tilt correction amount corresponding to the blur enlargement permissible amount as the use tilt correction amount to output it to the image signal processor  206 . 
     As described above, this embodiment sets, in the tilt correction, the use tilt correction amount equal to or lower than the tilt correction permissible amount calculated depending on the degree of necessity of the tilt correction for the object, and further resets the use tilt correction amount depending on the blur enlargement permissible amount for the image data. As a result, this embodiment can provide, when an captured image includes an object whose degree of necessity of the tilt correction is high and the image enlargement in the tilt correction is allowed, an image whose tilt is well corrected. On the other hand, this embodiment can provide, when an captured image includes an object whose degree of necessity of the tilt correction is low and the image enlargement in the tilt correction is limited, an image whose tilt may not be well corrected but whose image quality is good. 
     Also in this embodiment, as in Embodiment 1, more tilt correction permissible amounts and more blur enlargement permissible amounts may be set. 
     Embodiment 3 
     Although Embodiments 1 and 2 described the case where the tilt correction is performed using the object information and the detected acceleration or angular velocity, a third embodiment (Embodiment 3) of the present invention will describe control of a camera that performs the tilt correction in an automatic image capturing process (hereinafter simply referred to as “automatic image capturing”). In the camera that performs the automatic image capturing, image capturing is automatically performed without user&#39;s confirmation of captured images, so that the possibility that a lot of tilted captured images are generated is high. Therefore, it is desirable to perform the tilt correction on the captured images. 
     The camera of this embodiment has the same configuration as that of the camera  101  of Embodiment 1.  FIG. 5  is a flowchart of a control process performed in the camera performing the automatic image capturing. The controller  215  executes this process according to a computer program. 
     At S 501 , the controller  215  causes the image capturing signal processor  205  to generate an object recognition image from the image data to output the object recognition image to the object information detector  218 . The object information detector  218  recognizes, from the object recognition image, an object such as a person or a stationary/moving object to generate the object information. When recognizing a person, the object information detector  218  detects a face or body of the person. 
     In detection of the face, the object information detector  218  detects, in the object recognition image, an area (face area) that matches a face pattern prepared for determining a person&#39;s face. The object information detector  218  also calculates a reliability indicating certainty of the face. The reliability is calculated from, for example, the size of the face area in the object recognition image or the degree of matching with the face pattern. When recognizing the stationary/moving object, it is possible to recognize an object that matches a prepared pattern. 
     Next, at S 502 , the controller  215  calculates the tilt correction amount using the acceleration detected by the three-axis accelerometer  213 . The controller  215  may calculate at S 502  an absolute tilt angle of the camera using the acceleration detected by the three-axis accelerometer  213 , and use this absolute tilt angle at S 508  described later to calculate the tilt correction amount. 
     Next, at S 503 , the controller  215  performs zoom control for driving a zoom drive actuator (not illustrated) to move the magnification-varying lens  201 . The controller  215  performs the zoom control depending on the object information acquired at S 501 . 
     Next, at S 504 , the controller  215  determines whether or not a user&#39;s manual image capturing instruction has been input. The controller  215  proceeds to S 509  when the manual image capturing instruction has been input, and otherwise proceeds to S 505 . The manual image capturing instruction may be an instruction generated by fully pressing the shutter button  105  or by detecting a tap of the camera (body) by the three-axis accelerometer  213 . Alternatively, the manual image capturing instruction may be input through wireless communication from an external apparatus such as a smartphone that is operated by the user via the communication unit  211 . At S 509 , in response to the manual image capturing instruction, the controller  215  executes a manual image capturing process. 
     On the other hand, at S 505 , the controller  215  performs an automatic image capturing determination process. In the automatic image capturing determination process, the controller  215  determines whether or not the user will preview captured images through the display device. When the user will not preview the captured images, the controller  215  sets execution (ON) of the automatic image capturing. The case where the user will not preview the captured images is, for example, a case where the display device for preview is turned off, a case where the display device is provided in the camera, but the user is not looking at the display device, and a case where the display device is not provided in the camera. Further, when an image capturing instruction as a voice command is input, the controller  215  may set the execution of the automatic image capturing. 
     At S 506 , the controller  215  determines whether or not the execution of the automatic image capturing has been set at S 505 . The controller  215  proceeds to S 507  when the execution has been set, and otherwise ends the present process. 
     At S 507 , the controller  215  starts the automatic image capturing. Next, at S 508 , the controller  215  performs the correction amount switching process described in Embodiment 1 or 2, and causes the image signal processor  206  to perform the tilt correction on the captured image. 
     The controller  215  having proceeded from S 509  or S 508  to S 510  generates, using a state of the camera in image capturing and the captured images, information to be used for a learning process described later (hereinafter referred to as “learning information”). The learning information includes a zoom magnification in image capturing, an object recognition result in the captured image, a face detection result, number of faces included in the captured image, a degree of smiling, a degree of eye closure, a face angle, a face recognition (ID) number, and a line-of-sight angle of a person object. 
     The learning information may further include a determination result of an image capturing scene, an elapsed time from previous image capturing, an image capturing clock time, position information of the camera acquired by GPS or the like, a position change amount from previous image capturing, a voice level in image capturing, a person making a voice, the presence or absence of applause, and whether or not cheers are raised. 
     Moreover, the learning information may include camera shake information (acceleration, whether or not using a tripod), environmental information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet rays), and information indicating whether or not performing image capturing in response to the manual image capturing instruction. The controller  215  also calculates a score as an output of a neural network that quantifies a user&#39;s image preference. 
     This score becomes higher as the tilt-corrected image is closer to an image intended by the user. It can be said that the tilt-corrected image having a high score is corrected as intended by the user. Therefore, when setting the tilt correction permissible amount and the blur enlargement permissible amount depending on the object, the same setting as that for a high-score image makes it possible to perform the tilt correction more accurately as intended by the user. 
     The high-score image is, for example, a captured image to which a high score is manually added by the user, a captured image acquired by being transmitted from an external device capable of communication with the camera in response to a user&#39;s instruction, and a captured image stored in an external device capable of communication with the camera. In addition, the high-score image is, for example, a captured image uploaded to a server by the user, and a captured image whose parameters are changed (that is, edited) by the user. 
     The controller  215  that has generated the learning information at S 511  records the learning information as tag information to a captured image file or records it in the non-volatile memory  221 . 
     The image signal processor  206  selects an important object in the captured image, reads the learning information for the important object from the recorded learning information, and sets the tilt correction permissible amount and the blur enlargement permissible amount depending on the read learning information. 
     As described above, according to this embodiment, even the camera that performs the automatic image capturing can appropriately perform the tilt correction on the captured image depending on the object. Further, the camera provided with the above-described learning function can perform the tilt correction as a user intend in next and subsequent image capturing. 
     Although this embodiment does not perform the tilt correction when performing the manual image capturing, the tilt correction may be performed when the manual image capturing is performed (after S 509 ). In this case, since image capturing is more likely to be performed with the camera being tilted in the automatic image capturing than in the manual image capturing, the tilt correction permissible amount may be set larger in the automatic image capturing than in the manual image capturing. 
     For example, although each of the above-described embodiments described the tilt correction in still image capturing, the same tilt correction may be performed in moving image capturing. Further, although each of the above-described embodiments described the tilt correction in detail, blur correction may be performed together with the tilt correction. Moreover, when the camera is provided with an optical blur correction function that moves a lens in an image capturing optical system or an image sensor with respect to an object, the blur enlargement permissible amount in Embodiment 2 may be calculated based on a remaining blur correction amount after the optical blur correction. 
     According to each of the above-described embodiments, the tilt correction of the captured image can be appropriately performed depending on the object. 
     OTHER EMBODIMENTS 
     Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like. 
     While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions. 
     This application claims the benefit of Japanese Patent Application Nos. 2019-230974, filed on Dec. 20, 2019 and 2020-160318, filed on Sep. 25, 2020 which are hereby incorporated by reference herein in their entirety.