Patent Description:
When an object does not directly face a camera in, for example, shooting a tall building from the ground, the object in the captured image is distorted. As a technique for solving this, there is known a technique (to be referred to as "shift control" hereinafter) of causing a camera and an object to directly face each other, translationally moving (shifting) the image sensor (or optical system) with respect to the imaging plane, and then shooting the object.

<CIT> (patent literature <NUM>) discloses a technique of detecting the tilt of an object with respect to a camera using distance measurement data obtained based on image capturing, and performing correction for direct-facing. Patent literature <NUM> also discloses a technique of performing correction for direct-facing by rotation control of a lens unit and shift control of an image sensor. <CIT> (patent literature <NUM>) discloses a technique of detecting the distortion (angle) of the edge of an object in the vertical direction and performing shift control of an optical system to compensate for the detected distortion.

However, in patent literature <NUM>, the tilt of an object is detected based on the difference between a plurality of distance measurement data obtained at different image height positions in an image sensor. When the distance measurement precision is low, the distortion of the object may not be properly corrected. In patent literature <NUM>, shift control of the optical system is performed to compensate for the distortion of an edge, but fluctuations of the imaging range by shift control are not considered. The position of the object in the captured image may change during shift control, degrading the quality. <CIT> discloses an imaging apparatus to which a lens unit which includes a shift mechanism, a tilt mechanism, a revolving mechanism, and a tilt revolving mechanism, but cannot automatically detect their respective movement amounts can be attached and which includes a camera operation part through which a shift amount, a tilt amount, and a revolving angle set by lens operation are set in the camera by a user. When the lens unit is attached, display for urging manual operation setting is performed and the user sets the shift amount, the tilt amount and the revolving angle set in the lens unit in the camera through the camera operation part, to correct the exposure unevenness in the shutter running direction, according to the set values. <CIT> discloses a digital camera having an imaging sensor providing a digital representation of an image projected onto an imaging portion of the imaging sensor. A processor associated with the device identifies an image compression factor and progressively, over a selected region of the digital representation of the image, magnifies at least a portion of the digital representation of the image to reduce an apparent convergence distortion. Correction preferably occurs before lossy compression. The digital camera preferably includes a display to provide a visual display of the digital representation of the image to a user who may use a user input to manually correct apparent distortion by following prompts generated on the display. The digital camera also preferably includes a non-volatile memory configured to store the digital representation of the image.

The present invention provides an image capturing apparatus, a method of controlling an image capturing apparatus, and a computer program as specified in the appended claims.

The present invention performs more appropriate distortion correction while improving the quality of a captured image.

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

An image capturing system will be exemplified below as an image capturing apparatus according to the first embodiment of the present invention.

<FIG> is a block diagram for explaining the arrangement of the image capturing system. The image capturing system includes an image capturing apparatus and a monitoring apparatus <NUM>. The image capturing apparatus includes a camera unit <NUM> serving as an image capturing unit, an Auto Gain Controller (AGC) <NUM>, an Analog-to-Digital (A/D) converter <NUM>, a camera signal processor <NUM>, and a communication unit <NUM>. The image capturing apparatus further includes a driving amount decision unit <NUM>, a shift control unit <NUM>, a rotation control unit <NUM>, a shift driving unit <NUM>, and a rotation driving unit <NUM>.

The camera unit <NUM> includes an imaging optical system, a BandPass Filter (BPF) <NUM>, a color filter <NUM>, and an image sensor <NUM>. The imaging optical system includes a zoom lens <NUM> that moves along the optical axis to change the focal length, a focus lens <NUM> that moves along the optical axis to adjust the focus, and a stop unit <NUM> that adjusts the light quantity. Note that the imaging optical system may be integral with the image capturing apparatus or detachable from it.

Light having passed through the imaging optical system forms an object image as an optical image on the image sensor <NUM> via the BPF <NUM> and the color filter <NUM>. The BPF <NUM> may be removable from the optical path of the imaging optical system. The object image is photoelectrically converted by the image sensor <NUM>.

An analog electric signal (image capturing signal) output from the image sensor <NUM> is gain-adjusted by the AGC <NUM> and converted into a digital signal by the A/D converter <NUM>. The digital signal is input to the camera signal processor <NUM>. The camera signal processor <NUM> performs various image processes on the digital image capturing signal, generating a video signal.

The video signal is output to the monitoring apparatus <NUM> connected by wired or wireless communication to the image capturing apparatus via the communication unit <NUM>. The monitoring apparatus <NUM> receives an instruction from the user, and outputs control signals such as a command to the shift control unit <NUM> and the rotation control unit <NUM> via the communication unit <NUM>.

Based on either of a rotation driving amount when performing rotation control of the camera unit <NUM> and a shift driving amount when performing shift control of the image sensor <NUM>, the driving amount decision unit <NUM> decides the other driving amount. The rotation control is posture control on rotation movement in the pan direction and/or tilt direction. At this time, a decided driving amount is the other driving amount for performing a correction operation to compensate for a change of the imaging range generated by one control (rotation control or shift control). That is, the other driving amount is decided to suppress a change of the position of the object in the captured image. Details of decision of the driving amount in the driving amount decision unit <NUM> will be described later with reference to <FIG>.

The shift control unit <NUM> instructs the image sensor <NUM> about shift driving based on the driving instruction (shift driving amount) decided by the driving amount decision unit <NUM> or accepted via the communication unit <NUM>. The rotation control unit <NUM> instructs the camera unit <NUM> about rotation driving based on the rotation driving amount decided by the driving amount decision unit <NUM> or designated via the communication unit <NUM>.

The shift driving unit <NUM> drives the image sensor <NUM> based on the shift driving designated from the shift control unit <NUM>. The shift driving unit <NUM> is, for example, a motor. The rotation driving unit <NUM> drives the camera unit <NUM> based on the rotation driving designated from the rotation control unit <NUM>. The rotation driving unit <NUM> is, for example, a motor. The shift control unit <NUM> and the shift driving unit <NUM> perform designation of shift driving and driving to the image sensor <NUM>, but may perform designation of shift driving and driving to the optical system.

<FIG> is a view for explaining an object distortion. <FIG> is a schematic view for when the photographer looks up at a building from the ground and shoots it, and illustrates an image captured at that time. When the camera unit does not directly face an object <NUM> (that is, the object <NUM> (wall of the building) and an optical axis <NUM> are not perpendicular), the building is distorted like a trapezoid (object distortion), as illustrated in a captured image <NUM>.

<FIG> is a view for explaining shift control. <FIG> is a schematic view when the image capturing apparatus directly faces the same object <NUM> (building) as that in <FIG> and shoots it by shift control, and illustrates an image captured at that time. More specifically, an imaging plane <NUM> is shifted down (relatively to an optical system <NUM>) so that the optical axis <NUM> becomes perpendicular to the object <NUM> (wall of the building) and the object <NUM> falls within the imaging range. By this shift control, the object distortion is compensated (corrected) for as represented by a captured image <NUM>.

To achieve image capturing in which an object distortion is compensated (corrected) for like the captured image <NUM>, the camera unit <NUM> and the image sensor <NUM> need to be controlled appropriately. This requires information about an elevation angle <NUM> and an object distance <NUM>.

<FIG> is a view for explaining control for reducing an object distortion while maintaining the imaging range. More specifically, <FIG> shows a state in which shift control (shift amount x) for maintaining the imaging range is performed while performing rotation control (rotation amount α) of the camera unit to bring the optical axis <NUM> close to be perpendicular to the object <NUM> (wall of the building) from the shooting state in <FIG>.

That is, when the camera unit <NUM> is controlled to rotate (tilt) by only the small rotation amount α, the imaging range changes from an original imaging range <NUM> to an imaging range <NUM> by α rotation of the camera unit. For example, in a captured image within the imaging range <NUM>, the ground is captured in <NUM>/<NUM> of the image from the bottom and the top of the building is lost (not captured). To suppress such a change of the imaging range (maintain the imaging range <NUM>), the image sensor <NUM> is shifted down by the shift amount x within the imaging plane.

<FIG> is a view for explaining calculation of the driving amounts of rotation control and shift control. More specifically, a method of calculating the shift amount x of the image sensor <NUM> in accordance with the rotation amount α of the camera unit <NUM> to maintain the imaging range as shown in <FIG> will be explained.

A triangle OAB shown in <FIG> is an enlarged view of a triangle OAB shown in <FIG>, where α is the rotation amount of the camera unit <NUM>, α<NUM> is the vertical angle of view/<NUM>, x is the shift amount of the image sensor <NUM>, x<NUM> is the vertical sensor size/<NUM>, and l is the imaging position. At this time, the shift amount x of the image sensor when the camera unit <NUM> is rotated by α is given by equation (<NUM>): <MAT>.

In this case, equation (<NUM>) is established: <MAT>.

From equations (<NUM>) and (<NUM>), when the camera unit <NUM> is rotated by α, the shift amount x of the image sensor for suppressing a change of the imaging range can be given by equation (<NUM>) using the vertical sensor size and the imaging position: <MAT>.

That is, an object distortion can be reduced by performing rotation control and shift control using the relation of equation (<NUM>). In addition, a change of the imaging range during correction control (rotation control and shift control) can be suppressed (that is, the imaging range can be maintained).

<FIG> is a view showing a temporal change of the captured image in distortion correction. <FIG> shows, for example, a change of the captured image displayed on the display unit of the monitoring apparatus <NUM> from the start to completion of correction control (rotation control and shift control).

At the start of image capturing (before the start of correction control), a distortion is generated in which the upper portion of the building serving as an object becomes shorter than the lower portion (to be referred to as "upper distortion" hereinafter). In this state, shift control of the image sensor <NUM> synchronized with rotation control of the camera unit <NUM> is started using the relation of equation (<NUM>). Then, the distortion generated in the object is gradually corrected over time while maintaining the relative position of the object in the captured image. The correction to the upper distortion will be called "upper correction" hereinafter. Similarly, corrections on lower distortion, right distortion, and left distortion will be called lower correction, right correction, and left correction, respectively.

<FIG> is a view illustrating a distortion correction user interface (UI). For example, the UI can be provided as a physical button or a GUI on the display unit in the monitoring apparatus <NUM>. The user decides distortion correction (upper correction, lower correction, right correction, or left correction) to be executed in accordance with the distortion state of an object in a captured image displayed on the display unit of the monitoring apparatus <NUM>, and presses a corresponding button on the UI. Then, user instructions (driving instruction, driving start instruction, and driving stop instruction) including information of a corresponding shift direction are transmitted.

For a captured image as shown in <FIG>, "upper correction" is executed. Control of the camera unit <NUM> and image sensor <NUM> starts based on equation (<NUM>) at the start timing of pressing the button by the user, and stops at the end timing of pressing the button by the user.

While confirming the captured image changing over time, as shown in <FIG>, the user presses the button and can perform distortion correction of the object at high precision. For example, when the user presses an "upper correction" button too long and the correction becomes excessive, he/she can press a "lower correction" button.

Calculation of the shift amount given by equation (<NUM>) does not require information such as the elevation angle of the image capturing apparatus and the object distance. More specifically, pieces of information necessary to calculate a shift amount based on equation (<NUM>) are the rotation amount of the camera unit <NUM>, the vertical sensor size, and the imaging position. These pieces of information are information known for each image capturing apparatus or information obtained from control information. The above-described shift control is useful because it is not influenced by the detection precision of information such as the elevation angle and the object distance.

As described above with reference to <FIG>, a change of the imaging range during correction control can be suppressed by synchronously performing rotation control of the camera unit <NUM> and shift control of the image sensor <NUM>. That is, the object position with respect to the imaging range does not vary during correction control, only a distortion of the object is corrected, and thus the quality is high.

Synchronous control between rotation control of the camera unit <NUM> and shift control of the image sensor <NUM> will be further explained. In particular, two different synchronous control methods will be described.

<FIG> is a graph for explaining a control method in the first synchronous control. In the first synchronous control, rotation control instructions each for one step (predetermined amount) of the camera unit <NUM> are sequentially accepted to sequentially decide the shift amount of the image sensor <NUM> and perform synchronous control. That is, the camera unit <NUM> and the image sensor <NUM> are synchronously driven while finely designating their target control positions.

<FIG> shows the relationship between the rotation amount of the camera unit <NUM> and the shift amount of the image sensor <NUM> under a given image capturing condition. In this case, the rotation amount of one step of the camera unit <NUM> is α', the shift amount x of the image sensor <NUM> in accordance with a rotation amount from a reference position is calculated, and control is performed based on the calculation result.

<FIG> is a flowchart of processing in the first synchronous control. A processing corresponding to the flowchart can be realized by, for example, executing the corresponding program (stored in the auxiliary storage device <NUM> or the like) by the CPU <NUM> operating in the imaging device. The respective units of the image capturing apparatus perform the following operation.

In step S900, the image capturing apparatus obtains x<NUM> (vertical sensor size/<NUM>) which is necessary to calculate the shift amount of the image sensor <NUM> based on equation (<NUM>).

In step S901, the image capturing apparatus determines whether a distortion correction button (<FIG>) is pressed by a user operation. For example, the determination is made based on whether a control signal transmitted from the monitoring apparatus <NUM> and input via the communication unit <NUM> represents the press of the distortion correction button. If the distortion correction button is pressed, the process advances to step S902; if not, the determination continues.

Among four buttons shown in <FIG>, a button corresponding to the shape of a distortion to be corrected is selectively pressed by a user operation. The driving directions (shift directions) of the camera unit <NUM> and image sensor <NUM> are uniquely decided based on the type of selected correction. For example, when upper correction is selected, rotation control is controlled to a tilt down direction, and shift control is controlled to a down direction. When lower correction is selected, rotation control and shift control are controlled to directions opposite to those in upper correction. Further, when left correction is selected, rotation control is controlled to a pan right direction, and shift control is controlled to a right direction. When right correction is selected, rotation control and shift control are controlled to directions opposite to those in left correction.

In step S902, the image capturing apparatus obtains the imaging position l. The imaging position l is an imaging distance between O and B shown in <FIG>, and represents an imaging distance between the optical system <NUM> and the imaging plane <NUM>. The imaging position l can be obtained from a focus lens position used in control of the focus lens. It is enough to obtain the imaging position l only once between the time when the user presses the button and the time when the user stops pressing the distortion correction button.

In step S903, the driving amount decision unit <NUM> sets the rotation amount α' of one step of the camera unit <NUM>. As the rotation amount α' of one step is smaller, the effect of maintaining the imaging range is higher.

In step S904, the driving amount decision unit <NUM> sets the target rotation position α of the camera unit <NUM> in the rotation control unit <NUM>. The target rotation position α is set based on the rotation amount α' of one step set in step S903. The target rotation position α of the camera unit <NUM> is an amount obtained by adding only the rotation amount α' of one step every rotation of one step using, as a reference, the rotation position of the camera unit <NUM> at the start of distortion correction.

In step S905, the driving amount decision unit <NUM> decides the target shift amount x of the image sensor <NUM> based on the target rotation position α of the camera unit <NUM> set in step S904. This decision is made based on equation (<NUM>), and the values obtained or set in steps S900, S902, and S904 are used as necessary x<NUM>, I and α.

In step S906, the driving amount decision unit <NUM> sets, in the shift control unit <NUM>, the target shift amount x of the image sensor <NUM> calculated in step S905.

In step S907, the rotation control unit <NUM> and the shift control unit <NUM> control the rotation driving unit <NUM> and the shift driving unit <NUM>, respectively, and drive the camera unit <NUM> and the image sensor <NUM>. That is, the camera unit <NUM> and the image sensor <NUM> are driven based on the target rotation position α of the camera unit <NUM> set in step S904 and the target shift amount x of the image sensor <NUM> set in step S906.

In step S908, the rotation control unit <NUM> and the shift control unit <NUM> respectively determine whether the control position of either of rotation driving of the camera unit <NUM> and shift driving of the image sensor <NUM> has reached a mechanical driving end. If it is determined that the control position of either driving has reached the driving end, the distortion correction control ends. If it is determined that the control position of neither driving has reached the driving end, the process advances to step S909.

In step S909, the image capturing apparatus determines whether the press of the distortion correction button has ended. Similar to step S901, the determination is made based on whether a control signal transmitted from the monitoring apparatus <NUM> and input via the communication unit <NUM> represents the press of the distortion correction button. If the press has ended, the distortion correction control ends; if not, the process returns to step S904 to continue the distortion correction control.

If the distortion correction button is kept pressed, the processes in steps S904 to S908 are repetitively performed and distortion correction can be performed while maintaining the relative position of the object with respect to the imaging range. The shift driving amount of the image sensor <NUM> is calculated in accordance with rotation control of one step of the camera unit <NUM>. However, it is also possible to set the shift driving amount of one step of the image sensor <NUM> and calculate the rotation driving amount in accordance with the shift driving amount. Shift control may not target the image sensor <NUM>, but the optical system <NUM> may be controlled.

As described above, in the first synchronous control, the camera unit <NUM> and the image sensor <NUM> can be synchronously driven while finely designating their target control positions.

<FIG> is a graph for explaining a control method in the second synchronous control. In the second synchronous control, the camera unit <NUM> and the image sensor <NUM> are synchronously controlled by designating the driving speeds of rotation control of the camera unit <NUM> and shift control of the image sensor <NUM> at the start of control.

<FIG> and <FIG> are a flowchart of processing in the second synchronous control. A processing corresponding to the flowchart can be realized by, for example, executing the corresponding program (stored in the auxiliary storage device <NUM> or the like) by the CPU <NUM> operating in the imaging device. The respective units of the image capturing apparatus perform the following operation. Note that processes in steps S900 to S902, S907, and S909 are similar to those in the first synchronous control, and a description thereof will not be repeated.

In step S1100, the image capturing apparatus obtains a shift movable amount xlim. of the image sensor <NUM>. The shift movable amount xlim is a distance from the current position of the image sensor <NUM> to the mechanical driving end.

In step S1101, the image capturing apparatus obtains a rotation movable amount αlim of the camera unit <NUM>. The rotation movable amount αlim is an angle from the current position of the camera unit <NUM> to the mechanical driving end.

In step S1102, the driving amount decision unit <NUM> calculates a maximum shift driving amount xmax of the image sensor <NUM>. The maximum shift driving amount xmax is a shift driving amount of the image sensor <NUM> necessary to maintain the relative position of an object with respect to the imaging range when the camera unit <NUM> is driven to rotate only by the rotation movable amount αlim obtained in step S1101. The maximum shift driving amount xmax is calculated based on equation (<NUM>).

In step S1103, the driving amount decision unit <NUM> compares the shift movable amount xlim calculated in step S1100 and the maximum shift driving amount xmax calculated in step S1102. If xlim > xmax, the process advances to step S1104. In the case of xlim > xmax, when the camera unit <NUM> is controlled up to the mechanical driving end and shift control of the image sensor <NUM> necessary to maintain the relative position of the object is performed, the image sensor <NUM> does not collide against the mechanical driving end. If xlim ≤ xmax, the process advances to step S1106. In the case of xlim ≤ xmax, when the camera unit <NUM> is controlled up to the mechanical driving end, the image sensor <NUM> collides against the mechanical driving end and the relative position of the object cannot be maintained.

In step S1104, the driving amount decision unit <NUM> sets the target shift position of the image sensor <NUM> to a position obtained by adding only the maximum shift driving amount xmax calculated in step S1102 to a current position xnow of the image sensor <NUM>.

In step S1105, the driving amount decision unit <NUM> sets the target rotation position of the camera unit <NUM> to a position obtained by adding only the rotation movable amount αlim obtained in step S1101 to a current position αnow of the camera unit <NUM>.

In step S1106, the driving amount decision unit <NUM> calculates a maximum rotation driving amount αmax of the camera unit <NUM>. The maximum rotation driving amount αmax is a rotation driving amount of the camera unit <NUM> necessary to maintain the relative position of the object with respect to the imaging range when the image sensor <NUM> is controlled to shift only by the shift movable amount xlim obtained in step S1100. The maximum rotation driving amount αmax is calculated based on equation (<NUM>).

In step S1107, the driving amount decision unit <NUM> sets the target shift position of the image sensor <NUM> to a position obtained by adding only the shift movable amount xlim obtained in step S1100 to the current position xnow of the image sensor <NUM>.

In step S1108, the driving amount decision unit <NUM> sets the target rotation position of the camera unit <NUM> to a position obtained by adding only the maximum rotation driving amount αmax calculated in step S1106 to the current position αnow of the camera unit <NUM>.

In step S1109, the driving amount decision unit <NUM> sets a rotational speed vr of the camera unit <NUM> in the rotation control unit <NUM>. As the rotational control speed vr is lower, the user can more easily perform fine adjustment in designating the start and end of distortion correction.

In step S1110, the driving amount decision unit <NUM> sets a shift speed vs of the image sensor <NUM> in the shift control unit <NUM>. At this time, the shift speed vs is so set that the camera unit <NUM> and the image sensor <NUM> reach the target positions at the same time (that is, the driving times become equal).

In step S1111, the rotation control unit <NUM> and the shift control unit <NUM> respectively determine whether the camera unit <NUM> and the image sensor <NUM> have reached the target positions. If both the camera unit <NUM> and the image sensor <NUM> have reached the target positions, the distortion correction control ends. If at least either of the camera unit <NUM> and the image sensor <NUM> has not reached the target position, the process advances to step S909.

In step S1112, the image capturing apparatus performs distortion correction stop processing. In the stop processing, the control ends after the image sensor <NUM> is shifted to a shift position where the imaging range is maintained, with respect to the orientation of the camera unit <NUM> at a timing when a distortion correction end instruction is accepted.

When the linearity between the rotation amount of the camera unit <NUM> and the shift amount of the image sensor <NUM> is low, the relative position of the object with respect to the imaging range may not be completely maintained at a point before the camera unit <NUM> and the image sensor <NUM> reach the target positions. Thus, when distortion correction is ended by a user operation before the target control positions, the imaging range is adjusted by the stop processing in step S1112.

As described above, in the second synchronous control, the camera unit <NUM> and the image sensor <NUM> can be synchronously driven by designating the driving speeds of control of the camera unit <NUM> and image sensor <NUM> at the start of control.

In both the above-described first synchronous control and second synchronous control, the camera unit <NUM> and the image sensor <NUM> can be synchronized. However, the first synchronous control is superior to the second synchronous control because the maintenance effect of the relative position of an object during distortion correction control is high. To the contrary, the second synchronous control is superior to the first synchronous control in shooting conditions requiring high-speed control because the number of calculations and the number of driving operations are small (one).

Next, an example of the hardware arrangement of the image capturing apparatus will be described with reference to the block diagram of <FIG>. Note that the arrangement shown in <FIG> is merely an example of an arrangement applicable to the image capturing apparatus, and the arrangement of the image capturing apparatus can be properly modified/changed.

A CPU <NUM> executes processing using a computer program and data stored in a main storage <NUM>. The CPU <NUM> controls the operation of the overall image capturing apparatus, and executes or controls each process that is performed by the image capturing apparatus in the above description. For example, the CPU <NUM> executes processing using a computer program and data stored in the main storage <NUM>, implementing the functions of the respective functional units of the camera signal processor <NUM>, driving amount decision unit <NUM>, shift control unit <NUM>, and rotation control unit <NUM> shown in <FIG>.

The main storage <NUM> is a storage such as a Random Access Memory (RAM). The main storage <NUM> has areas for storing computer programs and data loaded from an auxiliary storage device <NUM>, images captured by the camera unit <NUM>, and various data received from the monitoring apparatus <NUM> via the communication unit <NUM>. Further, the main storage <NUM> has a work area used when the CPU <NUM> executes various processes. In this manner, the main storage <NUM> can properly provide various areas.

The auxiliary storage device <NUM> is a large-capacity storage such as a Hard Disk Drive (HDD), a Read Only Memory (ROM), or a Solid State Drive (SSD). The auxiliary storage device <NUM> saves an Operating System (OS), and computer programs and data for causing the CPU <NUM> to execute or control each process that is performed by the image capturing apparatus in the above description. The auxiliary storage device <NUM> also saves data received from the monitoring apparatus <NUM> via the communication unit <NUM>. The computer programs and data saved in the auxiliary storage device <NUM> are properly loaded to the main storage <NUM> under the control of the CPU <NUM>, and processed by the CPU <NUM>.

A driving unit <NUM> drives the camera unit <NUM> based on image capturing parameters received from the monitoring apparatus <NUM>. For example, the driving unit <NUM> corresponds to the shift driving unit <NUM> that performs shift control of the image sensor <NUM>, and the rotation driving unit <NUM> that performs rotation control of the camera unit <NUM>. Note that the control target by the driving unit <NUM> is not limited to a specific target, and may be another target (for example, the position of the camera unit <NUM>).

The camera unit <NUM> includes the image sensor and the optical system, and forms an image of an object on the image sensor using, as the center of image capturing, the intersection between the optical axis of the optical system and the image sensor. The image sensor is, for example, a Complementary Metal-Oxide Semiconductor (CMOS) or a Charged Coupled Device (CCD). The communication unit <NUM> performs data communication with the monitoring apparatus <NUM>.

As described above, according to the first embodiment, when performing shift control of the image sensor, rotation control (tilt or pan) of the camera unit is performed in synchronization with shift control of the image sensor. More specifically, rotation control is synchronously performed to compensate for a change of the imaging range caused by shift control. By this synchronous control, a change of the imaging range during correction control can be suppressed, and a high-quality image can be provided.

In a modification, an example will be explained in which the image capturing apparatus determines the shape of a distortion generated in an object based on a captured image and performs rotation control of the camera unit <NUM> and shift control of the image sensor <NUM> so as to correct the distortion shape.

<FIG> is a flowchart of processing executed by the image capturing apparatus according to the modification. A processing corresponding to the flowchart can be realized by, for example, executing the corresponding program (stored in the auxiliary storage device <NUM> or the like) by the CPU <NUM> operating in the imaging device.

In step S1200, the image capturing apparatus determines whether a distortion correction execution instruction from the user has been received. If the image capturing apparatus determines that the distortion correction instruction from the user has been received, the process advances to step S1201. If the image capturing apparatus determines that no instruction has been received, it keeps waiting for a distortion correction instruction.

In step S1201, the image capturing apparatus determines whether an object suffering a distortion (to be referred to as a distorted object hereinafter) exists in a captured image. If the image capturing apparatus determines that a distorted object exists, the process advances to step S1202. If the image capturing apparatus determines that no distorted object exists, the process advances to step S1203. As a distorted object presence/absence determination method, a method based on depth information and a method based on edge detection will be explained. However, the presence/absence of a distorted object may be determined by another method.

<FIG> is a view for explaining distortion shape determination based on depth information. In the distorted object presence/absence determination method based on depth information (for example, a depth image in which each pixel has a depth value at its position), when the object distance monotonously changes in a given object region, it is determined that the object does not directly face the image capturing apparatus and is a distorted object.

<FIG> is a view for explaining distortion shape determination based on edge detection. In the distorted object presence/absence determination method based on edge detection, when the edge interval in the vertical direction (or horizontal direction) is not constant in regard to a pair of edges detected in a given object region, it is determined that the object is a distorted object.

In step S1202, the image capturing apparatus sets, as a distortion correction target, an object determined as a distorted object in step S1201. In some scenes, a plurality of distorted objects may exist in a captured image. In this case, a correction target object is set by accepting selection of the correction target object from the user.

<FIG> is a view for explaining distortion shape determination when a plurality of objects exist. <FIG> is a view for explaining correction target decision when a plurality of objects exist. More specifically, <FIG> shows a screen on which selection of a correction target object is accepted from the user. <FIG> shows a display screen of a form in which a plurality of polygonal regions (e.g., <NUM> and <NUM>) obtained by dividing a region for respective objects determined as distorted objects in step S1201 are superposed and displayed on a captured image, and selection of one polygonal region is accepted from the user.

In step S1203, the image capturing apparatus notifies the user that no distorted object (correction target object) exists in the captured image.

In step S1204, the image capturing apparatus determines the distortion shape type (upper distortion, lower distortion, right distortion, or left distortion) of the correction target object. For the distortion shape determination, depth information-based determination and edge detection-based determination are possible.

For example, in depth information corresponding to a captured image <NUM> in <FIG>, the depth (object distance) to the building region in the captured image <NUM> changes from the near side to the far side in a direction from the lower end to upper end of the image. Because of perspective, an object looks large as the distance is short and small as it is long. From this, it can be determined that the distortion shape of the building of the captured image <NUM> is "upper distortion" (the lower portion of the image is large and the upper portion is small).

Similarly, in depth information corresponding to a captured image <NUM>, the depth (object distance) to the building region in the captured image <NUM> changes from the near side to the far side in a direction from the left end to right end of the image. It can be determined that the distortion shape of the building of the captured image <NUM> is "right distortion" (the left portion of the image is large and the right portion is small). In this manner, depth information can be used to determine a distortion shape based on a direction in which the depth in the object (building) within the captured image changes.

In a captured image <NUM> of <FIG>, the interval between a pair of edges decreases in a direction from the lower end to upper end of the image in regard to the pair of edges detected for the object in the captured image. In this case, it can be determined that the distortion shape of the object is "upper distortion". Further, in a captured image <NUM>, the interval between a pair of edges decreases in a direction from the left end to right end of the image in regard to the pair of edges detected for the object in the captured image. In this case, it can be determined that the distortion shape of the object is "right distortion". In this fashion, the distortion shape of the object can be determined based on a direction in which the edge interval between a pair of edges detected for the object changes.

In step S1205, based on the distortion shape determined in step S1204, the image capturing apparatus sets the driving directions of rotation control of the camera unit <NUM> and shift control of the image sensor <NUM> suited to each distortion shape.

<FIG> is a table showing the relationship between rotation control and shift control for each distortion shape. As shown in <FIG>, the driving directions of the camera unit <NUM> and image sensor <NUM> for correcting each distortion shape can be uniquely decided for each distortion shape.

In step S1206, the image capturing apparatus performs distortion correction driving. Assume that distortion correction driving (step driving) is performed by every predetermined amount and steps S1206 to S <NUM> are repeated. Note that this processing is similar to the processing (steps S902 to S907) described with reference to <FIG>, and a description thereof will not be repeated.

In step S1207, the image capturing apparatus calculates a current distortion amount generated in the correction target object. Even for the distortion amount calculation, depth information-based calculation and edge detection-based calculation are possible.

<FIG> is a view for explaining distortion amount calculation based on depth information. <FIG> exemplifies a change of the captured image during distortion correction driving in step S <NUM>. A captured image <NUM> represents that the difference in depth (object distance) between the upper and lower portions of the correction target object is large. In this case, the distortion amount is calculated to be a large value. In a captured image <NUM> obtained after performing distortion correction driving of one or more steps, the difference in depth between the upper and lower portions of the correction target object becomes smaller than that in the captured image <NUM>. Hence, the distortion amount is calculated to be a smaller value than for the captured image <NUM>. In a captured image <NUM> obtained after further performing distortion correction driving, the depths at the upper and lower portions of the correction target object are substantially equal. The distortion amount is therefore calculated to be a value close to <NUM>. When the distortion amount is calculated based on depth information, the difference in depth between a plurality of regions (for example, upper and lower portions) in a correction target object can be evaluated as a distortion amount.

<FIG> is a view for explaining distortion amount calculation based on edge detection. <FIG> exemplifies a change of the captured image during distortion correction driving in step S1206. A captured image <NUM> represents that the difference in edge interval between a pair of edges is large between the upper and lower portions of the correction target object. In this case, the distortion amount is calculated to be a large value. In a captured image <NUM> obtained after performing distortion correction driving of one or more steps, the difference in edge interval between the upper and lower portions of the correction target object becomes smaller than that in the captured image <NUM>. The distortion amount is calculated to be a smaller value than for the captured image <NUM>. In a captured image <NUM> obtained after further performing distortion correction driving, the edge intervals at the upper and lower portions of the correction target object is substantially equal. The distortion amount is therefore calculated to be a value close to <NUM>. When the distortion amount is calculated based on edge detection, the difference in edge interval between a plurality of regions (for example, upper and lower portions) in a correction target object can be evaluated as a distortion amount.

In step S1208, the image capturing apparatus determines whether the distortion amount calculated in step S1207 is equal to or smaller than a predetermined threshold. If the distortion amount is larger than the predetermined threshold, the distortion correction is not enough and the process returns to step S1206. If the distortion amount is equal to or smaller than the predetermined threshold, the process advances to step S1209.

In step S1209, the image capturing apparatus determines whether the distortion shape determination (step S1204) has been performed twice. This is because both a vertical distortion (upper or lower distortion) and horizontal distortion (right or left distortion) of the captured image sometimes exist. Since only either vertical distortion or horizontal distortion is corrected by the correction (steps S1204 to S <NUM>) based on one distortion shape determination, it is determined whether the distortion shape determination has been performed twice.

As described above, the image capturing apparatus can correct a distortion generated in an object of a captured image based on depth information corresponding to the captured image or the result of edge detection to the captured image.

Claim 1:
An image capturing apparatus comprising:
an image capturing means (<NUM>) including an imaging optical system (<NUM>, <NUM>) and an image sensor (<NUM>);
a rotation driving means (<NUM>) configured to drive the image capturing means in a pan direction and/or tilt direction;
a shift driving means (<NUM>) configured to drive at least one of the imaging optical system and the image sensor within a plane parallel to an imaging plane; and
a synchronous control means (<NUM>) configured to synchronize a rotation driving of the image capturing means and a shift driving of at least one of the imaging optical system and the image sensor so as not to change an imaging range of the image capturing means during performing a correction on distortion of an object in a captured image captured by the image capturing means,
wherein
the synchronous control means (<NUM>) is configured to decide a shift driving amount of the shift driving means based on a rotation driving amount of the rotation driving means, a size of the image sensor and an imaging distance from the imaging optical system to the image sensor.