Patent Description:
A known image capture apparatus can capture an omnidirectional image with a <NUM>-degree field of view (see <CIT>). Such an image capture apparatus is referred to as a <NUM>-degree camera or an omnidirectional camera.

For example, a <NUM>-degree camera provided with two circular fisheye lenses with a <NUM>-degrees field of view includes a function for cropping a rectangular region from an image captured using only one lens and generating an image like one captured with a wide-angle lens with a field of view that is less than <NUM> degrees.

In this case, electronic image stabilization can be performed irrespective of the rotation angle for the rotation (movement in the roll direction) of the camera about the optical axis. However, for rotation (movement in the pitch and yaw direction) of the camera about the axes orthogonal to the optical axis, there are restrictions on the rotation angle for which image stabilization can be performed.

Furthermore, <CIT> discloses a technology to encode <NUM>° video content in graphics processing architectures. In this regard, it is determined, on a per camera basis, an interest level with respect to panoramic video content, a subset of cameras in a plurality of cameras for which the interest level is below a threshold is identified, and power consumption in the subset of cameras is reduced. Still further, <CIT> refers to an image capturing system, wherein a processing circuitry obtains an image captured by a device such as a camera, and detects an attitude of the device. The processing circuitry corrects the captured image based on the attitude of the device and a designation including a point on the image and at least one of a region of the image and a direction.

It is an object of the present invention to provide an image capture apparatus that can generate an omnidirectional image using a plurality of lenses and a control method therefor that can enhance image stabilization when generating an image with a smaller field of view than an omnidirectional image.

According to the present invention, this object is achieved by an image capture apparatus according to claim <NUM>, a control method for an image capture apparatus according to claim <NUM>, and a computer-readable medium according to claim <NUM>.

Further features and advantageous modifications are shown in the dependent claims.

Note that in the embodiments described below, the present invention is embodied as an image capture apparatus such as a digital camera. However, the present invention can be implemented with any electronic device with an image capture function. Examples of such an electronic device include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, and the like), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are examples, and the present invention can be implemented with other electronic devices.

<FIG> are perspective views illustrating an example of the appearance of a digital camera <NUM> representing an example of an image capture apparatus according to an embodiment of the present invention. Herein, the side where a shutter button <NUM> is provided corresponds to the front side of the digital camera <NUM>. Thus, the front side of the example configuration is illustrated in <FIG>, and the rear side is illustrated in <FIG>. Also, <FIG> is a block diagram illustrating an example of the functional configuration of the digital camera <NUM>.

Hereinafter, an exemplary configuration of the digital camera <NUM> will be described using <FIG>. The digital camera <NUM> includes imaging lenses 103a and 103b, which are circular fisheye lenses with a <NUM>-degree field of view, on the front side and the rear side, respectively. The imaging lenses 103a and 103b, for example, include the same optical axis and are configured to have capture directions that are <NUM> degrees different from one another. In the present specification, the capture direction of the imaging lens 103a provided on the front side of the digital camera <NUM> is referred to as the front, and the capture direction of the imaging lens 103b provided on the rear side of the digital camera <NUM> is referred to as the rear.

The digital camera <NUM> is an omnidirectional camera or a <NUM>-degree camera that can generate an omnidirectional image with a <NUM>-degree horizontal field of view by combining or stitching together images formed by the imaging lenses 103a and 103b. Note that the omnidirectional image may be a <NUM>-degree image and may have a vertical field of view of less than <NUM> degrees like a <NUM>-degree image. In the present embodiment, the two imaging lenses 103a and 103b are circular fisheye lens with a horizontal field of view of <NUM> degrees or greater, and the digital camera <NUM> can generate a <NUM>-degree image.

Note that in this example, the two imaging lenses 103a and 103b cover a horizontal field of view of <NUM> degrees or greater, but a configuration may be used in which three or more imaging lenses with different capture directions cover a horizontal field of view of <NUM> degrees or greater.

The digital camera <NUM> includes a plurality of image capture units including a front camera <NUM> that captures images of the front and a rear camera <NUM> that captures images of the rear. The front camera <NUM> includes the imaging lens 103a and a barrier 102a (protective member) provided on the front side of the imaging lens 103a. The barrier 102a is a transparent member with a hemispherical shape, for example. The imaging lens 103a includes a focus lens that can move in the optical axis direction. A shutter 101a also functions as a diaphragm. The imaging lens 103a forms an optical image on an imaging surface of an image sensor 22a.

The image sensor 22a, for example, may be a known CCD or CMOS color image sensor including a primary color Bayer array color filter. The image sensor 22a includes a pixel array including a plurality of pixels in a two-dimensional array and a peripheral circuit for reading signals from the pixels. The pixels accumulate a charge corresponding to the incident light via photoelectric conversion. By reading a signal including voltage corresponding to the amount of charge accumulated in the exposure period from each pixel, a pixel signal group (analog image signal) representing an optical image formed on the imaging surface is obtained. An A/D converter 23a converts an analog image signal read out from the image sensor 22a into a digital image signal (image data).

Also, the front camera <NUM> includes a mechanism for moving the focus lens, a mechanism for detecting the position of the focus lens, and a mechanism for driving the shutter 101a. Note that the front camera <NUM> may include a movement mechanism for the image sensor 22a for implementing an optical blur correction function and/or a movement mechanism for a shift lens included in the imaging lens 103a.

The rear camera <NUM> has the same configuration as the front camera <NUM>, and thus description of the components is omitted. Note that hereinafter, the image sensors 22a and 22b are collectively referred to as the image sensors <NUM>. This also applies to other components included in the front camera <NUM> and the rear camera <NUM>. In the present embodiment, since a configuration is used in which one image capture unit uses one imaging lens, selectively using an image capture unit and selectively using an imaging lens have the same meaning. Thus, a description relating to enabling and disabling an image capture unit (camera) can be read as enabling and disabling image capture via an imaging lens. However, the present invention can be applied to configurations in which one image capture unit simultaneously uses a plurality of imaging lenses or switches between using a plurality of imaging lenses.

The image data output from the A/D converter <NUM> is written on memory <NUM> via both an image processing unit <NUM> and a memory control unit <NUM> or via only the memory control unit <NUM>.

The memory <NUM> is used as a buffer for the image data, work memory of the image processing unit <NUM>, video memory of a display unit <NUM>, and the like.

The image processing unit <NUM> applies predetermined image processing to the image data output by the A/D converter <NUM> or the memory control unit <NUM> or stored in the memory <NUM> and acquires or generates signals corresponding to the application, image data, various types of information. The image processing unit <NUM> may be, for example, a dedicated hardware such as an Application-Specific Integrated Circuit (ASIC) configured to implement a specific function. Alternatively, the image processing unit <NUM> may have a configuration for implementing a specific function by the processor, such as a Digital Signal Processor (DSP) or a Graphics Processing Unit (GPU), executing a piece of software.

The image processing to be applied to the image data by the image processing unit <NUM> includes, for example, preprocessing, color interpolation processing, correction processing, detection processing, data modification processing, evaluation value calculation processing, special effects processing, and the like.

Preprocessing may include signal amplification, reference level adjustment, defective pixel correction, and the like.

Color interpolation processing is processing executed when the image sensors <NUM> are provided with a color filter for interpolating values of color components that are not included in the pieces of pixel data forming the image data. Color interpolation processing is also referred to as demosaic processing.

Correction processing may include various processing including white balance adjustment, tone correction, correction (image restoration) of image degradation caused by an optical aberration in the imaging lenses 103a and 103b, color correction, and the like. Correction processing also includes processing for correcting significant distortion at the peripheral portion of an ultra wide angle lens such as a circular fisheye lens.

Detection processing may include processing for detecting a feature area (for example, a face area or a human body area) or movement thereof, processing for recognizing a person, and the like.

Data modification processing may include processing including region cropping (trimming), combining, scaling, encoding and decoding, header information generation (data file generation), and the like. Data modification processing also includes generating image data for display and image data for recording.

Evaluation value calculation processing may include processing including generating signals or evaluation values that are used in automatic focus detection (AF), generating evaluation values that are used in automatic exposure control (AE), and the like.

Special effects processing may include processing including adding a blur effect, changing color tone, relighting, and the like.

Note that these are examples of the processing that can be applied by the image processing unit <NUM>, and are not intended to limit the processing applied by the image processing unit <NUM>. Also, image processing that can be applied by the image processing unit <NUM> may be executed by a system control unit <NUM>.

In the present embodiment, the image processing unit <NUM> stitches together two circular fisheye images captured by the front camera <NUM> and the rear camera <NUM> and generates a <NUM>-degree image. The generated <NUM>-degree image is transformed into an image using equidistant cylinder transformation, and the positions of the pixels can be associated with the coordinates of the <NUM>-degree surface.

Also, the image processing unit <NUM> crops a region from the data of the circular fisheye image, applies distortion correction, and the like and generates image data for live view display in a rectangular shape suitable for the display unit <NUM> or an external display apparatus.

The system control unit <NUM>, for example, is a processor (CPU, MPU, microprocessor, or the like) that can execute programs. The system control unit <NUM>, by loading a program stored in a non-volatile memory <NUM> onto a system memory <NUM> and executing the program, controls the operations of the functional blocks of the digital camera <NUM> and implements the functions of the digital camera <NUM>.

The non-volatile memory <NUM> is electrically rewritable and stores a program executed by the system control unit <NUM>, various setting values of the digital camera <NUM>, GUI data, and the like. The system memory <NUM> is used as the main memory when the system control unit <NUM> executes a program. Note that the memory <NUM> and the system memory <NUM> may be different areas within a continuously memory space.

The shutter button <NUM> includes a switch SW1 <NUM> that turns ON with a half press and a switch SW2 <NUM> that turns ON with a full press. The system control unit <NUM> recognizes ON at the switch SW1 <NUM> as a still image capture preparation instruction and ON at the switch SW2 <NUM> as a still image capture start instruction. When ON at the switch SW1 <NUM> is detected, the system control unit <NUM> uses a signal, evaluation value, or the like generated by the image processing unit <NUM> and executes automatic focus detection (AF) and automatic exposure control (AE) for the front camera <NUM> and/or the rear camera <NUM>. Also, when ON at the switch SW2 <NUM> is detected, the system control unit <NUM> controls the shutters <NUM> in according with the exposure condition determined by AE processing, captures a still image, and executes recording processing. The still image data for recording generated by the image processing unit <NUM> is temporarily stored in the memory <NUM> before being recording on a recording medium <NUM> via an I/F <NUM> by the system control unit <NUM>.

Note that the shutter button <NUM> may include only one switch. In this case, when ON at the switch is detected, the system control unit <NUM> continuously executes image preparation operations and image capture processing.

The operation mode of the digital camera <NUM> can be changed by the user operating a mode change switch <NUM>. The operation modes include, for example, a mode for capturing a <NUM>-degree image, a mode for capturing a typical wide-angle image with a horizontal field of view of less than a <NUM>-degree (referred to as crop mode), a playback mode, a mode for operating in cooperation with an external apparatus, and the like. Note that the mode for capturing a wide-angle image may include a plurality of modes (<NUM>-degree mode, <NUM>-degree mode, and the like) corresponding to the horizontal field of view.

Note that the operation mode can be selected via a combination of an operation of the mode change switch <NUM> and an operation of another operation member. For example, the mode change switch <NUM> may be used to select the broad category of the operation mode, and then, from the more specific categories displayed on the display unit <NUM>, the operation mode may be selected.

A power switch <NUM> is a switch for instructing to turn ON or OFF the power supply of the digital camera <NUM>. The power supply control unit <NUM> controls these operations.

The power supply control unit <NUM> includes a battery detection circuit, a DC-DC converter, a switch circuit for switching between functional blocks that supply power, and the like and controls the power supply to the components of the digital camera <NUM> from a power supply unit <NUM>, which may be a battery or an AC adapter. The power supply control unit <NUM> detects the type of the power supply unit <NUM>. Also, when the power supply unit <NUM> is a battery, the power supply control unit <NUM> detects the type and remaining battery amount. The power supply control unit <NUM> can change the components that supply power and the power supplied in accordance with the state of the power switch <NUM> and control by the system control unit <NUM>.

A microphone <NUM> is pointed to the outside of the digital camera <NUM> and outputs audio signals to the system control unit <NUM>. When moving images are recording, image data and audio data are recorded.

An operation unit <NUM> is a generic term for an input device assembly (buttons, switches, dials, and the like) besides the mode change switch <NUM>, the shutter button <NUM>, and the power switch <NUM>. This includes a moving image recording switch, a menu button, a directional key, an enter key, and the like. When the display unit <NUM> is a touch display, the operation unit <NUM> may also be constituted by a touch-operable software button or key.

The display unit <NUM> is a liquid crystal display (LCD), for example. The display unit <NUM> may not be a touch display. The display unit <NUM> displays characters and images. By displaying moving images continuously captured immediately on the display unit <NUM>, the display unit <NUM> can function as an electronic viewfinder (EVF). Moving images displayed with a display apparatus functioning as an EVF are referred to as live view images. Note that by outputting a live view image to an external apparatus connected to a communication unit <NUM>, a display apparatus included in an external apparatus can function as an EVF.

A light-emitting unit <NUM> is a light-emitting diode (LED) that notifies the user of the status of the digital camera <NUM> and the like via the pattern or color of the emitted light.

A fixing portion <NUM> (<FIG>) provided on the bottom side of the digital camera <NUM> is a threaded hole where a tripod is attached, for example.

A system timer <NUM> outputs the time of an internal clock and measures time in response to a request from the system control unit <NUM>.

The communication unit <NUM> is an interface for wired or wireless communication with an external apparatus, such as an electronic device described below, an external display apparatus, or the like. The communication unit <NUM> is compatible with one or more wired or wireless communication standards and includes a connector, a transceiver, and the like appropriate for that standard. Representative standards the communication unit <NUM> may support include, but are not limited to, USB, HDMI (registered trademark), Bluetooth (registered trademark), wireless LAN, and the like.

An orientation detection unit <NUM>, for example, includes a gyro sensor and an acceleration sensor and outputs signals representing the orientation and movement of the digital camera <NUM> to the system control unit <NUM>. The orientation of the digital camera <NUM> is represented by the rotation angles (roll, pitch, and yaw) about angular axes, with the x-axis being parallel with the optical axis, the y-axis extending in the horizontal direction, and the z-axis extending in the vertical direction. The orientation of the digital camera <NUM> when capturing an image may be associated with the image data and recorded. Also, the detected orientation and movement of the digital camera <NUM> can be used in image stabilization and tilt correction.

The I/F <NUM> is an interface for writing data to the recording medium <NUM>, which is a memory card, hard disk, or the like, and reading data recorded on the recording medium <NUM>. The recording medium <NUM> may or may not be detachable from the digital camera <NUM>.

<FIG> are diagrams of a smartphone <NUM>, which is an example of an external apparatus that can cooperate with the digital camera <NUM> via communication using the communication unit <NUM>. <FIG> is a perspective view illustrating an example of the appearance, and <FIG> is a block diagram illustrating an example of the functional configuration. Note that the external apparatus is not limited to a smartphone and it is sufficient that the electronic device can communicate with the digital camera <NUM> and include a processor that can execute an application for cooperating with the digital camera <NUM>.

The configuration of the smartphone <NUM> will now be described.

An internal bus <NUM> connects each block allowing for data to be exchanged in both directions.

A CPU <NUM> is a processor that can execute programs and, by loading a program stored in a non-volatile memory <NUM> on a memory <NUM> and executing the program, controls the operations of the functional blocks of the smartphone <NUM> and implement the functions of the smartphone <NUM>.

The memory <NUM> is used as the main memory when the CPU <NUM> executes a program. A portion of the memory <NUM> is used as the video memory of a display <NUM>.

The non-volatile memory <NUM> is electrically rewritable and stores programs (OS and applications) executed by the CPU <NUM>, setting values for the smartphone <NUM>, GUI data, user data, and the like.

The display <NUM> is an LCD, for example, and is where the OS and applications display images and various types of information. The display <NUM> is a touch display including a touch panel 206a and can detect a touch operation on the display surface of the display <NUM>. The display <NUM> may be an external apparatus.

The image processing unit <NUM> applies image processing to image data stored in the non-volatile memory <NUM> and a recording medium <NUM>, image data obtained via an external I/F <NUM>, image data obtained via a communication I/F <NUM>, and the like on the basis of control by the CPU <NUM>.

The image processing to be applied to the image data by the image processing unit <NUM> may be similar to the image processing unit <NUM> of the digital camera <NUM>. When the smartphone <NUM> does not include a camera, generation of evaluation values used in AF and AE may not be executed. Also, the image processing that the image processing unit <NUM> can execute may be executed by the CPU <NUM>.

The image processing unit <NUM> can generate image data for Virtual Reality (VR) display corresponding to the movement of the smartphone <NUM> from ultra wide angle images (for example, images with a horizontal field of view greater than <NUM> degrees) such as an omnidirectional image in accordance with control by the CPU <NUM>. VR display is implemented by cropping a region of an area to be captured corresponding to a change in the orientation of the smartphone 200from an ultra wide angle image and displaying the generated cropped image data on the display <NUM>. With VR display, for example, by fixing the display <NUM> of the smartphone <NUM> in front of the eyes of the user using goggles or the like, the image displayed changes, tracking the movement of the head of the user. In this manner, the user can experience the feeling of being inside the virtual space represented by the ultra wide angle image.

Alternatively, by controlling the image processing unit <NUM> to crop a region of the same area to be captured from an ultra wide angle image to irrespective of the orientation of the smartphone <NUM>, electronic image stabilization can be implemented.

An operation unit <NUM> is a generic term for an input device assembly the user can use to give instructions to the smartphone <NUM>. Input devices typically included in the smartphone <NUM> include but are not limited to a button, a switch, and a touch panel. Also, the operation unit <NUM> may be constituted by a keyboard and mouse communicatively connected to the smartphone <NUM>. Note that in <FIG>, the touch panel 206a is illustrated separate to the display <NUM>. However, the touch panel 206a is actually built-in or attached to the display screen of the display <NUM>.

A power button 206b, volume buttons 206c and 206d, and a home button 206e are examples of input devices constituting the operation unit <NUM>. The power button 206b switches the power of the smartphone <NUM> ON and OFF. The volume buttons 206c and 206d are buttons for increasing or decreasing the volume output from an audio output unit <NUM>. The home button 206e is a button for displaying a specific screen provided by the OS on the display <NUM>.

A medium I/F <NUM> is an interface for accessing the recording medium <NUM>. When the recording medium <NUM> is a removable media such as a memory card, for example, the medium I/F <NUM> includes a slot where the recording medium <NUM> is inserted/removed. The CPU <NUM> can write data to the recording medium <NUM> via the medium I/F <NUM> and read out data from the recording medium <NUM>.

The external I/F <NUM> is an interface for wired or wireless communication with an external apparatus, such as the digital camera <NUM> or an external display apparatus. The external I/F <NUM> is compatible with one or more wired or wireless communication standards and includes a connector, a transceiver, and the like appropriate for that standard. Representative standards the communication unit <NUM> may support include, but are not limited to, USB, HDMI (registered trademark), Bluetooth (registered trademark), wireless LAN, and the like.

The communication I/F <NUM> is an interface for communications via a cellular network <NUM>. The communication I/F <NUM>, for example, may be a communication I/F compliant with a mobile communications standard established by the 3GPP, such as <NUM>, <NUM>, or <NUM> modem.

The audio output unit <NUM> outputs audio (audio based on moving images or music data, operation sounds, ringtones, various types of notification sounds, and the like). The audio output unit <NUM> includes an audio output terminal 212a that an earphone or the like can connect to and a speaker 212b but may output audio to an external apparatus via the external I/F.

An orientation detection unit <NUM>, for example, includes a gyro sensor and an acceleration sensor and outputs signals representing the orientation and movement of the smartphone <NUM> to the CPU <NUM>. The orientation of the smartphone <NUM> is represented by the rotation angles (roll, pitch, and yaw) about angular axes, with the x-axis being orthogonal to the display screen of the display <NUM>, the y-axis extending in the horizontal direction, and the z-axis extending in the vertical direction. The orientation detected by the orientation detection unit <NUM> can be used in the VR display described above.

<FIG> schematically illustrate operation control of the front camera <NUM> and the rear camera <NUM> according to the orientation when the digital camera <NUM> is operating in an operation mode (crop mode) for capturing still images with a horizontal field of view of less than <NUM> degrees.

<FIG> illustrates an example of the orientation (reference orientation) of the digital camera <NUM> when image capture starts, and <FIG> illustrates an example of the orientation of the digital camera <NUM> during image capture. In the state (upright state) illustrated in <FIG>, the roll, pitch, and yaw are all at <NUM> degrees. Also, <FIG> illustrates a state in which, from the state of <FIG>, only the pitch (rotation angle about the y-axis) has changed, with the rear camera <NUM> facing downward.

In crop mode, an image is generated for recording or display by cropping a partial area from an omnidirectional image or an ultra wide angle image. When the horizontal field of view of the image generated in crop mode can be covered by one of the imaging lens, there is no need for image capture using the other lens. This allows only the imaging lens and related circuit required for image capture to be used.

For example, if the horizontal field of view of the image generated in crop mode by the digital camera <NUM> according to the present embodiment is less than <NUM> degrees, by enabling only one of the front camera <NUM> or the rear camera <NUM>, power consumption can be reduced. In this example, the rear camera <NUM> is used when starting image capture so that the display unit <NUM> can be viewed when starting image capture. Also, an image with a horizontal field of view of <NUM> degrees and a vertical field of view of <NUM> degrees is generated.

Furthermore, electronic image stabilization is executed by changing the crop position of the image so that the same area to be captured as when starting image capture is maintained, even when the orientation of the digital camera <NUM> changes from the reference orientation (for example, the orientation when starting image capture) during image capture. Hereinafter, the area cropped from the capturable area in crop mode is referred to as a recording area, for the sake of convenience. Note that the use of the image data generated in crop mode is not limited to recording.

The orientation (reference orientation) of the digital camera <NUM> when starting image capture is an upright state. In this case, an arrow <NUM> indicates the capture direction (optical axis direction of the imaging lenses 103a and 103b). Accordingly, of a capturable area <NUM> (right semicircle) of the rear camera <NUM>, an area <NUM> indicated a gray color corresponds to the vertical recording area. Since the front camera <NUM> is disabled, a capturable area <NUM> (left semicircle) of the front camera <NUM> is not captured. The direction of a boundary line <NUM> between the capturable area <NUM> of the front camera <NUM> and the capturable area <NUM> of the rear camera <NUM> is aligned with a gravity direction <NUM>.

When the digital camera <NUM> changes from the orientation illustrated in <FIG> to the orientation illustrated in <FIG>, to implement image stabilization, cropping an area the same as the recording area <NUM> when starting image capture is required. However, of the recording area <NUM>, the hatched portion in <FIG> is not included in the capturable area <NUM> of the rear camera <NUM> and is included in the capturable area <NUM> of the front camera <NUM>. Thus, image stabilization cannot be implemented with only the image data obtained by the rear camera <NUM>.

In this manner, when image stabilization cannot be implemented with the area to be captured of one camera, the system control unit <NUM> enables another camera (in this example, the front camera <NUM>) with an area to be captured that includes the area required for image stabilization and executes image capture with the plurality of cameras. Accordingly, even when the digital camera <NUM> changes to the orientation of <FIG>, the recording area <NUM> can be cropped to allow image stabilization to be implemented.

On the other hand, when image stabilization can be implemented with the area to be captured of one camera, by disabling the other camera, power consumption can be reduced. Note that when the recording area approaches a boundary of the current capturable area, the camera that can capture the area required if the recording area exceeds the capturable area may be enabled. This can help avoid being unable to execute image stabilization while a camera additionally enabled is activated and an image is obtained.

Note that in the example described above using <FIG>, control is performed according to a change in the orientation in the pitch direction. However, this also applies to a change in the orientation in the yaw direction.

The operations of the system control unit <NUM> when in crop mode will be further described using the flowchart of <FIG>. These operations are executed when the digital camera <NUM> is operating in crop mode, image stabilization is set to enabled, and moving image capture is started. Note that the moving image capture may be for recording or may be for live view display, but in the example described below, the moving image capture is for recording. Step S400 may be executed by, but is not limited to, the operation of a moving image switch in a standby state when capturing for recording or by the detection of an enable image stabilization operation in a standby state when capturing for live view display.

Note that when remote operation by an external apparatus is enabled, step S400 may be executed by a start recording instruction received via the communication unit <NUM> from an external apparatus.

In step S400, the system control unit <NUM> obtains the orientation (reference orientation) of the digital camera <NUM> from the orientation detection unit <NUM>. The system control unit <NUM> determines the optical axis direction of the rear camera <NUM> or the front camera <NUM> as the capture direction on the basis of the obtained orientation. The capture direction is a unidirectional direction that is not affected by changes to the orientation of the digital camera <NUM> thereafter. The system control unit <NUM> stores the obtained orientation and the determined capture direction in the system memory <NUM>, for example.

In step S401, the system control unit <NUM> determines the recording area on the basis of the capture direction determined in step S400 and the settings of the digital camera <NUM>. In this example, in crop mode, the horizontal field of view is selectable and the vertical field of view is constant or determined according to the horizontal field of view. In this example, a horizontal field of view of <NUM> degrees is selected, and a vertical field of view of <NUM> degrees is determined.

The system control unit <NUM> associates the area determined as the recording area from the orientation and capture direction of the digital camera <NUM> when starting recording and the capturing field of view with the orientation of the digital camera <NUM> when starting image capture and stores these in the system memory <NUM>. The system control unit <NUM>, for example, can set, as the recording area, a region obtained by mapping a rectangular region with a size based on the horizontal field of view and the vertical field of view, centered on the capture direction, on an omnidirectional image according to the orientation of the digital camera <NUM>. Thus, the recording area corresponds to a specific region in an omnidirectional image the digital camera <NUM> can generate.

In step S402, the system control unit <NUM> starts moving image recording operations, and the processing of step S403 onward is continuously executed for each frame.

In step S403, the system control unit <NUM> obtains the orientation of the digital camera <NUM> from the orientation detection unit <NUM>.

In step S404, the system control unit <NUM> determines the camera to enable on the basis of the recording area and a change in the orientation from when starting image capture. This will be described below in detail.

In step S405, the system control unit <NUM> executes image capture equal to one frame using the enabled camera. Since only the rear camera <NUM> is enabled when starting image capture, the system control unit <NUM> executes image capture of one frame using only the rear camera <NUM>. The image processing unit <NUM> generates image data for recording and image data for display for <NUM> frame read out from the enabled camera and stores this in the memory <NUM>. Note that the operations relating to determining the image capture conditions and driving the focus lens and image sensor when capturing an image are known and thus not described.

In step S406, the system control unit <NUM> determines whether or not a plurality of cameras are enabled. This determination corresponds to a determination of whether or not combining images is required. The digital camera <NUM> according to the present embodiment includes the front camera <NUM> and the rear camera <NUM>, and if both cameras are enabled, the system control unit <NUM> executes step S407, and if only one camera is enabled, the system control unit <NUM> skips step S407 and executes step S408.

In step S407, the system control unit <NUM> instructs the image processing unit <NUM> to combine (stitch together) the image data captured by the enabled cameras. In response to the instruction, the image processing unit <NUM> combines (stitches together) the image data of the current frame stored in the memory <NUM> and generates data of a combined image with a continuous area to be captured. Note that the image processing unit <NUM> already knows what kind of positional relationship to stitch together the images obtained by the cameras. The image processing unit <NUM> stores the combined image data in the memory <NUM>. When the system control unit <NUM> generates the combined image data, step S408 is executed.

In step S408, the system control unit <NUM> instructs the image processing unit <NUM> to crop the image data of the recording area. The system control unit <NUM>, for example, reads out the orientation when starting image capture obtained in step S400 and the orientation obtained in step S403 and the recording area determined in step S401 from the system memory <NUM> and sends this and the cropping instruction to the image processing unit <NUM>. Note that, instead of orientation information, the position corresponding to the capture direction when starting image capture obtained when determining which camera to enable in step S404 may be sent to the image processing unit <NUM>.

The image processing unit <NUM> calculates the position of the recording area in the image obtained with the current orientation on the basis of a change in the orientation of the digital camera <NUM> from when starting image capture to present. Then, by cropping the recording area from the image data, the image processing unit <NUM> obtains image data with the same area to be captured as the recording area when starting image capture and stores this in the memory <NUM>. Note that a known technique can be used for the necessary distortion correction and the like to make the region cropped from the circular fisheye image a regular rectangular image.

Note that the image processing unit <NUM> executes the combining processing of step S407 and the cropping processing of step S408 for both the image data for recording and the image data for display. Also, the image processing unit <NUM> may execute the necessary distortion correction and the like to make the region cropped from the circular fisheye image a regular rectangular image.

Of the recording area image data obtained by cropping by the image processing unit <NUM>, the image data for recording is stored on the recording medium <NUM> by the system control unit <NUM> as wide-angle image data generated in crop mode. Also, the system control unit <NUM> executes guide display processing in step S409 for the image data for display before outputting this to the display unit <NUM> or an external apparatus. Note that data of a plurality of frames may be collectively recorded on the recording medium <NUM> in accordance with the encoding method or the like.

In step S409, the system control unit <NUM> executes guide display processing. This will be described below in detail.

In step S410, the system control unit <NUM> determines whether or not an image capture end operation has been performed. When an image capture end operation is determined to have been performed, operations relating to moving image recording end, and when this is not determined, processing on the next frame from step S403 is executed. The image capture end operation may be the operation of a moving image recording switch or an end recording instruction received via the communication unit <NUM> from an external apparatus.

Next, the operations of the system control unit <NUM> in step S404 will be described using the flowchart illustrated in <FIG>.

In step S500, the system control unit <NUM> obtains the difference between the reference orientation (in this example, the orientation when starting image capture) of the digital camera <NUM> and the current orientation of the digital camera <NUM> obtained in step S403. Then, the system control unit <NUM> calculates the position of the recording area in the image obtained in the current orientation on the basis of the difference in orientation.

Furthermore, the system control unit <NUM> determines whether or not the recording area corresponding to the current orientation is greater than the capturable area of the currently enabled camera. As illustrated in the example in <FIG>, the system control unit <NUM> detects when the recording area <NUM> exceeds the capturable area <NUM> of the enabled rear camera <NUM> (includes an area not included in the capturable area <NUM>). When the system control unit <NUM> determines that the recording area corresponding to the current orientation is greater than the capturable area of the currently enabled camera, step S501 is executed, and when this is not determined, step S503 is executed.

In step S501, of the cameras currently disabled, the system control unit <NUM> enables the camera with a recording area corresponding to the current orientation of the digital camera <NUM> included in the area to be captured, and the processing of step S404 ends. For example, in the example illustrated in <FIG>, the front camera <NUM> is enabled.

Steps S503 to S508 correspond to processing executed by the system control unit <NUM> for the cameras of the digital camera <NUM> that are currently enabled.

In step S504, the system control unit <NUM> obtains the capturable area.

In step S505, the system control unit <NUM> determines whether or not the capturable area obtained in step S504 includes at least a portion of the recording area corresponding to the current orientation of the digital camera <NUM> calculated in step S500. When the system control unit <NUM> determines that the capturable area includes at least a portion of the recording area, step S506 is executed, and when this is not determined, step S507 is executed.

In step S506, the system control unit <NUM> keeps the target camera enabled, and the processing of step S404 ends.

In step S507, the system control unit <NUM> disables the target camera, and the processing of step S404 ends. Being disabled is a state in which less power is consumed than when in an enabled state, and power supply to at least one portion may be continued. For example, the image sensor and the A/D converter may be put in a power saving state. This can also be said to be causing a circuit relating to the disabled imaging lens to be transition to a state in which less power is consumed than when the lens is enabled.

In this manner, the camera required to capture the recording area (in other words, the area to be captured required for image generation) corresponding to the current orientation is enabled, and the camera not required to capture the recording area corresponding to the current orientation is disabled. Accordingly, only the required camera is dynamically enabled in response to a change in the orientation of the digital camera <NUM>, allowing the power consumption to be reduced.

Next, the processing of step S409 will be described in detail using <FIG> and <FIG>. For the sake of convenience, in this example, the digital camera <NUM> operates in a cooperation mode with the smartphone <NUM>, and the image data for live view display is transmitted to the smartphone <NUM> via the communication unit <NUM>. Also, with the smartphone <NUM>, an application for cooperating with the digital camera <NUM> is executed, and a display based on the image data for live view display received from the digital camera <NUM> performed on the display <NUM>. Also, the user is holding the digital camera <NUM> and in a state where they can see the display <NUM> of the smartphone <NUM>. Note that the guide display operation described below may be executed with the digital camera <NUM> alone if the live view display can be seen via the display unit <NUM> of the digital camera <NUM>.

<FIG> is a diagram illustrating a specific example of a guide display executed in step S409. A live view image <NUM> is displayed on the display <NUM>. The guide display is a predetermined image, such as an indicator, icon, or the like, superimposed on a live view image when a plurality of cameras are enabled. In this example, the guide display includes a boundary line <NUM>, a number of cameras in use warning <NUM>, and a recommended direction display <NUM>.

The boundary line <NUM> represented by a broken line indicates the boundary of the capturable area of the camera and corresponds to the boundary line <NUM> in <FIG>. For the boundary line <NUM> to be displayed, a plurality of cameras may be required to capture the recording area, as illustrated in <FIG>. When one camera can capture the entire recording area, such as in <FIG>, the boundary line <NUM> is not displayed.

The number of cameras in use warning <NUM> is an icon indicating that a plurality of cameras are being used to capture the recording area. The boundary line <NUM> and the number of cameras in use warning <NUM> are both displayed. Note that another representation method, such as using a character string or a line display instead of an icon, may be used.

The recommended direction display <NUM> is an icon indicating how to change the orientation of the digital camera <NUM> to reduce the number of cameras required to capture the recording area. Looking at the examples illustrated in <FIG>, to change the orientation of <FIG> to the orientation of <FIG>, the digital camera <NUM> can be changed to almost an upright orientation. In this case, the recommended direction display <NUM> can be an icon representing the upright direction (direction to make the digital camera <NUM> upright). More specifically, the current orientation and the recommended orientation may be alternately displayed, or animation may be used in the display.

When the guide display requirements are met, the digital camera <NUM> (system control unit <NUM>) uses the image processing unit <NUM> to generate a live view display image with the guide display superimposed.

Next, the operations of the system control unit <NUM> in step S409 will be described using the flowchart illustrated in <FIG>.

In step S700, the system control unit <NUM> determines whether or not a plurality of cameras are enabled. When it is determined that a plurality of cameras are enabled, step S704 is executed, and when this is not determined, step S701 is executed.

In step S701, the system control unit <NUM> sets the number of cameras in use warning <NUM> to not be displayed.

In step S702, the system control unit <NUM> sets the boundary line <NUM> to not be displayed.

In step S703, the system control unit <NUM> sets the recommended direction display <NUM> to not be displayed. The system control unit <NUM> notifies the image processing unit <NUM> of the settings of steps S701 to S703.

Note that all of the guide displays (the boundary line <NUM>, the number of cameras in use warning <NUM>, and the recommended direction display <NUM>) may be collectively set to not be displayed.

In step S704, the system control unit <NUM> sets the number of cameras in use warning <NUM> to be displayed.

In step S705, the system control unit <NUM> calculates the difference in angle between the optical axis direction and the capture direction for each enabled camera and sets the camera with the smallest difference in angle as the main camera.

In step S706, the system control unit <NUM> sets the boundary line <NUM> of the capturable area of the main camera to be displayed.

In step S707, the system control unit <NUM> calculates the direction to orientate the digital camera <NUM> to reduce the difference in angle between the optical axis direction and the capture direction of the main camera.

In step S708, the system control unit <NUM> sets the recommended direction display <NUM> to be displayed. The system control unit <NUM> notifies the image processing unit <NUM> of the settings of steps S704, S706, and S708 and the direction calculated in step S707.

In step S709, the image processing unit <NUM> generates image data for live view display on the basis of the notification from the system control unit <NUM>.

When steps S701 to S703 are executed, the image processing unit <NUM> generates image data for normal live view display without a guide display superimposed.

On the other hand, when steps S704 to S708 are executed, the image processing unit <NUM> generates image data for live view display with a guide display superimposed. Note that for the guide displays, the superimposition position of the number of cameras in use warning <NUM> and the recommended direction display <NUM> and the image for the recommended direction display <NUM> corresponding to the notified direction are predetermined. Also, regarding the image data for live view display, the image processing unit <NUM> superimposes the boundary line <NUM> at the position corresponding to the boundary of the plurality of images used to generate a combined image in step S407.

Then, the image processing unit <NUM> stores the generated image data for live view display in the memory <NUM>. The system control unit <NUM> transmits the image data for live view display to the smartphone <NUM> via the communication unit <NUM>. The system control unit <NUM> can also output the image data for live view display to the display unit <NUM>.

As described above, according to the present embodiment, with an image capture apparatus that can generate an omnidirectional image using a plurality of image capture units, when generating an image with a smaller field of view than an omnidirectional image, only the image capture unit required for image generation are enabled. Also, when image stabilization cannot be implemented in the capturable area of an enabled image capture unit, the required image capture unit is enabled. Thus, power consumption can be reduced, and the area in which image stabilization can be performed can be increased.

In the configuration of the embodiment described above, a single image capture unit uses a single imaging lens. However, in the configuration of other similar embodiments, the number of imaging lenses used by a single image capture unit can be changed. In this case, image capture using a plurality of lenses may be performed sequentially while switching lenses. Also, operation relating to enabling and disabling the cameras in the embodiment described above is synonymous with enabling and disabling the lenses. A configuration in which the number of lenses used is increased in response to a change in the orientation of the digital camera achieves less of the effect of reducing power consumption compared to a configuration in which the number of image capture units used is increased but achieves a similar effect regarding an increase in the area where image stabilization can be performed.

Claim 1:
An image capture apparatus (<NUM>) that generates an omnidirectional image using a plurality of imaging lenses (103a, 103b) with different capture directions, comprising:
detecting means (<NUM>) for detecting an orientation of the image capture apparatus (<NUM>),
characterized by
identifying means (<NUM>) for, when the image capture apparatus (<NUM>) is in a crop mode as an operation mode for generating a first image that is a stabilized image with a smaller field of view than an omnidirectional image, identifying an area to be captured required to generate the stabilized first image on a basis of the initial position of the area to be captured within a capturable area (<NUM>, <NUM>) of one of the plurality of the imaging lenses when the imaging device is in a reference orientation and a difference between the reference orientation and a current orientation of the image capture apparatus detected by the detecting means; and
control means (<NUM>) for, when the image capture apparatus is in the operation mode, enabling image capture using one or more imaging lenses, from among the plurality of imaging lenses (103a, 103b) having a capturable area (<NUM>) which include at least a portion of the area to be captured and disabling image capture using one or more imaging lenses, from among the plurality of imaging lenses (103a, 103b), having a capturable area (<NUM>) which does not include any portion of the area to be captured.