Patent Description:
In recent years, hand-held cameras capable of capturing still images or moving images with a hemispherical or omnidirectional field of view are widely used. With such hand-held cameras, it is difficult to accurately control the position (angle) of the camera when shooting. In view of such a situation, there are known cameras capable of correcting the inclination of an image captured with the camera body inclined.

The technology for correcting the inclination of an image is disclosed, for example, in <CIT>. <CIT> discloses an image capturing apparatus having two fish-eye lenses that calculates a transformation parameter for transforming a fish-eye image into a holomorphic image, in accordance with an inclination angle detected by an acceleration sensor disposed inside the image capturing apparatus for detecting an inclination angle of the image capturing apparatus with reference to the vertical direction. However, the correction device disclosed in <CIT> fails to correct the rotational distortion of the image around the vertical direction, which is because only the correction for the inclination is made using the acceleration sensor in the apparatus of <CIT>.

Further, <CIT> discloses an omnidirectional imaging system including an image capturing unit, an image synthesizing unit that creates omnidirectional frame data, a posture detecting unit that detects posture information of the image capturing unit, a correction-amount calculating unit that generates correction-amount data for transforming the coordinates of the omnidirectional frame data based on the posture information, and an associating unit that associates the omnidirectional frame data with the correction-amount data to obtain an omnidirectional image. In the omnidirectional imaging system of <CIT>, the correction-amount calculating unit generates, for the omnidirectional frame data, the correction-amount data used to correct the inclination with reference to the vertical direction in the global coordinate and to remove a small vibration component from the horizontal plane vibration, thus to transmit the correction-amount data to the associating unit.

However, the technology disclosed in <CIT> adversely increases the load during the shooting because the posture information of the image capturing unit is detected and the correction-amount data is calculated during the shooting. Further, since only the correction amount data used to remove the small vibration component within the horizontal plane in the global coordinate system is stored in association with the omnidirectional frame data, the information regarding the small vibration component is unavailable in the system of <CIT>.

The patent publication <CIT> discloses the correction of image tilt of captured images based on an acceleration sensor and an angular velocity sensor.

It is difficult to capture an image with the stationary position and posture while holding the camera with a hand. If images are captured with the camera body inclined, the zeniths of the images are misaligned and the horizontal line is distorted unless the zenith correction is performed. If the camera body rotates due to, for example, camera shake, the image blurs along the horizontal line unless the rotation correction is performed.

In view of the above, there is provided image processing apparatus according to claim <NUM>.

Accordingly, the rotational blur of the image around the prescribed reference direction can be effectively corrected.

The aforementioned and other aspects, features, and advantages of the present disclosure will be better understood by reference to the following detailed description when considered in connection with the accompanying drawings.

In describing preferred embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this patent specification is not intended to be limited to the specific terminology so selected, and it is to be understood that each specific element includes all technical equivalents that have the same function, operate in a similar manner, and achieve a similar result. In the following embodiments, an image processing apparatus and an image processing system are described using an omnidirectional camera and an omnidirectional moving-image system. Hereinafter, the overall configuration of an omnidirectional moving-image system <NUM> according to an embodiment of the present disclosure will be described with reference to <FIG> and <FIG>.

<FIG> is a cross-sectional view of an omnidirectional camera <NUM> that constitutes the omnidirectional moving-image system <NUM> according to an embodiment of the present disclosure. The omnidirectional camera <NUM> in <FIG> includes an imaging body <NUM>, a housing <NUM> that holds the imaging body <NUM> and components such as a controller (a central processing unit (CPU <NUM>)) and a battery, and a shutter button <NUM> provided on the housing <NUM>. The imaging body <NUM> in <FIG> includes two image-forming optical systems 20A and 20B and two image sensors 22A and 22B. Examples of the image sensors 22A and 22B include charge-coupled devices (CCDs) and complementary metal oxide semiconductors (CMOSs). The image-forming optical systems 20A and 20B are hereinafter sometimes referred to collectively as an image-forming optical system <NUM>. The image sensors 22A and 22B are hereinafter sometimes referred to collectively as an image sensor <NUM>. Each of the image-forming optical systems 20A and 20B is configured as a fish-eye lens consisting of, for example, seven lenses in six groups. In the embodiment illustrated in <FIG>, the above-mentioned fish-eye lens has a full angle of view of greater than <NUM> degrees (= <NUM> degrees/n where n denotes the number of optical systems and n is <NUM>). Preferably, the fish-eye lens in <FIG> has an angle of view of <NUM> degrees or greater. One of such wide-angle image-forming optical systems <NUM> (20A and 20B) is combined with corresponding one of the image sensors <NUM> (22A and 22B) to constitute a wide-angle imaging optical system (<NUM> and <NUM>).

The relative positions of the optical elements (lenses, prisms, filters, and aperture stops) of the two image-forming optical systems 20A and 20B are defined with reference to the image sensors 22A and 22B. More specifically, these elements are positioned such that the optical axis of the optical element of each of the image-forming optical systems 20A and 20B meets the central portion of the light receiving area of corresponding one of the image sensors <NUM> at the right angle and such that the light receiving area serves as the image-forming plane of corresponding one of the fish-eye lenses.

In the embodiment illustrated in <FIG>, the image forming optical systems 20A and 20B have the same specification and are combined in directions reverse to each other such that the optical axes thereof match with each other. The image sensors 22A and 22B transform the light distribution of the received light into image signals, and sequentially output image frames to an image processing unit (an image processing block <NUM>) of the controller (the CPU <NUM>). As will be described later in detail, the images captured by the respective image sensors 22A and 22B are combined to generate an image over a solid angle of 4π steradian (hereinafter, such an image is referred to as a "spherical image"). The omnidirectional image is an image of all the directions that can be seen from an image capturing point. Thus-obtained consecutive frames of the omnidirectional image form an omnidirectional moving image. In the following embodiments, cases where an omnidirectional image and omnidirectional moving image are generated are described. In some embodiments, a full-circle image and a full-circle moving image, a panoramic image and a panoramic moving image obtained may be generated. Note that such a panoramic image and moving image are obtained by photographing <NUM> degrees only in a horizontal plane.

<FIG> is an illustration of a hardware configuration of the omnidirectional camera <NUM> that constitutes the omnidirectional moving-image system <NUM> according to an embodiment of the present disclosure. The omnidirectional camera <NUM> corresponds to an image processing apparatus or an image-capturing device according to an embodiment to be described.

The omnidirectional camera <NUM> includes the CPU <NUM> (a first CPU), a read only memory (ROM) <NUM>, an image processing block <NUM>, a moving-image compressing block <NUM>, a dynamic random access memory (DRAM) <NUM> that is connected thereto through a DRAM interface <NUM>, and a sensor <NUM> that is connected thereto through a sensor interface <NUM>.

The CPU <NUM> controls the operations of components of the omnidirectional camera <NUM>. The ROM <NUM> stores therein a control program described in a code readable by the CPU <NUM> and various kinds of parameters. The image processing block <NUM> is connected to a first image sensor 130A and a second image sensor 130B (corresponding to the image sensors 22A and 22B in <FIG>, respectively), and receives image signals of images captured by the image sensors 130A and 130B. The image processing block <NUM> includes, for example, an image signal processor (ISP), and applies, for example, shading correction, Bayer interpolation, white balance correction, and gamma correction to the image signals received from the image sensors 130A and 130B.

The moving-image compressing block <NUM> is a codec block for compressing and expanding a video such as video in MPEG-<NUM> AVC/H. <NUM> format. The DRAM <NUM> provides a storage area for temporarily storing data therein when various types of signal processing and image processing are applied.

The sensor <NUM> is a sensor for detecting three-axis acceleration components and three-axis angular velocity components. The detected acceleration component and angular velocity component are used to perform zenith correction of the omnidirectional image in the direction of gravity and rotation correction around the direction of gravity as described later. The sensor <NUM> may further include a sensor such as a geomagnetic sensor for obtaining, for example, an azimuth angle.

The omnidirectional camera <NUM> further includes a storage interface <NUM>, a universal serial bus (USB) interface <NUM>, and a serial block <NUM>. The storage interface <NUM> is connected to an external storage <NUM>. The storage interface <NUM> controls reading and writing to the external storage <NUM>, such as a memory card inserted in a memory card slot. The USB interface <NUM> is connected to a USB connector <NUM>. The USB interface <NUM> controls USB communication with an external device such as a smartphone via the USB connector <NUM>. The serial block <NUM> controls serial communication with an external device such as a smartphone and is connected to a wireless network interface card (NIC) <NUM>.

When the power is turned on by the operation of a power switch, the control program mentioned above is loaded into the main memory (the DRAM <NUM>). The CPU <NUM> controls the operations of the parts of the device and temporarily stores the data required for the control in the memory according to the program read into the main memory. This operation implements functional units and processes of the omnidirectional camera <NUM>, as will be described later.

<FIG> is an illustration of a hardware configuration of the information terminal <NUM> of the omnidirectional moving-image system <NUM> according to the present embodiment. The information terminal <NUM> is an example of an information processing apparatus according to an embodiment of the present disclosure.

The information terminal <NUM> in <FIG> includes a CPU <NUM>, a RAM <NUM>, an internal storage <NUM>, an input device <NUM>, a storage <NUM>, a display <NUM>, a wireless NIC <NUM>, and a USB connector <NUM>.

The CPU <NUM> controls the operations of each and entire component of the information terminal <NUM>. The RAM <NUM> provides the work area of the CPU <NUM>. The internal storage <NUM> stores a control program such as an operating system described with code that can be decrypted by the CPU <NUM> and an application that processes the information terminal <NUM> according to the present embodiment.

The input device <NUM> is an input device such as a touch screen and provides a user interface. The input device <NUM> accepts various instructions from the operator, for example, to correct the omnidirectional moving image. The storage <NUM> is a removable recording medium inserted, for example, into a memory card slot of the information terminal <NUM>, and records various types of data, such as image data in a video format and still image data. The display <NUM> displays the corrected omnidirectional moving image on the screen in response to the user operation. The wireless NIC <NUM> provides a wireless communication connection with an external device such as the omnidirectional camera <NUM>. The USB connector <NUM> provides a USB connection to an external device such as the omnidirectional camera <NUM>.

When the information terminal <NUM> is powered on and the power supply is turned on, the control program is read from the internal storage <NUM> and loaded into the RAM <NUM>. The CPU <NUM> control the operation of each part of the apparatus and temporarily stores the data required for the control in the memory, according to the control program read into the RAM <NUM>. This operation implements functional units and processes of the information terminal <NUM>, as will be described later.

Hereinafter, the omnidirectional moving-image correction function of the omnidirectional moving-image system <NUM> according to the present embodiment are described with reference to <FIG>.

<FIG> is an illustration of functional blocks related to an omnidirectional moving image correction function implemented on the omnidirectional moving-image system <NUM> according to the present embodiment of the present disclosure. As illustrated in <FIG>, the functional block <NUM> of the omnidirectional camera <NUM> includes an omnidirectional moving-image capturing unit <NUM>, a storage unit <NUM>, an omnidirectional moving-image correction unit <NUM> as a correction unit, a receiving unit <NUM>, and an image output unit <NUM> as an output unit. Further, a functional block <NUM> of the information terminal <NUM> includes an instruction receiving unit <NUM> and a display unit <NUM>.

The functional block <NUM> of the omnidirectional camera <NUM> is first described. The omnidirectional moving-image capturing unit <NUM> sequentially captures consecutive frames using the two image sensors 130A and 130B, and records the omnidirectional moving-image data <NUM> in the storage unit <NUM>. Further, the omnidirectional moving-image capturing unit <NUM> measures acceleration components in the three axial directions and angular velocity components in the three axial directions using the sensor <NUM>, and records the measured information as the metadata of the omnidirectional moving-image data <NUM>.

The storage unit <NUM> stores one or more pieces of moving-image data <NUM> recorded by the omnidirectional moving-image capturing unit <NUM>. The storage unit <NUM> serves as a part of a storage area of the external storage <NUM> or another storage in <FIG>. As illustrated in <FIG>, the omnidirectional moving-image data <NUM> includes omnidirectional frame data <NUM> as moving-image data captured by the omnidirectional camera <NUM>, zenith correction data <NUM> that is time-series data of the inclination angle of the omnidirectional camera <NUM> with respect to the reference direction of the omnidirectional camera <NUM> during shooting, and angle velocity data <NUM> that is time-series data of the angle velocity of the omnidirectional camera <NUM> during the shooting operation.

The omnidirectional frame data <NUM> includes a plurality of frames constituting each omnidirectional moving image during the time from the start to the end of an imaging operation. That is, the omnidirectional frame data <NUM> is moving-image data that constitutes the omnidirectional moving-image data <NUM>.

Hereinafter, a description will be given of how to generate an omnidirectional image and the generated omnidirectional image with reference to <FIG>, <FIG>, and <FIG>. <FIG> is an illustration of data structure of each image and the data flow of the image in generating an omnidirectional image. First, an image directly captured by each of the image sensors 130A and 130B is an image that roughly covers a hemisphere of the whole sphere as a field of view. Light that passes through the image-forming optical system <NUM> is focused on the light receiving area of the image sensor <NUM> to form an image according to a predetermined projection system. The image sensor <NUM> is a two-dimensional image sensor defining a planar area of the light receiving area. Accordingly, the image formed by the image sensor <NUM> is image data represented by a plane coordinate system. Such a formed image is a typical fish-eye image that contains an image circle as a whole in which each captured area is projected, as illustrated in a fish-eye image A and a fish-eye image B in <FIG>.

A plurality of fish-eye images of each frame captured by the plurality of image sensors <NUM> is subjected to distortion correction and synthesis processing to form an omnidirectional image (spherical image) for each frame. In the synthesis processing, an omnidirectional image, which constitutes a complementary hemispherical portion, is generated from each planar fish-eye image. Then, the two omnidirectional images including the respective hemispherical portions are joined together by matching the overlapping areas of the hemispherical portions, and the omnidirectional images are synthesized to generate a full spherical (omnidirectional) image including the whole sphere.

<FIG> is an illustration of a plane data structure of the image data of an omnidirectional image used in the omnidirectional moving-image system according to an embodiment of the present disclosure. <FIG> is an illustration of a spherical data structure of the image data of an omnidirectional image used in the omnidirectional moving-image system according to an embodiment of the present disclosure. As illustrated in <FIG>, the image data of the omnidirectional image is expressed as an array of pixel values where the vertical angle φ corresponding to the angle with reference to a certain axis and the horizontal angle θ corresponding to the angle of rotation around the axis are the coordinates. The vertical angle φ ranges from <NUM>° to <NUM>° (alternatively from -<NUM>° to +<NUM>°), and the horizontal angle θ ranges from <NUM>° to <NUM>° (alternatively from -<NUM>° to +<NUM>°).

As illustrated in <FIG>, the respective coordinate values (θ, φ) of the omnidirectional format (the spherical data structure of the image data) are associated with the points on the sphere that represents all directions from the photographing location. Thus, all directions are associated with the points on the omnidirectional images. The plane coordinates of the fish-eye image captured by a fish-eye lens are associated with the coordinates on the sphere of the omnidirectional image, which are included in a predetermined transformation table. The transformation table includes data prepared in advance by, for example, a manufacturer in accordance with a predetermined projection model based on design data of each lens optical system. The data of the transformation table is used for transforming a fish-eye image into an omnidirectional image.

In the description above, the omnidirectional frame data <NUM> is assumed to constitute an omnidirectional moving image. However, no limitation is intended thereby. There is no need for each frame to be recorded on the format of the synthesized omnidirectional image (synthesized omnidirectional image format) illustrated in <FIG>. In some embodiments, the omnidirectional frame data <NUM> is in any other form as long as the omnidirectional moving image can be constructed for reproduction.

For example, assuming that fish-eye images for each frame are subjected to distortion correction and synthesis processing to generate an omnidirectional image to be reproduced, moving-image data of two fish-eye images directly captured by the image sensors 130A and 130B (the respective pieces of moving-image data corresponding to the fish-eye image A and the fish-eye image B in <FIG>) is recorded as the omnidirectional frame data <NUM>. Alternatively, moving-image data of a cemented image formed by joining together the two fish-eye images A and B (that is moving-image data of one image formed by joining together the fish-eye image A and the fish-eye image B arranged side by side) may be recorded as the omnidirectional frame data <NUM>. In the following description, the omnidirectional frame data <NUM> is assumed to contain images of each frame in the distortion-corrected and synthesized omnidirectional image format, for convenience of description.

Further, the omnidirectional frame data <NUM> is not limited to moving-image data, and may be recorded in any form as long as a moving image can be reproduced. For example, the omnidirectional frame data <NUM> is recorded as moving-image data in which a plurality of frames are compressed in a certain codec such as H. <NUM>/MPEG-<NUM> advanced video coding (AVC) or H. <NUM>/High Efficiency Video Coding (HEVC). Further, although Joint Photographic Experts Group (JPEG) is a format that expresses a moving image as continuous still images, the moving-image data may be recorded as a continuous series of still images of a plurality of frames.

The following description is made with reference to <FIG>. The zenith correction data <NUM> is time-series data of the inclination angle of the omnidirectional camera <NUM> with respect to the reference direction of the omnidirectional camera <NUM> during the time from a start to an end of a shooting operation. The time series data is recorded in association with each frame of an omnidirectional image. The reference direction of the omnidirectional camera <NUM> is typically the direction of gravity in which acceleration of gravity is applied. The inclination angle with respect to the direction of gravity is generated based on the signal measured by the acceleration sensor included in the sensor <NUM>. The inclination angle is typically configured as a vector of acceleration dimension values. Since the acceleration sensor does not distinguish between gravity and inertial force, preferably the inclination angle obtained from the acceleration sensor of the sensor <NUM> may be corrected based on the signal measured by the angular velocity sensor.

In the embodiments of the present disclosure, the inclination angles correspond to the frames on a one-by-one basis, and the inclination angles and the frames are recorded synchronously. However, the rate of the inclination angle to be recorded may not be the same as the frame rate. When the rate of the inclination angle is not the same as the frame rate, an inclination angle corresponding to the frame on a one-by-one basis is obtained by performing resampling at the frame rate.

The angular velocity data <NUM> is time-series data of the angular velocities generated around three axes of the angular velocity sensor of the omnidirectional camera <NUM> during the time from a start to an end of a shooting operation, measured by the angular velocity sensor of the sensor <NUM>. The angular velocities may not be recorded in association with the frames. Typically, angular velocities faster than the frame rates are recorded in the angular velocity data <NUM>. In this case, by using the time stamp, the relations between the frames and the inclination angles are obtained at a later time. Alternatively, the angular velocities may be recorded in association with the frames of the omnidirectional moving images, respectively.

The omnidirectional moving-image correction unit <NUM> reads out the omnidirectional moving-image data <NUM> stored in the storage unit <NUM>, and applies zenith correction and rotation correction to each frame of the omnidirectional frame data <NUM> (in other words, the omnidirectional moving-image correction unit <NUM> rotates an image of each frame), thus outputting the corrected omnidirectional moving image data to another unit.

The following describes the zenith correction and the rotation correction with reference to <FIG> and <FIG>. <FIG> is an illustration of the zenith correction and the rotation correction applied to an omnidirectional image according to an embodiment of the present disclosure. <FIG> and <FIG> are illustrations of an omnidirectional image obtained by performing the zenith correction and the rotation correction according to an embodiment of the present disclosure. <FIG> is an illustration of a frame of a moving image before the zenith correction is made, and <FIG> is an illustration of a frame of the moving image after the zenith correction is made.

As described above, the image data of an omnidirectional image format is expressed as an array of pixel values where the vertical angle φ corresponding to the angle with reference to a certain axis z0 and the horizontal angle θ corresponding to the angle of rotation around the axis z0 are the coordinates. If no correction is made, the certain axis z0 is defined with reference to the omnidirectional camera <NUM>. For example, the axis z0 is defined as the central axis z0, which defines the horizontal angle θ and the vertical angle φ, passing through the center of the housing <NUM> from the bottom to the top where the top is the imaging body <NUM> side and the bottom is the opposite side of the omnidirectional camera <NUM> in <FIG>. Further, for example, the horizontal angle θ of an omnidirectional image is defined such that the direction of the optical axis of the optical element of one of the image-forming optical system 20A and 20B is positioned at the center of the corresponding image sensor <NUM> at the horizontal angle θ.

The zenith correction (correction in the direction of roll and the direction of pitch) is correction processing that corrects the omnidirectional images (<FIG>) captured with the central axis z0 actually inclined with respect to the reference direction Z (the direction of gravity) as illustrated in the left illustration of <FIG>, to an omnidirectional image (<FIG>) captured with the central axis z0 aligned with the reference direction Z as illustrated in the right illustration of <FIG>. The rotation correction is a correction (correction in Yaw direction) that reduces rotational distortion around the reference direction Z (in the horizontal angle θ direction in <FIG> and <FIG>) in the omnidirectional image to which the zenith correction has been made to have the central axis z0 aligned with the reference direction Z.

The operation of the omnidirectional moving-image correction unit <NUM> is described below in detailed with reference to <FIG>. More specifically, the omnidirectional moving-image correction unit <NUM> includes a camera front calculation unit <NUM> (a first calculation unit), a correction angle calculation unit <NUM> (a second calculation unit), a correction angle adjusting unit <NUM> (a third calculation unit), and an image rotating unit <NUM>.

Based on the zenith correction data <NUM> and the angular velocity data <NUM>, the camera front calculation unit <NUM> calculates time-series values of the front direction (the direction of the optical axis of one of the image-forming optical systems 20A and 20B) of the omnidirectional camera <NUM>. More specifically, the camera front calculation unit <NUM> first calculates the front direction V(<NUM>) (first front direction) of the omnidirectional camera <NUM> in capturing the leading frame (at the first time), based on a value T(<NUM>) of the inclination angle of the leading frame included in the zenith correction data <NUM>. After obtaining the front direction V(<NUM>) for the leading frame, the camera front calculation unit <NUM> calculates the time-series values of the front direction V(n) of the omnidirectional camera <NUM> over a plurality of times corresponding to the respective frames based on the angular velocity Gyro (n) of the angular velocity data <NUM>, starting from the initial value V(<NUM>). The front direction V(n) is obtained by integrating infinite small rotation corresponding to the angular velocity of each axis, starting from the front direction V(<NUM>) for the leading frame. The calculated time-series values of the front direction V(n) are transmitted to the correction angle calculation unit <NUM>.

Based on the inclination angle data T(m) of the zenith correction data <NUM>, the correction angle calculation unit <NUM> calculates the correction angle Angle (n) of rotation around the reference direction Z in capturing each frame, from the front direction V(n) of the omnidirectional camera <NUM> in capturing each corresponding frame. The calculated correction angle Angle (n) is transmitted to the correction angle adjusting unit <NUM>.

The correction angle adjusting unit <NUM> adjusts the calculated correction angle Angle (n) for each frame to achieve a successful natural correction operation. The obtained (calculated) correction angle Angle (n) is typically within a prescribed domain (for example, from -<NUM> degrees to +<NUM> degrees). Accordingly, the correction angle Angle (n) might change from the vicinity of +<NUM> degrees to the vicinity of -<NUM> degrees (or in the opposite direction from the vicinity of -<NUM> degrees to the vicinity of +<NUM> degrees) when viewed as a time-series values. In view of such a situation, the correction angle adjusting unit <NUM> detects a change of the correction angle Angle (n) from the vicinity of the upper limit to the vicinity of the lower limit (from the vicinity of the lower limit to the vicinity of the upper limit) of the domain based on the amount of change in the correction angle Angle (n) between continuously obtained correction angles. In other words, the correction angle adjusting unit <NUM> detects a change of the correction angle Angle (n) from an extreme to another extreme of the domain of the correction angle. Based on such a detection, the correction angle adjusting unit <NUM> adds or subtracts an adjustment value to or from the correction angle Angle (n) for each frame. With such a configuration, the domain of the correction angle is expanded and the continuity of the time-series correction angles. Further, the correction angle adjusting unit <NUM> (a filtering unit) performs a filtering process on the correction angles within the expanded domain to allow for passage of only frequency component within a predetermined frequency range, preferably a high frequency component. The corrected angle Angle Out (n) on which the filtering process has been performed is transmitted to the image rotating unit <NUM>.

Based on the zenith correction data <NUM>, the image rotating unit <NUM> performs the zenith correction on an omnidirectional image for each of the plurality of frames of the omnidirectional frame data <NUM>. At the same time, the image rotating unit <NUM> according to the present embodiment rotates the images based on the calculated correction angles Angle Out (n) for the plurality of frames, so as to reduce the rotational distortion around the reference direction Z within the range of the predetermined frequency.

When the image of each frame in the omnidirectional frame data <NUM> is an image of the synthesized omnidirectional image format, the image rotating unit <NUM> transforms the rotational coordinates of the omnidirectional image for each frame. When each frame in the omnidirectional frame data <NUM> includes a plurality of fish-eye images before the synthesis, the image rotating unit <NUM> applies the zenith correction and the rotation correction to each frame, and also transforms the plurality of fish-eye images into an omnidirectional image.

The receiving unit <NUM> receives various requests from the information terminal <NUM> communicable with the omnidirectional camera <NUM>. Upon receipt of a request to correct an omnidirectional moving image from the information terminal <NUM> (for example, a request to output a corrected omnidirectional moving image for reproduction), the receiving unit <NUM> transfers the request to the omnidirectional moving-image correction unit <NUM>. In response to the request to correct the omnidirectional moving image, the omnidirectional moving-image correction unit <NUM> starts performing the zenith correction and the rotation correction on the designated omnidirectional moving-image data <NUM> to output the corrected image data. The image output unit <NUM> outputs the image data of the omnidirectional moving image based on the plurality of frames on which the zenith correction and the rotation correction has been performed by the omnidirectional moving-image correction unit <NUM>, to the information terminal <NUM> of the request source. In this case, the image data may be encoded in a predetermined moving-image compression format based on the corrected omnidirectional images for the respective plurality of frames, and output as final moving-image data or as a series of still images.

The information terminal <NUM> is a terminal device such as a smartphone, a tablet computer, and a personal computer in which an application for communicating with the omnidirectional camera <NUM> to view and reproduce omnidirectional images is installed. The information terminal <NUM> receives various instructions from the operator via the application and issues various requests to the omnidirectional camera <NUM>. In response to accepting an instruction of the operator to correct the omnidirectional moving image designated by the operator (for example, an instruction to reproduce a moving image while correcting), the instruction receiving unit <NUM> of the information terminal <NUM> issues, to the omnidirectional camera <NUM>, a request to output moving-image data of a certain corrected omnidirectional moving image. The display unit <NUM> of the information terminal <NUM> displays an omnidirectional moving image on the display device such as the display <NUM> of the information terminal <NUM> based on the image data of the omnidirectional moving image output from the omnidirectional camera <NUM>.

Note that the information terminal <NUM> displays any desired type of image based on the corrected image data. For example, the omnidirectional image as is may be displayed on the display device. Alternatively, the omnidirectional image may be pasted on the spherical object, and images captured when the spherical object is observed with a camera of a predetermined viewing angle from a predetermined position are displayed as frames to display a moving image.

In the present embodiment, the zenith correction and rotation correction processes are actually performed using the resources of the omnidirectional camera <NUM>, not the information terminal <NUM>, and the correction result is output to the information terminal <NUM> to display a corrected image. With this configuration, regardless of the processing performance of the information terminal <NUM>, it is possible to stably reproduce moving-images reproduction while applying zenith correction and rotation correction to images.

In the embodiments of the present disclosure, the cases where the image data of the omnidirectional moving image is transmitted to the information terminal <NUM> are described. However, no limitation is intended thereby. When the omnidirectional camera <NUM> includes a display device, the display device of the omnidirectional camera <NUM> may display an omnidirectional moving image. Alternatively, the image data of the omnidirectional moving image may be stored as another file.

The rotation correction is described below in more detail according to an embodiment of the present disclosure with reference to <FIG> and <FIG>. <FIG> is a flowchart of an omnidirectional moving-image correction processing performed by the omnidirectional camera <NUM> that constitutes the omnidirectional moving-image system <NUM> according to an embodiment of the present disclosure. <FIG> and <FIG> are illustrations for explaining a method of calculating a front vector for the rotation correction according to an embodiment of the present disclosure.

In the processing illustrated in <FIG>, the omnidirectional camera <NUM> starts the processing in response to a receipt of the request to output moving-image data of a corrected certain omnidirectional moving image from the information terminal <NUM> that externally communicates with the omnidirectional camera <NUM>.

In step S101, the omnidirectional camera <NUM> reads the omnidirectional moving-image data <NUM> designated by the received request. In this case, the omnidirectional moving-image data <NUM> read by the omnidirectional camera <NUM> includes the omnidirectional frame data <NUM> that constitutes the omnidirectional moving-image data <NUM> and metadata including the zenith correction data <NUM> and the angular velocity data <NUM>.

In step S102, the omnidirectional camera <NUM> calculates the front direction V(<NUM>) of the omnidirectional camera <NUM> for the leading frame (in capturing the leading (first) frame) based on the inclination angle vector T(<NUM>) of the leading frame in the zenith correction data <NUM>. First, a quaternion Q (<NUM>) of the inclination angle vector T(<NUM>) of the leading frame and the direction of gravity vector G = (<NUM>, <NUM>, <NUM>) are generated. The inclination angle vector T(<NUM>) represents the rotational displacement between the global coordinate system for the leading (first) frame and the local coordinate system of the omnidirectional camera <NUM>. The rotational axis A(<NUM>) and the rotation angle θ(<NUM>) of the quaternion Q(<NUM>) are expressed by the following formulas (<NUM>) and (<NUM>). Then, the quaternion Q(<NUM>) represents the rotation of the rotation angle θ(<NUM>) about the rotation axis A(<NUM>) as illustrated in <FIG>. <MAT> <MAT>.

As illustrated in <FIG>, the global front vector (<NUM>, <NUM>, <NUM>) is rotated with the obtained quaternion Q(<NUM>) to obtain the initial value V(<NUM>) of the front direction vector of the omnidirectional camera <NUM>. In this case, assuming that the central axis z0 of the omnidirectional camera <NUM> is aligned with the Z axis (direction of gravity) of the global coordinate system and the front direction (the optical axis of one optical system) of the omnidirectional camera <NUM> is aligned with Y axis of the global coordinate system, the initial value V(<NUM>) of the front direction vector that indicates the front of the omnidirectional camera <NUM> in an inclined state.

In step S103, the omnidirectional camera <NUM> calculates time-series values of the front direction vector V(n) of the omnidirectional camera <NUM> over a plurality of time points corresponding to the respective frames, based on the angular velocity data <NUM>. The front direction vector V(n) at each time point is obtained by calculating the time difference between sampling processes in the angular velocity data <NUM> and integrating infinite small rotation corresponding to the angular velocity Gyro (n) about three axes for each sampling. Although sampling intervals might vary due to the load on the omnidirectional camera <NUM>, the time difference between sampling can be suitably obtained from the time stamp. In step S103, as illustrated in <FIG>, the time-series values of the front direction vector V(n) are calculated, starting from the front direction vector V(<NUM>) with which the leading (first) frame is captured.

In the loop of steps S104 to S110, the zenith correction and rotation correction are applied for each frame while calculating the correction angle for the rotation correction.

In step S105, the omnidirectional camera <NUM> searches the camera front direction vector V(n) corresponding to the frame of interest and calculates the correction angle for the rotation correction based on the zenith correction data <NUM>. Typically, the angular velocity data <NUM> has a sampling rate different from the frame rate, in which a front direction vector V close to the time stamp and corresponding to a frame of interest of the zenith correction data <NUM> is searched.

In this case, it is assumed that the front direction vector V(n) corresponds to the m-th frame. First, quaternion Q(m) is obtained from the inclination angle vector T(m) of the m-th frame. The rotation axis A(m) and the rotation angle θ(m) of the quaternion Q(m) are expressed by the following formulas (<NUM>) and (<NUM>) same as in the formulas (<NUM>) and (<NUM>). <MAT> <MAT>.

Then, as illustrated in <FIG>, the front direction vector V(n) corresponding to the m-th frame is rotated with the quaternion Q (m) to obtain the front direction vector V'(n). As a result, the direction in which the front face vector V(n) of the camera faces after zenith correction is obtained. Then, based on the obtained vector V'(n), the correction angle Angle(n) is obtained within the XY plane of the global coordinate, using the following formula.

In this case, the function atan2 (x coordinate and y coordinate) in the formula (<NUM>) above is a function that returns the arc tangent in the range of -<NUM> degrees to <NUM> degrees (-π to π).

In step S106, as a pre-process of the high-pass filter, the omnidirectional camera <NUM> adjusts the correction angle obtained in step S105 to maintain the continuity of the correction angles, and thus calculates a correction angle AngleIn (n) after the adjustment.

In the embodiments described above, the correction angle Angle (n) is obtained as a value within the range of -<NUM>° to +<NUM>° by the arc tangent. In such a case, the correction angle Angle (n) may change from the vicinity of +<NUM> degrees to the vicinity of -<NUM> degrees (or in the opposite direction from the vicinity of -<NUM> degrees to the vicinity of +<NUM> degrees) when viewed as a time-series values. In step S106, as indicated by the following pseudocode, the omnidirectional camera <NUM> (an addition-subtraction unit) detects a change of the correction angle Angle (n) from the vicinity of the upper limit to the vicinity of the lower limit (from the vicinity of the lower limit to the vicinity of the upper limit) of the domain based on the amount of change in the correction angle Angle (n) between continuously obtained correction angles. In other words, a change of the correction angle Angle (n) from an extreme to another extreme of the domain is detected. Based on such a detection, an adjustment value is added to or subtracted by the addition-subtraction unit from the correction angle Angle (n) for each frame. In the following pseudocodes, the "threshold" is a threshold for the amount of change of the correction angle Angle (n) to detect the change of the correction angle from the vicinity of the upper limit to the vicinity of the lower limit (from the vicinity of the lower limit to the vicinity of the upper limit) of the domain. The last code "Angle(<NUM>)" in the following formula is the initial value of the correction angle obtained by the above formula (<NUM>). Based on the correction angle Angle(<NUM>), the correction is performed. In the following pseudocodes, "(n - <NUM>)" corresponds to the frame immediately before the frame to which n corresponds.

In step S107, the omnidirectional camera <NUM> performs a high-pass filtering process on the adjusted correction angle AngleIn (n) for the frame of interest to allow for passage of the high-frequency component. Thus, the filtered correction angle AngleOut (n) after the filtering process is obtained. The filtered correction angle AngleOut (n) is obtained by the following formula. The symbol "P" in the following formula is a sampling period, and the symbol "Hc" is a cutoff frequency. In the following formula, (n + <NUM>) corresponds to the frame immediately after the frame corresponding to "n". <MAT> <MAT> <MAT>.

In step S108, in addition to the zenith correction, the omnidirectional camera <NUM> performs the rotation correction on the frame of interest in the omnidirectional frame data <NUM>, based on the calculated correction angle and a corresponding inclination vector T (m) in the zenith correction data <NUM>, so as to reduce the rotational distortion around the reference direction due to the high-frequency component. In step S109, the omnidirectional camera <NUM> outputs a frame of the omnidirectional image that has been subjected to the rotational transformation.

When the processes of steps S105 to S109 are performed on all the frames included in the omnidirectional moving-image data <NUM>, the loop of steps S104 to S110 ends and the correction processing ends.

When the omnidirectional camera <NUM> is properly placed in the vertical direction (the omnidirectional camera <NUM> is not inclined with reference to the reference direction (direction of gravity)), it is possible to shoot an omnidirectional image such that the zenith is aligned with the horizontal line, which is recognized by the shooter, as illustrated in <FIG>. In general, however, it is difficult to capture an image with the position and posture secured accurately while holding the camera with a hand. If an image is captured with the camera body inclined, the zeniths of the images are misaligned and the horizontal line is distorted unless the zenith correction is performed, as illustrated in <FIG>. If the camera body rotates due to, for example, camera shake, the image blurs along the horizontal line unless the rotation correction is performed.

However, in the embodiments according to the present disclosure, the images for all the frames are subjected to the transformation process (the correction processing) such that the reference direction Z such as the direction of gravity is aligned with the central axis z0 of the omnidirectional camera <NUM> and such that the rotational distortion of an image around the reference direction Z is reduced. With such a configuration, the viewer can view the corrected omnidirectional moving image without feeling uncomfortable.

As described above, the configuration according to the above-described embodiments can provide an image processing apparatus capable of effectively correcting rotational blur of an image around a prescribed reference direction, an image processing system incorporating the image processing apparatus, and carrier means.

Particularly, the rotation correction is performed using angular velocity data stored as metadata in shooting, which can reduce the load during the shooting. Further, it is possible to decide to not perform the rotation correction or to perform the rotation correction so as to prevent only a very small component, at the time of reproduction.

In the above-described embodiments, the omnidirectional camera <NUM> are described as an example of an image processing apparatus. With the configuration that performs the substantive processing of the zenith correction and the rotation correction using the resources of the omnidirectional camera <NUM>, a moving image can be stably reproduced while applying zenith correction and rotation correction to images, regardless of the processing performance of the apparatus that serves to reproduce the moving image. However, the image processing apparatus may not be an image-capturing device such as the omnidirectional camera <NUM>. In some other embodiments, an information processing apparatus such as a personal computer different from the omnidirectional camera <NUM> may be used as the image processing apparatus.

In the above-described embodiment, the cases where two partial images captured by the lens optical systems each having an angle of view greater than <NUM> degrees are superimposed and synthesized are described. However, no limitation is intended thereby. In some other embodiments, three or more partial images captured by one re more lens optical systems may be superimposed on each other and synthesized. Further, the configuration according to the above-described embodiments are applied to an imaging system equipped with a fish-eye lens. Alternatively, the configuration according to the embodiments of the present disclosure is applicable in an omnidirectional moving-image imaging system equipped with a super-wide-angle lens.

Further, in the above-described embodiments, the cases where the omnidirectional camera <NUM> is separate from the information terminal <NUM> in the omnidirectional moving-image system <NUM>. However, no limitation is intended thereby. In some embodiments, the omnidirectional camera <NUM> may be combined with the information terminal <NUM> in the omnidirectional moving-image system <NUM>.

Further, in the above-described embodiments, the omnidirectional camera <NUM> is not inclined when the direction of gravity is aligned with the central axis of the omnidirectional camera <NUM>. However, no limitation is intended thereby. Instead of the direction of gravity, for example, the horizontal direction or another desired direction may be set as a reference direction, and the inclination of the image may be corrected based on the inclination of a prescribed object, such as the omnidirectional camera <NUM> or the image sensor 130A or 130B, with reference to the reference direction.

The functional blocks as described above are implemented by a computer-executable program written by programming languages such as an assembler language, C, and object-oriented programming languages such as C++, C#, and Java (registered trademark). The program may be distributed via a telecommunication line as being stored in a computer-readable storage medium such as a ROM, an electrically erasable and programmable read only memory (EEPROM), an electrically programmable read only memory (EPROM), a flash memory, a flexible disk, a compact disc read only memory (CD-ROM), a compact disc rewritable (CD-RW), a digital versatile disk (DVD)-ROM, a DVD-RAM, a DVD-RW, a Blu-ray disc, a secure digital (SD) card, and a magneto-optical disc (MO). All or some of the functional units described above can be implemented, for example, on a programmable device such as a field programmable gate array (FPGA), or as an application specific integrated circuit (ASIC). To implement such functional units on the programmable device, circuit configuration data (bit stream data) to be downloaded to the programmable device can be distributed using a recording medium that stores data written in, for example, a hardware description language (HDL), Very High Speed Integrated Circuit Hardware Description Language (VHDL), or Verilog HDL.

The present invention can be implemented in any convenient form, for example using dedicated hardware, or a mixture of dedicated hardware and software. The present invention may be implemented as computer software implemented by one or more networked processing apparatuses. The processing apparatuses can compromise any suitably programmed apparatuses such as a general purpose computer, personal digital assistant, mobile telephone (such as a WAP or <NUM>-compliant phone) and so on. Since the present invention can be implemented as software, each and every aspect of the present invention thus encompasses computer software implementable on a programmable device. The computer software can be provided to the programmable device using any conventional carrier medium (carrier means). The carrier medium can compromise a transient carrier medium such as an electrical, optical, microwave, acoustic or radio frequency signal carrying the computer code. An example of such a transient medium is a TCP/IP signal carrying computer code over an IP network, such as the Internet. The carrier medium can also comprise a storage medium for storing processor readable code such as a floppy disk, hard disk, CD ROM, magnetic tape device or solid state memory device.

Each of the functions of the described embodiments may be implemented by one or more processing circuits or circuitry. Processing circuitry includes a programmed processor, as a processor includes circuitry. A processing circuit also includes devices such as an application specific integrated circuit (ASIC), DSP (digital signal processor), FPGA (field programmable gate array) and conventional circuit components arranged to perform the recited functions.

The illustrated apparatuses are only illustrative of one of several computing environments for implementing the embodiments disclosed herein. For example, in some embodiments, the image-capturing device includes a plurality of computing devices, e.g., a server cluster, that are configured to communicate with each other over any type of communications link, including a network, a shared memory, etc. to collectively perform the processes disclosed herein. Similarly, the image processing apparatus can include a plurality of computing devices that are configured to communicate with each other.

Claim 1:
An image processing apparatus (<NUM>) comprising:
a storage unit (<NUM>) to store moving-image data including a plurality of frames captured by an image-capturing device (<NUM>) communicable with the image processing apparatus (<NUM>), time-series data of an inclination angle of the image-capturing device with reference to a reference direction, and time-series data of angular velocities generated around three axes of an angular velocity sensor (<NUM>) of the image-capturing device (<NUM>); and
a correction unit (<NUM>) to, based on the time-series data of the angular velocities, rotate an image of each of the plurality of frames of the moving-image data to reduce a rotational distortion around the reference direction; and
an output unit (<NUM>) to output image data of the rotated image of each of the plurality of frames to an external device communicable with the image processing apparatus (<NUM>),
characterized in that
the image processing apparatus (<NUM>) further comprises
a camera front calculation unit (<NUM>) to calculate a first front direction of the image-capturing device (<NUM>) at a first time based on a value of the time-series data of the inclination angle at the first time, and after obtaining the first front direction, to calculate time-series values of the front direction of the image-capturing device (<NUM>) over a plurality of times corresponding to the plurality of frames based on the time-series data of the angular velocities, wherein the front direction is the direction of an optical axis of the image-capturing device (<NUM>);
a correction angle calculation unit (<NUM>) to calculate a correction angle for the rotation correction from the time-series values of the front direction of the image-capturing device (<NUM>) in a predetermined frame among the plurality of frames, based on the time-series data of the inclination angle, wherein the correction unit (<NUM>) rotates the image based on the calculated correction angle.