Digital photographing motion compensation system and method

A digital photographing system may include a lens unit detecting an image of a subject, and a first processor performing an image compensation data generation process and an optical compensation process in parallel. The optical compensation process may be a process of selectively controlling a movement of the lens unit based on motion data corresponding to a motion of a camera module. The image compensation data generation process may be a process of generating image compensation data corresponding to the image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Korean Patent Application No. 10-2014-0046885, filed on Apr. 18, 2014, entitled “Digital Photographing System and Controlling Method Thereof” and Korean Patent Application No. 10-2014-0076061, filed on Jun. 20, 2014, entitled “Digital Photographing System and Controlling Method Thereof”, both of which are incorporated by reference herein in their entireties.

BACKGROUND

Some embodiments of the present disclosure may relate to a digital photographing system and a method for controlling the same.

Generally, a digital photographing system may process an image received through an image pickup device using a digital signal processor, compress the processed image to generate an image file, and store the generated image file in a memory.

Further, the digital photographing system may display the image of the image file, which is received through the image pickup device or stored in a storage medium, on a display device such as an LCD. However, the digital photographing system such as a camera may be shaken due to motions (hand shaking, and the like) of a user or any disturbance when the user photographs a desired image. Due to the shaking, the image input through the image pickup device is shaken and thus the quality of the image may be degraded.

RELATED ART DOCUMENT

Patent Document

SUMMARY

Some embodiments of the present disclosure may provide a digital photographing system and a method for controlling the same capable of applying an optical compensation process and an image compensation process in parallel to compensate for a shaking or disturbance of a photographed image, based on motion data of a motion sensor about a motion of a camera when the motion of the camera occurs during a process of photographing a subject.

Some embodiments of the digital photographing system and the method for controlling the same may perform the optical compensation process and the image compensation data generation process in parallel, depending on whether the moving distance of the camera module in each axial direction is within a preset reference range, and may compensate for the shaking of the image due to the motion by selectively applying the optical compensation process under the predetermined condition. The moving distance of the camera module may be calculated based on the motion data corresponding to the motion of the camera module.

Some embodiments of the present disclosure may shorten the image compensation processing time due to the motion and secure the reliability of the image quality, by individually applying the optical compensation processor controlling the movement of the lens unit and the image compensation data generation process generating the image compensation data using the preset synchronization information under the predetermined condition, for the motion (hand shaking or horizontal/vertical movement) which may occur while the image (still or moving picture) is photographed.

DETAILED DESCRIPTION

The objects, features and advantages of the present disclosure will be more clearly understood from the following detailed description of the exemplary embodiments taken in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant descriptions thereof are omitted. Further, in the following description, the terms “first,” “second,” “one side,” “the other side” and the like are used to differentiate a certain component from other components, but the configuration of such components should not be construed to be limited by the terms. Further, in the description of the present disclosure, when it is determined that the detailed description of the related art would obscure the gist of the present disclosure, the description thereof will be omitted.

Hereinafter, a digital photographing system and a method for controlling the same according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

FIG. 1is a block diagram illustrating a digital photographing system according to an exemplary embodiment of the present disclosure,FIG. 2is a diagram illustrating a configuration of a camera module of the digital photographing system according to the exemplary embodiment of the present disclosure, andFIG. 3is a diagram illustrating a function of an auto-focusing processor of the digital photographing system according to the exemplary embodiment of the present disclosure.

As illustrated inFIGS. 1 and 2, a digital photographing system10according to an exemplary embodiment of the present disclosure may include a camera module140, a motion sensor100, a first processor110, an optical driver120, an optical driving module130, a second processor160, a memory170, and an auto-focusing processor180. The digital photographing system10may be included, for example, in mobile multi-functional devices, such as a digital camera, a cellular phone, and a tablet computer, or in a laptop computer, a desktop computer, or the like, but the digital photographing system10is not limited thereto.

The motion sensor100may be provided inside or outside the camera module140and may generate or output motion data corresponding to a motion of the camera module140. The motion sensor100may include an angular velocity sensor101and an acceleration sensor102. The angular velocity sensor101may sense a change in a rotation component (angular velocity) of the camera module140, for example, but not limited to, due to hand shaking or disturbance and the like. The acceleration sensor102may sense a change in a linear component (velocity), for instance, but not limited to, due to movement in a vertical or horizontal direction of the camera module140.

For example, 1) the angular velocity sensor101may be a gyro sensor which may sense the change in the angular velocity of motions of the camera module140in two directions of a yaw axis and a pitch axis to compensate for vertical and horizontal shaking or movement of the camera module140due to hand shaking of a user or disturbance, and 2) the acceleration sensor102may sense the change in the velocity in the horizontal (x axis) or vertical (y axis) direction of the camera module140due to the user's movement or disturbance, which corresponds to a linear component due to the motion of the camera module140.

The lens unit141may include a lens barrel141aand a position sensor142. The lens barrel141amay include an image sensor141band a lens group (not illustrated) which optically process light from a subject141cto detect or capture an image frame of an image such as still or moving picture of the subject141c. The position sensor142may sense a position change of the lens barrel141a(seeFIG. 3).

For instance, the lens group (not illustrated) may include at least one of a zoom lens, a focus lens or a compensation lens. The image sensor141bmay be, for example, but not limited to, a charge coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS) or any device converting an optical signal of light incident through the lens barrel141ainto an electrical analog signal. The position sensor142may sense the position change of the lens barrel141ato transmit current position information of the lens barrel141ato the first processor110. The position sensor142may be, for instance, but not limited to, a hall sensor (not illustrated) using a hall effect in which a voltage is changed depending on a magnetic field strength to detect a current position of the lens barrel141.

The optical driver120may generate a driving voltage and a control signal of the optical driving module130to move the lens unit141depending on a control signal input from the first processor110.

Further, the optical driver120may control a drive of the optical driving module130based on a switching operation corresponding to the control signal to control a moving range of the lens unit141. The optical driver120may be, for example, but not limited to, a motor drive integrated chip (IC). The optical driver120may be embedded in the first processor110.

In the embodiment, the optical driving module130may comprise first and second actuators131and132including a voice coil motor (VCM) or a piezoelectric device. The first actuator131may control a movement of the lens unit141in a vertical direction (y-axis direction), and the second actuator132may control a movement of the lens unit141in the horizontal direction (x-axis direction).

The first processor110may perform an image compensation data generation process and an optical compensation process in parallel or simultaneously. For example, the optical compensation process may control the movement of the lens unit141based on motion data, and the image compensation data generation process may generate image compensation data corresponding to the image, but not limited thereto. The image compensation data generation process and the optical compensation process may be simultaneously performed.

For instance, the optical compensation process may be selectively performed depending on whether a moving distance in a moving direction (each axial direction such as x-axis or y-axis direction) of the camera module140is within a preset reference range and detailed contents thereof will be described below. The moving distance of the camera module140may be calculated based on, for example, but not limited to, motion data (data for a change in acceleration or angular velocity) corresponding to a motion of the camera module140.

Further, when performing the image compensation data generation process, the first processor110may generate the image compensation data synchronized with each image frame of the image based on preset synchronization information. For instance, the image compensation data ID(seeFIG. 6) may include dummy data configured of null data, motion data depending on a change in acceleration, and metadata configured of focus information of the lens unit141.

For example, the first processor110may synchronize 1) the dummy data with the image frame acquired in a section or period in which the optical compensation process is performed and 2) the metadata with the image frame acquired in a section or period in which the optical driving process is not performed, by using the synchronization information (the motion data and the detection timing of the focus information of the lens unit141, seeFIG. 5).

Further, the optical driving processor or the first processor110may store the image compensation data IDin the memory170or transmit the image compensation data IDto the application processor or the second processor160.

When performing the optical compensation process, the first processor110may selectively perform the optical compensation process depending on whether the moving distance of the camera module140in the moving direction (each axial direction such as x-axis or y-axis direction) which may be calculated based on the motion data is within the preset reference range.

The motion data may include, for instance, but not limited to, data for the rotation component (angular velocity) of the camera module140due to hand shaking of a user or disturbance and the linear component (acceleration) for a motion of the camera module140in the horizontal or vertical direction.

When the moving distance of the camera module140is within the reference range, the first processor110may perform the optical compensation process controlling the movement of the lens unit141so as to be able to compensate for the motion of the camera module140. For example, the reference range may be a maximum movable range DXor DYof the lens unit in a horizontal (x-axis) direction or a vertical (y-axis) direction, but is not limited thereto.

In detail, the first processor110may control the movement of the lens barrel141abased on the position information of the lens barrel141awhich is transmitted from the position sensor142so as to be able to compensate for shaking of an image due to the motion of the camera module140. For example, the first processor110may generate a control signal to move the lens barrel141ain an opposite direction to the moving direction of the camera module140as much as the moving distance of the camera module140, and transmit the generated control signal to the optical driver120.

The optical driver120may generate the driving voltage and the control signal of the optical driving module130to move the lens unit141depending on or responding to the control signal transmitted from the first processor110. Further, the optical driver120may control the driving of the optical driving module130based on the switching operation corresponding to the control signal to move the lens unit141.

Therefore, it is possible to detect or capture the image frame obtained by compensating for the shaking or disturbance due to the motion of the camera module140using the lens unit141in the section or period in which the optical compensation process is performed.

The second processor160may use the image compensation data transmitted from the first processor110to calculate the moving pixel information on the display190between the respective image frames and then may compensate for the shaking or disturbance of the image due to the motion of the camera module140based on the calculated moving pixel information.

In detail, the second processor160may use the image compensation data such as dummy data and metadata acquired based on the synchronization information with the image frame configuring the image to calculate the moving pixel information between the respective image frames, and compensate for the shaking disturbance of the image due to the motion of the camera module140. For example, the second processor160may be an application processor which is equipped in a mobile phone, and the like and may include an image sensing processor (ISP).

The auto-focusing processor180may calculate focus information (including a focal length) on the respective image frames of the subject acquired through the lens unit141, and may transmit the focus information to the first processor110. The focus information may also be detected by a proximity sensor and/or an optical or acoustic means.

For instance, as illustrated inFIG. 3, the auto-focusing processor180may control a position of a focus f to move from f1to f2so as to definitely project an image of a subject on the image sensor141bwhen a position of a subject141cmoves from P1to P2or when a position of the lens barrel141amoves from L1to L2, and may calculate the focus information such as the focal length which is a distance between the focus f and the image of the subject projected on the image sensor141band then transmit the focus information to the first processor110.

The memory170may store the image compensation data generated by the first processor110and/or the synchronization information established or generated during calibration. The memory170may be, for example, but not limited to, a volatile memory such as a static random access memory (SRAM) and a dynamic random access memory (DRAM) or a non-volatile memory such as a read-only memory (ROM) and a flash memory.

The display190may be a display device visually outputting data on a screen and may be, for instance, a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED), and an organic light emitting diode (OLED), but is not necessarily limited thereto.

The first processor110, the application processor or the second processor160, and the auto-focusing processor180which are described above may include an algorithm for performing the foregoing functions and may be implemented by firmware, software, or hardware (for example, semiconductor chip or application-specific integrated circuit).

Hereinafter, the calibration (preprocessing) process of the digital photographing system according to the exemplary embodiments of the present disclosure will be described in more detail with reference toFIGS. 4 and 5.

FIG. 4is a flow chart illustrating the calibration process of the digital photographing system according to an exemplary embodiment of the present disclosure, andFIG. 5is a diagram illustrating that the synchronization information is detected by the calibration of the digital photographing system according to an exemplary embodiment of the present disclosure.

First, as illustrated inFIG. 4, the calibrations (preprocessing) of the digital photographing system according to the exemplary embodiment of the present disclosure may include at least one or more steps of 1) detecting, capturing or sensing the respective image frames of the moving picture (or still image) using the lens unit141(S10), 2) acquiring the motion data output from the motion sensor100(S20), and 3) calculating the synchronization information on acquisition timing of the image frame and the motion data (S30). Here, the calibration may be performed once at an early stage and then be stored in the memory170, the first processor110, or the like and thus may be repeatedly used.

For example, as illustrated inFIG. 5, 1) the respective image frames f1to fnconfiguring the moving picture be detected, captured or sensed by the lens unit141. Here, a photographing speed (detection speed and period TCof the respective image frames) of the moving picture may be variously configured depending on a photographing mode of the moving picture (for example, 30 frames per second (fps) or 40 fps).

2) The motion sensor100may output the motion data a1to an(preferably, data for the change in the angular velocity detected by the gyro sensor) due to or corresponding to the motion of the camera module140which may occur during the photographing process of the moving picture at a constant speed. Here, the output speed of the motion data a1to anmay be changed depending on the setting of the motion sensor100.

3) From the motion data a1to anoutput from the motion sensor100, detection timings t1to t4n+1at which motion data a5to a4n+1most accurately representing the movement of the image frames f1to fndue to the motion of the camera module140is detected may be calculated and the detection timings t1to t4n+1may have a predetermined period Ta. ΔT ofFIG. 5means a delay time before synchronization for the motion data which may occur due to a difference between the output speed of the motion data of the motion sensor100and the photographing speed of the image starts.

Here, the detection timing may be variously configured depending on the photographing mode (for example, 30 fps) of the moving picture, which is only an example, and therefore the detection timing is not limited to the description above.

Therefore, synchronization information (detection speed (for example, 30 fps) of the image frame of the moving picture and the detection timings t5to t4n+1of the motion data corresponding thereto) on extraction timing of the motion data and timing at which the respective image frames f1to fnof the moving picture may be acquired, is acquired by the calibration (preprocessing) process.

Hereinafter, the digital photographing system and the method for controlling the same according to exemplary embodiments of the present disclosure will be described in more detail with reference toFIGS. 1, 2 and 5 to 8.

FIG. 6is a diagram illustrating an image compensation data generation process of a digital photographing system according to an exemplary embodiment of the present disclosure,FIG. 7is a diagram illustrating a process of calculating moving pixel information between image frames configuring an image in the digital photographing system according to an exemplary embodiment of the present disclosure, andFIG. 8is a flow chart illustrating a method for controlling a digital photographing system according to an exemplary embodiment of the present disclosure.

First, as illustrated inFIGS. 1 and 8, the digital photographing system10and a method for controlling the same according to exemplary embodiments of the present disclosure may perform 1) photographing (detecting) the respective image frames configuring the image (still or moving picture) of the subject141cby the lens unit141(S100). The detection speed and period of the image frame may be variously set depending on the photographing mode of the image. The photographed image frame of the moving picture may be transmitted to the second processor160through the lens unit141.

Next, 2) when the motion (hand shaking, horizontal or vertical movement or any disturbance) of the camera module140occurs while the image of the subject141cis photographed, captured or sensed (detected) by the lens unit141, the motion data for the change in the angular velocity (rotation component in a yaw-axis or pitch-axis direction) due to the hand shaking among the motions and the acceleration (change in the velocity in the horizontal (x-axis) or vertical (y-axis) direction) caused by the linear motion in the horizontal or vertical direction among the motions may be output from the motion sensor100(S110).

Next, 3) the first processor110may perform the image compensation data generation process which may generate the image compensation data (configured of the dummy data and the metadata) synchronized with the image frame of the moving picture based on the synchronization information (synchronization information (detection timing of the motion data) corresponding to the photographing speed (for instance, 30 fps or 50 fps) of the current moving picture, among the detection timings t5to 54n+1of the motion data) which is preset by the calibration (S120).

Further, 4) the first processor110may use the motion data for the angular velocity (rotation component in a yaw-axis or a pitch-axis direction) and the acceleration (change in the velocity in the horizontal (x-axis) or vertical (y-axis) direction) which are applied from the motion sensor100to calculate the actual moving direction and the moving distance of the camera module140in the horizontal (x-axis) or vertical (y-axis) direction, thereby determining whether to perform the optical compensation process (S130).

For example, the first processor110may determine whether the moving distance of the camera module140in the horizontal (x-axis) or vertical (y-axis) direction is within the preset reference range (for instance, but not limited to, maximum movable range of the lens unit141in a horizontal (DX) direction and a vertical (DY) direction).

Further, i) when the moving distance of the camera module140is within the reference range, the first processor110may receive the current position information of the lens barrel141afrom the position sensor142. ii) the first processor110may control the movement of the lens barrel141abased on the position information to compensate for the shaking of the image depending on the motion of the camera module140. For example, the first processor100may generate the control signal to move the lens unit141in the opposite direction to the motion direction of the camera module140as much as the actually moving distance of the camera module140and transmit the generated control signal to the optical driver120.

iii) The optical driver120may generate the driving current and the control signal of the optical driving module130based on the control signal, and then use the control signal to control the driving of the optical driving module (for instance, but not limited to, voice coil motor or piezoelectric device)130, thereby performing the optical driving process controlling the moving range of the lens unit141.

Here, the first actuator131may control the movement of the lens unit141in the vertical direction (y-axis direction), and the second actuator132may control the movement of the lens unit141in the horizontal direction (x-axis direction).

The first processor110may synchronize i) the dummy data with the image frame (hand shaking compensated image frame by the optical compensation process) of the image acquired in the section in which the optical compensation is performed, based on the synchronization information such as detection timing of the motion data corresponding to the photographing speed of the currently photographing moving picture which is set in the calibration in the generating of the image compensation data (S140).

Further, the first processor may synchronize ii) the metadata with the image frame of the image acquired in the section in which the optical compensation is not performed (S150). Here, the dummy data may be configured of the null data (negligible value or meaningless value) and the metadata may be configured of the motion data (for example, data for the change in the acceleration) of the respective image frames of the photographing moving picture and the focus information (such as focal length) of the lens unit141.

The auto-focusing processor180may calculate the focus information (for example, focal length) of the lens unit141at the image frame acquisition timing t1to t4n+1to transmit the focus information (for instance, focal length) to the first processor110.

For example, as illustrated inFIG. 6, 1) the respective image frames f1to fnof the moving picture which are detected, captured or sensed at a constant photographing speed (for example, 30 fps) by the lens unit141may be transmitted to the second processor160, and 2) the first processor110may determine whether the moving distance of the camera module140in the horizontal (x-axis) or vertical (y-axis) direction is within the preset reference range (for instance, but not limited to, maximum movable range of the lens unit141in the horizontal (DX) and vertical (DY) directions as illustrated inFIG. 2) based on the motion data (e.g. data for the change in the acceleration and the change in the angular velocity) corresponding to the motion of the camera module140to determine whether to perform the optical driving process.

3) The first processor110may perform the image compensation process, which may generate the image compensation data IDconfigured of the dummy data and metadata synchronized with the image frame based on the synchronization information (for instance, but not limited to, detection timings t5, t9, and t13to t4n+1of the motion data) corresponding to the photographing speed of the moving picture, in parallel.

For example, i) the dummy data D1, D2, D3, . . . , Dnmay be synchronized based on the synchronization information (for instance, detection timings t5, t9, t13to t4n+1) for the image frame of the image acquired in the section or period (for example, section a or when the moving distance of the camera module140in the horizontal (x-axis) or vertical (y-axis) direction is within the preset reference range) in which the optical compensation is performed.

ii) The metadata may be synchronized with the image frame of the image acquired in a section (for instance, section b or when the moving distance of the camera module140in the horizontal (x-axis) or vertical (y-axis) direction exceeds or is out of the preset reference range) in which the optical compensation is not performed based on the synchronization information (for example, metadata M4, M5, and M6to Mn(motion data (data for the change in the acceleration) for the respective image frame of the photographing moving picture) based on the detection timings t5, t9, t13to t4n+1and the focus information (such as focal length of the lens unit141), may be synchronized.

iii) Therefore, depending on whether the optical driving process is performed while all the image frames f1to fnof the image are detected, sensed or captured by the lens unit141, the image compensation data ID=D1, D2, D3, . . . , M4, M5, and M6to MNfor the image generated by the image compensation data generation process may be configured of the dummy data and the metadata synchronized with the image frame based on the preset synchronization information corresponding to the photographing speed of the moving picture.

Further, when the photographing of the image ends, the first processor110may store the image compensation data for the image in the memory170or transmit the image compensation data to the second processor160.

Next, the second processor160may use the image compensation data transmitted from the first processor110to calculate the moving pixel information between the respective image frames configuring the image (S160).

For example, the second processor160may use the image frame configuring the image and the motion data for the change in the acceleration of the motion sensor100and the metadata M4, M5, and M6to MNincluding the focus information of the lens unit141which are synchronized at the acquisition timing of the respective image frames to calculate the moving pixel information between the respective image frames.

For instance, as illustrated inFIG. 7, the moving pixel information between the image frames in the second processor160may be calculated by the following Equations 1 to 4, in which Wdrepresents a horizontal distance or width of the image sensor141b, a field of view (FOV) represents a viewing angle, RSrepresents a screen resolution (graphic setting value) of a display, δprepresents the number of moving pixels of the respective image frames on the display, and δSrepresents the moving distance of the camera module140.

Here, the moving pixel information between the image frames which is calculated using the motion data (data acquired by the acceleration sensor102) for the change in the acceleration and the focal length of the lens unit141may include the number of moving pixels and the moving direction.
Wd=2*f*tan(FOV/2)|  [Equation 1]
δp:δS=RS:Wd[Equation 2]
δS=∫∫(δ2s/δt2)  [Equation 3]
δS=δd*RS/(2*f*tan(FOV/2))  [Equation 4]

Next, since the moving pixel information between the image frames calculated by the second processor160may be stored in the memory170and the like, and the compensated image may be output through a dedicated player and the like by using the image and the moving pixel information corresponding to the image, it is possible to more reduce power consumption and processing time for image compensation than a post-compensation method for photographing an image and then using an image processing technique.

As described above, the digital photographing system according to the exemplary embodiment of the present disclosure may apply the optical compensation process and the image compensation data generation process in parallel depending on the motion (hand shaking, horizontal/vertical movement or any disturbance) which may occur while the still or moving picture is photographed, thereby shortening the image compensation processing time by the motion and securing the reliability of the image compensation.

Although the preferred embodiments of the present disclosure have been disclosed for illustrative purposes, they are for specifically explaining the present disclosure and thus a digital photographing system and a method for controlling the same according to the present disclosure are not limited thereto, but those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the disclosure as disclosed in the accompanying claims.

Accordingly, any and all modifications, variations or equivalent arrangements should be considered to be within the scope of the disclosure, and the detailed scope of the disclosure will be disclosed by the accompanying claims.