Apparatus and method for compensating for image change caused by optical image stabilization motion

An electronic device is disclosed. Moreover, various embodiment found through the disclosure are possible. An electronic device may include a lens assembly including, one or more lenses, an image sensor, an image stabilizer, and a processor. The lens assembly may be arranged to from a first angle between an optical axis of at least some lenses of the one or more lenses, and a surface of the image sensor. The processor may be configured to change an angle of the lens assembly through the image stabilizer in response to shaking of the electronic device, obtain an image through the image sensor, in a state that the angle of the lens assembly is changed, correct, based at least on the first angle and a second angle corresponding to the changed angle of the lens assembly, at least a portion of the image, which is distorted, by the second angle, and display the corrected image through a display electrically connected with the electronic device.

TECHNICAL FIELD

Embodiments disclosed in the disclosure relate to an apparatus and a method for compensating the variation of images caused by optical image stabilization (OIS) motion.

BACKGROUND ART

An electronic device may obtain an image distorted when a problem, such as the handshaking of a user, is caused during the shooting of the image. The electronic device may perform image correction to compensate for the handshaking of the user. The image correction may include, for example, optical image stabilization (OIS) correction.

The OIS correction may refer to correction that a stabilizer moves a lens or an image sensor in a direction of compensating for the shaking of the electronic device when the shaking of the electronic device occurs while the camera module is capturing an image. The motion of the lens or the image sensor through the OIS correction may be referred to as the OIS motion.

DISCLOSURE

Technical Problem

Although the compensating for the shaking of the electronic device is achieved through the OIS correction, when an angle of the lens is changed, an angle formed between an optical axis of the lens and the image sensor may be changed to cause an artifact by the changed angle within the image. The phenomenon, in which the artifact is caused within the image due to the OIS correction, may be referred to as “wobbling”.

Various embodiments of the disclosure are to provide an apparatus for measuring an angle of a lens, which is changed due to the OIS correction, and compensating for distortion caused by the OIS correction using the measured angle, and a method for the same.

Technical Solution

According to an embodiment disclosed in the disclosure, an electronic device may include a lens assembly including one or more lenses, an image sensor, an image stabilizer, and a processor. The lens assembly may be arranged to from a first angle between an optical axis of at least some lenses of the one or more lenses, and a surface of the image sensor. The processor may be configured to change an angle of the lens assembly through the image stabilizer in response to shaking of the electronic device, obtain an image through the image sensor, in a state that the angle of the lens assembly is changed, correct, based at least on the first angle and a second angle corresponding to the changed angle of the lens assembly, at least a portion of the image, which is distorted, by the second angle, and display the corrected image through a display electrically connected with the electronic device.

According to an embodiment disclosed in the disclosure, an electronic device may include an image sensor, a lens assembly including one or more lenses, disposed above the image sensor to have a first specified angle between at least some lenses of the one or more lenses and a surface of the image sensor to provide an image to the image sensor such that the image sensor obtains the image at the first specified angle, an image stabilizer which is able to correct at least partially shaking of the electronic device by changing an angle formed between the optical axis and the surface of the image sensor, and a processor. The processor may be configured to change the angle of the lens assembly to a second specified angle depending on the shaking, using the image stabilizer, obtain an image of an external object at the second specified angle using the image sensor, correct, based partially on the first specified angle and the second specified angle, at least a portion of the image distorted by the second specified angle, and display the corrected image through a display electrically connected with the electronic device.

According to an embodiment disclosed in the disclosure, a method of an electronic device may include changing an angle of a lens assembly, to change a first angle between an optical axis of at least some lenses and a surface of an image sensor, in response to shaking of the electronic device, obtaining an image through the image sensor, after changing the angle of the lens assembly, correcting at least a portion of the image distorted by the second angle, based on the first angle and the second angle corresponding to the angle of the changed angle of the lens assembly, and displaying the corrected image through a display electrically connected with the electronic device.

Advantageous Effects

According to embodiments disclosed in the disclosure, the electronic device may measure the motion of the lens, which is caused due to the OIS correction, thereby compensating for image distortion caused by the OIS correction.

According to embodiments disclosed in the disclosure, the electronic device may improve the performance of the camera module by compensating for the image distortion caused by the OIS correction.

Besides, a variety of effects directly or indirectly understood through the disclosure may be provided.

In the following description made with respect to the accompanying drawings, similar components will be assigned with similar reference numerals.

MODE FOR INVENTION

Hereinafter, various embodiments of the disclosure may be described with reference to accompanying drawings. However, those of ordinary skill in the art will understand that the disclosure is not limited to a specific embodiment, and modifications, equivalents, and/or alternatives on the various embodiments described herein can be variously made without departing from the scope and spirit of the disclosure.

FIG. 2is a block diagram200illustrating the camera module180according to various embodiments.

Referring toFIG. 2, the camera module180may include a lens assembly210, a flash220, an image sensor230, an image stabilizer240, memory250(e.g., buffer memory), or an image signal processor260. The lens assembly210may collect light emitted or reflected from an object whose image is to be taken. The lens assembly210may include one or more lenses. According to an embodiment, the camera module180may include a plurality of lens assemblies210. In such a case, the camera module180may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies210may have the same lens attribute (e.g., view angle, focal length, auto-focusing, f number, or optical zoom), or at least one lens assembly may have one or more lens attributes different from those of another lens assembly. The lens assembly210may include, for example, a wide-angle lens or a telephoto lens.

The flash220may emit light that is used to reinforce light reflected from an object. According to an embodiment, the flash220may include one or more light emitting diodes (LEDs) (e.g., a red-green-blue (RGB) LED, a white LED, an infrared (IR) LED, or an ultraviolet (UV) LED) or a xenon lamp. The image sensor230may obtain an image corresponding to an object by converting light emitted or reflected from the object and transmitted via the lens assembly210into an electrical signal. According to an embodiment, the image sensor230may include one selected from image sensors having different attributes, such as a RGB sensor, a black-and-white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor230may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer240may move the image sensor230or at least one lens included in the lens assembly210in a particular direction, or control an operational attribute (e.g., adjust the read-out timing) of the image sensor230in response to the movement of the camera module180or the electronic device101including the camera module180. This allows compensating for at least part of a negative effect (e.g., image blurring) by the movement on an image being captured. According to an embodiment, the image stabilizer240may sense such a movement by the camera module180or the electronic device101using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module180. According to an embodiment, the image stabilizer240may be implemented, for example, as an optical image stabilizer.

The memory250may store, at least temporarily, at least part of an image obtained via the image sensor230for a subsequent image processing task. For example, if image capturing is delayed due to shutter lag or multiple images are quickly captured, a raw image obtained (e.g., a Bayer-patterned image, a high-resolution image) may be stored in the memory250, and its corresponding copy image (e.g., a low-resolution image) may be previewed via the display device160. Thereafter, if a specified condition is met (e.g., by a user's input or system command), at least part of the raw image stored in the memory250may be obtained and processed, for example, by the image signal processor260. According to an embodiment, the memory250may be configured as at least part of the memory130or as a separate memory that is operated independently from the memory130.

The image signal processor260may perform one or more image processing with respect to an image obtained via the image sensor230or an image stored in the memory250. The one or more image processing may include, for example, depth map generation, three-dimensional (3D) modeling, panorama generation, feature point extraction, image synthesizing, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor260may perform control (e.g., exposure time control or read-out timing control) with respect to at least one (e.g., the image sensor230) of the components included in the camera module180. An image processed by the image signal processor260may be stored back in the memory250for further processing, or may be provided to an external component (e.g., the memory130, the display device160, the electronic device102, the electronic device104, or the server108) outside the camera module180. According to an embodiment, the image signal processor260may be configured as at least part of the processor120, or as a separate processor that is operated independently from the processor120. If the image signal processor260is configured as a separate processor from the processor120, at least one image processed by the image signal processor260may be displayed, by the processor120, via the display device160as it is or after being further processed.

According to an embodiment, the electronic device101may include a plurality of camera modules180having different attributes or functions. In such a case, at least one of the plurality of camera modules180may form, for example, a wide-angle camera and at least another of the plurality of camera modules180may form a telephoto camera. Similarly, at least one of the plurality of camera modules180may form, for example, a front camera and at least another of the plurality of camera modules180may form a rear camera.

FIG. 3Aillustrates an operation of obtaining an image in a state that optical image stabilization (OIS) correction is not performed, according to various embodiments.

Referring to reference number301ofFIG. 3A, the camera module180may include a lens310(e.g., at least some lenses of the lens assembly210ofFIG. 2) and an image sensor330(e.g., the image sensor230ofFIG. 2). AlthoughFIG. 3Aillustrates that the camera module180merely includes the lens310and the camera module180, the camera module180may further include other components not illustrated inFIG. 3A. For example, the camera module180may further include a motion sensor, which detects the shaking of the electronic device101, inside or outside the camera module180, or a Hall sensor which senses the motion of the lens310. For another example, the camera module180may further include an image stabilizer (e.g., the image stabilizer240ofFIG. 2) which controls the motion of the lens310.

According to an embodiment, the lens310may collect light reflected from a subject and transmit the light to the image sensor330. The image sensor330may obtain an image by converting the light transmitted from the lens310into an electrical signal.

According to an embodiment, the lens310may be arranged such that an optical axis315of the lens310is positioned at the center (e.g., reference numeral335) of the image sensor330and the angle formed between the optical axis315and the image sensor330is a first angle342(e.g., a right angle). In the state that the lens310is arranged at the angle and in the direction as illustrated in reference numeral301, the light reflected from the subject may be transmitted to the image sensor330through the lens310.

Referring to reference numeral302ofFIG. 3A, the image sensor330may generate an image350using the light received from the lens310. The image350may include a plurality of pixels. The electronic device101may convert the pixels into two dimensional coordinates so as to transform or compensate the image. For example, the electronic device101may convert an index of a pixel, which is positioned in reference numeral352, into two dimensional coordinates (x, y).

FIG. 3Billustrates an operation of obtaining an image while OIS correction is performing, according to various embodiments. InFIG. 3B, it may be assumed that the electronic device101is shaken by the handshaking of a user or another cause, when the electronic device101is in a shooting mode to obtain the image.

Referring to reference numeral303ofFIG. 3B, when the camera module180is shaken in a direction the same as a direction in which the electronic device101is shaken, the image sensor330and the lens310may be shaken in the same direction. The image stabilizer (e.g., the image stabilizer240ofFIG. 2) may control the motion of the lens310such that the lens310moves in a direction opposite to the direction in which the electronic device101is shaken. For example, the lens310may move in the direction illustrated in reference numeral345under the control of the image stabilizer. The operation that the lens310moves under the control of the image stabilizer may be referred to as the OIS correction.

According to an embodiment, when the lens310moves, the optical axis315may move by an angle at which the lens310moves. For example, an optical axis315-1may correspond to the lens310(e.g., a lens310-1) before the OIS correction is performed, and an optical axis315-2may correspond to the lens310(e.g., the lens310-2) after the OIS correction is performed. When the optical axis315moves, because the optical axis315deviates from the center of the image sensor330, an image of the subject may be distorted by the angle at which the lens310(or the optical axis315) moves. In the disclosure, the angle corresponding to the moving of the lens310due to the OIS correction may be referred to as a second angle or OIS motion. The second angle may, for example, correspond to an angle which is formed between the lens (e.g., the lens310-1) before the OIS correction is performed and the lens310(e.g., the lens310-2) moving by an angle illustrated in reference numeral345due to the OIS correction. For another example, the second angle may correspond to an angle (e.g., reference numeral344) between a virtual plane332normal to the optical axis315-2and parallel to the lens310-2subject to the OIS correction and the image sensor330.

Referring to reference numeral304ofFIG. 3B, because the image sensor330generates an image using the light received from the lens310subject to the OIS correction, at least a portion of the image350may be distorted. For example, the coordinates (x, y) as in reference numeral352ofFIG. 3Amay be transformed to coordinates (x′, y′) as in reference numeral354ofFIG. 3B.

FIG. 4illustrates a block diagram of an electronic device including an image stabilizer, according to various embodiments.

Referring toFIG. 4, the electronic device101includes a camera module420(e.g., the camera module180ofFIG. 1), a sensor module440(e.g., the sensor module176ofFIG. 1), and a processor460(e.g., the processor120ofFIG. 1or the image signal processor260ofFIG. 2). According to an embodiment, the electronic device101may not include at least one of the components illustrated inFIG. 4or may further include at least additional component. For example, the electronic device101may further include a display (e.g., the display device160ofFIG. 1) to output a corrected image. For another example, the electronic device101may further include a memory (e.g., the memory130ofFIG. 1or the memory250ofFIG. 2) including instructions used to control other components.

According to an embodiment, the camera module420may obtain an image based on the light reflected from the subject and may perform the OIS correction while the image is obtained. The camera module420may include a lens assembly422(e.g., the lens assembly210ofFIG. 2or the lens310ofFIG. 3), an image sensor424(e.g., the image sensor230inFIG. 2or the image sensor330ofFIG. 3) and an image stabilizer426(e.g., the image stabilizer ofFIG. 2).

According to an embodiment, the image sensor424may obtain an image of a subject using light collected by the lens assembly422and transmit the image to the processor460. According to an embodiment, the image stabilizer426may control the lens assembly422to move in response to the motion of the electronic device101sensed by a sensor embedded in the image stabilizer426(or attached to an outer portion of the image stabilizer426). According to an embodiment, the sensor embedded in the image stabilizer426may sense motion, such as handshaking, in a high frequency band.

According to an embodiment, one sensor module440or a plurality of sensor modules440may be provided. For example, the sensor module440may include a gyro sensor or an acceleration sensor to sense the motion of the electronic device101. The sensor module440may be a sensor separate from the sensor embedded in the image stabilizer426. In the disclosure, data on the motion of the electronic device101may be referred to gyro data, and data (e.g., the second angle) on the motion of the lens assembly422may be referred to as OIS motion data. For example, the gyro data and the OIS motion data may include an angle value (unit: radian or degree). According to an embodiment, the sensor module440may be embedded in the electronic device101or may be disposed outside the electronic device101. According to an embodiment, the sensor module440may transmit the measured gyro data to the processor460.

According to an embodiment, the processor460may receive an image subject to the OIS correction from the camera module180, and may correct the image based on the first angle formed between the optical axis and the image sensor424before the OIS correction is performed and the second angle corresponding to the motion of the lens assembly422. The processor460may include an image frame obtaining module462, an OIS motion determining module464, and an image correcting module468. The image frame obtaining module462, the OIS motion determining module464, and the image correcting module468may be a hardware module or a software module.

According to an embodiment, the processor460may obtain at least one image, which is subject to the OIS correction, from the camera module420through the image frame obtaining module462.

According to an embodiment, the processor460may determine the second angle, which corresponds to the motion of the lens assembly422by the OIS correction, through the OIS motion determining module464. For example, the OIS motion determining module464may measure the second angle through the Hall sensor (not illustrated) embedded separately from the sensor module440. For another example, when the electronic device101does not include the Hall sensor, the OIS motion determining module464may measure a third angle (e.g., gyro data) corresponding the motion of the electronic device101through the sensor module440(e.g., the gyro sensor or acceleration sensor), and may estimate the second angle by using the third angle and a high frequency pass filter (HPF) or a band pass filter. For another example, the OIS motion determining module464may estimate the second angle, based on the difference between the angle corresponding to the motion of at least one feature point in the image obtained through the image frame obtaining module462and the third angle. In the disclosure, an angle corresponding to the motion of at least one feature point in the image may be referred to as a fourth angle or an image motion. According to an embodiment, the OIS motion determining module464may determine a fourth angle based on a position, a distance, or a direction in which at least one feature point in an image moves, by comparing two or more images with each other. The feature point may be replaced with, for example, an edge, feature, a key point, an interesting point, or a corner.

According to an embodiment, the processor460may correct an image based on at least one of the first angle (e.g., the first angle342ofFIG. 3A) formed between the optical axis and the image sensor, and the second angle corresponding to the motion of the lens assembly422by the OIS correction, through the image correcting module468. For example, the image correcting module468may determine variations of pixels in the image based on the second angle, and transform the image based on the variations of pixels. According to an embodiment, the image correcting module468may transform an image based on at least one of an affine transform or a perspective transform.

FIG. 5illustrates a flowchart of operations of an electronic device to correct an image based on a changed angle of a lens assembly, according to various embodiments. The operations illustrated inFIG. 5may be implemented by the electronic device101or the processor460.

Referring toFIG. 5, in operation505of a method500, the processor460may change the angle of the lens assembly422depending on the shaking of the electronic device101. For example, the processor460may control the lens assembly422to move in a direction opposite to the direction, in which the electronic device101is shaken, through the image stabilizer426. In operation510, the processor460may obtain an image through the image sensor424after the angle of the lens assembly422is changed.

In operation515, the processor460may correct a distorted image, by the second angle, based on at least one of the first angle (e.g., the first angle342ofFIG. 3A), which is formed between the optical axis of the lens assembly422and the image sensor424before the OIS correction is performed, and the second angle (e.g., reference numeral344ofFIG. 3B) corresponding to the motion of the lens assembly422by the OIS correction. According to an embodiment, the second angle may be measured by a Hall sensor. When the electronic device101does not include the Hall sensor, the processor460may estimate the second angle based on the third angle corresponding to the motion of the electronic device101. For example, the processor460may estimate the second angle based on the third angle using the high pass filter. For another example, the processor460may estimate the second angle based on the difference between the fourth angle corresponding to the motion of at least one feature point in the image and the third angle.

In operation520, the processor460may display the corrected image through a display (e.g., the display device160ofFIG. 1). Through the above-described method500, the electronic device101may compensate for the distortion of the image by the OIS motion and may improve the performance of the camera module420.

FIG. 6illustrates a flowchart of operations of an electronic device to determine variations of pixels based on a changed angle of a lens assembly, according to various embodiments. Operations illustrated inFIG. 6may refer to operations implemented in more detail as compared to operation515ofFIG. 5.

Referring toFIG. 6, in operation605, the processor460may determine a distortion variation representing a distortion extent of pixels forming an image, based on the second angle. For example, the processor460may determine the distortion variation of the pixels by comparing pixels on an image plane before the OIS correction is performed with pixels on an image plane subject to the OIS correction by the second angle. According to an embodiment, the distortion variation may be varied depending on pixels.

In operation610, the processor460may transform the image based on the determined distortion variation of the pixels. According to an embodiment, the image transform may include at least one of an affine transform or a perspective transform. According to an embodiment, the processor460may compensate for the distortion of the image using the distortion variation and the transform matrix of the determined pixels.

FIG. 7illustrates three dimensional coordinates showing a pixel variation resulting from a changed angle of a lens assembly, according to various embodiments. InFIG. 7, it may be assumed that the lens assembly422moves in a pitch direction.

Referring toFIG. 7, in 3D coordinates 700, an origin point705may correspond to a camera center. The camera center may refer to, for example, the center of the lens assembly422. A first image plane710may be referred to as an image plane in which an image of a subject is formed before OIS correction is performed. A z-axis may refer to an optical axis (e.g., the optical axis315ofFIG. 3A) normal to the first image plane710. A second image plane720may refer to a virtual plane (e.g., the plane332ofFIG. 3B) normal to an optical axis (e.g., the optical axis315-2ofFIG. 3B) of the lens assembly422after the angle of the lens assembly422is changed by the OIS correction. Even if the angle of the lens assembly422and the angle of the optical axis are changed by the OIS correction, the image of the subject is formed on the first image plane710, so pixels forming the image may be distorted by the difference between the first image plane710and the second image plane720.

According to an embodiment, when the angle of the lens assembly422is changed, the angle of the image plane, in which the image of the subject is formed, is changed, so the angle of the first image plane710may be assumed as being changed corresponding to the motion of the lens assembly422. For example, the first image plane710may be rotated by a second angle715(e.g., reference numeral344ofFIG. 3) in the pitch direction. A point a (x, y, z) included in the first image plane710may be moved to a′ (x′, y′, z′) to correspond to the motion of the lens assembly422. The relationship between the points a and a′ may be expressed as in Following Equation 1.
x′=x
y′=ycos θ
z′=z−ysin θ  Equation 1

In Equation 1, θ may refer to the second angle715. Although Equation 1 represents an example that the first image plane710rotates in a specified direction, a sign may be changed depending on the direction in which the first image plane710rotates.

To represent an angle of the light, which is incident into the origin705through the point a (x, y, z), and an angle of the light, which is incident into the origin705through the point a′ (x′, y′, z′), an angle θx′, and an angle θ may be expressed as in Equation 2 by taking into consideration Equation 1.
θx=atan(x/z)
θx′=atan(x/z′)=atan(x/(z−ysin θ))  Equation 2

In Equation 2, θx may refer to an angle of the light incident into the origin705through the point a (x, 0, z) and θx′may refer to an angle of the light incident into the origin705through the point a′ (x′, y0, z′),

The second image plane720on three dimensional coordinates may be expressed as two dimensional coordinates on a focal length. When the focal length corresponds to a z value, the coordinates of the point a′ (x′, y′) on the second image plane720may be expressed as in Equation 3 by taking into consideration Equation 2.
img(x)=f×tan θx′=f×(x/(z−ysin θ))
img(y)=f×y×cos θ/(z−ysin θ)  Equation 3

In Equation 3, image (x) and image (y) may refer to x′ and y′, respectively, and ‘f’ may refer to the focal length.

The relationship between the focal length, and a point (x′, y′), and the coordinates of the image may be expressed as in Equation 4.
x/z=xi/f
y/z=yi/fEquation 4

According to Equation 3 and Equation 4, the coordinates (x′, y′) on the second image plane720, which are present on the focal length may be expressed as in following Equation 5.

FIG. 8illustrates an operation of determining a pixel variation using a transform matrix, according to various embodiments.

Referring toFIG. 8, coordinates of a first image810(e.g., the first image710ofFIG. 7) at a focal length (e.g., z or f) and coordinates of a second image820formed by distorting the first image810by the OIS correction may be expressed as in following Equation 6.

In Equation 6, α and β may refer to variations of pixels when the first image810and the second image820are expressed through the transform matrix. The electronic device101may transform the distorted second image820into the first image810through α and β. For example, the electronic device101may transform an image through the affine transform or the perspective transform.

The α and β may be expressed as in following Equation 7 based on Equation 5 and Equation 6.

When the lens assembly422moves in a yaw direction, the α and β may be expressed as in following Equation 8.

According to an embodiment, the electronic device101may directly measure the second angle715(e.g., θ) through the Hall sensor or may estimate the second angle based on the third angle corresponding to the motion of the electronic device101. For example, the electronic device101may measure the third angle through the gyro sensor or the acceleration sensor, and may estimate the second angle using the third angle and the high pass filter. An embodiment in which the electronic device101estimates the second angle through the high pass filter may be described with reference toFIGS. 9 to 10. For another example, the electronic device101may determine a fourth angle representing the motion of at least one feature point in the image, and may estimate the second angle based on the difference between the fourth angle and the third angle. An embodiment in which the electronic device101estimates the second angle based on the fourth angle may be described with reference toFIG. 11.

FIG. 9illustrates a high pass filter circuit to estimate a changed angle of a lens assembly, according to various embodiments.

Referring toFIG. 9, the processor460may estimate a second angle y [n] by using a third angle x[n] and a high pass filter900. The high pass filter H(z) may be expressed as in following Equation 9.

In Equation 9, z may refer to a parameter of z-transform, and a0, a1, and b1 may mean filter coefficients, respectively. The filter coefficient may be determined based on, for example, a limit range of an angle at which the lens assembly422may move in the OIS correction. The processor460may estimate the second angle y[n] based on Equation 9 representing the high pass filter900, as in following Equation 10.
y[n]=a0x[n]+a1x[n]−1+b1y[n]−1Equation 10

FIG. 10illustrates a graph showing an estimated angle of a lens assembly using a high pass filter, according to various embodiments.

Referring toFIG. 10, a graph1001represents an estimation result of a second angle (e.g., OIS motion data) in the pitch direction, and a graph1002may represent an estimation result of the second angle in the yaw direction. Although not illustrated inFIG. 10, an estimation result of the second angle in a roll direction may be similarly represented. In the graph1001and the graph1002, a horizontal axis may represent time (unit: second (Sec.)), and a vertical axis may represent an angle (unit: degree).

In the graph1001and the graph1002, hall sensor data1030may represent a measurement angle of the second angle through the Hall sensor. The OIS motion (that is, the second angle1020) estimated from the third angle1010through the high pass filter900may have a value the same as or similar to the value of the OIS motion (that is, the Hall sensor data1030), which is measured by the Hall sensor. The electronic device101may more clearly estimate the second angle from the third angle measured through the gyro sensor or the acceleration sensor without separately using the Hall sensor.

FIG. 11illustrates an operation of estimating a changed angle of a lens assembly based on an angle corresponding to a motion of an electronic device and an angle corresponding to the motion of a feature point in an image, according to various embodiments.

Referring toFIG. 11, the electronic device101may obtain an image1110after an OIS operation is performed through the camera module420. The electronic device101may determine a motion vector (MV)1115corresponding to at least one feature point in an image1110by comparing the image1110with another image obtained from a previous frame through the camera module420. The electronic device101may measure a gyro motion vector (MVg)1117corresponding to the motion of the electronic device101through the sensor module440while the image1110is obtained. AlthoughFIG. 11illustrates that the electronic device101determines the MV1115and the MVg1117having a linear vector form, the electronic device101may determine the third angle corresponding to the motion of the electronic device101by dividing the MV1115by the focal length, or may determine a fourth angle corresponding the motion of at least one feature point in the image by dividing the MVg1117by the focal length. When the image1110is obtained after the OIS correction is performed, the size of the MV1115is reduced by an extent, in which the lens assembly422moves, through the OIS correction, so the electronic device101may estimate the second angle based on the difference between the MVg1117and the MV1115. For example, the second angle may be expressed as in following Equation 11.
θ=(MVg−MV)/focal length  Equation 11

FIG. 12illustrates an operation of compensating an image in unit of a line, according to various embodiments.

Referring toFIG. 12, when the image sensor424includes a CMOS sensor, the electronic device101may read out light in order from the upper most line1202-0to the lower most line1202-M (M is a natural number equal to or greater than ‘1’) while obtaining an image1201. The operation that the electronic device101reads out the light in unit of a line may be referred to as a rolling shutter operation.

According to an embodiment, the electronic device101may compensate for image distortion in unit of a line through the OIS correction. For example, the electronic device101may calculate variations of the pixels in order from the top line1202-0to the bottom line1202-M to correct the image1201, or may calculate the variation in the reverse order.

As described above, an electronic device (e.g., the electronic device101ofFIG. 1) may include a lens assembly (e.g., the lens assembly422ofFIG. 4) including one or more lenses, an image sensor (e.g., the image sensor424ofFIG. 4), an image stabilizer (e.g., the image stabilizer426ofFIG. 4), and a processor (e.g., the processor460ofFIG. 4). The lens assembly may be arranged to from a first angle between an optical axis of at least some lenses of the one or more lenses, and a surface of the image sensor. The processor may be configured to change an angle of the lens assembly through the image stabilizer in response to shaking of the electronic device, obtain an image through the image sensor, in a state that the angle of the lens assembly is changed, correct, based at least on the first angle and a second angle corresponding to the changed angle of the lens assembly, at least a portion of the image, which is distorted, by the second angle, and display the corrected image through a display electrically connected with the electronic device.

According to an embodiment, the processor may be configured to determine a distortion variation representing an extent in which pixels forming the image are distorted, based on the second angle, and transform the image based on the distortion variation.

According to an embodiment, the processor may be configured to perform at least one of an affine transform or a perspective transform, as a portion of transforming of the image.

According to an embodiment, the electronic device may further include a Hall sensor to sense a motion of the lens assembly. The processor is configured to measure the second angle using the Hall sensor.

According to an embodiment, the electronic device may further include a motion sensor (e.g., the sensor module440ofFIG. 4) to sense a motion of the electronic device. The processor may be configured to: measure a third angle corresponding to the motion of the electronic device using the motion sensor, and estimate the second angle based on the third angle, using a high pass filter.

According to an embodiment, a filter coefficient of the high pass filter may be determined based on a limit range of an angle at which the lens assembly moves.

According to an embodiment, the electronic device may further include a motion sensor (e.g., the sensor module440ofFIG. 4) to sense a motion of the electronic device. The processor may be configured to measure a third angle corresponding to the motion of the electronic device, through the motion sensor, determine a fourth angle corresponding to the motion of at least one feature point in the image, and estimate the second angle based on the third angle and the fourth angle.

According to an embodiment, the image sensor may include a complementary metal oxide semiconductor (CMOS), and the processor may be configured to obtain the image in unit of a line through the image sensor, and correct the image in unit of the line based partially on the first angle and the second angle.

As described above, an electronic device (e.g., the electronic device101ofFIG. 1) may include an image sensor (e.g., the image sensor424ofFIG. 4), a lens assembly (e.g., the lens assembly422ofFIG. 4) including one or more lenses, disposed above the image sensor to have a first specified angle between at least some lenses of the one or more lenses and a surface of the image sensor to provide an image to the image sensor such that the image sensor obtains the image at the first specified angle, an image stabilizer (e.g., the image stabilizer426ofFIG. 4) which is able to correct at least partially shaking of the electronic device by changing an angle formed between the optical axis and the surface of the image sensor, and a processor (e.g., the processor460ofFIG. 4). The processor may be configured to change the angle of the lens assembly to a second specified angle depending on the shaking, using the image stabilizer, obtain an image of an external object at the second specified angle using the image sensor, correct, based partially on the first specified angle and the second specified angle, at least a portion of the image distorted by the second specified angle, and display the corrected image through a display electrically connected with the electronic device.

According to an embodiment, the processor may be configured to determine a distortion variation representing an extent in which pixels forming the image are distorted, based on the second angle, and transform the image based on the distortion variation.

According to an embodiment, the electronic device may further include a Hall sensor to sense a motion of the lens assembly, and the processor may be configured to measure the second specified angle using the Hall sensor.

According to an embodiment, the electronic device may further include a motion sensor to sense a motion of the electronic device, and the processor may be configured to measure a third specified angle corresponding to the motion of the electronic device, through the motion sensor and estimate the second angle based on the third specified angle, using a high pass filter.

According to an embodiment, a filter coefficient of the high pass filter may be determined based on a limit range of an angle at which the lens assembly moves.

According to an embodiment, the electronic device may further include a motion sensor to sense a motion of the electronic device, and the processor may be configured to measure a third specified angle corresponding to the motion of the electronic device, through the motion sensor, determine a fourth specified angle corresponding to a motion of at least one feature point in the image, and estimate the second angle based on the third specified angle and the fourth specified angle.

According to an embodiment, the image sensor may include a CMOS sensor, and the processor may be configured to obtain the image in unit of a line through the image sensor, and correct the image in unit of the line based partially on the first specified angle and the second specified angle.

As described above, a method of an electronic device may include changing an angle of a lens assembly, to change a first angle between an optical axis of at least some lenses and a surface of an image sensor, in response to shaking of the electronic device, obtaining an image through the image sensor, after changing the angle of the lens assembly, correcting at least a portion of the image distorted by the second angle, based on the first angle and the second angle corresponding to the angle of the changed angle of the lens assembly, and displaying the corrected image through a display electrically connected with the electronic device.

According to an embodiment, in the method, the correcting of the at least a portion of the image may include determining a distortion variation representing an extent in which pixels forming the image is distorted, based on the second angle, and transforming the image based on the distortion variation.

According to an embodiment, the method may further include measuring the second angle through the Hall sensor of the electronic device.

According to an embodiment, the method may include measuring a third angle corresponding to a motion of the electronic device, through a motion sensor of the electronic device, and estimating the second angle based on the third angle, using a high pass filter.

According to an embodiment, the method may include measuring a third angle corresponding to a motion of the electronic device, through a motion sensor of the electronic device, determining a fourth angle corresponding to a motion of at least one feature point in the image, and estimating the second angle based on the third angle and the fourth angle.