Patent Description:
Recently, there has been provided an image stabilization function capable of preventing the afterimage of the captured image, which is caused due to the hand shake or the unintentional movement of a user. The image stabilization may include a video digital image stabilization (VDIS) scheme or a digital image stabilization (DIS) scheme. The published patent application <CIT> discloses an electronic device including two cameras with one among the cameras having a rotatable mirror, wherein the electronic device provides a shaking compensation function and a distortion correction.

An electronic device may provide a function of tracking an object by moving a lens. When the lens is moved to track the object, an image may be distorted and blurred due to the movement of the object. To correct the distortion and blurring of the image, after the distortion of the image resulting from the movement of the lens is corrected, the corrected image is transmitted to an image stabilization module, such that the blurring of the image is corrected, for example, digital image stabilization (DIS). The above method may include a plurality of operations of outputting images that may cause degradation to image quality.

Various embodiments of the disclosure may provide an electronic device configured to simultaneously correct the distortion and the blurring of the image to correspond to the movement of a field of view (FOV) in a camera system configured to move the FOV of a camera or a lens using a reflective member, for example, a prism or a mirror.

According to the invention, there is provided an electronic device according to claim <NUM>.

According to another aspect of the invention, there is provided a method for operating an electronic device according to claim <NUM>.

According to various embodiments of the disclosure, the electronic device may minimize degradation of the image quality and reduce a current consumption by correcting the blurring and the distortion caused by the movement of the FOV of the camera in the camera system configured to move the FOV of the camera or the lens using the reflective member.

In the following description made with respect to the accompanying drawings, similar components will be assigned with similar reference numerals.

Hereinafter, various embodiments of the disclosure may be described with reference to accompanying drawings. However, those of ordinary skill in the art will understand that the disclosure is not limited to a specific embodiment, modifications can be made without departing from the scope defined by the appended claims.

Referring to <FIG>, the electronic device <NUM> in the network environment <NUM> may communicate with an electronic device <NUM> via a first network <NUM> (e.g., a short-range wireless communication network), or at least one of an electronic device <NUM> or a server <NUM> via a second network <NUM> (e.g., a long-range wireless communication network). According to an embodiment, the electronic device <NUM> may include a processor <NUM>, memory <NUM>, an input module <NUM>, a sound output module <NUM>, a display module <NUM>, an audio module <NUM>, a sensor module <NUM>, an interface <NUM>, a connecting terminal <NUM>, a haptic module <NUM>, a camera module <NUM>, a power management module <NUM>, a battery <NUM>, a communication module <NUM>, a subscriber identification module(SIM) <NUM>, or an antenna module <NUM>. In some embodiments, at least one of the components (e.g., the connecting terminal <NUM>) may be omitted from the electronic device <NUM>, or one or more other components may be added in the electronic device <NUM>. In some embodiments, some of the components (e.g., the sensor module <NUM>, the camera module <NUM>, or the antenna module <NUM>) may be implemented as a single component (e.g., the display module <NUM>).

<FIG> is a block diagram <NUM> illustrating the camera module <NUM> according to various embodiments.

Referring to <FIG>, the camera module <NUM> may include a lens assembly <NUM>, a flash <NUM>, an image sensor <NUM>, an image stabilizer <NUM>, memory <NUM> (e.g., buffer memory), and an image signal processor <NUM>. The lens assembly <NUM> may collect light emitted or reflected from an object whose image is to be taken. The lens assembly <NUM> may include one or more lenses.

According to an embodiment, the camera module <NUM> may include a plurality of lens assemblies <NUM>. In such a case, the camera module <NUM> may form, for example, a dual camera, a <NUM>-degree camera, or a spherical camera. Some of the plurality of lens assemblies <NUM> may have the same lens attribute (e.g., view angle, focal length, auto-focusing, f number, or optical zoom), or at least one lens assembly may have one or more lens attributes different from those of another lens assembly. The lens assembly <NUM> may include, for example, a wide-angle lens or a telephoto lens.

The flash <NUM> may emit light that is used to reinforce light reflected from an object. According to an embodiment, the flash <NUM> may include one or more light emitting diodes (LEDs) (e.g., a red-green-blue (RGB) LED, a white LED, an infrared (IR) LED, or an ultraviolet (UV) LED) or a xenon lamp. The image sensor <NUM> may obtain an image corresponding to an object by converting light emitted or reflected from the object and transmitted via the lens assembly <NUM> into an electrical signal. According to an embodiment, the image sensor <NUM> may include an image sensor selected from image sensors having different attributes, such as a RGB sensor, a black-and-white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same attribute, or a plurality of image sensors having different attributes. Each image sensor included in the image sensor <NUM> may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer <NUM> may move the image sensor <NUM> or at least one lens included in the lens assembly <NUM> in a particular direction, or control an operational attribute (e.g., adjust the read-out timing) of the image sensor <NUM> in response to the movement of the camera module <NUM> or the electronic device <NUM> including the camera module <NUM>. This allows compensating for at least part of a negative effect (e.g., image blurring) by the movement on an image being captured. According to an embodiment, the image stabilizer <NUM> may sense such a movement by the camera module <NUM> or the electronic device <NUM> using a gyro sensor or an acceleration sensor disposed inside or outside of the camera module <NUM>. According to an embodiment, the image stabilizer <NUM> may be implemented, for example, as an optical image stabilizer.

The memory <NUM> may store, at least temporarily, at least part of an image obtained via the image sensor <NUM> for a subsequent image processing task. For example, if image capturing is delayed due to shutter lag or multiple images are quickly captured, a raw image obtained (e.g., a Bayer-patterned image, a high-resolution image) may be stored in the memory <NUM>, and its corresponding copy image (e.g., a low-resolution image) may be previewed via the display module <NUM>. Thereafter, if a specified condition is met (e.g., by a user's input or system command), at least part of the raw image stored in the memory <NUM> may be obtained and processed, for example, by the image signal processor <NUM>. According to an embodiment, the memory <NUM> may be configured as at least part of the memory <NUM> or as a separate memory that is operated independently from the memory <NUM>.

The image signal processor <NUM> may perform one or more image processing with respect to an image obtained via the image sensor <NUM> or an image stored in the memory <NUM>. The one or more image processing may include, for example, depth map generation, three-dimensional (3D) modeling, panorama generation, feature point extraction, image synthesizing, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor <NUM> may perform control (e.g., exposure time control or read-out timing control) with respect to at least one (e.g., the image sensor <NUM>) of the components included in the camera module <NUM>. An image processed by the image signal processor <NUM> may be stored in the memory <NUM> for further processing, or may be provided to an external component (e.g., the memory <NUM>, the display module <NUM>, the electronic device <NUM>, the electronic device <NUM>, or the server <NUM>) outside the camera module <NUM>.

According to an embodiment, the image signal processor <NUM> may be configured as at least part of the processor <NUM>, or as a separate processor that is operated independently from the processor <NUM>. If the image signal processor <NUM> is configured as a separate processor from the processor <NUM>, at least one image processed by the image signal processor <NUM> may be displayed, by the processor <NUM>, via the display module <NUM> as it is or after being further processed.

According to an embodiment, the electronic device <NUM> may include a plurality of camera modules <NUM> having different attributes or functions. In such a case, at least one of the plurality of camera modules <NUM> may form, for example, a wide-angle camera and at least another of the plurality of camera modules180 may form a telephoto camera. Similarly, at least one of the plurality of camera modules <NUM> may form, for example, a front camera and at least another of the plurality of camera modules <NUM> may form a rear camera.

<FIG> illustrates a block diagram of an electronic device, according to an embodiment of the disclosure.

Referring to <FIG>, according to an embodiment, an electronic device <NUM> (e.g., the electronic device <NUM> of <FIG>) may include a processor <NUM> (e.g., the processor <NUM> of <FIG>), a memory <NUM> (e.g., the memory <NUM> of <FIG>), a display <NUM> (e.g., the display module <NUM> of <FIG>), a sensor circuit <NUM> (e.g., the sensor module <NUM> of <FIG>), a camera module <NUM> (e.g., the camera module <NUM> of <FIG>), and/or a communication circuit <NUM> (e.g., the communication module <NUM> of <FIG>).

For example, the processor <NUM> may be operatively connected with the memory <NUM>, the display <NUM>, the sensor circuit <NUM>, the camera module <NUM>, and the communication circuit <NUM>. The memory <NUM> may store one or more instructions that may be executed by the processor <NUM> to perform various operations of the electronic device <NUM>.

According to an embodiment, the display <NUM> includes a plurality of pixels. For example, the electronic device <NUM> may display an image, which is obtained through the camera module <NUM>, on the display <NUM>. The display <NUM> may include a touch screen to sense a touch input. The electronic device <NUM> may identify an object (or a subject) in the displayed image, based on an input to the display <NUM>. For example, a user may capture a moving picture while tracking a relevant object through a touch input to the displayed object.

The sensor circuit <NUM> senses the movement of the electronic device <NUM>. For example, the sensor circuit <NUM> includes at least one sensor (e.g., a motion sensor, a gyro sensor, an acceleration sensor, and/or an inertial sensor) to sense movement information of the electronic device <NUM>. The electronic device <NUM> senses the movement information of the electronic device <NUM> using the sensor circuit <NUM> and controls the camera module <NUM>, based on the movement information and the movement of the object. For example, the electronic device <NUM> may move a lens (e.g., a lens unit <NUM> of <FIG>) or a reflective member or device (e.g., a reflective member or device <NUM> of <FIG>) of a second camera <NUM>, based on the movement information of the electronic device <NUM>. The second camera <NUM> is configured to provide an optical image stabilization function or an object tracking function by moving the reflective member <NUM>. However, embodiments are not limited thereto. According to various embodiments, the second camera <NUM> may be configured to provide the optical image stabilization function by moving the lens unit <NUM>.

The camera module <NUM> includes a plurality of cameras. The camera module <NUM> includes a first camera <NUM> and the second camera <NUM>. The first camera <NUM> has a first field of view (FOV). For example, the first camera <NUM> may be a wide camera, and may include at least one wide-angle lens. The second camera <NUM> has a second FOV. The second FOV is narrower than the first FOV. For example, the second camera <NUM> may be a tele-camera, and may include at least one telephoto lens.

According to an embodiment, the camera module <NUM> may further include a third camera <NUM>. The third camera <NUM> may have a third FOV. The third FOV may be wider than the first FOV. For example, the third camera <NUM> may be an ultra-wide angle camera and may include at least one ultra wide angle lens. According to various embodiments, the camera module <NUM> may not include the third camera <NUM>.

According to an embodiment, the first camera <NUM>, the second camera <NUM>, and the third camera <NUM> may be disposed in the electronic device <NUM> to face the same direction with respect to the electronic device <NUM>. For example, the first camera <NUM>, the second camera <NUM>, and the third camera <NUM> may be disposed on a rear surface of the electronic device <NUM>. At least one of the first camera <NUM>, the second camera <NUM>, and the third camera <NUM> may include a component (e.g., an optical image stabilizer (OIS) driving module) for image stabilization. The second camera <NUM> may be configured to move the central direction of the second FOV. Hereinafter, an operation of moving the position of the FOV (second FOV) of the second camera <NUM> may be referred to as an operation of moving the central axis of the second FOV of the second camera <NUM> by rotating the reflective member (e.g., the reflective member <NUM> of <FIG>), which is included in the second camera <NUM>, within a specific range. Moving the central axis of the second FOV of the second camera <NUM> will be described later with reference to <FIG>.

According to an embodiment, the camera module <NUM> may further include a distance sensor <NUM>. The distance sensor <NUM> may include at least one sensor to measure a distance between the camera module <NUM> and the object. For example, the distance sensor <NUM> may include a time of flight (ToF) sensor, a laser sensor, an infrared sensor, and/or a light detection and ranging (LiDAR) sensor but the invention is not restricted thereto. The electronic device <NUM> may be configured to perform auto focusing (AF) using the distance sensor <NUM>.

According to an embodiment, the communication circuit <NUM> may be configured to support short-range wireless communication and/or long-range wireless communication. For example, the communication circuit <NUM> may support short-range wireless communication such as Bluetooth (e.g., Bluetooth legacy and/or Bluetooth low energy (BLE)), a neighbor awareness network (NAN), Wi-Fi direct, and/or an ultra-wide band (UWB).

Components of the electronic device <NUM> illustrated in <FIG> are provided for the illustrative purpose, and embodiments are not limited thereto. The electronic device may further include components (at least one of the components of the electronic device <NUM> of <FIG>) not illustrated in <FIG>. For example, the electronic device <NUM> may further include a battery and a housing. Hereinafter, the operations of the electronic device <NUM> will be described by making reference to components of the electronic device <NUM> described with reference to <FIG>. For example, the operations of the electronic device <NUM> may be performed by the processor <NUM>.

<FIG> illustrates an electronic device, according to an embodiment.

Referring to <FIG>, according to an embodiment, the electronic device <NUM> may include a housing <NUM>, an input device <NUM> (e.g., the input module <NUM> of <FIG>), the display <NUM>, and the camera module <NUM>.

According to an embodiment, the housing <NUM> may form an outer appearance of the electronic device <NUM>. For example, the housing <NUM> may include a front surface <NUM>, a rear surface <NUM> facing a direction opposite to that of the front surface <NUM>, and a side surface <NUM> to surround a space between the front surface <NUM> and the rear surface <NUM>. According to various embodiments, the housing <NUM> may be a structure forming some of the front surface, the rear surface, and the side surface.

According to an embodiment, the housing <NUM> may be provided to have the display <NUM> and the camera module <NUM> disposed in the housing <NUM>. For example, the housing <NUM> may be configured such that at least a portion of the display <NUM> is visually exposed through the front surface <NUM> and at least a portion of the camera module <NUM> is visually exposed through the rear surface <NUM>. According to various embodiments, other components (e.g., the processor <NUM>, the memory <NUM>, the sensor circuit <NUM>, and the communication circuit <NUM> of <FIG>) of the electronic device <NUM> may be disposed in the housing <NUM>.

According to an embodiment, the input device <NUM> may be disposed on the side surface <NUM> of the housing <NUM>. The input device <NUM> may be referred to as a key input device. For example, the input device <NUM> may include one or more side keys or side buttons. The input device <NUM> may include a power key <NUM> and/or a volume key <NUM>. According to various embodiments, the electronic device <NUM> may not include some or the entire portion of the input device <NUM>, and the input device <NUM> not included may be implemented in another form, such as a soft key, on the display <NUM>.

According to an embodiment, the camera module <NUM> may include the first camera <NUM>, the second camera <NUM>, the third camera <NUM>, the distance sensor <NUM>, a flash <NUM>, and a fourth camera <NUM>. For example, the first camera <NUM>, the second camera <NUM>, and the third camera <NUM> may be rear cameras, and the fourth camera <NUM> may be a front camera.

According to an embodiment, the first camera <NUM>, the second camera <NUM>, the third camera <NUM>, the distance sensor <NUM>, and/or the flash <NUM> may be visually exposed through at least a portion of a camera decorating member <NUM> disposed on the rear surface <NUM> of the housing <NUM>. The camera decorating member <NUM> may include a plurality of transparent areas formed of a transparent member. For example, the first camera <NUM>, the second camera <NUM>, the third camera <NUM>, and the distance sensor <NUM> may be configured to receive light from the outside or obtain an external image through the transparent area of the camera decorating member <NUM>. For example, the flash <NUM> may be configured to emit light to the outside through the transparent area of the camera decorating member <NUM>.

Although arrangement and configuration of rear cameras illustrated in <FIG> are provided for the illustrative purpose, embodiments of the disclosure are not limited thereto. For example, the camera decorating member <NUM> may further include a microphone hole). At least some of the illustrated components may be omitted. According to various embodiments, the camera decorating member <NUM> may be formed to protrude to a specific height from the rear surface <NUM> of the housing <NUM>, or may form substantially the same plane as the rear surface <NUM>.

According to an embodiment, the fourth camera <NUM> may be visually exposed through the front surface <NUM> of the housing <NUM> through a portion of the display <NUM>. For example, the fourth camera <NUM> may be configured to receive light from the outside or obtain an external image through the camera hole formed in a portion of the display <NUM>. According to various embodiments, the fourth camera <NUM> may be provided in the form of a punch-hole camera or an under display camera (UDC).

<FIG> illustrates a second camera, according to an embodiment. <FIG> illustrates a second camera, according to an embodiment.

<FIG> is an exploded perspective view of the second camera <NUM>.

<FIG> is a view schematically illustrating a cross-section of the second camera <NUM>. For example, <FIG> may be a cross-sectional view illustrating a cross section of the second camera <NUM>, which is taken in a direction parallel to an x-axis.

Referring to <FIG> and <FIG>, according to an embodiment, the second camera <NUM> may include a camera housing <NUM>, the lens unit <NUM>, an image sensor assembly <NUM>, the reflective member <NUM>, a guide member <NUM>, and a fixing member <NUM>.

According to an embodiment, the second camera <NUM> may have a structure (e.g., a folded-type camera) in which a direction (e.g., a z-axis direction) of external light incident onto the second camera <NUM> is perpendicular to an optical axis OA of a lens included in the lens unit <NUM>. For example, the lens or an image sensor <NUM> of the second camera <NUM> may be disposed to be substantially perpendicular to the rear surface (e.g., the rear surface <NUM> of the housing <NUM> of <FIG>) of the electronic device <NUM>, and the second camera <NUM> is configured to change an optical path through the reflective member <NUM>.

According to an embodiment, the camera housing <NUM> may include an upper housing <NUM> and a lower housing <NUM>. The reflective member <NUM> may be positioned inside the lower housing <NUM>, and may receive light from the outside of the camera housing <NUM> through an opening <NUM> formed in the upper housing <NUM>. The reflective member <NUM> may reflect and/or refract light, which is incident in the z-axis direction, in the x-direction (e.g., in the direction of the lens unit <NUM>). The reflective member <NUM> may be fixed to the guide member <NUM> by the fixing member <NUM>. For example, the reflective member <NUM> may be referred to as the lens assembly (e.g., the lens assembly <NUM> of <FIG>) of the second camera <NUM>, together with the lens unit <NUM>. The lens assembly of the second camera <NUM> may be understood as a prism type lens assembly. For example, in the second camera <NUM>, the optical path onto the lens unit <NUM> and the image sensor <NUM> may be changed depending on the movement of the prism (e.g., the reflective member <NUM>). Hereinafter, in the second camera <NUM>, the movement and/or rotation of the reflective member <NUM> may refer to an operation of moving the second FOV of the second camera <NUM>.

According to various embodiments, the reflective member <NUM> may include a mirror or a prism, and the reflective member <NUM> is not limited to the above-described examples. For example, the reflective member <NUM> may be implemented by using various components to change the traveling path of light by refracting and/or reflecting the light.

According to an embodiment, the guide member <NUM> may be positioned inside the lower housing <NUM>. The guide member <NUM> may be configured to guide rotation of the reflective member <NUM>. For example, the guide member <NUM> may include at least one magnetic member (e.g., a magnet and/or an electromagnet). The electronic device <NUM> may move the guide member <NUM> by adjusting the magnetism of the magnetic member positioned in the lower housing <NUM>. For example, the guide member <NUM> may include at least one gear. The electronic device <NUM> may move the guide member <NUM> by controlling a driver positioned in the lower housing <NUM> and operated together with the gear of the guide member <NUM>. For example, the electronic device <NUM> may rotate the reflective member <NUM> about at least one rotation axis (e.g., a first rotation axis <NUM> and a second rotation axis <NUM> of <FIG>) perpendicular to the optical axis OA by moving the guide member <NUM>. The electronic device <NUM> may perform an image stabilization function (e.g., OIS correction) and/or an object tracking function.

According to an embodiment, the lens unit <NUM> may include a plurality of lens. For example, the electronic device <NUM> may perform a focusing function (e.g., an AF function) by moving the plurality of lenses of the lens unit <NUM> in a direction parallel to the optical axis OA. The optical axis OA may be defined as a line linking the center of the lens to the focus. The lens unit <NUM> may transmit light refracted by the reflective member <NUM> to the image sensor assembly <NUM> or the image sensor <NUM> (e.g., the image sensor <NUM> of <FIG>). The image sensor assembly <NUM> may convert an optical signal received through the lens unit <NUM> into an electrical signal, and may transmit the converted electrical signal to the processor (e.g., the image signal processor <NUM> of <FIG>).

Hereinafter, the path through which the external light reaches the image sensor <NUM> of the second camera <NUM> will be described with reference to <FIG>.

According to an embodiment, the second camera <NUM> may receive external light through the opening <NUM> of the camera housing <NUM>. The external light may be incident on the reflective member <NUM> through the opening <NUM>. For example, the reflective member <NUM> may be positioned to receive the external light, preferably under the opening <NUM> of the camera housing <NUM> (e.g., in the -z-axis direction). According to various embodiments, the opening <NUM> may be aligned with the transparent area formed in the camera decorating member (e.g., the camera decorating member <NUM> of <FIG>) in the z-axis direction. The external light may be incident on the reflective member <NUM> in the -z-axis direction through the transparent area and the opening <NUM> of the camera decorating member <NUM>. Light incident on the reflective member <NUM> is refracted in the +x-axis direction. The light refracted in the +x-axis direction may pass through the lens unit <NUM> and may enter the image sensor <NUM>. Light passing through the lens of the lens unit <NUM> may be focused on the image sensor <NUM> of the image sensor assembly <NUM>.

As described above with reference to <FIG>, the electronic device <NUM> may move the lens assembly (e.g., the reflective member <NUM> and/or the lens unit <NUM>) of the second camera <NUM>. The electronic device <NUM> may move the reflective member <NUM> and/or the lens unit <NUM> of the second camera <NUM>. For example, the movement of the lens unit <NUM> in the optical axis OA direction may be an operation for performing an auto-focus function of the second camera <NUM>. The movement (e.g., rotation) of the reflective member <NUM> may be an operation for performing an image stabilization function and/or an object tracking function of the second camera <NUM>. Hereinafter, a movement (e.g., rotation) of the reflective member <NUM> of the second camera <NUM> according to an embodiment is described with reference to <FIG> and <FIG>.

<FIG> illustrates a rotation operation of a reflective member of a second camera according to an embodiment.

Referring to <FIG> and <FIG>, according to an embodiment, the electronic device <NUM> may perform an image stabilization function and/or an object tracking function by rotating the reflective member <NUM> of the second camera <NUM> within a specific range about rotation axes <NUM> and <NUM>. For example, the second camera <NUM> may move or change the field of view (e.g., the second FOV) or the line of vision of the second camera <NUM>, as the optical path incident on the lens unit <NUM> is changed by the rotation operation of the reflective member <NUM>.

According to an embodiment, the reflective member <NUM> may be configured to rotate in a specified range about the first rotation axis <NUM> and the second rotation axis <NUM> perpendicular to the optical axis OA of the lens. The first rotation axis <NUM> and the second rotation axis <NUM> may be substantially perpendicular to each other. For example, the optical axis OA may be a virtual axis extending parallel to the x-axis. The first rotation axis <NUM> may be a virtual axis extending parallel to the z-axis. The second rotation axis <NUM> may be a virtual axis extending parallel to the y-axis.

According to an embodiment, an operation in which the reflective member <NUM> rotates about the first rotation axis <NUM> may be understood as a yaw tilt driving or yawing operation. The electronic device <NUM> may perform a first rotation R1 for the reflective member <NUM>. The first rotation R1 may refer to an operation of rotating the reflective member <NUM> in a first rotation direction (e.g., counterclockwise when viewed based on <FIG>) about the first rotation axis <NUM>. The electronic device <NUM> may perform a second rotation R2 for the reflective member <NUM>. The second rotation R2 may refer to an operation of rotating the reflective member <NUM> in a second rotation direction which is opposite to the first rotation direction (e.g., clockwise when viewed based on <FIG>) about the first rotation axis <NUM>.

According to an embodiment, an operation in which the reflective member <NUM> rotates about the second rotation axis <NUM> may be understood as a pitch tilt driving or pitching operation. The electronic device <NUM> may perform a third rotation R3 for the reflective member <NUM>. The third rotation R3 may refer to an operation of rotating the reflective member <NUM> in a third rotation direction (e.g., counterclockwise when viewed based on <FIG>) about the second rotation axis <NUM>. The electronic device <NUM> may perform a fourth rotation R4 for the reflective member <NUM>. The fourth rotation R4 may refer to an operation of rotating the reflective member <NUM> in a fourth rotation direction which is opposite to the third rotation direction (e.g., clockwise when viewed based on <FIG>) about the second rotation axis <NUM>.

Although <FIG> illustrates that the first rotation axis <NUM> and the second rotation axis <NUM> pass through the reflective member <NUM> such that the first rotation axis <NUM> and the second rotation axis <NUM> are positioned inside the reflective member <NUM>, embodiments are not limited thereto. For example, the first rotation axis <NUM> or the second rotation axis <NUM> may be positioned outside the reflective member <NUM> without overlapping the reflective member <NUM>. According to various embodiments, the first rotation axis <NUM> and the second rotation axis <NUM> may be formed by the guide member (e.g., the guide member <NUM> of <FIG>).

The reflective member <NUM> may be configured to rotate about an axis (e.g., an x axis) parallel to the optical axis OA. For example, an operation in which the reflective member <NUM> rotates about the x axis may be a roll tilt driving or rolling operation.

According to an embodiment, when the electronic device <NUM> moves, for example, by being shaken while the camera module <NUM> (or the second camera <NUM>) is obtaining the image, the electronic device <NUM> (or the image stabilizer <NUM> of <FIG>) may perform an OIS correction operation to compensate for the movement such as shaking of the electronic device <NUM>. For example, the electronic device <NUM> may perform the OIS correction operation by rotating the reflective member <NUM> of the second camera <NUM> about the first rotation axis <NUM> and/or the second rotation axis <NUM> to cope with the shaking of the electronic device <NUM>, which is sensed through the sensor circuit (e.g., the sensor circuit <NUM> of <FIG>). However, the above-described OIS correction manner is provided for the illustrative purpose. According to various embodiments, the electronic device <NUM> may perform the OIS correction operation in a sensor shift manner to move the image sensor <NUM>.

The electronic device <NUM> senses the movement such as shaking of the electronic device <NUM> using the sensor circuit <NUM> (e.g., a motion sensor <NUM> of <FIG>). According to embodiments, information corresponding to the movement of the electronic device <NUM> is referred to as first movement information or gyro data. In addition, according to embodiments, information, which corresponds to the movement of some components (the lens unit <NUM> and/or the reflective member <NUM>) of the lens assembly, resulting from the OIS correction is referred to as second movement information or OIS movement data. In addition, according to embodiments, information, which corresponds to the movement of some components (the lens unit <NUM> and/or the reflective member <NUM>) of the lens assembly, resulting from object tracking is referred to as third movement information or object tracking movement data. According to an embodiment, the first movement information, the second movement information, or the third movement information may be expressed as the change in an angle, or a vector over time.

<FIG> illustrates a digital image stabilization (DIS) correction operation of an electronic device according to an embodiment.

Referring to <FIG>, according to an embodiment, the electronic device <NUM> may generate an image <NUM> by performing the DIS correction (or video digital image stabilization (VDIS) correction) operation, based on at least one of a plurality of image frames <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>,. , and <NUM>-M (M is a natural number equal to or greater than '<NUM>'), which are obtained through the camera module <NUM>, and first movement information <NUM> (e.g., the gyro data) corresponding to the movement of the electronic device <NUM>. For example, the DIS correction operation may be referred to as electronic image stabilization (EIS) correction operation. The number of image frames is not limited to an example illustrated in <FIG>. For example, the camera module <NUM> may obtain one or more image frames.

According to an embodiment, the electronic device <NUM> may identify a blurring degree of an image by comparing the plurality of image frames <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>,. , and <NUM>-M with each other through a difference image analysis (DIA) scheme. For example, the electronic device <NUM> may identify a feature point in the image frames <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>,. , and <NUM>-M and may identify the blurring degree of the image, based on a position, a distance, or a direction in which the feature point moves in the image frames. For example, the feature point may be used interchangeably with an edge, a feature, a key point, an interesting point, or a corner.

According to embodiments, information indicating the movement of the feature point in the image frame may be referred to as fourth movement information or image movement. According to an embodiment, the fourth movement information may refer to information corresponding to the change between the plurality of image frames. For example, the electronic device <NUM> may determine the fourth movement information corresponding to the movement of one feature point by comparing two image frames (e.g., an N-th frame and an (N+<NUM>)-th frame) with each other.

According to an embodiment, the electronic device <NUM> may perform DIS correction operation using the fourth movement information and the first movement information <NUM>. The electronic device <NUM> may obtain the first movement information <NUM> corresponding to the movement of the electronic device <NUM> through the motion sensor (e.g., the sensor circuit <NUM> of <FIG>). For example, the first movement information <NUM> may indicate movement information of the electronic device <NUM> that is shaken in at least one of a roll direction, a yaw direction, or a pitch direction.

According to various embodiments, the electronic device <NUM> may perform the DIS correction operation based on the first movement information (e.g., gyro data), the second movement information (e.g., the OIS movement data) and/or the third movement information (e.g., the object tracking movement data), together with the fourth movement information (e.g., the image movement). For example, the electronic device <NUM> may perform the DIS correction operation based on fifth movement information, which is obtained by removing the second movement information (e.g., the movement information of the lens assembly (the reflective member <NUM>) of the second camera <NUM>) from the first movement information, or sixth movement information obtained by additionally removing the third movement information (e.g., the movement information of the lens assembly (the reflective member <NUM>) of the second camera <NUM> through the object tracking) from the fifth movement information (e.g., the fifth movement information = first movement information -second movement information).

<FIG> illustrates a block diagram of an image stabilization system of an electronic device according to an embodiment.

Referring to <FIG>, according to an embodiment, the electronic device <NUM> may include the camera module <NUM>, the motion sensor <NUM> (e.g., the sensor circuit <NUM> of <FIG>) and a processor <NUM> (e.g., the processor <NUM> of <FIG>).

According to an embodiment, the camera module <NUM> may convert light, which is reflected from an object, into an electrical signal to obtain an image (e.g., a plurality of image frames or image data) of the object, and may transmit the image to the processor <NUM>. The camera module <NUM> performs the OIS correction operation to correspond to the movement of the electronic device <NUM>, when the electronic device <NUM> is shaken while the image is being obtained. The camera module <NUM> includes a lens assembly (e.g., the lens assembly <NUM> of <FIG>), an image sensor (e.g., the image sensor <NUM> in <FIG>), and an optical image stabilizer (e.g., the image stabilizer <NUM> of <FIG>).

According to an embodiment, the camera module <NUM> is configured in the form of a multi-camera system including the first camera <NUM> and the second camera <NUM>. For example, the camera module <NUM> includes the first camera <NUM> having the first FOV and the second camera <NUM> having the second FOV narrower than the first FOV. The camera module <NUM> may be configured such that the second camera <NUM> moves within the first FOV of the first camera <NUM>. For example, the central axis of the second FOV of the second camera <NUM> may be rotated within a specified angular range in the first FOV of the first camera <NUM>. An operation, in which the second camera <NUM> moves within the first FOV, will be described below with reference to <FIG>.

According to an embodiment, the camera module <NUM> may transmit a synchronization signal to the processor <NUM>. For example, the camera module <NUM> may be configured to transmit the synchronization signal to an OIS control module <NUM> to identify the first movement information and/or the second movement information synchronized with an image which is input to the processor <NUM>. For example, the camera module <NUM> may be configured to transmit the synchronization signal to the OIS control module <NUM> at a time or a time point at which the image sensors of the first camera <NUM> and/or the second camera <NUM> obtain the image. The synchronization signal may include information on the current time. For example, the camera module <NUM> may be connected to the OIS control module <NUM> by vertical synchronization (VSync). The synchronization signal may be a vertical synchronization signal, but is not limited thereto. According to various embodiments, the camera module <NUM> may obtain an image and/or output a synchronization signal through an image sensor included in the camera module <NUM>.

According to an embodiment, the synchronization signal may include at least one of a first synchronization signal output at a time point at which the first camera <NUM> obtains the image and a second synchronization signal output at a time point at which the second camera <NUM> obtains the image. For example, only the first camera <NUM> may transmit the first synchronization signal to the OIS control module <NUM>, and an image obtained by the second camera <NUM> may be connected in synchronization with movement information (e.g., the first movement information) of the electronic device to be matched to the first synchronization signal of the first camera <NUM>.

According to an embodiment, the motion sensor <NUM> may measure the first movement information corresponding to the movement of the electronic device <NUM> during at least a portion of a period in which the camera module <NUM> obtains the image. The period may be a time window including a time point of the event of image capturing. The motion sensor <NUM> may include, for example, a gyro sensor or an acceleration sensor. The motion sensor <NUM> may be embedded in the camera module <NUM> or may be disposed outside of the camera module <NUM>. According to an embodiment, the motion sensor <NUM> may transmit the first movement information to the processor <NUM>. For example, the motion sensor <NUM> transmits the first movement information to the OIS control module <NUM>. According to various embodiments, the motion sensor <NUM> may consecutively transmit the first movement information to the OIS control module <NUM> while the first camera <NUM> or the second camera <NUM> is obtaining an image.

According to an embodiment, the processor <NUM> may receive an image, which is OIS-corrected, from the camera module <NUM> and may receive the first movement information from the motion sensor <NUM>. The processor <NUM> may perform a DIS correction operation and a distortion correction operation with respect to an image obtained through the second camera <NUM>, based on the first movement information and the movement information of the lens assembly (e.g., the reflective member <NUM>) of the second camera <NUM>. The processor <NUM> may include the OIS control module <NUM>, a program module (kernel <NUM>), a hardware abstraction layer (HAL) <NUM>, and an image stabilization module <NUM>.

According to an embodiment, the OIS control module <NUM> may control the OIS of the camera module <NUM>. For example, the OIS control module <NUM> may control the OIS of the first camera <NUM> and/or the second camera <NUM> in the direction for correcting the shaking of the electronic device <NUM> to correspond to the first movement information. The OIS control module <NUM> may generate OIS movement information of the first camera <NUM> and/or the second camera <NUM> for OIS correction, based on the first movement information. According to embodiments, the processor <NUM> may correct shaking with respect to an image, which is obtained by the second camera <NUM>, and the OIS movement information of the second camera <NUM> considered to correct the shaking may be referred to as the second movement information (or OIS movement data).

According to an embodiment, the OIS control module <NUM> may receive the first movement information from the motion sensor <NUM> and may receive a synchronization signal from the camera module <NUM>. The OIS control module <NUM> may identify the first movement information corresponding to an image obtained from the camera module <NUM> using information on the time at which the synchronization signal is transmitted. For example, the OIS control module <NUM> may detect the first movement information received at a specific time, at which a synchronization signal is transmitted from the camera module <NUM>, of a plurality of first movement information received from the motion sensor <NUM>. The detected first movement information may be information corresponding to movement such as shaking of the electronic device <NUM> while the camera module <NUM> obtains the image at the specific time. The OIS control module <NUM> may identify the first movement information corresponding to the obtained image, and may transmit the first movement information and the OIS movement information, which corresponds to the first movement information, to the HAL <NUM>.

According to an embodiment, the OIS control module <NUM> may control the movement of the second FOV of the second camera <NUM>. For example, the OIS control module <NUM> may move the lens assembly (e.g., the reflective member <NUM>) of the second camera <NUM> for OIS correction and object tracking. As described above, the second camera <NUM> may move the second FOV of the second camera <NUM> by rotating the reflective member <NUM>.

According to an embodiment, the OIS control module <NUM> may generate the information for the movement (e.g., rotation) of the reflective member <NUM> of the second camera <NUM>, and may rotate the reflective member <NUM> of the second camera <NUM> based on the movement information. The OIS control module <NUM> may generate second movement information, which corresponds to the movement of the reflective member <NUM> and is used to perform the OIS correction operation, and third movement information, which corresponds to the movement of the reflective member <NUM> and is used for the object tracking operation. For example, the OIS control module <NUM> may generate second movement information based on the first movement information to perform the OIS correction. For example, the OIS control module <NUM> may generate the third movement information to perform the tracking of the object, based on the position and/or amount of movement of the region of interest (ROI). The ROI may be an image area containing a object to be tracked.

According to an embodiment, the kernel <NUM> may receive an image obtained through the camera module <NUM>. The kernel <NUM> may generate image time information on a time when the image is obtained from the camera module <NUM>.

The hardware abstract layer (HAL) <NUM> may receive the first movement information and the OIS movement information (e.g., second movement information) from the OIS control module <NUM>. The HAL <NUM> may receive first movement time information on the time at which the first movement information is received, from the OIS control module <NUM>. The HAL <NUM> may match an image obtained at a specific time to the first movement information and the OIS movement information (e.g., the second movement information) corresponding to the image, by synchronizing the image, image time information, the first movement information, the first movement time information, and the OIS movement information.

According to an embodiment, the image stabilization module <NUM> may correct an image, which is, for example, blurred or distorted, obtained through the second camera <NUM>. For example, the image stabilization module <NUM> may perform the DIS correction operation for the image, based on the first movement information, the second movement information, and/or the third movement information, and may correct the distortion caused by the lens assembly (e.g., the reflective member <NUM>) after performing the DIS correction operation. For example, the image stabilization module <NUM> may include the distortion correcting module <NUM>. The distortion correcting module <NUM> may correct the distortion based on a correction position and a distortion variation calculated by the image stabilization module <NUM>.

Output information (e.g., the correction position, or the pixel position (coordinates)) necessary for the DIS correction is calculated based on data obtained by removing the OIS correction and/or a movement amount (or a rotation amount of the reflective member <NUM>) of the second camera <NUM>, which results from the object tracking, from the movement information of the electronic device, and the shaking and the distortion may be corrected through one warping operation, by performing a correction using the output information, which is calculated through the DIS correction, as the input of the distortion information.

<FIG> illustrates capturing operations of a first camera and a second camera according to an embodiment.

Referring to <FIG>, the electronic device <NUM> according to an embodiment may be configured to move the second camera (e.g., the second camera <NUM> of <FIG>) within the FOV of the first camera (e.g., the first camera <NUM> of <FIG>). For example, the electronic device <NUM> may perform an OIS correction and/or object tracking operation by moving the second camera <NUM> within the FOV of the first camera <NUM>.

According to an embodiment, the first camera <NUM> and the second camera <NUM> may obtain an image of an external object through the rear surface (e.g., the rear surface <NUM> of <FIG>) of the housing (e.g., the housing <NUM> of <FIG>) in the electronic device <NUM>. For example, the first camera <NUM> and the second camera <NUM> may obtain an image of an object located in the rear direction of the electronic device <NUM>. The camera decorating member <NUM> may be positioned on the rear surface <NUM> of the housing <NUM>, and the camera decorating member <NUM> may be configured to support viewing angles of the first camera <NUM> and the second camera <NUM> through a plurality of transparent areas. For example, the first camera <NUM> and the second camera <NUM> may receive light incident from the outside of the housing <NUM> through the transparent area of the camera decorating member <NUM>.

A first FOV <NUM> of the first camera <NUM> is wider than a second FOV <NUM> of the second camera <NUM> by a specific size. For example, the first camera <NUM> may be a wide-angle camera, and the second camera <NUM> may be a telephoto camera. The electronic device <NUM> may move the lens (e.g., the reflective member <NUM> of <FIG>) of the second camera <NUM> such that the second FOV <NUM> of the second camera <NUM> may move within the first FOV <NUM> of the first camera <NUM>. For example, the electronic device <NUM> may move a central axis <NUM> of the second FOV by rotating (e.g., yawing or pitching of <FIG>) the reflective member <NUM> of the second camera <NUM>.

According to an embodiment, the electronic device <NUM> may move the second central axis <NUM> of the second FOV <NUM> to a first position <NUM> by rotating the reflective member <NUM>. In addition, the electronic device <NUM> may move the second central axis <NUM> of the second FOV <NUM> to a second position <NUM> by rotating the reflective member <NUM>. For example, when a counterclockwise direction about a first central axis <NUM> of the first FOV <NUM> is defined as a positive (+) angle, and a clockwise direction about the first central axis <NUM> of the first FOV <NUM> is defined as a negative (-) angle, the second central axis <NUM> may be positioned at the first position <NUM> to form an included angle having a positive size, with respect to the first central axis <NUM>, and positioned at the second position <NUM> to form an included angle having a negative size, with respect to the first central axis <NUM>. According to various embodiments, when the range of the first FOV <NUM> is <NUM>°, the second camera <NUM> may be configured such that the central axis <NUM> of the second FOV <NUM> moves in the range from +<NUM>° to -<NUM>°.

According to various embodiments, the first FOV <NUM> of the first camera <NUM> may have the first central axis <NUM> which is fixed, and the second FOV <NUM> of the second camera <NUM> may have the second central axis <NUM> which is movable within the range of the first FOV <NUM> through a rotation operation of the reflective member <NUM> of the second camera <NUM>. However, embodiments are not limited thereto. The first camera <NUM> may also be configured such that the first central axis <NUM> of the first FOV <NUM> moves.

According to an embodiment, the electronic device <NUM> may capture a moving picture while tracking an object. For example, the electronic device <NUM> may provide an image having an object positioned at the center of the image by moving the lens assembly (e.g., the reflective member <NUM>) of the second camera <NUM>. The electronic device <NUM> may provide the image having the object positioned at the center of the image by moving the reflective member <NUM> of the second camera <NUM>, enlarging the image, and/or cropping the image.

According to various embodiments, the electronic device <NUM> may identify movement of an object from images obtained through the first camera <NUM>, and may move the lens of the second camera <NUM> based on the identified movement. The direction in which the second FOV <NUM> of the second camera <NUM> faces may be changed (e.g., the central axis <NUM> may be moved to the first position <NUM> or the second position <NUM>), as the lens moves. However, a method for tracking the object is not limited to the above example.

According to embodiments, as the second camera <NUM> may move largely within the range of the first FOV <NUM> of the first camera <NUM>, an image obtained through the second camera <NUM> may be distorted (e.g., perspective distortion). According to an embodiment, the electronic device <NUM> may more accurately correct the distortion of the image by applying rectification based on a correction position calculated by considering the movement information (e.g., gyro data) of the electronic device <NUM> and the movement information (e.g., the OIS movement data and/or movement data for object tracking) of the lens (or the reflective member <NUM>) of the second camera <NUM>.

<FIG> illustrates a distortion correction operation of an electronic device according to an embodiment.

Referring to <FIG> and <FIG>, according to an embodiment, the second camera <NUM> may distort (e.g., perspective distortion) an image as the lens assembly (in particular, the reflective member <NUM> moves) moves. For example, when the reflective member <NUM> of the second camera <NUM> moves, the image sensor <NUM> generates the image by using the light that has passed through the lens unit <NUM> after the path is changed by the moved reflective member <NUM>, and thus at least a portion of the image may be distorted.

According to an embodiment, the second camera <NUM> may be configured such that the reflective member <NUM> rotates about the first rotation axis <NUM> (e.g., yawing) or rotates about the second rotation axis <NUM> (e.g., pitching), in the state that the image sensor <NUM> is not moved. As the reflective member <NUM> rotates, when the central axis of the second FOV of the second camera <NUM> moves, the central axis of the second FOV is out of the center of the image sensor <NUM>. Accordingly, the image may be distorted to correspond to the rotation of the reflective member <NUM> of the second camera <NUM>. For example, when the reflective member <NUM> of the second camera <NUM> is in a basic state, the image may be a rectangular shape. However, when the reflective member <NUM> of the second camera <NUM> moves, the image may be distorted to be in a shape different than the rectangular shape, in particular in a trapezoidal shape.

According to an embodiment, referring to <FIG>, when the center of the second FOV <NUM> of the second camera <NUM> is matched to the center of the first FOV <NUM> of the first camera <NUM>, an image <NUM> obtained by the second camera <NUM> may not be distorted and may have, for example, a rectangular shape. When the center of the second FOV <NUM> of the second camera <NUM> is offset from the center of the first FOV <NUM> of the first camera <NUM>, an image <NUM> obtained by the second camera <NUM> may be distorted to be in, for example, a trapezoidal shape. For example, the image may be distorted in a different shape depending on a direction in which the second FOV <NUM> of the second camera <NUM> moves, and the distortion degree may increase, as the second FOV <NUM> of the second camera <NUM> is away from the center of the first FOV <NUM>.

According to an embodiment, as the reflective member <NUM> of the second camera <NUM> relatively slightly moves, a object in the image may be slightly distorted or inclined. Accordingly, the correction for the image may not be necessary. For example, the reflective member <NUM> of the second camera <NUM> may move to a specific reference value or less for the OIS correction operation. In this case, the correction for the distortion of the image is not necessary.

According to an embodiment, as the movement of the reflective member <NUM> of the second camera <NUM> exceeds the specific range, a object in the image may be largely distorted or inclined. For example, the reflective member <NUM> of the second camera <NUM> may move to a specific reference value or more for the object tracking operation. In this case, the correction for the distortion of the image is necessary.

According to an embodiment, the electronic device <NUM> may convert the first image <NUM> distorted in a trapezoidal shape into a second image <NUM> having a rectangular shape by performing a distortion correction operation. For example, the electronic device <NUM> may correct the distortion of the image based on first correction information HRecti to correct the distortion. The first correction information HRecti may be a transform matrix. The electronic device <NUM> may convert the distorted first image <NUM> to the second image <NUM> through the transform matrix. For example, the electronic device <NUM> may determine a distortion variation of pixels by comparing pixels of the first image <NUM> distorted with pixels of the second image <NUM>, and may compensate for the distortion of the image by using the determined distortion variation and the transform matrix.

According to embodiments, the electronic device <NUM> may correct the distorted image through the distortion correcting module <NUM> included in the image stabilization module <NUM>. The distortion correcting module <NUM> performs the correction by using coordinates of an output pixel, which is calculated through the DIS correction operation, as an input of the distortion correction.

<FIG> illustrates an image stabilization operation and a distortion correction operation of an electronic device according to an embodiment.

<FIG> may be a view illustrating an operation of performing a blurring correction operation and a distortion correction operation of an image, which is obtained through the second camera <NUM>, based on the first movement information and the second movement information, when the reflective member (e.g., the reflective member <NUM> of <FIG>) of the second camera <NUM> for the object tracking does not rotate.

Referring to <FIG>, the electronic device <NUM> may obtain a first frame <NUM> and a second frame <NUM> using the second camera <NUM>. The frame may mean an image frame. The second frame <NUM> (e.g., the N-th frame) may be a frame subsequent to the first frame <NUM> (e.g., the (N-<NUM>)-th frame) on a time domain. The first frame <NUM> and the second frame <NUM> may be consecutive image frames of a plurality of image frames obtained through the second camera <NUM>.

According to an embodiment, while the first frame <NUM> and the second frame <NUM> including an object <NUM> are being obtained through the second camera <NUM>, the electronic device <NUM> (or the camera module <NUM>) may be shaken due to external factors such as the user's hand shaking. The electronic device <NUM> calculates output information (e.g., a correction position) necessary for a DIS correction operation, based on the first movement information and the second movement information, and performs a distortion correction operation, based on the calculated correction position.

According to an embodiment, the processor <NUM> may perform a correction operation including the first correction operation and the second correction operation for the image frames <NUM> and <NUM> obtained using the second camera <NUM> through the image stabilization module. For example, the first correction operation may be referred to as the distortion correction operation, and the second correction operation may be referred to as the DIS correction operation.

According to an embodiment, the processor <NUM> may receive the first frame <NUM> and the second frame <NUM>, which are OIS-corrected to correspond to the first movement information, from the second camera <NUM>. As described above, the processor <NUM> may detect the first movement information and the second movement information synchronized with the first frame <NUM> and the second frame <NUM>, based on the synchronization signal received from the second camera <NUM> at the time point at which the second camera <NUM> obtains the first frame <NUM> and the second frame <NUM>. The processor <NUM> may perform the DIS correction operation based on the first movement information and the second movement information. In this case, the processor <NUM> may perform the DIS correction operation based on the fifth movement information obtained by removing the second movement information from the first movement information. For example, the processor <NUM> may generate second correction information HVDIS for the DIS correction operation, based on the fifth movement information, and may perform the DIS correction operation by using the second correction information HVDIS. For example, the second correction information HVDIS may be a matrix.

According to an embodiment, the processor <NUM> may calculate distortion variation amounts of the first frame <NUM> and the second frame <NUM> to generate the first correction information HRecti for the distortion correction operation, and may perform the distortion correction operation by using the output information (e.g., the position of the output pixel) calculated based on the second correction information HVDIS. as an input for the distortion correction operation. The processor <NUM> may output images obtained by performing the DIS correction operation (e.g., second correction) and the distortion correction operation (e.g., first correction) based on Equation <NUM>.

According to an embodiment, the image may include a plurality of pixels. The electronic device <NUM> may convert information on pixels into two-dimensional coordinates to correct an image. For example, the electronic device <NUM> may convert the position of an arbitrary pixel into coordinates x and y. In Equation <NUM>, coordinates x and y may refer to positions of pixels included in the first frame <NUM> and/or the second frame <NUM>, and coordinates x and y may be changed to coordinates X and Y through image correction based on the first correction information HRecti and the second correction information HVDIS.

According to an embodiment, a movement trajectory (e.g., a first movement trajectory <NUM>) of the second camera <NUM> may be caused due to a movement such as the shaking of the electronic device <NUM>. The electronic device <NUM> may rotate the reflective member <NUM> of the second camera <NUM> for an OIS correction operation. Accordingly, the image obtained through the second camera <NUM> may be distorted. For example, the first frame <NUM> and the second frame <NUM> may be distorted in a trapezoidal shape, and internal grid patterns of the first frame <NUM> and the second frame <NUM> may be tilted or distorted. In this case, the objects <NUM> included in the frames <NUM> and <NUM> may be distorted in a shape different from a real shape. However, the rotation amount of the reflective member <NUM> through the OIS correction operation is not large. Accordingly, the image distortion based on the rotation amount may be relatively low.

According to an embodiment illustrated in <FIG>, the object <NUM> may be fixed without relatively moving with respect to the camera module <NUM> (or the electronic device), and the camera module <NUM> may be shaken by the user's hand shaking. The image frames <NUM> and <NUM> obtained by the second camera <NUM> may be distorted by rotation of the reflective member <NUM> for correcting the user's hand shaking. For example, the reflective member <NUM> is not rotated other than the purpose of the correction of the user's hand shaking. Accordingly, the first frame <NUM> and the second frame <NUM> are distorted in substantially the same form. In addition, the positions of the object <NUM> on the first frame <NUM> and the second frame <NUM> may be different from each other due to the shaking of the camera module <NUM>.

According to an embodiment, the DIS correction operation may be a method of securing a margin area by cropping a specified first output area <NUM> and a specified second output area <NUM> from the first frame <NUM> and the second frame <NUM>. The margin area may refer to an area obtained by excluding the output areas <NUM> and <NUM> from the first frame <NUM> and the second frame <NUM>. For example, the output areas <NUM> and <NUM> may be determined based on at least one of a specified number of pixels, a specified field of view, or a specified viewing angle. The DIS correction operation may be performed by adjusting positions of the output areas <NUM> and <NUM> in the first frame <NUM> and the second frame <NUM>, based on the first movement information and the second movement information. For example, the processor <NUM> (e.g., the image stabilization module <NUM>) may adjust the positions of the output areas <NUM> and <NUM> within the margin area, based on the first movement information and the second movement information.

As illustrated in <FIG>, the DIS correction may be understood as an operation of correcting a first movement trajectory <NUM> of the camera to a first correction trajectory <NUM>. The first correction trajectory <NUM> may refer to a camera trajectory in which shaking is corrected. For example, the first correction trajectory <NUM> is a trajectory (or position) of the camera after determining the stabilization path, and the determining the stabilization path may be referred to as camera path planning. According to an embodiment, the DIS correction operation may be performed, based on a camera trajectory (e.g., the first movement trajectory <NUM>) before determining the stabilization path and a camera trajectory (e.g., the first correction trajectory <NUM>) after determining the stabilization path. For example, a difference between the first movement trajectory <NUM> and the first correction trajectory <NUM> may refer to a DIS correction amount. As the DIS correction operation is performed, the object <NUM> included in the first frame <NUM> and the second frame <NUM> may be placed at substantially the same position in the first output area <NUM> and the second output area <NUM>.

According to an embodiment, the distortion correction operation may be a method of securing a margin area by cropping a specified third output area <NUM> and a specified fourth output area <NUM> from the first output area <NUM> and the second output area <NUM>. For example, the third output area <NUM> may be set to have a smaller size than the first output area <NUM>, and the fourth output area <NUM> may be set to have a smaller size than the second output area <NUM>. As the distortion correction is performed, the grid patterns inside the first output area <NUM> and the second output area <NUM> may be corrected to be straight, and the shape of the output image may be corrected to a rectangular shape.

<FIG> may be a view illustrating an image stabilization operation and a distortion correction operation for an image, based on the first movement information, the second movement information, and the third movement information, when the reflective member (e.g., the reflective member <NUM> of <FIG>) of the second camera <NUM> for the object tracking moves.

For example, <FIG> is a view illustrating a correction operation based on the movement of the reflective member <NUM> for the object tracking. In the following description, the duplicated description of the description made with reference to <FIG> is omitted.

Referring to <FIG>, the electronic device <NUM> may obtain a first frame <NUM> and a second frame <NUM> by tracking an object using the second camera <NUM>. For example, the electronic device <NUM> may capture a moving picture while tracking an object by moving the reflective member <NUM> of the second camera <NUM>. As described above, as the movement amount (e.g., rotation amount) of the reflective member <NUM> is increased, the image may be significantly distorted, and the movement trajectory of the camera may be largely changed. For example, when the movement of the reflective member <NUM> for the object tracking is absent, the movement trajectory of the camera is not largely changed as illustrated in <FIG>. However, when the movement of the reflective member <NUM> for the object tracking is present, the movement trajectory of the camera may be significantly largely changed as illustrated in <FIG>.

According to an embodiment illustrated in <FIG>, the camera module <NUM> may move the FOV of the second camera <NUM> to track the object, and may be shaken due to the user's hand shaking. The image frames <NUM> and <NUM> obtained by the second camera <NUM> may be distorted by rotating the reflective member <NUM> for correcting the user's hand shaking and for object tracking. For example, the reflective member <NUM> is rotated due to the object tracking. Accordingly, the first frame <NUM> and the second frame <NUM> are distorted in mutually different shapes. In addition, the positions of an object <NUM> on the first frame <NUM> and the second frame <NUM> may be different from each other, as the camera module <NUM> is shaken.

According to an embodiment, the electronic device <NUM> may calculate a correction position necessary for the DIS correction operation, based on the first movement information, the second movement information, and the third movement information, and may perform the distortion correction operation, based on the calculated correction position.

According to an embodiment, the processor <NUM> may receive the first frame <NUM> and the second frame <NUM>, which are OIS-corrected to correspond to the first movement information, from the second camera <NUM>. The processor <NUM> may perform the DIS correction operation based on the first movement information, the second movement information and the third movement information. In this case, the processor <NUM> may perform the DIS correction operation, based on the sixth movement information obtained by removing the second movement information and the third movement information from the first movement information. For example, the processor <NUM> may generate second correction information HVDIS for the DIS correction operation, based on the sixth movement information, and may perform the DIS correction operation based on the second correction information.

According to an embodiment, the processor <NUM> may calculate distortion variation amounts of the first frame <NUM> and the second frame <NUM> to generate the first correction information HRecti for the distortion correction, and may perform the distortion correction by using the correction position information calculated based on the second correction information HVDIS as an input of the distortion correction operation. The processor <NUM> may output images obtained by performing the DIS correction operation (e.g., the second correction operation) and the distortion correction operation (e.g., the first correction operation) based on Equation <NUM> described with reference to <FIG>. For example, as illustrated in <FIG>, when the movement of the reflective member <NUM> for the tracking of the object <NUM> is made, the second correction information HVDIS may be changed, when comparing with the case of <FIG>.

As described above, the DIS correction operation may be understood as an operation of correcting a second movement trajectory <NUM> of the camera to a second correction trajectory <NUM>. The second correction trajectory <NUM> may refer to a camera trajectory in which shaking is corrected. For example, the second correction trajectory <NUM> is a trajectory (or position) of the camera after determining the stabilization path. According to an embodiment, the DIS correction operation may be performed, based on a camera trajectory (e.g., the second movement trajectory <NUM>) before determining the stabilization path and a camera trajectory (e.g., the second correction trajectory <NUM>) after determining a stabilization path. For example, a difference between the second movement trajectory <NUM> and the second correction trajectory <NUM> may refer to a DIS correction amount. When comparing with the embodiment of <FIG>, according to the embodiment of <FIG>, as the rotation of the reflective member <NUM> for the object tracking operation is additionally made, the change in the second movement trajectory <NUM> is greater than the change in the first movement trajectory <NUM>. Accordingly, the DIS correction amount is relatively increased. As the DIS correction is performed, the object <NUM> included in the first frame <NUM> and the second frame <NUM> may be placed at substantially the same position in a first output area <NUM> and a second output area <NUM>.

According to an embodiment, the distortion correction may be a method of securing a margin area by cropping a specified third output area <NUM> and a specified fourth output area <NUM> from the first output area <NUM> and the second output area <NUM>. For example, the third output area <NUM> may be set to be in a smaller size than that of the first output area <NUM>, and the fourth output area <NUM> may be set to be in a smaller size than that of the second output area <NUM>. As the distortion correction is performed, the lattice pattern inside the first output area <NUM> and the second output area <NUM> may be corrected to be straight, and the shape of the output image may be corrected to be in a rectangular shape.

It is to be understood that if an element (e.g., a first element) is referred to, with or without the term "operatively" or "communicatively", as "coupled with," "coupled to," "connected with," or "connected to" another element (e.g., a second element), it means that the element may be coupled with the other element directly (e.g., wired), wirelessly, or via a third element.

Claim 1:
An electronic device comprising:
a camera module (<NUM>) comprising a first camera (<NUM>) having a first field of view, FOV, (<NUM>) and a second camera (<NUM>) having a second FOV (<NUM>) narrower than the first FOV (<NUM>), the second FOV (<NUM>) is positioned in the first FOV (<NUM>);
a motion sensor (<NUM>); and
a processor (<NUM>) electrically connected to the camera module (<NUM>) and the motion sensor (<NUM>),
wherein the second camera (<NUM>) comprises:
at least one lens (<NUM>);
a reflective member (<NUM>) configured to rotate relative to the at least one lens (<NUM>); and
an image sensor (<NUM>) configured to generate an electrical signal based on light incident through the reflective member (<NUM>) and the at least one lens (<NUM>), and
wherein the processor (<NUM>) is configured to:
obtain an image using the camera module (<NUM>);
obtain, from the motion sensor (<NUM>), first movement information of the electronic device which is shaken, to correspond to the obtaining of the image;
obtain second movement information of the reflective member (<NUM>) of the second camera (<NUM>) corresponding to an optical image stabilizer, OIS, correction operation or third movement information of the reflective member (<NUM>) of the second camera (<NUM>) corresponding to an object tracking operation during at least a portion of a period in which the image is obtained;
characterized in that the processor is further configured to:
perform an image stabilization operation with respect to the image by removing second movement information or third movement information from the first movement information;
obtain output information by performing the image stabilization operation; and
output an image corrected by correcting distortion of the image, based on the output information and a distortion variation resulting from the second movement or the third movement information.