Patent Description:
In recent years, high dynamic range (HDR) technology, which expands a range of brightness closer to what a person actually sees by making bright places brighter and dark places darker in digital images is being spotlighted.

However, in the case of the existing HDR technology, several images with different exposures had to be captured and then synthesized the images into one image in order to acquire one image. Thus, it is pointed out that the time required to acquire a desired image may be delayed, and motion artifacts may occur as a result of synthesis if a subject moves or a camera shakes during a process of capturing an image as a limitation.

Accordingly, there is a need for a technology capable of securing visibility of a subject in various lighting environments such as night, bad weather, backlight, etc. while overcoming the limitations of the prior art.

In addition, when a camera provided in a car, especially an autonomous vehicle, captures an image, it is a challenge to secure driving stability by smoothly recognizing objects that appear in front of the car even in various lighting environments encountered during the car driving.

<CIT> discloses an image capturing and processing unit obtaining high-dynamic range (HDR) data by synthesizing two images captures with different exposures. An image generating unit compresses the HDR data to generate two low-dynamic range (LDR) images, one high luminance compressed image LDR IMAGE <NUM> and one low luminance compressed image LDR IMAGE <NUM>. Regions of LDR IMAGE <NUM> and LDR IMAGE <NUM> are weighted, aiming at representing detail in each divided region as distinctly as possible with high contrast, before combining the weighted LDR images.

<CIT> discloses systems and methods for enabling a client device to request video streams with different bit depth remappings for different viewing conditions.

<CIT> relates to a device for detecting and storing digital pictures comprising an image detector for the generation of digital picture data and a picture processing device with a picture data memory.

<CIT> relates to feature extraction technique based on edge extraction.

The disclosure is devised to overcome the problems described above and provide an electronic apparatus and a method for controlling the same, which can secure high visibility of a subject by acquiring, from an original image, a plurality of sub images having a smaller bit number than the original image, and acquiring a synthesized image based on information about the shapes of objects included in the plurality of sub images.

The solution is set out in the appended set of claims.

The disclosure may have several embodiments, and the embodiments may be modified variously. In the following description, specific embodiments are provided with accompanying drawings and detailed descriptions thereof. However, it should be understood that the present disclosure is not limited to the specific embodiments described hereinafter, but various modifications, equivalents, and/or alternatives of the embodiments will be apparent to the skilled person. In relation to explanation of the drawings, similar drawing reference numerals may be used for similar constituent elements.

In describing exemplary embodiments, detailed description of relevant known functions or components may be omitted if it would obscure the description of the subject matter.

The terms used herein are solely intended to explain a specific exemplary embodiment, and not to limit the scope of the present disclosure.

The terms "have", "may have", "include", and "may include" used in the exemplary embodiments of the present disclosure indicate the presence of corresponding features (for example, elements such as numerical values, functions, operations, or parts), and do not preclude the presence of additional features.

In the description, the term "A or B", "at least one of A or/and B", or "one or more of A or/and B" may include all possible combinations of the items that are enumerated together. For example, the term "A or B" or "at least one of A or/and B" may designate (<NUM>) at least one A, (<NUM>) at least one B, or (<NUM>) both at least one A and at least one B.

The expression "<NUM>", "<NUM>", "first", or "second" as used herein may modify a variety of elements, irrespective of order and/or importance thereof, and only to distinguish one element from another. Accordingly, without limiting the corresponding elements.

When an element (e.g., a first element) is "operatively or communicatively coupled with/to" or "connected to" another element (e.g., a second element), an element may be directly coupled with another element or may be coupled through the other element (e.g., a third element).

Meanwhile, when an element (e.g., a first element) is "directly coupled with/to" or "directly connected to" another element (e.g., a second element), no other element (e.g., a third element) exists between an element and another element.

In the description, the term "configured to" may be changed to, for example, "suitable for", "having the capacity to", "designed to", "adapted to", "made to", or "capable of" under certain circumstances. The term "configured to (set to)" does not necessarily mean "specifically designed to" in a hardware level.

Under certain circumstances, the term "device configured to" may refer to "device capable of" doing something together with another device or components. For example, "a processor configured (or set) to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing the operation, or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing corresponding operations by executing one or more software programs stored in a memory device.

In the embodiments disclosed herein, a term 'module' or 'unit' refers to an element that performs at least one function or operation. The 'module' or 'unit' may be realized as hardware, software, or combinations thereof. In addition, a plurality of 'modules' or 'units' may be integrated into at least one module and may be realized as at least one processor in an integrated manner except for 'modules' or 'units' that should be realized in specific hardware.

Further, various elements and areas in the drawings are schematically drawn. Therefore, the technical ideas are not limited by a relative size or interval drawn in the accompanying drawings.

The example embodiments of the disclosure will be described in greater detail below in a manner that will be understood by one of ordinary skill in the art.

<FIG> is a view illustrating a process of acquiring from an original image to a synthesized image according to an embodiment.

As illustrated in <FIG>, the electronic apparatus according to an embodiment of the disclosure acquires an original image <NUM>. Specifically, the electronic apparatus acquires an original image <NUM> having a first bit number for each pixel through an image sensor.

For example, the original image <NUM> may be an original image <NUM> having a <NUM>-bit number per pixel. As such, in describing various embodiments of the disclosure, an image generated through an image sensor may be referred to the original image <NUM>, and hereinafter, the original image <NUM> having the <NUM>-bit number per pixel may be referred to as a <NUM>-bit original image.

Meanwhile, the electronic apparatus according to an embodiment of the disclosure acquires a plurality of sub images <NUM> based on the original image <NUM>. Specifically, the electronic apparatus acquires a plurality of sub images <NUM> having a second bit number smaller than the first bit number for each pixel based on the original image <NUM>.

For example, the plurality of sub images <NUM> may be the first sub image and the second sub image <NUM> having an <NUM>-bit number per pixel. As described above, in describing various embodiments of the disclosure, an image acquired from the original image <NUM> having a number of bits smaller than the number of bits of the original image <NUM> may be referred to as a sub image <NUM>, and hereinafter, a sub image <NUM> having an <NUM>-bit number for each pixel may be referred to as an <NUM>-bit sub image <NUM>, for convenience.

Various objects may be included in the plurality of sub images, and among the plurality of sub images, a specific sub image may include more information on the shapes of objects compared to the other sub images. In addition, even in the same sub image, the presence or absence of information on the shapes of objects and the amount of information on the shapes of the objects may be different for each of a plurality of areas included in the sub image.

Meanwhile, in each of the plurality of sub images <NUM>, the electronic apparatus according to an embodiment of the disclosure identifies information on the shapes of objects included in the plurality of sub images <NUM>.

In other words, the electronic apparatus may extract features on the shapes of objects included in the plurality of sub images <NUM> and identify information on the shapes of the objects based on the extracted features. Specifically, the electronic apparatus may identify the presence or absence of information on the shapes of objects and the amount of information on the shapes of the objects for each of a plurality of areas included in the plurality of sub images <NUM>.

Meanwhile, the electronic apparatus according to an embodiment of the disclosure acquires a synthesized weight for each area for at least two sub images <NUM> among the plurality of sub images <NUM> based on the identified information.

Specifically, as a result of identifying information on the shapes of objects included in the plurality of sub images <NUM>, the electronic apparatus assigns a higher synthesized weight on an image including more information on the shapes of objects in a specific area among the plurality of sub images <NUM>.

Meanwhile, the electronic apparatus according to an embodiment of the disclosure acquires a synthesized image <NUM> based on the acquired synthesized weight for each area. As such, in describing various embodiments of the disclosure, an image synthesized based on a synthesized weight for each identified area is referred to as a synthesized image <NUM>.

According to an embodiment of the disclosure as described above, a plurality of sub images <NUM> may be acquired from the original image <NUM>, and the synthesized image <NUM> capable of securing visibility for all areas included in the image from the plurality of sub images <NUM> may be acquired.

Meanwhile, in describing the disclosure, the first bit number (e.g., <NUM>-bit number) may be determined according to the amount of information acquired by the image sensor, and the second bit number (e.g., <NUM>-bit number) may be determined in terms of optimization.

Specifically, in the image sensor, it is necessary to acquire information having a higher bit number compared to <NUM>-bit in order to sense information having a wide dynamic range.

Meanwhile, optimization to reduce the amount of computation is required for an image processing process for an original image acquired through the image sensor, and particularly an image processing process performed through a deep learning-based algorithm. Therefore, the image processing process is generally optimized based on <NUM>-bit, which is the smallest unit that can access a memory.

The disclosure considers the description above, and according to the disclosure, after acquiring the <NUM>-bit original image <NUM> including information of a wide dynamic range, the <NUM>-bit sub image <NUM> may be acquired within a range that can minimize loss of information from the original image, and the image processing process optimized based on the <NUM>-bit may be performed.

In addition, even if the <NUM>-bit synthesized image <NUM> is acquired from the <NUM>-bit original image through the process described above, as much information as it can satisfy the range of expressions that can be recognized by the human eye is preserved.

Hereinafter, various embodiments according to the disclosure will be described, focusing on the configuration of the electronic apparatus according to the disclosure, with reference to <FIG> and <FIG>.

<FIG> is a block diagram illustrating a structure of an electronic apparatus, according to an example embodiment.

As shown in <FIG>, the electronic apparatus <NUM> according to an embodiment of the disclosure includes an image sensor <NUM>, a processor <NUM>, and a memory <NUM>.

The image sensor <NUM> may convert light entering through a lens into an electrical image signal. In addition, the processor <NUM> may acquire an original image of a subject through the image sensor <NUM>.

Meanwhile, the image sensor <NUM> may be a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor, but the image sensor <NUM> according to the disclosure is not limited thereto.

The memory <NUM> may include at least one command related to the electronic apparatus <NUM>. In addition, an operating system (O/S) for driving the electronic apparatus <NUM> may be stored in the memory <NUM>. In addition, various software programs or applications for operating the electronic apparatus <NUM> may be stored in the memory <NUM> according to various embodiments of the disclosure.

In addition, the memory <NUM> may include various software modules for operating the electronic apparatus <NUM> according to various embodiments of the disclosure, and the processor <NUM> may execute various software modules stored in the memory <NUM> to operate an operation of the electronic apparatus <NUM> according to various embodiments of the disclosure.

In particular, according to various embodiments of the disclosure, the memory <NUM> may store software based on various algorithms related to edge detection, such as a Laplacian mask, a Sobel mask, a Roberts mask, a Prewitt mask, a Canny mask, or the like. Such edge detection will be described below.

According to various embodiments of the disclosure, the memory <NUM> may store various types of image data. Specifically, the memory <NUM> may store data related to an original image, a plurality of sub images, a synthesized image, or the like according to the disclosure. Meanwhile, the memory <NUM> may include a semiconductor memory such as a flash memory, or a magnetic storage medium such as a hard disk, or the like.

The processor <NUM> may control the overall operation of the electronic apparatus <NUM>. Specifically, the processor <NUM> may be connected to the image sensor <NUM> and the memory <NUM> of the electronic apparatus <NUM> to control the overall operation of the electronic apparatus <NUM>.

The processor <NUM> may be realized in various methods. For example, the processor <NUM> may be at least one of an application-specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, hardware finite state machine (FSM), and a digital signal processor (DSP).

The processor <NUM> may include read-only memory (ROM), random access memory (RAM), a graphic processing unit (GPU), a central processing unit (CPU), and a bus. The ROM, the RAM, the GPU and the CPU may be connected with each other through the bus.

In particular, according to various embodiments of the disclosure, the processor <NUM> acquires an original image having the first bit number for each pixel through the image sensor <NUM>. The first bit number may be determined from a viewpoint of acquiring an original image having a wide dynamic range. In other words, according to the disclosure, the processor <NUM> may acquire an original image having a higher bit number than a bit number per pixel suitable for an image processing process through a so-called high dynamic range image sensor <NUM>.

For example, the processor <NUM> may acquire an original image having <NUM>-bit, <NUM>-bit, <NUM>-bit, or <NUM>-bit number per pixel through the image sensor <NUM>.

When the original image is acquired, the processor <NUM> acquires a plurality of sub images having a second bit number smaller than the first bit number for each pixel based on the original image. The second-bit number may be determined in terms of optimization of an image processing process. In other words, according to the disclosure, the processor <NUM> acquires a plurality of sub images having a bit number per pixel in a smallest unit accessible to the memory <NUM> based on the acquired original image.

For example, when the original image acquired through the image sensor <NUM> has <NUM>-bit, <NUM>-bit, <NUM>-bit, or <NUM>-bit per pixel, the processor <NUM> may acquire the plurality of sub images having an <NUM>-bit number per pixel may be acquired.

In other words, according to the disclosure, the processor <NUM> may acquire an original image including information of the wide dynamic range, and then acquire the plurality of sub images within a range capable of minimizing loss of information from the original image, and perform an optimized image processing process based on the acquired plurality of sub images.

Meanwhile, various objects may be included in the plurality of sub images acquired from the original image, and among the plurality of sub images, a specific sub-image may include more information on the shapes of the objects than other sub images. In addition, even in the same sub image, the presence or absence of information on the shapes of objects and the amount of information on the shapes of the objects may be different for each of a plurality of areas included in the sub image.

When a plurality of sub images are acquired, the processor <NUM> identifies information on the shapes of objects included in the plurality of sub images from each of the plurality of sub images.

Meanwhile, in image processing, various methods exist for extracting various features included in a specific image and obtaining various information from the extracted features. Particularly, according to various embodiments of the disclosure, the processor <NUM> may extract features of the shapes of objects included in the plurality of sub images, and identify information on the shapes of objects based on the extracted features.

When information on the types of objects included in the plurality of sub images is identified, the processor <NUM> acquires a synthesized weight for each area of at least two of the plurality of sub images based on the identified information. Further, the processor <NUM> obtains a synthesized image based on the acquired synthesized weight for each area.

Specifically, as a result of identifying information on the types of objects included in the plurality of sub images in each of the plurality of sub images, the electronic apparatus assigns a relatively high synthesized weight to an image including more information on the types of objects than other images in a specific area among the plurality of sub images.

For example, as a result of identifying information on the types of objects included in the plurality of sub images in each of the plurality of sub images, when the first sub image among the plurality of sub images for a specific area contains twice as much information on the shapes of the objects as compared to the second sub image among the plurality of sub images, in obtaining a synthesized image, the processor <NUM> may synthesize the first sub image and the second sub image by assigning a synthesized weight of the first sub image to twice a synthesized weight of the second sub image.

According to the electronic apparatus <NUM> as described above, a plurality of sub images having a smaller bit number than the original image may be acquired from the original image, and a synthesized image capable of securing visibility for an entire area included in the image based on information on the shapes of the objects included in the plurality of sub images, and the problem of motion artifacts of the prior art may be solved simultaneously.

Meanwhile, the case of the original image having the <NUM>-bit number, the <NUM>-bit number, the <NUM>-bit number, or the-<NUM>-bit number per pixel, and the plurality of sub images having the number of <NUM>-bit number per pixel have been described as an example in the above, but is not limited to an image having a specific number of bits for each pixel.

In other words, within a range capable of achieving the object of the disclosure, the bit number per pixel of the original image, the plurality of sub images, and further the synthesized image according to the disclosure may be variously determined. However, hereinbelow, it is described that, for convenience, the original image mainly has bit number per pixel, and the plurality of sub images and synthesized images acquired therefrom have an <NUM>-bit number per pixel.

<FIG> is a block diagram illustrating a detailed configuration of an electronic apparatus according to an embodiment.

As illustrated in <FIG>, the electronic apparatus <NUM> according to an embodiment of the disclosure may include not only an image sensor <NUM>, a memory <NUM>, and a processor <NUM>, but also a communicator <NUM> and an outputter <NUM>, a sensor <NUM>, and a user interface unit <NUM>. However, such a configuration is exemplary, and it is obvious that a new configuration may be added or some configurations may be omitted in addition to the above configuration in performing the disclosure.

The image sensor <NUM> may convert light entering through a lens into an electrical image signal, and the processor <NUM> may acquire an original image of a subject through the image sensor <NUM>.

The memory <NUM> may include at least one command related to the electronic apparatus <NUM>. The memory <NUM> may store software based on various algorithms related to edge detection, and the memory <NUM> may store data related to an original image, a plurality of sub images, and a synthesized image according to the disclosure.

The processor <NUM> may control the overall operation of the electronic apparatus <NUM>. Specifically, the processor <NUM> may be connected to the image sensor <NUM>, the memory <NUM>, the communicator <NUM>, the outputter <NUM>, the sensor <NUM>, and the user interface unit <NUM> of the electronic apparatus <NUM> to control the overall operation of the electronic apparatus <NUM>.

Detailed descriptions of the other image sensor <NUM>, memory <NUM>, and processor <NUM> have been described above in the description of <FIG>, and redundant descriptions will be omitted.

Meanwhile, the communicator <NUM> may communicate with the external electronic apparatus (not illustrated) or the external server (not illustrated). The communicator <NUM> may include at least one of a Wi-Fi chip <NUM>, a Bluetooth chip <NUM>, a wireless communication chip <NUM>, and a near field communication (NFC) chip <NUM>.

In particular, the Wi-Fi chip <NUM> and the Bluetooth chip <NUM> may communicate in the Wi-Fi method and the Bluetooth method, respectively. When the Wi-Fi chip <NUM> or the Bluetooth chip <NUM> is used, a variety of connection information such as an SSID, etc. may be exchanged first, and after communication connection using the same, various types of information may be transmitted and received.

The wireless communication chip <NUM> refers to a chip that performs communication according to the various communication standards such as IEEE, Zigbee, 3rd Generation (<NUM>), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), and so on. The NFC chip <NUM> indicates chip to operate in NFC method using <NUM> bandwidth among various RF-ID frequency bandwidths such as <NUM>, <NUM>, <NUM>, <NUM>~<NUM>, and <NUM>.

In particular, in various embodiments of the disclosure, the communicator <NUM> may receive a signal regarding an original image by performing communication with an external electronic apparatus (not illustrated) or a server (not illustrated). Further, the processor <NUM> may acquire the original image through the communicator.

In other words, an embodiment that the processor <NUM> acquires an original image through the image sensor <NUM> has been described in the above, but the original image according to the disclosure may be acquired from an external electronic apparatus (not shown) or a server (not illustrated) through the communicator.

Specifically, the processor <NUM> may acquire an original image having a first bit number for each pixel through the communicator <NUM>, and based on the original image, a plurality of sub images having a second bit number smaller than the first bit number for each pixel based on the original image.

In addition, the processor <NUM> identifies information on the shapes of objects included in the plurality of sub images from each of the plurality of sub images, and based on the identified information, the processor <NUM> acquires a synthesized weight for each area of at least two sub images among a plurality of sub images, and further acquires a synthesis image based on the acquired synthesized weight for each area.

In addition, various embodiments of the disclosure may be applied not only when the original image is acquired through the image sensor <NUM> of the electronic apparatus, but also when the original image is acquired through the communicator <NUM> of the electronic apparatus.

Meanwhile, it has been described the image sensor <NUM> is included in some components of the electronic apparatus <NUM> in the above, but the disclosure is not necessarily limited thereto.

In other words, in order to reduce power consumption, unlike a general electronic apparatus or a photographing device, only the image sensor may be separated and implemented as an external image sensor (not illustrated). In this case, the processor <NUM> may acquire the original image from the external image sensor (not illustrated) through the communicator <NUM>, and further, a synthesized image may be acquired through the process described above. The outputter <NUM> may output various functions that can be performed by the electronic apparatus. The inputter <NUM> may include at least one of a display <NUM>, a speaker <NUM>, and an indicator <NUM>.

The display <NUM> may output image data under the control of the processor <NUM>. In particular, in an embodiment of the disclosure, when the processor <NUM> may acquire an original image, acquire a plurality of sub images based on the original image, and acquire a synthesize image based on a plurality of sub images, the processor <NUM> may control the display <NUM> to display the original image, the plurality of sub images, and the synthesized image.

In addition, the display <NUM> may display an image or a user interface pre-stored in the memory <NUM> under the control of the processor <NUM>.

Meanwhile, the display <NUM> may be implemented as a liquid crystal display panel (LCD), organic light emitting diodes (OLED), or the like, and the display <NUM> may also be implemented as a flexible display, a transparent display, etc. depending on a case. However, the display <NUM> according to the disclosure is not limited to a specific type.

The speaker <NUM> may output audio data under the control of the processor <NUM>, and the indicator <NUM> may be lit under the control of the processor <NUM>.

Specifically, in an embodiment of the disclosure, when the processor <NUM> may acquire an original image, acquire a plurality of sub images based on the original image, and acquire a synthesized image based on the plurality of sub images, the processor <NUM> may control the speaker <NUM> or the indicator <NUM> to output a voice or turn on an LED whenever each image is acquired.

The sensor <NUM> may detect various types of input. Specifically, the sensor <NUM> may be a touch sensor that senses a user touch, and the electronic apparatus according to the disclosure may include various sensors such as a motion sensor, a temperature sensor, a humidity sensor, an illuminance sensor, or the like.

The user interface unit <NUM> is an element that senses a user interaction for controlling an overall operation of the electronic apparatus <NUM>. Specifically, the user interface unit <NUM> may be composed of a camera, a microphone, a remote-control signal receiver, or the like. Meanwhile, the user interface unit <NUM> may be implemented in a form included in the display <NUM> as a touch screen.

Meanwhile, as described above, when the original image is acquired through the image sensor <NUM>, the processor <NUM> may control the display <NUM> to display the acquired original image.

In addition, when a user command for selecting at least one of the plurality of areas included in the displayed original image is input through the user interface unit <NUM>, the processor <NUM> may acquire synthesized weight for each area of at least two sub images among a plurality of sub images based on the information on the shapes of objects included in at least one area selected from the plurality of sub images.

As described above, an embodiment of inputting a user command for selecting at least one of a plurality of areas included in the original image will be described in detail with reference to <FIG>.

Meanwhile, a process of acquiring from an original image to a synthesized image according to an exemplary embodiment of the disclosure will be described in more detail below with reference to <FIG>.

<FIG> is a view illustrating an original image according to an embodiment. <FIG> and <FIG> are views illustrating two sub images separated from an original image illustrated in <FIG>.

As described above, the disclosure is not limited to an image having a specific bit number per pixel, but hereinafter, the original image has a <NUM>-bit number per pixel, and a sub image acquired from the original image has an <NUM>-bit number per pixel. This will be explained using an example.

As described above, the processor <NUM> may acquire a <NUM>-bit original image through the image sensor <NUM>, and the processor <NUM> may acquire a plurality of <NUM>-bit sub images smaller than <NUM>-bit based on the original image.

Specifically, the plurality of sub images may include a most significant bit (MSB) side image and a least significant bit (LSB) side image acquired based on the original image.

The MSB side image is the most significant bit-side image, for example, based on a bit of the largest digit in the <NUM>-bit original image, an <NUM>-bit image that can be acquired based on information from the <NUM>-bit to the <NUM>-bit.

In addition, the LSB side image is the least significant bit side image, for example, based on a bit of the smallest digit in the <NUM>-bit original image, an <NUM>-bit image that can be acquired based on the information from the <NUM>-bit to the <NUM>-bit that can be acquired based on information from the <NUM>-bit to the <NUM>-bit.

The MSB side image that can be acquired from the <NUM>-bit original image as shown in <FIG> is illustrated in <FIG>, and the LSB side image that can be acquired from the <NUM>-bit original image as illustrated in <FIG> is shown in <FIG>.

As shown in <FIG> and <FIG>, the MSB side image and the LSB side image acquired from the same original image may have different areas of high visibility for objects included in the image.

Specifically, a first area <NUM> of <FIG> indicating the MSB side image is expressed too dark, thereby making hard to easily identify the object included in the first area <NUM>, whereas an object included in a second area <NUM> of <FIG> may be easily identified relative to the first area <NUM>.

In addition, an object included in a third area <NUM> of <FIG> representing the LSB side image may be identified relatively easily, whereas a fourth area <NUM> of <FIG> is expressed too brightly and thus an object included in the fourth area may not be easily identified relative to the third area <NUM>.

Accordingly, when synthesized weights for the second area <NUM> of <FIG> and the third area <NUM> of <FIG> are assigned higher than synthesized weights of the first area <NUM> of <FIG> and the fourth area <NUM> of <FIG>, the synthesized image acquired as a result of the synthesis may secure high visibility for all areas included in the image compared to the original image.

Meanwhile, in the above, the image illustrated in <FIG> and <FIG> is divided into two areas and specified as the first area <NUM>, the second area <NUM>, the third area <NUM>, and the fourth area <NUM> for convenience. However, specifically, the visibility of objects included in each image may vary for each pixel included in the image. Also, the visibility of objects included in each image may vary according to various lighting environments.

<FIG> are views illustrating a process of identifying information on the shapes of objects included in sub images in each of the two sub images illustrated in <FIG> and <FIG>.

According to various embodiments of the disclosure, in each of the plurality of sub images, the processor <NUM> identifies information on the shapes of objects included in the plurality of sub images. Specifically, the processor <NUM> uses edge detection, and may use various other methods such as histogram feature detection, image high frequency analysis, image variance analysis, corner detection or the like to identify the information on the shapes of objects included in the plurality of sub images.

For example, the processor <NUM> may detect an edge in which a brightness of an image changes from a high value to a low value or from a low value to a high value in the plurality of sub-images to identify information on the shapes of objects included in the plurality of sub images.

In the edge detection, various methods such as a method using a Laplacian mask, a Sobel mask, a Roberts mask, a Prewitt mask, or a Canny mask may be used, and the disclosure is not limited to a specific method.

<FIG> and <FIG> are views illustrating a process of identifying information on the shapes of objects included in the sub-images by performing edge detection using the Laplacian mask for each of the two sub images illustrated in <FIG> and <FIG>.

Specifically, <FIG> is a view illustrating the result of performing the edge detection using the Laplacian mask on the MSB side image illustrated in <FIG>, and <FIG> is view illustrating the result of performing the edge detection using the Laplacian mask on the LSB side image illustrated in <FIG>.

Referring to 6A and 6B, it may be identified that the amount of the detected edge is smaller in a first area <NUM> of <FIG> compared to a third area <NUM> of <FIG>, and the amount of the detected edge is larger in a second area <NUM> of <FIG> compared to a fourth area <NUM> of <FIG>. Here, the large amount of detected edge means that there is a large amount of information about the shapes of objects.

In the case of synthesizing by giving synthesized weights for the second area <NUM> of <FIG> and the third area <NUM> of <FIG> higher than synthesized weights for the first area <NUM> of <FIG> and the fourth area <NUM> of <FIG>, the synthesized image acquired as a result of the synthesis may secure high visibility for all areas included in the image.

Meanwhile, <FIG> and <FIG> are view illustrating a process of identifying information on the shapes of objects included in the sub images by performing edge detection using Sobel masks for each of the two sub images shown in <FIG> and <FIG>.

Specifically, <FIG> is a view illustrating a result of performing edge detection using the Sobel mask on the MSB side image shown in <FIG>, and <FIG> is a view illustrating the result of performing the edge detection using the Sobel mask on the LSB side image shown in <FIG>.

Referring to <FIG> and <FIG>, blocks included in a second area <NUM> of <FIG> are displayed relatively brighter than the blocks included in a first area <NUM> of <FIG>. On the other hand, blocks included in a fourth area <NUM> of <FIG> are relatively darker compared to the blocks included in a third area <NUM> of <FIG>. Here, the area which is expressed relatively brighter means that there is more information on the shapes of the objects than the area expressed relatively darker.

Accordingly, in the case of synthesizing by giving synthesized weights for the second area <NUM> of <FIG> and the third area <NUM> of <FIG> higher than synthesized weights for the first area <NUM> of <FIG> and the fourth area <NUM> of <FIG>, the synthesized image acquired as a result of the synthesis may secure high visibility for all areas included in the image.

Meanwhile, the processor <NUM> may identify information on the shapes of objects included in the plurality of sub images based on a histogram of oriented gradient (HOG) value. Specifically, the processor <NUM> may divide the areas of the plurality of sub images into cells of a predetermined size, and for each cell, the processor may obtain a histogram of the directions of edge pixels with a slope value equal to or greater than a certain value, and then obtain a vector connecting these histogram bin values to identify information on the shapes of objects included in the plurality of sub-images.

In addition, the processor <NUM> may identify information on the shapes of objects included in the plurality of sub images based on a high-frequency image analysis. Specifically, the processor <NUM> may identify information on the shapes of objects included in the plurality of sub images by analyzing a high-frequency component having a large change rate of a pixel value among frequency components included in the plurality of sub images.

In addition, the processor <NUM> may identify information on the shapes of objects included in the plurality of sub images based on image variance analysis. Specifically, the processor <NUM> may identify information on the shapes of objects included in the plurality of sub images by statistically analyzing a distribution of image data included in the plurality of sub images.

Further, the processor <NUM> may identify information on the shapes of objects included in the plurality of sub images through various methods including a corner detection method.

In the above, various methods for identifying information on the types of objects included in the plurality of sub images have been described. In addition to the above-described methods, various methods for identifying information on the shapes of objects may be used within a range that can achieve the object of the disclosure.

Meanwhile, the process of identifying information on the types of objects included in the plurality of sub images by various methods as described above may be performed for each pixel included in the plurality of sub images, but for the efficiency and speed of the image processing process. As will be described below, the processor <NUM> may identify information on the shapes of objects included in the plurality of sub images for each block after performing blocking for each pixel.

For example, the processor <NUM> may block 1280x720 pixels included in the entire plurality of sub images to be included in 32x18 blocks.

In addition, the processor <NUM> may identify information on the shapes of objects included in the plurality of sub images for each block according to the blocking. Specifically, the processor <NUM> may identify information on the shapes of objects included in the plurality of sub images for each block according to blocking by using various methods as described above, such as edge detection, or the like.

Meanwhile, in order to reduce the computational amount of image processing, the processor <NUM> may acquire a grayscale image for each of the plurality of sub images prior to performing blocking on each pixel, and block each pixel included in the acquired grayscale image.

Specifically, the processor <NUM> may convert the plurality of sub images acquired as a color image into grayscale images, respectively. In the process of acquiring the grayscale image, a method of using color luminance has been mainly used, but the process of acquiring the grayscale image according to the disclosure is not limited to a specific method.

<FIG> is a view illustrating a result of acquiring a grayscale image of the MSB side image, and performing edge detection after blocking each pixel included in the acquired grayscale image, as illustrated in <FIG>.

In addition, <FIG> is a view illustrating a result of acquiring a grayscale image of the LSB side image, and performing edge detection after blocking each pixel included in the acquired grayscale image, as illustrated in <FIG>.

Referring to <FIG>, blocks included in a second area <NUM> of <FIG> are displayed relatively brighter than the blocks included in a first area <NUM> of <FIG>. Further, referring to <FIG>, blocks included in a fourth area <NUM> of <FIG> are relatively dark compared to the blocks included in a third area <NUM> of <FIG>.

Here, blocks expressed in dark indicate that there is little information on the shapes of objects in the area included in the blocks, and blocks expressed in bright indicate that there is much information about the shapes of objects in the area included in the blocks.

In other words, as a result of performing edge detection, the processor <NUM> may identify the presence or absence of information on the shapes of objects included in the image for each block, as well as the amount of information on the shapes of the objects.

In the above, the images illustrated in <FIG> are divided into two areas and specified for convenience, but specifically, the feature that the visibility of objects included in each image may vary for each pixel included in the image has been described above.

Meanwhile, the method of identifying information on the shapes of objects after converting the plurality of sub images into grayscale images and blocking each pixel included in the converted grayscale image has been described in the above, the process of converting and blocking grayscale images is not necessary to achieve the object of the present disclosure.

In particular, if the method is applied after performing the blocking process depending on the method for identifying information on the shapes of objects, it may be difficult to identify the information on the shapes of objects, or rather, the amount of computation of the overall image processing may increase. Thus, whether or not to perform the blocking process should be determined in relation to a specific method for identifying information on the shapes of objects.

Based on the information identified as described above, the processor <NUM> may acquire a synthesized weight for each area of the MSB side image and the LSB side image, which will be described in detail with reference to <FIG> and <FIG>.

<FIG> is a graph illustrating a synthesized weight for two sub images according to an embodiment of the disclosure, and <FIG> is a view illustrating an acquired synthesized image based on a synthesized weight for each area of the two sub images shown in <FIG> and <FIG>.

When information on the shapes of objects included in the plurality of sub images is identified, the processor <NUM> may acquire a synthesized weight for each area of at least two sub images among the plurality of sub images based on the identified information.

As described above, the shapes of objects included in the plurality of sub images for each block are identified after a grayscale image for each of the plurality of sub images is acquired, and each pixel included in the acquired grayscale image is blocked, the processor <NUM> may acquire a synthesized weight for each block for at least two of the plurality of sub images based on the information identified for each block.

The synthesized weight for each block as described above may be referred to as a dynamic curve for each block as shown in <FIG>. Here, an x-axis refers to data of an input original image, and a y-axis refers to data of an output synthesized image. For example, when a <NUM>-bit original image is input, the x-axis represents a value from <NUM> to <NUM>, and when an <NUM>-bit synthesized image is output, the y-axis represents a value from <NUM> to <NUM>.

Specifically, in the case of the graph shown in a first block <NUM> of <FIG>, a slope of a graph corresponding to an input initial value is formed larger than that of a second block <NUM>. This indicates that in the case of the area corresponding to the first block <NUM>, the synthesize weight for the LSB side image is higher than the synthesized weight for the MSB side image as compared to the case of the area corresponding to the second block <NUM>.

<FIG> illustrates a synthesized image acquired through a process from description of <FIG> to description of <FIG>.

Specifically, the processor <NUM> may acquire a <NUM>-bit original image as shown in <FIG> through the image sensor <NUM>, and based on the acquired original image, and acquire an <NUM>-bit MSB side image as shown in <FIG> and an <NUM>-bit LSB side image as shown in <FIG>.

Further, the processor <NUM> may acquire a grayscale image from each of the acquired MSB side image and LSB side image, block each pixel included in the acquired grayscale image, and also identify information on the shapes of objects included in the MSB side image and the LSB side image for each block by performing edge detection.

Also, the processor <NUM> may acquire a synthesized weight for each block with respect to the MSB side image and the LSB side image based on the information identified for each block. In addition, the processor <NUM> may acquire a synthesized image as illustrated in <FIG> based on the identified synthesized weight for each block.

As illustrated in <FIG>, the synthesized image acquired according to the above-described process may have high visibility for a subject included in the entire area of the image. In addition, since the synthesized image as shown in <FIG> is acquired based on one original image acquired through the image sensor <NUM>, the problem of motion artifacts pointed out as a limitation of the prior art may be solved according to the disclosure.

<FIG> are views exemplarily illustrating another plurality of sub images according to an exemplary embodiment of the disclosure.

Like the images illustrated in <FIG> and <FIG>, the images illustrated in <FIG> correspond to <NUM>-bit sub images acquired from the same <NUM>-bit original image. Hereinafter, the image illustrated in <FIG> is referred to as a first sub-image, the image illustrated in <FIG> is referred to as a second sub-image, and the image illustrated in <FIG> is referred to as a third sub-image.

The first sub-image corresponds to an MSB side image, that is, an image acquired based on information from the <NUM>-bit to the <NUM>-bit based on the bit having the largest digit in the <NUM>-bit original image.

The second sub-image corresponds to an image acquired based on information from the <NUM>-bit to the <NUM>-bit, based on the bit having the largest digit in the <NUM>-bit original image.

The third sub-image corresponds to an image acquired based on the information from the <NUM>-bit to the <NUM>-bit based on the bit having the largest digit in the <NUM>-bit original image.

Meanwhile, although not illustrated, the processor <NUM> may acquire a fourth sub-image acquired based on information from the <NUM>-bit to the <NUM>-bit based on the bit having the largest digit in the <NUM>-bit original image.

In addition, the processor <NUM> may acquire a fifth sub-image, that is, an LSB side image, which is acquired based on the information from the <NUM>-bit to the <NUM>-bit, based on the bit of the largest digit in the <NUM>-bit original image.

As described above, the processor <NUM> may acquire the first to fifth sub images of the <NUM>-bit, that is, an <NUM>-bit multi-layer image, from the same <NUM>-bit original image.

Referring to <FIG>, high visibility may be secured in the first sub-image in the case of a first area <NUM> including a crosswalk among objects included in a plurality of sub images, high visibility may be secured in the second sub-image in the case of a second area <NUM>, and high visibility may be secured in the third sub-image in the case of a third area <NUM> including pedestrians and signs,.

According to an embodiment of the disclosure, the processor <NUM> may acquire a synthesized image having high visibility in all areas included in the image based on the plurality of sub images as described above.

Specifically, the processor <NUM> may acquire a <NUM>-bit original image through the image sensor <NUM>, acquire five <NUM>-bit sub images based on the original image, and information on the shapes of objects included in the sub images in each of the five sub images.

Further, the processor <NUM> may acquire a synthesized weight for each area of at least two sub images among the plurality of sub images based on the identified information as above, and acquire a synthesized image based on the acquired synthesized weight for each area.

In other words, in the description of <FIG>, an exemplary embodiment of acquiring two sub images, that is, the MSB side image and the LSB side image from the original image, has been described, but as described in the description of <FIG>, a synthesized image may be acquired based on at least two sub images among the five <NUM>-bit sub images acquired from the <NUM>-bit original image.

Meanwhile, as shown in <FIG>, since the area with high visibility for the object included in the image for each sub-image is different, as the number of sub images that are the basis of the synthesis increases, the visibility of overall synthesized image may be higher.

However, depending on lighting environment, overall visibility of the synthesized image may not be as high as the number of sub images that are the basis of the synthesis increases. Accordingly, in various embodiments of the disclosure, the number of sub images that are the basis for synthesis should be determined by taking into account various lighting environments and an increase in image processing speed according to an increase in the number of sub images.

<FIG> is a view illustrating an input of a user command for selecting at least one of a plurality of areas included in an original image, according to an embodiment of the disclosure.

As described above, when the original image is acquired through the image sensor <NUM>, the processor <NUM> may control the display <NUM> to display the acquired original image.

In addition, a user command for selecting at least one area <NUM> from among a plurality of areas included in the original image displayed on the display <NUM> may be input through the user interface unit <NUM>. Here, the at least one area <NUM> selected by the user may correspond to an area where the user desires to secure visibility.

When the user command for selecting at least one area <NUM> from among the plurality of areas included in the original image is input, the processor <NUM> may acquire a synthesized weight for each area with respect to at least two of the sub images among the plurality of sub images based on information on the shapes of objects included in the selected at least one area <NUM>.

Specifically, the processor <NUM> may assign a higher synthesized weight to a sub image having a larger amount of information on the shapes of objects included in at least one area <NUM> selected by the user, compared to other sub images. Accordingly, the processor <NUM> may acquire a synthesized image capable of securing higher visibility for the area selected by the user command.

Meanwhile, the user command may be input through a user's touch interaction with a touch screen. In this case, the at least one area <NUM> selected by the user command may be an area within a predetermined radius from a point where the user's touch interaction is input, or an area including a plurality of pixels within a predetermined horizontal length and a predetermined vertical length at the point where the user's touch interaction is input.

In the above, the case where the user command is input through the user's touch interaction has been exemplified, but the user command may be input in various ways through various user interface units <NUM>.

In the above, the case where the original image is displayed on the display <NUM> and the user command for selecting at least one of the plurality of areas included in the original image is input through the user interface unit <NUM> has been exemplified. However according to another embodiment of the disclosure, the plurality of sub images acquired from the original image are displayed on the display <NUM>, and the user command that selects at least one of the plurality of areas included in at least one sub image among the plurality of sub images may be input through the user interface unit <NUM>.

In this case, when the user command for selecting at least one of the plurality of areas included in at least one of the plurality of sub images is input through the user interface unit <NUM>, the processor <NUM> may acquire a synthesized weight for each area with respect to at least two sub images among the plurally of sub images.

According to an embodiment of the disclosure as described above, since it is possible to acquire a synthesized image by assigning a high synthesized weight of a sub-image capable of securing high visibility to the area selected by the user command, the visibility of the user's desired area may be further improved.

<FIG> is a view illustrating an embodiment in which an electronic apparatus according to the disclosure is implemented as a part of a vehicle.

The electronic apparatus according to the disclosure may be implemented as an independent electronic apparatus such as a camera, a smartphone, or the like, and may be implemented as a part of a vehicle, a robot, or the like.

In particular, as illustrated in <FIG>, the electronic apparatus according to the disclosure may be implemented as a camera <NUM> that is a part of a vehicle. The vehicle may be an autonomous vehicle. In addition, the camera <NUM> may be implemented in a form coupled to a rearview mirror inside the vehicle to recognize an object appearing in front of the vehicle while driving. Meanwhile, the camera <NUM> may include the image sensor <NUM>, the memory <NUM>, and the processor <NUM> as described above.

Specifically, according to an embodiment of the disclosure, the processor <NUM> may acquire an original image including an object appearing in front of the vehicle through the image sensor <NUM>.

The processor <NUM> may acquire a plurality of sub images having a second bit number smaller than the first bit number for each pixel, based on the original image having the first bit number for each pixel.

In addition, the processor <NUM> may identify information on the shapes of objects included in the plurality of sub images from each of the plurality of sub images, acquire, based on the identified information, a synthesized weight for each area for at least two sub images among the plurality of sub images, and acquire a synthesized image based on the synthesized weight for each of the acquired area.

In addition, various embodiments of the disclosure as described above with reference to <FIG> may be similarly applied even when the electronic apparatus according to the disclosure is implemented as a part of a vehicle.

As described above, when the electronic apparatus according to the disclosure is implemented as a part of a vehicle, especially an autonomous vehicle, objects appearing in front of a vehicle such as lanes, pedestrians, vehicles, or the like even in various driving environments such as nighttime, bad weather, backlight, or the like may be smoothly recognized, and thus driving stability of the vehicle may be improved.

<FIG> is a flowchart schematically illustrating a method of controlling an electronic apparatus according to an embodiment of the disclosure.

According to an embodiment of the disclosure, the electronic apparatus may acquire an original image having a first bit number for each pixel through the image sensor <NUM> (S1201).

Meanwhile, when the original image is acquired, a plurality of sub images having a second bit number smaller than the first bit number for each pixel may be acquired based on the original image (S1202). The plurality of sub images may include a most significant bit (MSB) side image and a least significant bit (LSB) side image acquired based on the original image.

When the plurality of sub images are acquired, information on the shapes of objects included in the plurality of sub images may be identified from each of the plurality of sub images (S1203). The identification of information may be performed based on a result of edge detection, a histogram of oriented gradient (HOG) value, or the like.

When information on the shapes of objects included in the plurality of sub images is identified, a synthesized weight for each area of at least two sub images among the plurality of sub images may be acquired based on the identified information (S1204).

Meanwhile, the method of controlling the electronic apparatus according to an embodiment of the disclosure may further include displaying the acquired original image when the original image is acquired through the image sensor.

When a user command for selecting at least one of the plurality of areas included in the displayed original image is input, a synthesized weight for each area of at least two sub images of the plurality of sub images the plurality of sub images may be acquired based on the information on the shapes of objects included in at least one area selected among the plurality of sub images.

Meanwhile, when the synthesized weight for each area of at least two sub images among the plurality of sub images is acquired, the synthesized image may be acquired based on the acquired synthesized weight for each area (S1205).

<FIG> is a detailed flowchart illustrating a method of controlling an electronic apparatus according to an embodiment of the disclosure.

According to an embodiment of the disclosure, when a <NUM>-bit original image is acquired (S1301), an MSB side image and an LSB side image of an <NUM>-bit may be acquired based on the <NUM>-bit original image (S1302).

When the MSB side image and the LSB side image are acquired, a grayscale image for each of the acquired MSB side and LSB side images may be acquired (S1303). In other words, it may reduce the amount of calculation of image processing by converting the MSB side image and the LSB side image acquired as a color image into the grayscale image.

When the grayscale image is acquired, each pixel included in the acquired grayscale image may be blocked (S1304). For example, the efficiency and speed of image processing may be improved by blocking 1280x720 pixels included in each of the MSB side image and the LSB side image of the grayscale to be included in 32x18 blocks.

When each pixel is blocked, information on the shapes of objects included in the MSB side image and the LSB side image may be identified for each block (S1305). The identification of each block information may be performed based on various methods, such as edge detection, histogram feature detection, corner detection, or the like.

When information on the shapes of objects for each block is identified, a synthesized weight for each block for the MSB side image and the LSB side image may be acquired based on the identified information (S1306). Specifically, a higher synthesized weight may be assigned to an image capable of securing high visibility for a specific area of the MSB side image or the LSB side image in acquiring the synthesized weight for each block.

When the synthesized weight for each block is acquired, a synthesis image may be acquired based on the acquired synthesized weight for each block (S1307). In other words, it is possible to acquire a synthesized image having high visibility for all areas included in the image compared to the original image.

The method of controlling an electronic device according to the above-described various exemplary embodiments may be realized as a program and provided in the user terminal device. In particular, the program including a method for controlling a display apparatus according to exemplary embodiments may be stored in a non-transitory computer readable medium and provided therein.

The non-transitory computer readable recording medium refers to a medium that stores data and that can be read by devices. In detail, the above-described various applications or programs may be stored in the non-transitory computer readable medium, for example, a compact disc (CD), a digital versatile disc (DVD), a hard disc, a Blu-ray disc, a universal serial bus (USB), a memory card, a read only memory (ROM), and the like, and may be provided.

According to various embodiments of the disclosure as described above, a plurality of sub images having a smaller bit number than an original image are acquired from the original image, and a synthesized image capable of securing visibility for all included areas may be acquired based on information on the shapes of objects included in the plurality of sub images. In addition, since the synthesized image according to the disclosure is acquired based on one original image, the problem of motion artifacts, which is pointed out as a limitation of the prior art, may be solved according to the disclosure.

Claim 1:
An electronic apparatus (<NUM>) comprising:
an image sensor (<NUM>);
a display (<NUM>); and
a user interface unit (<NUM>),
a memory (<NUM>) including at least one command; and
a processor (<NUM>) configured to control the electronic apparatus being coupled to the image sensor and the memory,
wherein the processor is configured to:
acquire, through the image sensor (<NUM>), one original image (<NUM>) having a first bit number for each pixel,
based on the original image (<NUM>) being acquired through the image sensor, control the display to display the acquired original image,
acquire, from the original image (<NUM>), a plurality of sub images (<NUM>) each having a second bit number for each pixel that is smaller than the first bit number for each pixel,
identify an amount of information on shapes of objects included in the plurality of sub images from each of the plurality of sub images, wherein information on the shapes of objects included in the plurality of sub images is identified through an edge detection,
based on a user command for selecting at least one area (<NUM>) among a plurality of areas included in the displayed original image being imputed through the user interface unit (<NUM>), acquire a synthesized weight for each area with respect to two sub images among the plurality of sub images based on the identified amount of the information by assigning a higher synthesized weight to a sub image having a larger amount of information on the shapes of objects included in the at least one area (<NUM>) selected by the user, compared to other sub images of the plurality of sub images, and
acquire a synthesized image (<NUM>) based on the synthesized weight for each of the acquired area.