Patent ID: 12211175

DETAILED DESCRIPTION

Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings.

FIG.1is a block diagram illustrating an image processing system100according to an example embodiment.

The image processing system100may be included in an electronic device or implemented as a separate electronic device. The electronic device may be, for example, a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. The portable electronic device may be a laptop computer, a mobile phone, a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an e-book, a wearable device, or the like.

Referring toFIG.1, the image processing system100may include an image sensor110, an image signal processor120, a memory130, and a display device140.

The image sensor110may convert an optical signal of the subject OBJECT incident through the optical lens LS into an electrical signal or an image (i.e., image data). The image sensor110may include, for example, a pixel array including a plurality of two-dimensionally arranged sensing pixels and a sensing circuit, and the pixel array and the sensing circuit may be integrated into one semiconductor chip. The pixel array may convert received optical signals into electrical signals. The pixel array, for example, may be implemented with a photoelectric conversion element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and may be implemented with various types of photoelectric conversion devices. The sensing circuit may convert an electrical signal provided from the pixel array into an image, and generate the converted image as the input image signal IIMG. The input image signal IIMG may be an input to the image signal processor120in the following description.

The image signal processor120may image-process the input image signal IIMG provided from the image sensor110to generate an output image signal OIMG. For example, the image signal processor120may image-process the input image signal IIMG based on set scaling, white balance, various parameters, and the like. The output image signal OIMG may be a color space image such as an RGB image or a YUV image. A size, for example, a resolution, of the output image signal OIMG may be the same as that of the input image signal IIMG. The output image signal OIMG may be stored in the memory130. The memory130may be a volatile memory such as a dynamic random access memory (DRAM) or a static RAM (SRAM), or a non-volatile memory such as a phase change RAM (PRAM), a resistive RAM (ReRAM), or a flash memory. The output image signal OIMG stored in the memory130may be later used in the image processing system100or stored in a storage device.

Also, the image signal processor120may generate a scaled image by decreasing or increasing the size of the output image signal OIMG. For example, the image signal processor120may generate the scaled image by scaling the size, that is, the resolution of the converted image to match the resolution of the display device140. The image signal processor120may provide the scaled image to the display device140.

Power consumed by the image signal processor120for image processing and data band-width increase according to the high-pixel tendency for manufacturing the image sensor110. Therefore, to address these problems, a method of reducing the size, i.e., resolution, of the input image signal IIMG before image processing may be used. However, when the amount of data of the input image signal IIMG is reduced to reduce power consumption and data band-width, image quality loss occurs during the scaling process.

To minimize deterioration (loss) of image quality, according to one or more embodiments of the disclosure, the image signal processor120, the operating method of the image signal processor120, and the application processor200including the image signal processor120extract information on image quality deterioration due to scaling and image processing from the input image signal IIMG to generate an image information signal IMG_IF in addition to the process of reducing the data amount of the input image signal IIMG and image processing, and generate the output image signal OIMG by using the image information signal IMG_IF, thereby minimizing deterioration in image quality of the output image signal OIMG. Accordingly, in the final image processing of the high-definition input image signal IIMG, it is possible to minimize power consumption and data band-width while minimizing image quality degradation.

FIG.2is a diagram schematically illustrating an image signal processor120according to an example embodiment.

The image signal processor120ofFIG.2may include a decomposition circuit121, an image processing engine122, and a recomposition circuit124.

Referring toFIG.2, the decomposition circuit121may generate a first image signal IMG1and an image information signal IMG_IF by downscaling the input image signal IIMG and/or generating correction information. For example, the first image signal IMG1may be a down-scaled signal of the input image signal IIMG. Accordingly, the data amount of the first image signal IMG1may be less than the data amount of the input image signal IIMG. Alternatively, the decomposition circuit121may divide the input image signal IIMG for each frequency band, and generate the first image signal IMG1having a low frequency component by applying a low pass filter. Similarly, the data amount of the first image signal IMG1may be less than that of the input image signal IIMG. The image information signal IMG_IF may be generated by extracting a high frequency component of the input image signal IIMG. The high-frequency component may be correction information capable of correcting image quality deterioration due to scaling and image processing.

The image processing engine122may receive the first image signal IMG1and perform various image processing on the first image signal IMG1to generate the second image signal IMG2. The image processing engine122may include a plurality of image modules with high power consumption and high computational amount.

As described above, before image processing is performed by the image processing engine122, the decomposition circuit121generates the first image signal IMG1having a data amount smaller than that of the input image signal IIMG. In addition, by image processing the first image signal IMG1in the image processing engine122, power consumption and the amount of calculation in the image processing engine122may be reduced.

The recomposition circuit124may recompose the second image signal IMG2and the image information signal IMG_IF output from the image processing engine122to generate the output image signal OIMG. By reconstructing the second image signal IMG2and the image information signal IMG_IF, image quality deterioration may be minimized. As described above, the image information signal IMG_IF may be a high frequency signal. For example, the image information signal IMG_IF may include information related to an edge of the input image signal IIMG.

FIG.3is a block diagram schematically illustrating an image signal processor according to an example embodiment.

Referring toFIG.3, the image signal processor120may include a decomposition circuit121, an image processing engine122, a first upscaling circuit123, and a recomposition circuit124.

The decomposition circuit121of the image signal processor120may include a downscaling circuit125, a second upscaling circuit126, and a correction information generating circuit127. Although the second upscaling circuit126and the correction information generating circuit127are illustrated as separate components inFIG.3, embodiments are not limited thereto, and in another example embodiment, the second upscaling circuit126may be included in the correction information generating circuit127. In addition, although the recomposition circuit124and the first upscaling circuit123are illustrated as separate components inFIG.3, embodiments are not limited thereto, and the first upscaling circuit123may be included in the recomposition circuit124.

In an example embodiment, the downscaling circuit125, the image processing engine122, the recomposition circuit124, the first upscaling circuit123, the second upscaling circuit126, and the correction information generating circuit127may be implemented as hardware. However, embodiments are not limited thereto, and the downscaling circuit125, the image processing engine122, the recomposition circuit124, the first upscaling circuit123, the second upscaling circuit126, and the correction information generating circuit127may be implemented by a combination of hardware and software.

The input image signal IIMG may be input to the downscaling circuit125and the correction information generating circuit127. The downscaling circuit125may generate the first image signal IMG1by downscaling the input image signal IIMG. Accordingly, the resolution of the first image signal IMG1may be less than the resolution of the input image signal IIMG. For example, the resolution of the first image signal IMG1may be 640×480, and the resolution of the input image signal IIMG may be 800×600. As another example, the downscaling circuit125may include a low pass filter through which the first image signal IMG1having a low frequency component of the input image signal IIMG may be generated.

When the resolution of the input image signal IIMG is reduced through the downscaling circuit125, since the power consumption and the band-width gain are the largest, a low pass filter through scaling has been described as an example embodiment, but embodiments may be implemented by a low-pass filter capable of implementing a method other than scaling (e.g., a method of reducing a data amount other than downscaling).

The first image signal IMG1may be input to the image processing engine122and the second upscaling circuit126. The second upscaling circuit126may generate the fourth image signal IMG4by upscaling the first image signal IMG1. The resolution of the fourth image signal IMG4may be the same as the resolution of the input image signal IIMG. For example, the resolution of the fourth image signal IMG4and the resolution of the input image signal IIMG may be 800×600.

The fourth image signal IMG4and the input image signal IIMG may be input to the correction information generating circuit127. The correction information generating circuit127may extract correction information related to image quality deterioration caused by scaling and image processing in the image processing engine122from the fourth image signal IMG4and the input image signal IIMG to generate an image information signal IMG_IF. The image information signal IMG_IF may be a high frequency signal. For example, the image information signal IMG_IF including information for compensating for loss due to image processing and scaling in the image processing engine122may be a signal including high frequency information related to an edge of the input image signal IIMG. That is, since the image information signal IMG_IF includes information related to image quality degradation that occurs while the input image signal IIMG undergoes scaling and image processing, the recomposition circuit124may use the image information signal IMG_IF to minimize deterioration in image quality of the output image signal OIMG.

The image processing engine122may perform various image processing on the first image signal IMG1to generate the second image signal IMG2. Since image quality may be deteriorated during the image processing process, the second image signal IMG2may be an image signal with deteriorated image quality.

The first upscaling circuit123may generate the third image signal IMG3by upscaling the second image signal IMG2. Accordingly, the resolution of the input image signal IIMG and the third image signal IMG3may be the same. For example, the resolution of the input image signal IIMG and the resolution of the third image signal IMG3may be 800×600. Also in this process, image quality deterioration due to scaling may occur. Accordingly, the third image signal IMG3may be an image signal whose image quality is degraded by image processing and scaling, and the third image signal IMG3may have a lower image quality than the input image signal IIMG.

The recomposition circuit124may reconstruct the third image signal IMG3and the image information signal IMG_IF to generate the output image signal OIMG. As described above, the third image signal IMG3may be an image signal with reduced image quality, and the recomposition circuit may perform correction to minimize deterioration in image quality of the third image signal IMG3through correction information extracted from the input image signal IIMG and the fourth image signal IMG4included in the image information signal IMG_IF.

The output image signal OIMG may be scaled again to fit the resolution of the display device140such as an electronic device including the image signal processor120.

FIG.4is a block diagram schematically illustrating a decomposition circuit of an image signal processor according to an example embodiment.

Referring toFIG.4, the decomposition circuit121may include a downscaling circuit125, a second upscaling circuit126, and a correction information generating circuit127, and the correction information generating circuit127may include at least one of an adaptive filter127-2, a noise reduction filter127-4, a brightness enhancement circuit127-1, and a sharpness enhancement circuit127-3.

The input image signal IIMG and the fourth image signal IMG4may pass through the brightness enhancement circuit127-1and the adaptive filter127-2, and the adaptive filter127-2may filter the input image signal IIMG and the fourth image signal IMG4that have passed through the brightness enhancement circuit127-1to generate one image signal, but embodiments are not limited thereto.

As described above, the correction information generating circuit127generates the image information signal IMG_IF by performing the correction information generating process to extract as much as possible information related to image quality degradation of the input image signal IIMG that occurs in image processing and scaling. In the process of generating the correction information, deterioration of image quality that may occur in the process of scaling and image processing may be mainly related to luminance and/or sharpness. Accordingly, in various embodiments, the correction information generating circuit127may include a brightness enhancement circuit127-1and/or a sharp enhancement circuit127-3.

FIG.5is a block diagram schematically illustrating a decomposition circuit of an image signal processor according to an example embodiment.

Referring toFIG.5, the decomposition circuit121may include a downscaling circuit125, a second upscaling circuit126, and a correction information generating circuit127. The correction information generating circuit127may include a first brightness enhancement circuit127-11, a second brightness enhancement circuit127-12, and a differential circuit127-5.

FIG.5is one example embodiment in which the image information signal IMG_IF is generated by extracting information on image quality deterioration of the third image signal IMG3by image processing and scaling from the input image signal IIMG and the fourth image signal IMG4. The differential circuit127-5may differentiate the input image signal IIMG and the fourth image signal IMG4that have respectively passed through the first brightness enhancement circuit127-11and the second brightness enhancement circuit127-12, and a signal obtained by differentiating the input image signal IIMG and the fourth image signal IMG4that have respectively passed through the first brightness enhancement circuit127-11and the second brightness enhancement circuit127-12may be combined with the input image signal IIMG that has passed through the sharp enhancement circuit127-3and the second brightness enhancement circuit127-12. Finally, it may pass through the noise reduction filter127-4, and the correction information generation circuit127may generate the image information signal IMG_IF. The image information signal IMG_IF generated by the correction information generating circuit127through the above process may be a high frequency signal. For example, the image information signal IMG_IF may be a signal including information related to an edge of the input image signal IIMG.

FIG.6is a block diagram schematically illustrating a recomposition circuit of an image signal processor according to an example embodiment.

Referring toFIG.6, the image information signal IMG_IF generated by the correction information generating circuit127is input to the recomposition circuit124. The recomposition circuit124may include at least one of a radial correction circuit128and a gain control circuit129.

The recomposition circuit124may serve to correct the image information signal IMG_IF, and the radial correction circuit128and the gain control circuit129may be implemented as hardware.

The recomposition circuit124may reconstruct the image information signal IMG_IF and the third image signal IMG3to generate the output image signal OIMG. The recomposition circuit124may perform at least one function of a radial correction and a gain control on the image information signal IMG_IF to minimize the deterioration of the image quality of the output image signal OIMG.

The image may have more noise from the center to the periphery due to the characteristics of the lens. To adjust for such noise, the recomposition circuit124may include a radial correction circuit128. In addition, the intensity of the image information signal IMF IF may be adjusted to minimize deterioration of image quality of the third image signal IMG3through the gain control circuit129.

Referring toFIG.6, the recomposition circuit124is shown as a separate configuration from the first upscaling circuit123but it will be understand that the first upscaling circuit123may be included in the recomposition circuit124. Even when the recomposition circuit124includes the first upscaling circuit123, the first upscaling circuit123may up-scale the second image signal IMG2to generate the third image signal IMG3.

FIG.7is a block diagram schematically illustrating an image signal processor according to an example embodiment.

Referring toFIG.7, the image signal processor120may include a decomposition circuit121, an image processing engine122, a first upscaling circuit123, a recomposition circuit124, and a data conversion unit150.

The data conversion unit150may data-convert the raw image signal RIMG generated by the image sensor110to generate the input image signal IIMG. The resolution of the input image signal IIMG and the resolution of the raw image signal RIMG may be the same by the data conversion unit150, and the data amount of the input image signal IIMG may be greater than the data amount of the raw image signal RIMG. Alternatively, the color space of the raw image signal RIMG and the input image signal IIMG may be different due to data conversion of the data conversion unit150. For example, the raw image signal RIMG may have a Bayer pattern, and the input image signal IIMG may have an RGB or YUV pattern. Converting a Bayer pattern to an RGB pattern may increase the amount of data.

FIGS.8A and8Bshow a Bayer pattern that a raw image signal may have according to an example embodiment.

Referring toFIGS.8A and8B, the Bayer pattern may mean a pattern intersectingly arranged such that green is 50% and red and blue are 25%, respectively, according to human visual characteristics.

Referring toFIG.8A, the pixel group PGa may be configured in a 2×2 Bayer pattern. The pixel group PGa may include a first green pixel Gr, a red pixel R, a second green pixel Gb, and a blue pixel B, and the first green pixel Gr and the second green pixel Gb may be arranged in a diagonal direction, and the red pixel R and the blue pixel B may be arranged in a diagonal direction.

Referring toFIG.8B, the pixel group PGb may be configured in a 4×4 Bayer pattern. The pixel group PGb may include four first green pixels Gr, red pixels R, second green pixels Gb, and blue pixels B, respectively. In addition to this, the pixel group may be configured with Bayer patterns of various sizes.

FIG.9is a block diagram schematically illustrating an image signal processor according to an example embodiment.

Referring toFIG.9, the image signal processor120may include a decomposition circuit121, an image processing engine122, a first upscaling circuit123, a recomposition circuit124, and a memory160.

The decomposition circuit121may generate the first image signal IMG1and the image information signal IMG_IF from the input image signal IIMG. The image information signal IMG_IF generated by the decomposition circuit121may be generated after passing through the brightness enhancement circuit127-1of the correction information generating circuit127that may be included in the decomposition circuit121.

The image information signal IMG_IF includes correction information for minimizing image quality deterioration caused by image processing and scaling, and may be input to the recomposition circuit124together with the third image signal IMG3. As described above, the image information signal IMG_IF may be transmitted directly from the decomposition circuit121to the recomposition circuit124, but as shown inFIG.9, the image information signal IMG_IF may be stored in the memory160inside the image signal processor120and then transmitted to the recomposition circuit124. In this case, the image information signal IMG_IF and the third image signal IMG3transmitted from the memory160may be simultaneously input to the recomposition circuit124. However, embodiments are not limited thereto, and the image information signal IMG_IF and the third image signal IMG3may be input to the recomposition circuit124at a time interval intended by a user.

For example, the memory160may be a volatile memory such as a dynamic random access memory (DRAM) or a static RAM (SRAM), or a non-volatile memory such as a phase change RAM (PRAM), a resistive RAM (ReRAM), or a flash memory.

FIG.10is a block diagram schematically showing an image signal processor according to an example embodiment, andFIG.11is a block diagram schematically illustrating a high frequency decomposition circuit of the image signal processor ofFIG.10.

Referring toFIG.10, the image signal processor120may include a downscaling circuit125, a first gamma correction circuit127-5, a second gamma correction circuit127-6, an upscaler127-7, an image processing engine122, a HF decomposition circuit127-80, and an up-scaler with HF recomposition170.

The downscaling circuit125may generate the first image signal IMG1by downscaling the input image signal IIMG. The first image signal IMG1may be input to each of the image processing engine122, the first gamma correction circuit127-5, and the second gamma correction circuit127-6. The image processing engine122generates a second image signal IMG2, the first gamma correction circuit127-5generates a fifth image signal IMG5, and the second gamma correction circuit127-6generates the seventh image signal IMG7. The upscaler127-7may generate the sixth image signal IMG6by upscaling the fifth image signal IMG5. The high frequency decomposition circuit127-80may generate the image information signal IMG_IF by performing Gaussian filtering and differentiation between the sixth image signal IMG6and the seventh image signal IMG7. The up-scaler with HF recomposition170may generate the output image signal OIMG by reconstructing and upscaling the second image signal IMG2and the image information signal IMG_IF. The resolution of the output image signal OIMG may be the same as the resolution of the input image signal IIMG, and the resolution of the output image signal OIMG may be adjusted after passing through the recomposition circuit124to be displayed on the display. The image processing engine122may perform image processing desired by the user.

The first gamma correction circuit127-5and the second gamma correction circuit127-6may serve to correct the overall luminance of the image to alleviate non-linear characteristics of hardware. That is, the first gamma correction circuit127-5and the second gamma correction circuit127-6may compensate for a loss related to luminance of the first image signal IMG1that may be caused by downscaling in the downscaling circuit125.

The resolution of the sixth image signal IMG6generated by the upscaler127-7may be the same as the resolution of the input image signal IIMG.

Referring toFIG.11, the high frequency decomposition circuit127-80may include a first Gaussian filter127-81, a second Gaussian filter127-82, and a plurality of differential circuits127-83,127-84, and127-85. The Gaussian filter is a filtering technique that uses a filter mask generated by approximating a Gaussian distribution function, and the Gaussian distribution may have a normal distribution shape, and the Gaussian filter may serve to remove noise. The sizes of the first Gaussian filter127-81and the second Gaussian filter127-82may be different. For example, the size of the first Gaussian filter127-81may be 5×5, and the size of the second Gaussian filter127-82may be 3×3.

The HF decomposition circuit127-80may generate Medium HF, Fine HF through the differential process of the seventh image signal IMG7and the seventh image signals IMG7that have passed through the first Gaussian filter127-81and the second Gaussian filter127-82as inFIG.11by the differential circuits127-84and127-85. Residual HF may be generated by differentiating the sixth image signal IMG6and the seventh image signal IMG7, and a high-frequency component Merged HF including correction information may be extracted through Fine HF, Medium HF, and Residual HF.

The positions of the upscaler127-7and the downscaling circuit125may be changed depending on the image signal processor120, an operating method of the image signal processor120, and a design in the application processor200, and the position of the downscaling circuit125may also be changed.

FIG.12is a flowchart illustrating a method of operating an image signal processor according to an example embodiment.

Referring toFIG.12, the image signal processor120ofFIG.1may receive an input image signal IIMG (S10). For example, the image sensor110ofFIG.1may generate an input image signal IIMG, and the input image signal IIMG may be input to the image signal processor120.

The image signal processor120may generate the first image signal IMG1by downscaling the input image signal IIMG (S20). For example, as a result of downscaling the input image signal IIMG, the data amount of the first image signal IMG1may be less than the data amount of the input image signal IIMG.

The image signal processor120may image-process the first image signal IMG1to generate a second image signal (S30). For example, the first image signal IMG1may be image-processed by the image processing engine122including a plurality of image processing modules.

The image signal processor120may generate the third image signal IMG3by upscaling the second image signal IMG2(S40). For example, by upscaling, the resolution of the third image signal IMG3may be greater than the resolution of the second image signal IMG2and may be the same as the resolution of the input image signal IIMG.

The image signal processor120may generate the fourth image signal IMG4by upscaling the first image signal IMG1(S50). For example, by upscaling, the resolution of the input image signal IIMG and the resolution of the fourth image signal IMG4may be the same.

The image signal processor120may generate an image information signal IMG_IF by extracting information on image quality deterioration of the third image signal IMG3from the input image signal IIMG (S60). For example, for the fourth image signal IMG4and the input image signal IIMG, by performing at least one of brightness enhancement, sharp enhancement, adaptive filtering, and noise reduction filtering, the image information signal IMG_IF may be generated by extracting information on image quality degradation of the third image signal IMG3. The generating of the image information signal IMG_IF by extracting the information on the image quality deterioration of the third image signal IMG3is not limited to the above processes, and may include a plurality of different processing operations for extracting image information related to image quality degradation occurring for various reasons in the scaling and image processing operations. In addition, the image information signal IMG_IF may include a high-frequency component of the raw image for correcting image quality deterioration caused by scaling and/or image processing.

The image signal processor120may generate the output image signal OIMG by reconstructing the third image signal IMG3and the image information signal IMG_IF (S70). For example, the generating of the output image signal OIMG may include performing at least one of radial correction and gain control processing on the image information signal IMG_IF.

Through the above series of operations, various image processing may be performed by minimizing power consumption, data band-width, and image quality degradation.

FIG.13is a flowchart illustrating a method of operating an image signal processor according to an example embodiment.

Referring toFIG.13, the image signal processor120ofFIG.1may receive a raw image signal RIMG (S11). For example, the image sensor110ofFIG.1may generate a raw image signal RIMG, and the raw image signal RIMG may be input to the image signal processor120.

The image signal processor120may convert the raw image signal RIMG into an input image signal IIMG having a color space different from that of the raw image signal RIMG (S12). For example, the input image signal IIMG generated by the transformation may have a larger amount of data than the raw image signal RIMG. In addition, the raw image signal RIMG may be a Bayer pattern, and the input image signal IIMG may be an RGB or YUV pattern.

The image signal processor120may generate the first image signal IMG1by downscaling the input image signal IIMG (S20). For example, as a result of downscaling the input image signal IIMG, the data amount of the first image signal IMG1may be less than the data amount of the input image signal IIMG.

The image signal processor120may image-process the first image signal IMG1to generate a second image signal (S30). For example, the first image signal IMG1may be image-processed by the image processing engine122including a plurality of image processing modules.

The image signal processor120may generate the third image signal IMG3by upscaling the second image signal IMG2(S40). For example, by upscaling, the resolution of the third image signal IMG3may be greater than the resolution of the second image signal IMG2and may be the same as the resolution of the input image signal IIMG.

The image signal processor120may generate the fourth image signal IMG4by upscaling the first image signal IMG1(S50). For example, by upscaling, the resolution of the input image signal IIMG and the resolution of the fourth image signal IMG4may be the same.

The image signal processor120may generate an image information signal IMG_IF by extracting information on image quality deterioration of the third image signal IMG3from the fourth image signal IMG4and the input image signal IIMG (S60). For example, for the fourth image signal IMG4and the input image signal IIMG, by performing at least one of brightness enhancement, sharp enhancement, adaptive filtering, and noise reduction filtering, the image information signal IMG_IF may be generated by extracting information on image quality degradation of the third image signal IMG3. The generating of the image information signal IMG_IF by extracting the information on the image quality deterioration of the third image signal IMG3is not limited to the above processes, and may include a plurality of different processing steps for extracting image information related to image quality degradation occurring for various reasons in the scaling and image processing steps in the image processing process. In addition, the image information signal IMG_IF may include a high-frequency component of the raw image for correcting image quality deterioration caused by scaling and/or image processing.

The image signal processor120may generate the output image signal OIMG by reconstructing the third image signal IMG3and the image information signal IMG_IF (S70). For example, the generating of the output image signal OIMG may include processing at least one of radial correction and gain control processing on the image information signal IMG_IF.

Through the above series of operations, various image processing may be performed by minimizing power consumption, data band-width, and image quality degradation.

FIG.14is a block diagram illustrating an application processor according to an example embodiment.

Referring toFIG.14, the application processor200may include a main processor210, a random access memory (RAM)220, a compression encoder230, an image signal processor120, a non-volatile memory interface250, a camera interface260, a memory interface270, and a display interface280. Each of the components210,220,230,120,250,260,270, and280of the application processor200may transmit and receive data to and from each other through a bus290.

The main processor210may control the overall operation of the application processor200. The main processor210may be implemented as, for example, a CPU, a microprocessor, and the like, and according to an example embodiment, the main processor210may be implemented as one computing component having two or more independent processors (or cores), that is, a multi-core processor. The main processor210may process or execute programs and/or data stored in the RAM220(or ROM).

The RAM220may temporarily store programs, data, and/or instructions. According to an example embodiment, the RAM220may be implemented as a dynamic RAM (DRAM) or a static RAM (SRAM). The RAM220may be input/output through the interfaces250,260,270, and280, or may temporarily store an image generated by the image signal processor120or the main processor210.

In an example embodiment, the application processor200may further include a read only memory (ROM). The ROM may store continuously used programs and/or data. The ROM may be implemented as an erasable programmable ROM (EPROM) or an electrically erasable programmable ROM (EEPROM).

The non-volatile memory interface250may interface data input from the non-volatile memory device255or data output to the non-volatile memory. The non-volatile memory device255may be implemented as, for example, a memory card (e.g., MMC, eMMC, SD, and micro SD).

The camera interface260may interface data (e.g., a raw image signal RIMG or an input image signal IIMG) input from the camera265located outside the application processor200. The camera265may generate data for an image captured by using a plurality of light sensing elements. The raw image signal RIMG received through the camera interface260may be provided to the image signal processor120or stored in the memory130through the memory interface270.

The memory interface270may interface data input from the memory130external to the application processor200or data output to the memory130. According to an example embodiment, the memory130may be implemented as a volatile memory such as DRAM or SRAM or a non-volatile memory such as ReRAM, PRAM or NAND flash.

The display interface280may interface data (e.g., an output image signal OIMG) output to the display device140. The display device140may output image data through a display such as a liquid-crystal display (LCD) or active matrix organic light emitting diode (AMOLED).

The compression encoder230may encode an image to output an encoded image, that is, a compressed image. The compression encoder230may encode the converted image output from the image signal processor120or the converted image stored in the memory130. In an example embodiment, the compression encoder230may be a JPEG module, and the JPEG module may output a JPEG format image. The JPEG format image may be stored in the non-volatile memory device255.

As performing image processing on an image, for example, a raw image signal RIMG or an input image signal IIMG, provided from a camera (or image sensor110) to generate an image-processed image signal, the image signal processor120may store the image-processed image in the memory130, or scale the converted image to provide the scaled image to the display device140.

FIG.15is a block diagram illustrating an image signal processor and a memory of an application processor according to an example embodiment.

The application processor200may include an image signal processor120and a memory160. The image signal processor120may include a decomposition circuit121, an image processing engine122, and a recomposition circuit124.

The image information signal IMG_IF generated by the decomposition circuit121included in the image signal processor120may be stored in the memory160located outside the image signal processor120, and may be transmitted to the recomposition circuit124according to the timing at which the second image signal IMG2is input to the recomposition circuit124.

For example, the memory160may be a volatile memory such as a dynamic random access memory (DRAM) or a static RAM (SRAM), or a non-volatile memory such as a phase change RAM (PRAM), a resistive RAM (ReRAM), or a flash memory.

FIG.16is a block diagram illustrating an image sensor including an image signal processor according to an example embodiment.

The image sensor110may convert an optical signal of an object incident through the optical lens LS into image data. The image sensor110may be mounted on an electronic device having an image or light sensing function. For example, the image sensor110may be mounted on the electronic device such as digital still cameras, digital video cameras, smartphones, wearable devices, Internet of Things (IoT) devices, tablet Personal Computers (PCs), Personal Digital Assistants (PDAs), Portable Multimedia Players (PMPs), navigation devices, and the like. In addition, the image sensor110may be mounted on an electronic device provided as a component for vehicles, furniture, manufacturing facilities, doors, various measuring devices, and the like.

Referring toFIG.16, the image sensor110may include a pixel array10, a readout circuit11, and an image signal processor120. In an example embodiment, the pixel array10, the readout circuit11, and the image signal processor120may be implemented as a single semiconductor chip or semiconductor module. In an example embodiment, the pixel array10and the readout circuit11may be implemented as one semiconductor chip, and the image signal processor120may be implemented as another semiconductor chip.

The pixel array10, for example, may be implemented with a photoelectric conversion element such as a Charge Coupled Device (CCD) or a Complementary Metal Oxide Semiconductor (CMOS), and may be implemented with various types of photoelectric conversion devices. The pixel array10includes a plurality of sensing pixels PXs that convert a received optical signal (light) into an electrical signal, and the plurality of sensing pixels PXs may be arranged in a matrix. Each of the plurality of sensing pixels PXs includes a light sensing element. For example, the light sensing element may include a photo diode, an organic photo diode, a photo transistor, a port gate, or a pinned photo diode.

The readout circuit11may convert electrical signals received from the pixel array10into image data. The readout circuit11may amplify electrical signals and analog-digitize the amplified electrical signals. The image data generated by the readout circuit11may include a plurality of pixels corresponding to the plurality of sensing pixels PXs of the pixel array10. Here, the sensing pixels PXs of the pixel array10are physical structures that generate a signal according to the received light, and the pixels included in the image data represent data corresponding to the sensing pixels PXs. The readout circuit11may constitute a sensing core together with the pixel array10.

The data conversion unit150ofFIG.7may be added between the readout circuit11and the image signal processor120. The data conversion unit150may convert the raw image signal RIMG output from the readout circuit11into an input image signal IIMG having the same resolution as the raw image signal RIMG and having a larger amount of data. Alternatively, the data conversion unit150may convert the raw image signal RIMG into an input image signal IIMG having a different color space. For example, the raw image signal RIMG may have a Bayer pattern, and the input image signal IIMG may have an RGB or YUV pattern.

The image signal processor120may perform image processing on image data output from the readout circuit11, that is, raw image data. For example, the image signal processor120may perform image processing such as bad pixel correction, remosaic, and noise removal on the image data.

To minimize image quality deterioration due to scaling and image processing that may occur during the image processing process, the image signal processor120may include a decomposition circuit121, an image processing engine122, and a recomposition circuit124as shown inFIG.2. The decomposition circuit121may include a downscaling circuit125, a second upscaling circuit126, and a correction information generating circuit127as shown inFIG.7. The correction information generating circuit127may generate an image information signal IMG_IF including information capable of correcting image quality deterioration. The recomposition circuit124may generate an output image signal OIMG with reduced image quality by reconstructing the third image signal IMG3whose image quality is lower than that of the input image signal IIMG through the image information signal IMG_IF.

FIG.17is a block diagram illustrating a portable terminal including an image signal processor according to an example embodiment.

Referring toFIG.16, the portable terminal1000according to the example embodiment may include an image processing system100, a wireless transmission/reception unit1200, an audio processing unit1300, a non-volatile memory device1500, a user interface1600, and a controller1700.

The image processing system100may include a lens1110, an image sensor110, a display device140, a memory130, and an image signal processor120. As shown in the example embodiment, the image signal processor120may be implemented as a part of the controller1700.

The image signal processor120may generate a converted image by performing image processing on an image provided from the image sensor110, for example, an input image signal IIMG (or a raw image signal RIMG), and in this case, minimize power consumption, data band width, and deterioration of image quality in the image processing process according to embodiments of the disclosure. In addition, the converted image according to the image processing of embodiments of the disclosure may be stored in the memory130, or the converted image may be scaled to provide the scaled image to the display device140.

The wireless transmission/reception unit1200includes an antenna1210, a transceiver1220, and a modem1230. The audio processing unit1300may include an audio processor1310, a microphone1320, and a speaker1330. The non-volatile memory device1500may be provided as a memory card (e.g., MMC, eMMC, SD, and micro SD) or the like.

The user interface1600may be implemented with various devices capable of receiving user input, such as a keyboard, a curtain key panel, a touch panel, a fingerprint sensor, and a microphone. The user interface1600may receive a user input and provide a signal corresponding to the received user input to the controller1700.

The controller1700may control the overall operation of the portable terminal1000and may be provided as a system-on-chip (SoC) that drives an application program, an operating system, and the like. A kernel of an operating system driven in the SoC may include an I/O scheduler and a device driver for controlling the nonvolatile memory device1500.

While example embodiments of the disclosure

been particularly shown and described, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the following claims.