Image evaluation method that can quantify images distorted by artifacts, computer program performing the method, and computing device

An image evaluation method, including: obtaining a test image including a first lattice pattern formed by image edges; aligning the test image using the image edges to generate an aligned image including a second lattice pattern formed by aligned image edges; generating a compressed image by compressing the aligned image; and generating a quantified result by quantifying a per-pixel difference between the compressed image and the aligned image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. § 119 to Korean Patent Application No. 10-2021-0113481 filed on Aug. 26, 2021, and Korean Patent Application No. 10-2022-0002228 filed on Jan. 6, 2022, in the Korean Intellectual Property Office, the disclosures of which are incorporated by reference herein in their entireties.

BACKGROUND

The disclosure relates to a technology for optimizing parameters of an image signal processor by using image evaluation and an evaluation result, and more particularly, to an image evaluation method, a computer program, and a computing device capable of quantifying distortion of an image due to at least one of noise and artifacts, and optimizing the parameters of the image signal processor by using a quantified result.

2. Description of Related Art

An image processing device includes a complementary metal-oxide-semiconductor (CMOS) substrate on which light receiving elements (e.g., photodiodes) are formed, and a color filter array formed on the CMOS substrate.

The image processing device generates a color image by processing incomplete color image data corresponding to output signals of the light receiving elements that receive color signals passing through the color filter array.

In this case, the image processing device uses an image signal processor (ISP) for the purpose of generating a color image by processing incomplete color image data. However, when the ISP processes a high-frequency image (e.g., an image containing image edges whose brightness changes sharply or an image containing complex patterns), unexpected artifacts may be generated, thereby causing the degradation of quality of the image processed by the ISP.

SUMMARY

Provided are an image evaluation method capable of extracting an artifact-free reference image from a distorted image including a lattice pattern and quantifying (or express the quantify of) the distortion from a difference between the extracted reference image and the image, for the purpose of providing a reference making it possible to determine how well an image restored through the process of restoring an image is restored, a computer program performing the method, and a computing device executing the computer program.

In accordance with an aspect of the disclosure, an image evaluation method includes obtaining a test image including a first lattice pattern formed by image edges; aligning the test image using the image edges to generate an aligned image including a second lattice pattern formed by aligned image edges; generating a compressed image by compressing the aligned image; and generating a quantified result by quantifying a per-pixel difference between the compressed image and the aligned image.

In accordance with an aspect of the disclosure, a computer-readable storage medium is configured to store instructions which, when executed by at least one processor, cause the at least one processor to: obtain a test image including a first lattice pattern formed by image edges; align the test image using the image edges to generate an aligned image including a second lattice pattern formed by aligned image edges: generate a compressed image by compressing the aligned image; and generate a quantified result by quantifying a per-pixel difference between the compressed image and the aligned image.

In accordance with an aspect of the disclosure, an image evaluation method includes obtaining a test image including a first lattice pattern formed by image edges, noise, and artifacts; aligning the test image using the image edges to generate an aligned image including a second lattice pattern formed by aligned image edges; removing the noise and the artifacts from the aligned image to generate a reference image including the second lattice pattern; and generating a quantified result by quantifying a per-pixel difference between the reference image and the aligned image.

In accordance with an aspect of the disclosure, a computer-readable storage medium is configured to store instructions which, when executed by at least one processor, cause the at least one processor to: obtain a test image including a first lattice pattern formed by image edges, noise, and artifacts; align the test image using the image edges to generate an aligned image including a second lattice pattern formed by aligned image edges; remove the noise and the artifacts from the aligned image to generate a reference image including the second lattice pattern; and generate a quantified result by quantifying a per-pixel difference between the reference image and the aligned image.

In accordance with an aspect of the disclosure, a computing device includes a memory device configured to store a test image including a first lattice pattern formed by image edges, noise, and artifacts; and a processor configured to evaluate the test image output from the memory device, wherein the processor is further configured to: align the test image using the image edges to generate an aligned image including a second lattice pattern formed by aligned image edges; remove the noise and the artifacts from the aligned image to generate a reference image including the second lattice pattern; and quantify a per-pixel difference between the reference image and the aligned image to generate a quantified result.

In accordance with an aspect of the disclosure, a computing device includes a memory device configured to store a test image including a first lattice pattern formed by image edges, wherein the test image is generated by an image signal processor based on first parameters; and at least one processor configured to: align the image edges to obtain aligned image edges; generate an aligned image including a second lattice pattern formed by the aligned image edges; generate a reference image by compressing the aligned image; generate second parameters based on a difference between the reference image and the aligned image.

DETAILED DESCRIPTION

FIG.1is a block diagram of a computing device executing a computer program capable of performing an image evaluation method according to an embodiment of the present disclosure. Referring toFIG.1, a computing device100includes an image sensor110, an image signal processor (ISP)120, a memory device130, a processor140, and a display device160. In embodiments, one or more of the elements of the computing device100may be for example included in, or referred to as, a camera module.

The computing device100may be a personal computer (PC) or a mobile device, and the mobile device may be a smartphone, a laptop computer, a mobile Internet device (MID), a wearable computing device, a web server, or an image processing device including ISP120.

The processor140executes an image evaluation computer program150. The processor140may be a central processing unit (CPU) or an application processor. The image evaluation computer program150performs operations to be described with reference toFIGS.2to5for the purpose of evaluating or quantifying the performance of the ISP120.

According to embodiments, the processor140may further execute an ISP tuning computer program155. The ISP tuning computer program155performs a function of tuning the ISP120by using first parameters PMT, which may be for example pre-tuning parameters, a function of generating second parameters PMT2by using a quantified result value IDV generated by the image evaluation computer program150, and a function of tuning the ISP120again by using the second parameters PMT2. The first parameters PMT may be changed to or updated with the second parameters PMT2by the processor140.

Parameters of the ISP120may be optimized by the image evaluation computer program150and the ISP tuning computer program155.

According to embodiments, the ISP tuning computer program155may set or program the first parameters PMT or the second parameters PMT2to (or in) the ISP120by using setting values STV input from the outside or the quantified result value IDV generated by the image evaluation computer program150.

The first parameters PMT may include at least two parameters of the following: a parameter for Bayer transformation, a parameter for demosaicing, a parameter for noise reduction, a parameter for image sharpening, a parameter for image blurting, a parameter for dead pixel correction, a parameter for black level compensation, a parameter for lens shading correction, a parameter for anti-aliasing noise filtering, a parameter for auto white balance (AWB) gain control, a parameter for gamma correction, a parameter for edge enhancement, a parameter for false color suppression, a parameter for hue/saturation control, and a parameter for brightness and contrast control.

Also, the second parameters PMT2may include at least two of the parameters listed above.

The image evaluation computer program150may be recorded on or stored in a recording medium or storage medium readable by a computing device and may be connected with hardware (e.g., the processor140) to evaluate the performance of the ISP120or quantify or express the quantity of distortion included in an image distorted by artifacts. In embodiments, the recording medium may be a non-transitory recording medium.

The ISP tuning computer program155is recorded on a recording medium readable by the computing device and is connected with the hardware to tune, set or optimize the parameters of the ISP120. In embodiments the recording medium may be a non-transitory recording medium.

An example in which the image evaluation computer program150and the ISP tuning computer program155are independent computer programs is illustrated inFIG.1, but the image evaluation computer program150performing an image evaluation function or a function of quantifying image distortion, and the ISP tuning computer program155performing a function of tuning the parameters of the ISP120, may be implemented by one computer program.

The image sensor110generates the raw data RDATA or RDATA2associated with an object, which may be for example a subject for photography. The image sensor110may be implemented with a complementary metal-oxide semiconductor (CMOS) image sensor.

Before tuning, the ISP120converts the first raw data RDATA output from the image sensor110into the test image TIM by using the first parameters PMT.

After the second parameters PMT2corresponding to the quantified result value IDV are set in the ISP120, the image sensor110generates the second raw data RDATA2associated with the object. The ISP120converts the second raw data RDATA2output from the image sensor110into a second test image TIM2by using the second parameters PMT2.

According to embodiments, the test image TIM or TIM2may be stored in the memory device130and may then be transferred to the processor140. In embodiments, the test image TIM or TIM2may be directly transferred from the ISP120to the processor140.

The display device160may display at least one of the test image TIM, a difference image DIFF, and the quantified result value IDV under control of the processor140or the image evaluation computer program150. The quantified result value IDV may be a scalar value.

The display device160may be a light emitting diode (LED) display device, an organic light emitting diode (OLED) display device, or an active matrix OLED display device (AMOLED) display device.

FIG.2is a flowchart describing an image evaluation method performed by an image evaluation computer program running on a computing device illustrated inFIG.1, andFIG.3is a conceptual diagram describing an image evaluation method according to an embodiment of the present disclosure.

Before tuning, the ISP120converts the first raw data RDATA output from the image sensor110into the test image TIM by using the first parameters PMT.

Referring toFIGS.1and2, and section (a) ofFIG.3, in operation S110, the image evaluation computer program150receives the est image TIM, which may be for example a chessboard image, from the memory device130or the ISP120. As shown in section (a) ofFIG.3, the test image TIM may include a first lattice pattern LP1, in which first lattices LT1formed by image edges are included. Referring toFIGS.1and2, and section (b) ofFIG.3, image evaluation computer program150accurately aligns a portion or the whole of the test image TIM by using the image edges of the first lattice pattern LP1included in the test image TIM, and generates an aligned image AIM including a second lattice pattern LP2including second lattices LT2depending on a result of the alignment.

In embodiments, an image edge, which may be referred to as a boundary line, may refer to a point or a boundary line at which image brightness of a digital image sharply changes, or a point or a boundary line having discontinuities. An oval EDG included inFIG.3illustrates example image edges EG included in each of images TIM, AIM, and CIM.

According to embodiments, the first lattice pattern LP1may refer to a lattice pattern before alignment, and the second lattice pattern LP2may refer to a lattice pattern after alignment.

The image evaluation computer program150aligns the image edges (e.g., TXEG and TYEG as shown in section (a) ofFIG.4) included in the first lattice pattern. LPI skewed on the test image TIM of section (a) ofFIG.3such that the image edges are horizontal and vertical to each other as illustrated in section (b) ofFIG.3, and generates the aligned image AIM including the second lattice pattern LP2in which the second lattices LT2formed by aligned image edges XEG and YEG are included. The aligned image AIM may correspond to a portion or the whole of the test image TIM.

When viewed in an X-axis direction, which may be a horizontal direction, and a Y-axis direction, which may be a vertical direction, because the first lattice pattern LP1included in the test image TIM illustrated in section (a) ofFIG.3and section (a) ofFIG.4is skewed, as illustrated in section (a) ofFIG.3, the first lattices LT1may be different in size and shape.

For example, when the image sensor110photographing the object (e.g., a chessboard) including the first lattice pattern LP1is not parallel to the object, the first lattice pattern LP1of the test image TIM generated by the image sensor110may be skewed.

However, when viewed in the X-axis direction (or horizontal direction) and the Y-axis direction (or vertical direction), because the second lattice pattern LP2of the aligned image AIM illustrated in section (b) ofFIG.3is not skewed, the second lattices LT2illustrated in section (b) ofFIG.3are identical in size and shape. The second lattice pattern LP2is a lattice pattern including image edges detected from the first lattice pattern LP1through section (b) ofFIG.4to section (i) ofFIG.4.

As illustrated in section (b) ofFIG.3, the second lattice pattern LP2may include a plurality of second lattices LT2, the first image edges XEG arranged in the X-axis direction, and the second image edges YEG arranged in the Y-axis direction, and the first image edges XEG and the second image edges YEG are perpendicular to each other.

For example, the second lattice LT2formed by two corresponding image edges of the first image edges XEG and two corresponding edges of the second image edges YEG is in the shape of a quadrangle. In embodiments, a lattice may be referred to as a cell, a mesh, or a grid.

The image evaluation computer program150may accurately align the test image TIM including the first lattice pattern LP1by using Radon transform and may generate the aligned image AIM including the second lattice pattern LP2formed as a result of the alignment.

The image evaluation computer program150may extract a quadrilateral region of interest ROI from the test image TIM, may warp the extracted quadrilateral ROI, and may generate the aligned image AIM including the second lattice pattern LP2depending on a result of the warping. The warping is may be referred to as image warping.

FIG.4is a diagram describing a method for generating an aligned image according to an embodiment of the present disclosure.

The alignment may refer to a process of generating the aligned image AIM illustrated in section (i) ofFIG.4from the test image TIM illustrated in section (a) ofFIG.4, and warping may refer to two processes including a process of obtaining four points PT1, PT2, PT3, and PT4illustrated in section (h) ofFIG.4and a process of generating the aligned image AIM illustrated in section (i) ofFIG.4.

Referring toFIGS.1to4, the image evaluation computer program150detects the X-axis direction image edges TXEG and Y-axis direction image edges TYEG from the test image TIM including the first lattice pattern LP1illustrated in section (a) ofFIG.4by using at least one edge detection algorithm, which may be at least one of a plurality of edge detection algorithms. For example, the image evaluation computer program150detects image edges by performing edge filtering in the X-axis direction and edge filtering in the Y-axis direction on the test image TIM.

The edge filtering or the edge detection algorithm detects the image edges by determining whether the brightness and/or the intensity of an image changes sharply or dramatically.

Examples of edge detection algorithms include a Canny edge detector, a Sobel operator, a Laplace operator, a Prewitt operator, a Scharr operator, and the like, but are not limited thereto.

For example, the image evaluation computer program150may detect X-axis direction image edges and Y-axis direction image edges of each of the first lattices LT1included in the first lattice pattern LP1by applying the Sobel operator in each of the X-axis direction and the Y-axis direction. The Sobel operator is may be referred to as a Sobel-Feldman operator or a Sobel filter.

According to embodiments, section (b) ofFIG.4illustrates an example image EFX including image edges detected in the X-axis direction with respect to the test image TIM by using an edge detection algorithm (e.g., the Sobel operator), and section (e) ofFIG.4illustrates an example image EFY including image edges detected in the Y-axis direction with respect to the test image TIM by using the edge detection algorithm.

According to embodiments, section (c) ofFIG.4illustrates an example X-axis direction sinogram XSI, and section (f) illustrates an example Y-axis direction sinogram YSI.

In embodiments, the image evaluation computer program150may create or calculate the X-axis direction sinogram XSI, which may correspond to the image EFX edge-filtered in the X-axis direction (as shown in section (b) ofFIG.4), and may create or calculate the Y-axis direction sinogram YSI, which may correspond to the image EFY edge-filtered in the Y-axis direction (as shown in section (e) ofFIG.4).

For example, the image evaluation computer program150that uses the Radon transform creates the X-axis direction sinogram XSI illustrated in section (c) ofFIG.4from the image EFX edge-filtered in the X-axis direction and creates the Y-axis direction sinogram YSI illustrated in section (f) ofFIG.4from the image EFY edge-filtered in the Y-axis direction.

The image evaluation computer program150filters the X-axis direction sinogram XSI to generate a first filtered sinogram FSX illustrated in section (d) ofFIG.4and filters the Y-axis direction sinogram YSI to generate a second filtered sinogram FSY illustrated in section (g) ofFIG.4.

For example, the image evaluation computer program150creates the first filtered sinogram FSX by extracting points, each of which has a high intensity, from among points arranged in each row of the X-axis direction sinogram XSI and creates the second filtered sinogram FSY by extracting points, each of which has a high intensity, from among points arranged in each row of the Y-axis direction sinogram YSI. In embodiments, a high intensity may refer to, for example, a highest intensity from among intensities, or a relatively high intensity in comparison with other intensities.

Section (h) ofFIG.4illustrates an image DEIM including image edges detected from the test image TIM by using an edge detection algorithm. The image evaluation computer program150generates the image DEIM including the image edges detected from the test image TIM by using the first filtered sinogram FSX illustrated in section (d) ofFIG.4and the second filtered sinogram FSY illustrated in section (g) ofFIG.4.

The image evaluation computer program150generates the image DEIM including the four points PT1, PT2, PT3, and PT4by using two points PX1and PX2of points included in the first filtered sinogram FSX and two points PY1and PY2included in the second filtered sinogram FSY.

For example, the image evaluation computer program150determines the first point PT1by using the two points PX1and PY1, determines the second point PT2by using the two points PX1and PY2, determines the third point PT3by using the two points PX2and PY2, and determines the fourth point PT4by using the two points PX2and PY1.

The image evaluation computer program150warps the image DEIM including the detected image edges by using the four points PT1, PT2, PT3, and PT4and generates the aligned image AIM including the four points PT1, PT2, PT3, and PT4depending on a result of the warping.

For example, referring to section (b) ofFIG.3, to generate the aligned image AIM including the second lattice pattern LP2composed of 8×8 lattices LT2, the image evaluation computer program150may select the two first points PX1and PX2from a first group of points of the first filtered sinogram FSX, may select the two second points PY1and PY2from a second group of points of the second filtered sinogram FSY, and may generate the image DEIM including the four points PT1, PT2, PT3, and PT4selected.

For example, each point included in the first filtered sinogram FSX illustrated in section (d) ofFIG.4and each point included in the second filtered sinogram FSY illustrated in section (g) ofFIG.4may correspond to image edges in the actual image DEIM or AIM, respectively.

For example, when the first point PX1is the lowest point of a first group of points, the first point PX2is the highest point of the first group of points, the second point PYI is the lowest point of a second group of points, the second point PY2is the highest point of the second group of points, then the image DEM including the four points PT1, PT2, PT3, and PT4has the largest size compared to images each including four different points.

A size of a quadrilateral ROI targeted for warping is determined depending on whether the image evaluation computer program150selects any two first points from a first group of points and selects any two second points from a second group of points.

For example, based on the image evaluation computer program150selecting two first points from the remaining points other than the lowest point and the highest point of a first group of points, and selecting two second points from the remaining points other than the lowest point and the highest point of a second group of points, the image evaluation computer program150extracts a quadrilateral ROI corresponding to the four points thus selected, and warps the extracted quadrilateral ROI to generate the aligned image AIM.

Returning toFIG.3, in operation S120, the image evaluation computer program150compresses the image AIM aligned through operation S110by using truncated singular value decomposition and generates the compressed image CIM including the second lattice pattern LP2formed by the second lattices LT2illustrated in section (c) ofFIG.3.

Each of the test image TIM and the aligned image AIM may be an image distorted by noise and artifacts, but the compressed image CIM may be an image which does not include the noise and artifacts, that is, a distortion-free image.

Image distortion includes distortion in which a shape (e.g., a lattice pattern) changes depending on an image viewing angle, and distortion in which an outline, a contour, or a boundary line of an image edge or a two-dimensional point (or an apex in a three-dimensional image) is unclear due to imperfect performance of the ISP120.

In operation S130, the image evaluation computer program150quantifies a difference between the compressed image CIM illustrated in section (c) ofFIG.3and the aligned image AIM illustrated in section (b) ofFIG.3in units of a pixel and generates the difference image DIFF and the quantified result value IDV depending on a quantified result. In embodiments, quantifying a difference in units of a pixel may mean determining or calculating a difference for each pixel, or for example determining or calculating a per-pixel difference.

FIGS.5A and5Bare a conceptual diagrams illustrating a method for quantifying a difference between a compressed image and an aligned image for each pixel.

In embodiments, an aligned image AIMa illustrated inFIG.5Amay be an image including 4×4 pixels and a compressed image CIMa illustrated inFIG.5Bmay be an image including 4×4 pixels. The aligned image AIMa including the 4×4 pixels may correspond to a portion of the aligned image AIM illustrated in section (b) ofFIG.3, and the compressed image CIMa including the 4×4 pixels may correspond to a portion of the compressed image CIM illustrated in section (c) ofFIG.3.

The image evaluation computer program150calculates a difference between pixel values of pixels targeted for comparison in units of a pixel.

A pixel is the smallest element of an image. Each pixel corresponds to one pixel value.

For example, in an 8-bit gray scale image, a pixel value (i.e., a value of a pixel) ranges in value from 0 to 255. A pixel value at each point corresponds to the intensity of light photons striking each point. Each pixel has a value that is proportional to a light intensity measured or received at a corresponding location.

In embodiments, a pixel value is not limited to the above definition, and may be defined in a different manner. However, for convenience of description, each of the images CIM, CIMa, AIM, and AIMa may be described herein as an 8-bit gray scale image.

As illustrated inFIGS.5A and5Bas an example, the image evaluation computer program150calculates differences between pixel values of the 16 pixels included in the compressed image CIM and pixel values of the 16 pixels included in the aligned image AIMa in a pixelwise manner, and thus, 16 difference values are generated.

For example, the image evaluation computer program150calculates a difference between a pixel value VP2_11of a pixel P2_11and a pixel value VP1_11of a pixel P1_11, a difference between a pixel value VP2_14of a pixel P2_14and a pixel value VP1_14of a pixel P1_14, a difference between a pixel value VP2_41of a pixel P2_41and a pixel value VP1_41of a pixel P1_41, and a difference between a pixel value VP2_44of a pixel P2_44and a pixel value VP1_44of a pixel P1_44.

For convenience of description, the method for calculating 4 differences is described above as an example, but a method for calculating 12 differences, which is not described, may be sufficiently understood from the method for calculating 4 differences.

The image evaluation computer program150calculates the differences between pixel values of pixels included in the compressed image CIM and pixel values of pixels included in the aligned image AIM in units of a pixel and generates the difference image DIFF by using the differences.

Referring to section (d) ofFIG.3, the image evaluation computer program150generates the difference image DIFF including the differences calculated in units of a pixel.

The difference image DIFF includes bright points BR and dark points DK.

Each of the bright points BR indicates that a calculated difference between corresponding pixel values is relatively large, and each of the dark points DK indicates that a calculated difference between corresponding pixel values is relatively small.

The difference image MIT illustrated in section (d) ofFIG.3is expressed by using a black and white image, but in embodiments the image evaluation computer program150may express the difference image DIFF by using a color image.

When the image evaluation computer program150expresses the difference image DIFF by using a color image, the image evaluation computer program150may increase brightness of a corresponding point as a difference between two corresponding pixel values increases and may decrease brightness of a corresponding point as a difference between two corresponding pixel values decreases.

For example, as the brightness of a bright point BR becomes brighter, the degree of image distortion corresponding to the bright point BR becomes larger; as the brightness of a dark point DK becomes darker, the degree of image distortion corresponding to the dark point DK becomes smaller.

The image evaluation computer program150calculates the differences between the pixel values of the pixels included in the compressed image CIM and the pixel values of the pixels included in the aligned image AIM in units of a pixel and generates the quantified result value IDV by using the differences.

Referring to section (e) ofFIG.3, the image evaluation computer program150generates the quantified result value IDV, which may be referred to as an image distance, by using the differences calculated in units of a pixel.

For example, the image evaluation computer program150generates the quantified result value IDV by quantifying the differences between the pixel values of the pixels included in the compressed image CIM and the pixel values of the pixels included in the aligned image AIM in units of a pixel by using the L1 norm, L2 norm, or structural similarity index measure (SSIM).

The L1 norm may also be referred to as the Manhattan distance or Taxicab geometry, and the L2 norm may also be referred to as the Euclidean distance.

That is, the image evaluation computer program150may generate the quantified result value IDV having a scalar value by using the L1 norm, L2 norm, or SSIM.

In operation S130, the image evaluation computer program150may generate at least one of the difference image DIFF and the quantified result value IDV based on a difference between two corresponding pixel values, as described with reference toFIGS.5A and5B.

The image evaluation computer program150may send at least one of the test image TIM, the difference image DIFF, and the quantified result value IDV to the display device160, and the display device160may display at least one of the test image TIM, the difference image DIFF, and the quantified result value IDV. Accordingly, the user of the computing device100may visually see at least one of the test image TIM, the difference image DIFF, and the quantified result value IDV through the display device160.

The ISP tuning computer program155may receive the quantified result value IDV generated by the image evaluation computer program150, may generate the second parameters PMT2by using the quantified result value IDV, and may tune the ISP120by using the second parameters PMT2. Accordingly, the first parameters PMT set in the ISP120are updated with the second parameters PMT2.

After the second parameters PMT2are set in the ISP120(or are programmed or updated in the ISP120), the image sensor110photographs an object to generate the second raw data RDATA2.

The ISP120receives the second raw data RDATA2, processes the second raw data RDATA2by using the second parameters PMT2, and generates the second test image TIM2corresponding to a processing result. The second test image TIM2may be sent to the memory device130or the image evaluation computer program150.

The image evaluation computer program150generates a difference image and/or a quantified result value from the second test image TIM2using the method described with reference toFIGS.1to5B. The ISP tuning computer program155may generate third parameters by using the quantified result value and may update the second parameters PMT2set in the ISP120with the third parameters.

For example, the image evaluation computer program150may generate the aligned image AIM including the second lattice pattern LP2formed by the aligned image edges XEG and YEG by aligning the test image TIM including the first lattice pattern LP1including the first lattices LT1formed by the image edges TXEG and TYEG, noise, and artifacts in operation S110, may generate the reference image, which may correspond to the compressed image CLM, including only the second lattice pattern LP2by removing the noise and artifacts among the noise, the artifacts, and the second lattice pattern LP2included in the aligned image AIM in operation S120, and may generate a quantified result (e.g., the difference image DIFF and/or the quantified result value IDV) by quantifying differences of the reference image and the aligned image AIM in units of a pixel in operation S130.

In other words, the image evaluation computer program150may generate the aligned image AIM by using one source image TIM, may generate the compressed image by compressing the aligned image AIM, and may generate a quantified result (e.g., the difference image DIFF and/or the quantified result value IDV) by quantifying differences of the reference image and the aligned image AIM in units of a pixel by using the compressed image CLM as a reference image from which the image distortion is absent. Accordingly, the image evaluation computer program150may quantify the degree of image distortion of the test image TIM and may generate a quantified result (e.g., the difference image DIFF and/or the quantified result value IDV).

FIG.6is a flowchart describing an image evaluation method and a method for tuning parameters of an image processing processor by using the image evaluation method, according to an embodiment of the present disclosure.

Referring toFIGS.1to6, in operation S210, the image evaluation computer program150tunes the ISP120by using the first parameters PMT before testing the performance of the ISP120.

In operation S220, the ISP120receives the first raw data RDATA from the image sensor110and converts the first raw data RDATA into the test image TIM by using the first parameters PMT.

In operation S230, the image evaluation computer program150quantifies a difference between the compressed image CIM and the aligned image AIM in units of a pixel and may generate a quantified result (e.g., the difference image DIFF and/or the quantified result value IDV).

According to embodiments, the user of the computing device100may visually check the difference image DIFF and/or the quantified result value IDV displayed through the display device160and may decide whether to perform additional ISP tuning.

When the user of the computing device100inputs setting values STV indicating additional tuning for the ISP120to the ISP tuning computer program155by using an input device (e.g., a keyboard or a touch screen) (Yes in operation S240), in operation S210, the ISP tuning computer program155may generate the second parameters PMT2corresponding to the setting values STV and may tune the ISP120by using the second parameters PMT2.

When the user of the computing device100inputs the setting values STV, which indicate that the additional tuning for the ISP120is not required, to the ISP tuning computer program155by using the input device (No in operation S240), the ISP tuning computer program155may terminate the method in response to the setting values STV.

According to embodiments, the ISP tuning computer program155may compare the quantified result value DV output from the image evaluation computer program150with a reference value in operation S240; depending on a comparison result (Yes in operation S240or No in operation S240), the ISP tuning computer program155may continue to perform the tuning on the ISP120in operation S210or may terminate the method.

According to embodiments, the ISP tuning computer program155may be programmed to generate the second parameters PMT2corresponding to the quantified result value IDV when the quantified result value IDV is greater than the reference value.

However, according to embodiments, the ISP tuning computer program155may be programmed to generate the second parameters PMT2corresponding to the quantified result value IDV when the quantified result value IDV is smaller than the reference value.

In operation S240, the ISP tuning computer program155determines whether to perform additional tuning on the ISP120by using the setting values STV input by the user or the quantified result value IDV output from the image evaluation computer program150.

When the additional tuning is required (Yes in operation S240), in operation S210, the ISP tuning computer program155generates the second parameters PMT2by using the setting values STV or the quantified result value IDV and tunes the ISP120by using the second parameters PMT2.

However, when the additional tuning is not required (No in operation S240), the ISP tuning computer program155may terminate the method.

After the ISP120is tuned based on the second parameters PMT2, in operation S220, the ISP120receives the second raw data RDATA2from the image sensor110and converts the second raw data RDATA2into the second test image TIM2by using the second raw data RDATA2.

In operation S230, the image evaluation computer program150generates an aligned image from the second test image TIM2as described with reference to (b) ofFIG.3, generates a compressed image from the aligned image as described with reference to (c) ofFIG.3, quantifies a difference between the compressed image and the aligned image in units of a pixel as described with reference to section (d) and section (e) ofFIG.3, and generates a difference image and a quantified result value corresponding to a quantifying result.

In operation S240, the image evaluation computer program150determines whether to perform additional tuning for the ISP120, for example by determining whether the additional tuning is required or otherwise desired. For example, in embodiments the image evaluation computer program150may determine whether to perform additional tuning by comparing a difference corresponding to the difference image to a threshold difference, or by comparing the quantified result value to a threshold value. When the image evaluation computer program150determines to perform additional tuning for the ISP120(Yes in operation S240), in operation S210, the ISP tuning computer program155receives a quantified result value from the image evaluation computer program150, generates third parameters by using the quantified result, value, and again tunes the ISP120by using the third parameters. When the image evaluation computer program150determines not to perform the additional tuning (No in operation S240), the ISP tuning computer program155may terminate the method.

Operation S210to operation S240ofFIG.6may be repeatedly performed until it is determined not, to additionally tune the ISP120, for example until there is no need to additionally tune the ISP120, or for example until optimal parameters are set in the ISP120.

FIGS.7to10are block diagrams of image evaluation systems, according to embodiments. In embodiments, one or more of the ISP340, the ISP341, and the ISP520may correspond to the ISP120discussed above. In embodiments, the display device350may correspond to the display device160discussed above. In embodiments, the image sensor310may correspond to the image sensor110discussed above. In embodiments, one or more of the processor320, the processor321, and the processor530may correspond to the processor140discussed above. In embodiments, the memory device305may correspond to the memory device130discussed above discussed above.

FIG.7is a block diagram of an image evaluation system in which parameters corresponding to a quantified result and raw data are exchanged over an Internet, according to an embodiment of the present disclosure.

Referring toFIG.7, an image evaluation system200A includes a computing device300A and a server500A communicating with each other over a communication network. The server500A may be a computing device that executes the image evaluation computer program150performing an image evaluation operation, and the ISP tuning computer program155generating the parameters PMT2for tuning an ISP340included in the computing device300A.

The communication network to be describedFIGS.7to10may be an Internet or a Wi-Fi network.

The raw data RDATA output from image sensor310, which may be for example a digital camera, are sent to a communication device510of the server500A over a communication device330and the communication network, under control of a processor320.

An ISP520of the server500A may be tuned by parameters, and the ISP520converts the raw data RDATA from the computing device300A into the test image TIM by using the parameters.

The image evaluation computer program150executed by a processor530of the server500A generates the quantified result value IDV. The ISP tuning computer program155generates the parameters PMT2by using the quantified result value IDV generated by the image evaluation computer program150and sends the parameters PMT2to the communication device330of the computing device300A using a communication device510and the communication network. The processor320of the computing device300A tunes the ISP340of the computing device300A by using the parameters PMT2received through the communication device330. The ISP340tuned by the parameters PMT2may convert raw data output from the digital camera310into an image by using the parameters PMT2.

FIG.8is a block diagram of an image evaluation system in which raw data and a quantified result are exchanged over an Internet, according to an embodiment of the present disclosure.

Referring toFIG.8, an image evaluation system200B includes a computing device300B and a server500B communicating with each other over the communication network. The server500B may be a computing device that executes the image evaluation computer program150performing an image evaluation operation.

The raw data RDATA output from the image sensor310, which may be for example a digital camera, are sent to the communication device510of the server500B over the communication device330and the communication network, under control of a processor321.

The ISP520of the server500B may be tuned by parameters, and the ISP520converts the raw data RDATA from the computing device300B into the test image TIM by using the parameters.

The image evaluation computer program150executed by a processor531of the server500B generates the quantified result value IDV. The quantified result value IDV is sent to the communication device330of the computing device300B over the communication device510and the communication network.

The ISP tuning computer program155executed by the processor321of the computing device300B generates the parameters PMT2by using the quantified result value IDV received through the communication device330and tunes the ISP340of the computing device300B by using the parameters PMT2. The ISP340tuned by the parameters PMT2may convert raw data output from the digital camera310into an image by using the parameters PMT2. The quantified result value IDV may be displayed on a display device350under control of the processor321.

FIG.9is a block diagram of an image evaluation system in which a test image generated by an image processing processor and a quantified result are exchanged over an Internet, according to an embodiment of the present disclosure.

Referring toFIG.9, an image evaluation system200C includes a computing device300C and a server500C communicating with each other over the communication network. The server500C may be a computing device that executes the image evaluation computer program150performing an image evaluation operation.

The raw data RDATA output from the or image sensor310, which may be for example a digital camera, are sent to an ISP341. The ISP341tuned by the first parameters PMT receives the raw data RDATA and converts the raw data RDATA into the test image TIM by using the first parameters PMT.

The test image TIM is sent to the communication device510of the server500C through the communication device330and the communication network, under control of the processor321.

The image evaluation computer program150executed by the processor531of the server500C processes the test image TIM to generate the quantified result value IDV. The quantified result value IDV is sent to the communication device330of the computing device300C over the communication device510and the communication network.

The ISP tuning computer program155executed by the processor321of the computing device300C generates the parameters PMT2by using the quantified result value IDV received through the communication device330and tunes the ISP341of the computing device300C by using the parameters PMT2. The ISP341tuned by the second parameters PMT2may convert raw data output from the digital camera310into an image by using the second parameters PMT2. The quantified result value IDV may be displayed on the display device350under control of the processor321.

FIG.10is a block diagram of an image evaluation system in which a test image stored in a memory device and a quantified result are exchanged over an Internet, according to an embodiment of the present disclosure.

Referring toFIG.10, an image evaluation system200D includes a computing device300D and a server500D communicating with each other over the communication network. The server500D may be a computing device that executes the image evaluation computer program150performing an image evaluation operation.

The test image TIM stored in a memory device305is sent to the communication device510of the server500D over the communication device330and the communication network, under control of the processor320.

The image evaluation computer program150executed by the processor531of the server500D processes the test image TIM and generates the quantified result value IDV and the difference image DIFF corresponding to a quantified result. At least one of the difference image DIFF and the quantified result value IDV is sent to the communication device330of the computing device300D over the communication device510and the communication network.

At least one of the difference image DIFF and the quantified result value IDV received through the communication device330may be displayed on the display device350under control of the processor320of the computing device300D.

As described with reference toFIGS.1to10, the image evaluation computer program150may obtain an artifact-free reference image from an aligned image by using the phenomenon that an image becomes a lattice pattern, for example a chessboard pattern image, when the image is compressed, or when the image is highly compressed or extremely compressed, through feature value decomposition, for example truncated singular value decomposition (SVD), may detect artifacts, which may be for example structural artifacts, included in the image by using the artifact-free reference image, and may generate a quantified result by quantifying a detection result. Herein, the structural artifacts may mean artifacts generated by an ISP.

The image evaluation computer program150may generate an artifact-free reference image (e.g., a compressed image) from a single image (e.g., a test image) by using SVD image compression.

The image evaluation computer program150may create a reference image in which the image sharpness, referring to the degree to which a shape of an outline included in an image appears clearly, is invariant, and may generate a quantified result by quantifying a difference the reference image and an image including artifacts.

The image evaluation computer program150may quantify structural distortion included in the image and may generate a quantified result.

The image evaluation computer program150may align a test image including a lattice pattern formed by image edges more finely by using the Radon transform and may create an aligned image including the aligned image edges.

The image evaluation computer program150may quantify the detail of an image distorted by artifacts to perform quantitative evaluation on the image.

According to an embodiment of the present disclosure, an image evaluation method and a computer program performing the image evaluation method may accurately align a test image (e.g., a chessboard image) including a first lattice pattern formed by image edges by using the image edges.

According to an embodiment of the present disclosure, an image evaluation method and a computer program performing the image evaluation method may quantify, or express the quantity of, distortion appearing in a test image (e.g., chessboard image) and thus may match the degree of distortion, which a human may perceive with respect to the test image, and the degree of quantified distortion.

According to an embodiment of the present disclosure, an image evaluation method and a computer program performing the image evaluation method may generate a test image by using parameters set in an image processing processor before tuning, may accurately align the test image to generate an aligned image, may compress the aligned image to generate a compressed image, may quantify a difference between the compressed image and the aligned image in units of a pixel, may generate new parameters by using a quantified result, and may again tune the image processing processor by using the new parameters.

Accordingly, according to an embodiment of the present disclosure, an image evaluation method and a computer program performing the image evaluation method may optimize parameters of an image processing processor.