Patent ID: 12235434

BEST MODE

The present invention will now be described more fully with reference to the accompanying drawings and contents disclosed in the drawings. However, the present invention should not be construed as limited to the exemplary embodiments described herein.

The terms used in the present specification are used to explain a specific exemplary embodiment and not to limit the present inventive concept. Thus, the expression of singularity in the present specification includes the expression of plurality unless clearly specified otherwise in context. It will be further understood that the terms “comprise” and/or “comprising”, when used in this specification, specify the presence of stated components, steps, operations, and/or elements, but do not preclude the presence or addition of one or more other components, steps, operations, and/or elements thereof.

It should not be understood that arbitrary aspects or designs disclosed in “embodiments”, “examples”, “aspects”, etc. used in the specification are more satisfactory or advantageous than other aspects or designs.

In addition, the expression “or” means “inclusive or” rather than “exclusive or.” That is, unless otherwise mentioned or clearly inferred from context, the expression “x uses a or b” means any one of natural inclusive permutations.

In addition, as used in the description of the invention and the appended claims, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless context clearly indicates otherwise

In addition, terms such as “first” and “second” are used in this specification and the accompanying claims merely to describe a variety of constituent elements, but the constituent elements are not limited by the terms. The terms are used only for the purpose of distinguishing one constituent element from another constituent element.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present invention, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Meanwhile, in the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention unclear. The terms used in the specification are defined in consideration of functions used in the present invention, and can be changed according to the intent or conventionally used methods of clients, operators, and users. Accordingly, definitions of the terms should be understood on the basis of the entire description of the present specification.

FIG.1is a diagram for explaining the principle of image acquisition according to an embodiment of the present invention.

Referring toFIG.1, the present invention includes a first process110of acquiring source images111of a sample or an object and a second process120of performing image processing on the source images.

The first process110for acquiring the source images111may be performed by, for example, an optical system300shown inFIG.3. Here, when the optical system300includes a light-emitting element array, for example, an LED array, the number of the source images111may be the same as the number of light-emitting elements included in the light-emitting element array.

When the light-emitting elements are arranged on the light-emitting element array in a matrix form, the matrix form consists of N rows and M columns. Accordingly, the number of the source images111may be N×M. Here, N and M may be integers of 3 or more and may be the same number or different numbers.

Hereinafter, for convenience of description, the case where N and M are the same number will be mainly described as an example. In other words, the light-emitting element array may include a plurality of light-emitting elements arranged in a matrix structure such as 3×3, 5×5, 7×7, 9×9, 11×11, 13×13, or 15×15.

The light-emitting elements arranged in the light-emitting element array may be arranged in various forms as needed as well as a matrix structure. However, for convenience of explanation, a light-emitting element array in which light-emitting elements are arranged in a matrix structure will be mainly described in the present specification.

The second process120may be carried out by the apparatus shown inFIG.2.

The second process120includes an FPMP125and a ripple removal process123. In addition, although not shown inFIG.1, the second process120may further perform processes shown inFIGS.8to11.

Through the second process120, a high-resolution image130or140may be generated from a low-resolution or low-magnification image.

Hereinafter, the second process120and an apparatus for performing the second process120and a method thereof are described in detail with reference toFIGS.2to19.

FIG.2is a diagram for explaining the configuration of an image acquirement apparatus according to an embodiment.

Referring toFIG.2, an image acquirement apparatus200includes an image acquirer210, a storage220, and a controller230.

The image acquirer210acquires N×M source images photographed using a light source of each of a plurality of light-emitting elements arranged in the light-emitting element array.

For example, when the plural light-emitting elements are arranged in a 3×3 form, or when only 9 light-emitting elements are used based on the light-emitting elements located at the center of the light-emitting element array, 9 source images may be acquired.

The storage220stores source images.

The controller230may include at least one processor. Here, the controller740may be connected to at least one computer-readable storage in which instructions or programs are recorded.

The controller230includes at least one processor configured to select some images from among a plurality of source images, to reconstruct a first image from among the some images using Fourier Ptychographic Microscopy Process (FPMP), to detect a partial region including a detection target object from the first image, to acquire partial-region images corresponding to the partial region from each of the source images, and to reconstruct a second image from the partial-region images using FPMP.

Here, the some images selected from among the plural source images may be n×m (n is an integer less than N and m is an integer less than M) images. Here, n×m images may be selected based on a central image using a central light-emitting element, located in the center of the light-emitting element array, as a light source.

Here, “a central image using a central light-emitting element, located in the center of the light-emitting element array” may be, for example, an image photographed using a central light-emitting element “13” of a light-emitting element array210ofFIG.6as a light source. Referring toFIG.6, an image photographed using the central light-emitting element “13” as a light source is an image marked with “I13” among the source images110. The relationship between a light-emitting element used as a light source and an image is described in detail with reference toFIGS.6and7.

For example, N and M may be any one of 7, 9, 11, 13 and 15, n and m may be 3 or 5.

Here, the first image may be a low-resolution, large-area image.

Here, the second image reconstructed from partial-region images may be a high-resolution image of the partial region.

Accordingly, the controller230may select n×m (n is an integer less than N and m is an integer less than M) images from among the source images based on the central image using the central light-emitting element, located at the center of the light-emitting element array, as a light source and may reconstruct a low-resolution, large-area image from the n×m images using FPMP.

In addition, the controller230may include at least one processor configured to detect a partial region including a detection target object from a low-resolution, large-area image, to acquire partial-region images matching the partial region from each of the source images, and to reconstruct a high-resolution image for the partial region using FPMP.

In addition, the controller230may include at least one processor configured to determine overlapping images, in which a dark field and a bright field are present in a preset ratio, from among the plural source images, to apply FPMP to the remaining images, except for the overlapping images, of the plural source images, and to generate a reconstructed image.

The ripple removal process123shown inFIG.1includes performing FPMP on the remaining images except for overlapping images.

An image reconstructed through conventional FPMP may include ripples in an outer region thereof as shown inFIG.16. However, through the ripple removal process123according to an embodiment of the present invention, an image from which ripples are removed may be acquired as shown inFIG.17.

For example, the image acquirement apparatus200may apply FPMP to 3×3 images to reconstruct a low-resolution, large-area image, may select a partial region containing an object from the low-resolution, large-area image, and, after separating partial-region images, which correspond to the partial region, from all source images, may perform FPMP on the partial-region images.

For example, a target sample of the source image may be a blood image, and the object may be a feature point, such as white blood cells, which can be identified in the image.

In the case of a traditional microscope system, an object is detected after taking a sample with a low-magnification lens, and the object is observed again using a high-magnification lens. However, according to an embodiment of the present invention, a high-resolution image of an object may be reconstructed through image processing of source images photographed with a low-magnification lens. When the image acquirement apparatus according to an embodiment of the present invention is applied to a microscope system, there is an advantage in that the system may be configured with only a low-magnification lens.

FIG.3is a diagram for explaining an application example and source image acquirement of the present invention.

Referring toFIG.3, the image acquirement apparatus200according to an embodiment of the present invention may acquire source images photographed by the optical system300.

For example, the optical system300may be a microscope system.

The optical system300includes a light source part310, a sample plate301capable of positioning a sample, an optical part320, and an image sensor330.

The light source part310may include a light-emitting diode array (LED array) including a plurality of light-emitting elements312arranged in at least one or more columns. Each of the plural light-emitting elements312may emit light in a different color or the same color. In the light source part310, each of the plural light-emitting elements312may simultaneously or sequentially emit light to radiate light to the sample plate301disposed in parallel with the light source part310.

The optical part320may include a first lens322and a second lens324.

The optical part320may form an enlarged image of a sample by light irradiated from the light source part310to the sample and may provide the formed image to the image sensor330.

The first lens322is a lens configured to project an enlarged image of a sample formed by light irradiated to the sample and may be an objective lens.

The second lens324is a lens configured to transmit an enlarged image projected through the first lens322to the image sensor330and may be a tube lens.

The respective optical axes of the first lens322and the second lens324may coincide with each other.

The image sensor330may generate at least one image by capturing an image of a sample which has passed through the optical part320. For example, the image sensor330may include a Charge-Coupled Device (CCD) sensor, a Complementary Metal Oxide Semiconductor (CMOS) sensor, or the like.

FIG.4is a diagram for explaining a Fourier ptychographic microscopy technique.

When light is irradiated to a target object at an inclined angle as inFIG.4(A)using the optical system shown inFIG.3, the image sensor330acquires an image in the form of a shadow depending on a direction in which the light is irradiated. In addition, when light is vertically irradiated to the target object as inFIG.4(B), the image sensor330acquires an image in the form of almost no shadow. In addition, when light is irradiated to the target object at an inclined angle opposite toFIG.4(A)as shown inFIG.4(C), the image sensor330may acquire an image in the form of a shadow at a location different from the image acquired inFIG.4(A).

For example, when usingFIG.2(B)(when there are few shadows) as a reference area, the shadowed area inFIG.4(A) or4(C)appears as a partially shifted area compared to the reference area as shown in the drawings. In a process of synthesizing various image information converted to a spatial frequency plane as described above, there is an advantage in that a resolution is primarily increased.

In addition, phase information may be calculated from excess information in the overlapping region during the image reconstruction process, and 3D information may be acquired using this.

When the FPM technology is used, a low-cost light source may be used compared to a general digital holography technology, and an optical system having a simple structure may be configured, thereby reducing equipment cost and reducing the volume of the optical system.

However, since the amount of computation required for restoration may increase as the number of source images used in FPMP increases, a method for reducing the amount of computation may be required.

In addition, since an image reconstructed through FPMP may include ripples, a method for solving the ripple issue may be required.

FIG.5illustrates an example of an image generated through a Fourier ptychographic microscopy technique.

FIG.5(A)illustrates an example of source images, andFIG.5(B) illustrates images reconstructed using the source images.

Referring toFIGS.5(A)and (B), for example, it can be seen that the center of the reconstructed images is reconstructed with high resolution, compared to the center of the source images.

FIG.6is a diagram for explaining an indexing relationship between an optical element arrangement and a source image according to an embodiment.

Referring toFIG.6, the optical system300or the image acquirement apparatus200may assign an index to each of the source images110to match the position of each of the light-emitting elements arranged in the light-emitting element array210.

In the example shown inFIG.6, a No. 13 light-emitting element13located at the center of the light-emitting element array210may be a central light-emitting element located on the central axis of the optical part320.

The image I13is an image photographed in a state in which the No. 13 light-emitting element is turned on and the other light-emitting elements are turned off. Similarly, an image I1is an image photographed in a state in which only a No. 1 light-emitting element is turned on, and an image I25is an image photographed in a state in which only a 25th light-emitting element is turned on.

Accordingly, unlike the embodiment shown inFIG.6, a total of 49 (7×7) source images may be acquired when a 7×7 light-emitting element array is used.

In the present specification, an image using a light-emitting element, such as the No. 13 light-emitting element, located at the center of the light-emitting element array as a light source is referred to as “a central image.” Accordingly, “a central image” inFIG.6provided as an embodiment is the image I13.

An identifier or index for each source image may be linearly given as shown inFIG.6. In addition, considering the No. 13 light-emitting element as a (3, 3) component of the matrix, an index for the image I13may be expressed as I(3,3).

In addition, an index representing a group of images for use in the ripple removal process123according to an embodiment of the present invention is also possible.

For example, in the example shown inFIG.6, an index (3, 3) indicating an image group may be used when an index of (1, 1) is assigned to the image I13that is a central image.

Here, the image group corresponding to the index (3, 3) refers to images photographed using each of light-emitting elements corresponding to a “3×3 light-emitting elements matrix” as a light source.

In the embodiment shown inFIG.6, the 3×3 light-emitting element matrix means is Nos. 7, 8. 9, 12, 14, 17, 18, and 19 light-emitting elements except for the No. 13 light-emitting element that is a central light-emitting element.

Similarly, a 5×5 light-emitting element matrix means the case wherein light-emitting elements, arranged outside by 2 rows and 2 columns from the No. 13 light-emitting element as a central light-emitting element, are used as a light source.

Accordingly, in the embodiment shown inFIG.6, images included in the image group corresponding to the image group index (3, 3) are I7, I8. I9, I12, I14, I17, I18, and I19.

In addition, in the embodiment shown inFIG.6, images included in the image group index (5, 5) are I1, I2, I3, I4, I5, I6, I10, I11, I15, I16, I20, I21, I22, I23, I24, and I25.

The image group index will be further described with reference toFIG.7.

FIG.7is a diagram for explaining an indexing manner of source images according to an embodiment.

Squares shown inFIG.7represent source images matched to arrangement positions of light-emitting elements, respectively.

An embodiment shown inFIG.7may be a case wherein the light source part310ofFIG.3is composed of 49 light-emitting elements arranged in 7×7.

In addition, the embodiment shown inFIG.7may be a case wherein 7×7 light-emitting elements, based on the central light-emitting element, among 225 light-emitting elements arranged in 15×15 in the light source part310ofFIG.3are used.

Here, the number of source images may be determined according to performance (e.g., resolution, resolution) of an image to be used for final reading. For example, when a required resolution of a finally acquired image is 1.5 μm, 49 or 81 source images may be acquired using 7×7 or 9×9 light-emitting elements.

When generating a low-resolution, large-area image of a sample, 9 source images601,631,632,633,634,635,636,637,639, and631, which include a central image601, matching 3×3 light-emitting elements, may be used.

That is, n×m (n is an integer less than N and m is an integer less than M) images selected to reconstruct a low-resolution, large-area image from the plural source images may be 9 source images601,631,632,633,634,635,636,637,639, and631.

Meanwhile, in the present specification, “overlapping images” means images in which a dark field and a bright field exist at a preset ratio.

“Overlapping images” may be determined individually or assigned as a group.

When “overlapping images” is designated as a group, they may be images belonging to an image group index (2k+1, 2k+1).

InFIG.7, images corresponding to an image group index (5, 5) are (5, 5)_1 to (5, 5)_16 and are a total of 16 images. In addition, images belonging to an image group index (7, 7) are 24 images denoted by reference numeral710.

In summary, source images may be distinguished by indexes matching the respective positions of N×M light-emitting elements arranged in the matrix form. In addition, an index assigned to a central image may be (1,1), and an index assigned to an image group using light-emitting elements, arranged outside by k rows or k columns from the central light-emitting element, as a light source may be (2k+1, 2k+1), where k=1,2,3 . . . (M−1)/2, and an image group corresponding to the (2k+1, 2k+1) is composed of {(2k+1)×(2k+1)}−{(2k−1)×(2k−1))} images.

FIG.8is a flowchart for explaining a method of acquiring an image according to an embodiment.

The method shown inFIG.8may be performed through the optical system300and the image acquirement apparatus200ofFIG.3. More specifically, steps820to860inFIG.8, steps920to930inFIG.9, steps1031to1307inFIG.10and steps1131to1135inFIG.11may be performed by the controller230inFIG.2. However, for convenience of explanation, the performing subject inFIGS.8to11is referred to as “apparatus.”

Referring toFIG.8, in step810, the apparatus acquires a plurality of source images by sequentially irradiating a sample with a plurality of light-emitting element light sources arranged in the light-emitting element array.

A sample may be placed on the sample plate301, and the apparatus may determine a measurement position of the sample. Here, the measurement position of the sample may be a part marked with reference numeral1230or1240determined through a positioning algorithm as shown inFIG.12.

After determining the measurement position, the optical system300may perform a procedure for adjusting a lens focus of the image sensor330or the optical part320.

In step810, the apparatus may determine the number of light-emitting elements to be used as a light source according to a required resolution, etc., for example, N×M light-emitting elements arranged in a matrix structure (N and M are integers of 3 or more) may be used.

Accordingly, in step810, the apparatus may determine the measurement position of the sample, and may acquire N×M source images of the sample using a light source of each of N×M (N and M are integers of 3 or more) light-emitting elements arranged in a matrix structure.

In addition, in step810, the apparatus may confirm a resolution required depending upon the type of an object, and may determine the N and M based on the resolution.

In step820, the apparatus selects some images from among the plural source images.

Here, “some images” may be n×m (n is an integer less than N and m is an integer less than M) images among the source images based on a central image using a central light-emitting element, located at the center of the matrix structure, as a light source.

For example, “some images” may be 9 images I7to I9, I12to I13, and I17to I19matching the 3×3 light-emitting element matrix inFIG.6.

In step830, the apparatus reconstructs a first image from secondary images using FPMP.

Here, the first image may be a low-resolution, large-area image.

Accordingly, through steps820to830, the apparatus may select n×m (n is an integer less than N and m is an integer less than M) images from the source images and may perform FPMP on the n×m images to reconstruct a low-resolution, large-area image.

In step830, the apparatus may determine overlapping images, in which a dark field (DF) and a bright field (BF) exist with a preset ratio, among some images, and may perform FPMP on remaining images, except for the overlapping images, among the some images.

Here, the “preset ratio” may be determined in a range of 50 to 90% DF and 10 to 50% BF. For example, an image with DF of 82% and BF of 18% may be determined as “overlapping image”.

In step840, the apparatus detects a partial region including a detection target object from the first image.

Here, the object may have feature points that can be distinguished with the naked eye on the magnification of the sample, and may exist at multiple positions in one low-resolution, large-area image. As the partial region, a plurality of regions or only one region may be selected. When a plurality of partial regions are selected, high-resolution image reconstruction may be performed for each of the partial regions. Selection of the partial region may be performed using a candidate group detection algorithm using data learned through machine learning or deep learning. Machine learning or deep learning for selection of the partial region may use known techniques. The machine learning or deep learning is out of the scope of the present invention, so a detailed description thereof is omitted.

In step850, the apparatus acquires partial-region images corresponding to the partial region from the respective source images.

That is, unlike the process of generating a low-resolution, large-area image, partial-region images are acquired from the respective source images.

For example, when a 15×15 light-emitting element matrix is used, 225 source images are acquired, and even when a low-resolution, large-area image is reconstructed from 3×3 images, a partial region image may be composed of 225 source images acquired from all source images.

However, when the ripple removal process123is applied in the process of performing FPMP, the number of partial region images or the number of “some images” may be changed.

In step860, the apparatus reconstructs a second image from the partial-region images using FPMP. Here, the second image may be a high-resolution image of the partial region.

A high-resolution image of the partial region may be displayed through a display (not shown) of the image acquirement apparatus200for reading by an expert, or may be transmitted to a separate server or terminal apparatus.

In steps850and860, the apparatus may determine overlapping images, in which a dark field (DF) and bright field (BF) exist at a preset ratio, among the source images, may acquire partial-region images of remaining images, except for the overlapping images, among the source images, and may perform FPMP on the partial-region images acquired from the remaining images except for overlapping images.

FIG.9is a flowchart for explaining a method of acquiring an image according to another embodiment.

Referring toFIG.9, in step910, the apparatus acquires a plurality of source images by sequentially irradiating a sample with a plurality of light-emitting element light sources arranged in a matrix structure.

In step920, the apparatus determines overlapping images, in which a dark field and a bright field exist in a preset ratio, among the plural source images.

In step930, the apparatus applies FPMP to remaining images, except for the overlapping images, among the plural source images, and generates a reconstructed image.

For example, when overlapping images among the source images shown inFIG.7are (5, 5)_1, (5, 5)_2, (5, 5)_4, (5, 5)_5, (5, 5)_6, (5, 5)_8, (5, 5)_9, (5, 5)_10, (5, 5)_12, (5, 5)_13, (5, 5)_14, and (5, 5)_16, “remaining images” are (5, 5)_3, (5, 5)_7, (5, 5)_11, (5, 5)_15, 601, 631˜639 and 24 images belonging to reference numeral710. Accordingly, when there are 12 overlapping images, the number of remaining images is 37.

Here, indexing information for each of the “overlapping images” may be maintained and the image file may be removed from the storage220inFIG.2. In this way, it is possible to reduce the storage capacity of source images to create a reconstructed image.

Accordingly, the storage space of the apparatus may be efficiently used through the ripple removal process123according to an embodiment of the present invention, and when transmitting stored data to the outside, a transmission capacity may be reduced.

Step930of creating a reconstructed image may include the steps shown inFIG.10or the steps shown inFIG.11.

FIG.10is a flowchart for explaining an image reconstruction method according to an embodiment.

Referring toFIG.10, in step1031, the apparatus selects some images from among the remaining images based on a central image using the central light-emitting element located at the center of the light-emitting element array as a light source.

In step1033, the apparatus performs FPMP on some images and reconstructs a low-resolution, large-area image.

In step1035, the apparatus detects a partial region including a detection target object from the low-resolution, large-area image, and acquires partial-region images matching the partial region from the respective remaining images.

In step1037, the apparatus performs FPMP on the partial-region images and reconstructs a high-resolution image on the partial region.

The reconstructed image may be transferred from the apparatus to a running program or application.

FIG.11is a flowchart for explaining a method of reconstructing an image from which ripples are removed according to an embodiment.

Referring toFIG.11, in step1131, the apparatus confirms a central image using a central light-emitting element, located at the center of the matrix structure, as a light source, and checks an image group corresponding to an index (2k+1, 2k+1) assigned to an outer row and column image group from the central image based on an index (1,1) assigned to the central image.

For example, the central light-emitting element may be a No. 6 light-emitting element inFIG.6.

For example, “k” may be “2”, and images belonging to an image group (5, 5) may be 16 images indicated by (5, 5)_1 to (5, 5)_16 inFIG.7.

In step1133, the apparatus checks a DF and BF ratio of each image belonging to an image group corresponding to (2k+1, 2k+1).

In step1135, when, among images belonging to an image group, there are two or more images in which DF and BF exist in a preset ratio, the apparatus may delete the image group corresponding to (2k+1, 2k+1) from the source images.

Steps1131to1135are included in a ripple removal process, and an execution result may be transferred to a currently running program or application.

If, among the images belonging to the image group, there is no image in which DF and BF exist in a preset ratio or images below a certain number exist, the apparatus increases k value by 1 and then may perform steps1131to1135. For example, when steps1133and1135are performed for the image group (5, 5), steps1133and1135may be performed for an image group (7, 7) in which k value is increased by 1.

FIG.12is a diagram illustrating an example of the sample and sample plant inFIG.3.

A sample1201or1202prepared on a sample plate301or302may be, for example, whole blood, plasma, serum, saliva, ocular fluid, cerebrospinal fluid, sweat, urine, milk, ascites fluid, synovial fluid, peritoneal fluid, a liquid sample of cell lysate, or a solid sample of a skin system sub-organ of tissue hair or the nail.

The optical system300may set a position1230or1240to be measured through a positioning algorithm.

The optical system300may acquire an image using light-emitting elements as a light source for the selected location.

FIG.13illustrates an example of a low-resolution, large-area image according to an embodiment.

FIG.13illustrates an example of a low-resolution, large-area image of a blood sample reconstructed using some of all source images.

FIG.13(A)shows an image of one of light-emitting elements (for example, a light-emitting element present at the center of the light-emitting elements) taken with a light source, andFIG.13(B)shows an example of a low-resolution, large-area image acquired by performing FPM on images taken with 3×3 or 5×5 light-emitting elements as a light source.

InFIG.13, partial regions corresponding to reference numerals1310and1320indicate leukocytes that are detection targets.

FIG.14is a diagram for explaining a method of acquiring a partial region image according to an embodiment.

FIG.14(A)illustrates nine source images corresponding to a 3×3 light-emitting element matrix. Accordingly, reference numeral1415denotes a partial region of a central image acquired by using a central light-emitting element as a light source. For example, reference numeral1415may be a region matching reference numeral1310inFIG.13.

FIG.14(B)shows partial-region images1411,1412,1413,1414,1415,1416,1417,1418and1419respectively matching partial regions of 9 source images.

As described above, since the method of acquiring an image according to an embodiment of the present invention reconstructs a high-quality small-area image by using a partial region image from a source image, the amount of computation and the capacity of storage may be reduced.

FIG.15is a diagram showing an example of a high-resolution image reconstruction for a partial region according to an embodiment.

Referring toFIG.15, it can be seen that high-quality small-area images (1510,1520) can be reconstructed from a low-resolution, large-area image shown inFIG.13.

FIG.16shows an example of an image reconstructed through general FPMP.FIG.17shows an example of an image reconstructed through APFMP according to an embodiment.

An image reconstructed through general FPMP may include ripples in an outer region of an image as shown inFIG.16. However, through the ripple removal process123according to an embodiment of the present invention, it is possible to acquire an image in which ripples are removed as shown inFIG.17.

Here, the “overlapping images” removed by the ripple removal process123or excluded by FPMP may be learned through machine learning or deep learning depending on a sample or the type of sample. Through this, the image corresponding to the image group index (2k+1, 2k+1) may be removed from the source images without detecting the DF and FB ratio of the source images depending on the sample or the sample type and the number of source images used.

For example, for a specific sample A, an image group (5, 5) may be removed from the source images or, for a specific sample B, an image group (7, 7) may be removed from the source images, or if the number of source images is more than 15×15, all image groups (5, 5), (9, 9), (13, 13) may be removed.

As described above, when “overlapping image” to be removed from the source images is determined in advance, the image acquirement process according to the embodiment of the present invention may proceed more quickly.

FIG.18shows an example of source images matched to respective positions of light-emitting elements.FIG.19is a diagram for explaining an example of remaining images used in an APFMP process according to an embodiment.

FIG.18illustrates 7×7 source images acquired from the same samples as those shown inFIG.16or17.

The image acquirement apparatus200may detect the DF and BF ratio for each of 49 source images and may remove images corresponding to the image group index (5, 5) from the source images.

FIG.19shows a state in which the images corresponding to the image group index (5, 5) have been removed.

The image acquirement apparatus200may apply FPMP to 33 source images of “remaining images” except for 16 “overlapping images” from 49 source images.

As such, by excluding “overlapping images” and applying FPMP, the image quality of a reconstructed image may be improved by removing ripples, and the amount of computation and processing speed may also be improved by reducing the number of FPMP target images.

FPMP according to an embodiment of the present invention, which has improved processing speed and image quality compared to general FPMP, may be referred to as advanced FPMP.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be achieved using one or more general purpose or special purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executing on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may include a plurality of processing elements and/or a plurality of types of processing elements. For example, the processing apparatus may include a plurality of processors or one processor and one controller. Other processing configurations, such as a parallel processor, are also possible.

The software may include computer programs, code, instructions, or a combination of one or more of the foregoing, configure the processing apparatus to operate as desired, or command the processing apparatus, either independently or collectively. In order to be interpreted by a processing device or to provide instructions or data to a processing device, the software and/or data may be embodied permanently or temporarily in any type of a machine, a component, a physical device, a virtual device, a computer storage medium or device, or a transmission signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored in one or more computer-readable recording media.

The methods according to the embodiments of the present disclosure may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium can store program commands, data files, data structures or combinations thereof. The program commands recorded in the medium may be specially designed and configured for the present disclosure or be known to those skilled in the field of computer software. Examples of a computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, or hardware devices such as ROMs, RAMs and flash memories, which are specially configured to store and execute program commands. Examples of the program commands include machine language code created by a compiler and high-level language code executable by a computer using an interpreter and the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the present disclosure has been described with reference to limited embodiments and drawings, it should be understood by those skilled in the art that various changes and modifications may be made therein. For example, the described techniques may be performed in a different order than the described methods, and/or components of the described systems, structures, devices, circuits, etc., may be combined in a manner that is different from the described method, or appropriate results may be achieved even if replaced by other components or equivalents.

Therefore, other embodiments, other examples, and equivalents to the claims are within the scope of the following claims.