Patent Description:
Hyperspectral imaging is a combination of spectral analysis and image processing, for simultaneously obtaining spatial information and spectral information with respect to an object that is being measured. Hyperspectral imaging devices are capable of identifying the state, configuration, features, variations, etc. of an object through data obtained by a hyperspectral image sensor, so as to identify a material, and/or measure a degree of defect of a product, etc. However, in hyperspectral imaging systems, the resolution (e.g., the spatial resolution and/or spectral resolution) of an image sensor may be decreased due to an effect of crosstalk between pixels occurring as the pixel size of the image sensor is decreased. Therefore, in order for an image sensor to obtain image data having a high resolution, various image processing methods for compensating for the effect of crosstalk have been attempted. <CIT> discloses a color conversion matrix forming method, color conversion table forming method, and program. A method of easily obtaining a color conversion matrix serving as a foundation of color reproduction at a high speed irrespective of its degree is provided. The method has: a step of constructing nth-degree matrix coefficients regarding a first color space of a plurality of color patch data sets and obtaining a pseudo inverse matrix to the obtained matrix; and a step of forming a color conversion matrix from a product of a matrix constructed by color data regarding a second color space of the color patch data sets and the pseudo inverse matrix. <CIT> discloses a spectral reflectivity reconstruction method combining principal component analysis and regularized polynomial. The spectral reflectivity reconstruction method comprises the steps of collecting spectral data of training samples by employing a multi-spectral imaging system, carrying out dimensionality reduction on the spectral data of the training samples by means of a principal component analysis method to reduce the spectral data quantity of the training sample, and constructing a training sample set through the spectral data of the training samples. The publication of<NPL>" discloses a colorimetric and multispectral image acquisition using model-based and empirical device characterization. The focus is high quality image acquisition in colorimetric and multispectral formats, and aims to combine the spatial resolution of digital images with the spectral resolution of color measurement instruments. <CIT> discloses a multispectral reconstruction method and device based on L4 norm optimization.

According to an aspect of the present invention, there is provided a device and a method for processing spectrum data of an image sensor.

In one embodiment of the invention, according to claim <NUM>, there is described a method of processing spectrum data, the method comprising: obtaining spectrum response signals corresponding to channels of spectrum data of light, wherein a light source irradiates the light of a plurality of different wavelength bands toward an object, and the spectrum data is obtained from the object by an image sensor; determining a set of bases corresponding to the obtained spectrum response signals; performing, based on the determined set of bases, a change of basis on at least one basis included in the determined set of bases by using a pseudo inverse with respect to the spectrum data obtained by the image sensor, wherein the changed bases provide orthogonality for the spectrum response signal of each channel; and generating, by using the pseudo inverse, reconstructed spectrum data from the spectrum response signals on which the change of basis has been performed, wherein the pseudo inverse provides elements truncated by a truncation matrix in that for bases of which a singular value is determined to be smaller than a predetermined magnitude threshold value, the respective singular value is replaced with the threshold value or with a lowest value among the singular values greater than or equal to the threshold value.

The determining the set of bases may include obtaining a principal component with respect to a spectrum response signal of each channel by using a principal component analysis (PCA) algorithm.

The set of bases may include bases, a number of which is less than or equal to a number of the channels.

The performing the change of basis may include: determining at least one basis having a singular value, which is less than a threshold value, among the bases corresponding to the obtained spectrum response signals; and replacing the singular value of the determined at least one basis with a certain singular value.

The certain singular value may be the threshold value or a lowest value among singular values of other bases, in the bases corresponding to the obtained spectrum response signals, each having a singular value greater than or equal to the threshold value.

The method may further include generating a hyperspectral image, based on the generated reconstructed spectrum data.

The generating the hyperspectral image may include generating an RGB image, which is color-converted from the reconstructed spectrum data by using a color matching function.

The channels of spectrum data may include three or more channels.

The threshold value may be determined based on a singular value of an n-th (n being a natural number) singular value, among singular values of the bases corresponding to the obtained spectrum response signals in an order from smallest to greatest. In another embodiment of the invention, according to claim <NUM>, there is described a device for processing spectrum data, the device comprising: a light source configured to irradiate light of a plurality of different wavelength bands toward an object; an image sensor configured to obtain spectrum response signals corresponding to channels of spectrum data of light, the spectrum data being obtained from the object by the image sensor; and a processor configured to: determine a set of bases corresponding to the obtained spectrum response signals; perform, based on the determined set of bases, a change of basis on at least one basis included in the determined set of bases by using a pseudo inverse with respect to the spectrum data obtained by the image sensor, wherein the changed bases provide orthogonality for the spectrum response signal of each channel; and generate, by using the pseudo inverse, reconstructed spectrum data from the spectrum response signals on which the change of basis has been performed, wherein the pseudo inverse provides elements truncated by a truncation matrix in that for bases of which a singular value is determined to be smaller than a predetermined magnitude threshold value, the respective singular value is replaced with the threshold value or with a lowest value among the singular values greater than or equal to the threshold value.

The set of bases may be determined by obtaining a principal component with respect to a spectrum response signal of each channel by using a principal component analysis (PCA) algorithm.

The processor may be further configured to determine at least one basis having a singular value less than a threshold value among the bases corresponding to the obtained spectrum response signals, and replace the singular value of the determined at least one basis with a certain singular value.

The processor may be further configured to generate a hyperspectral image, based on the generated reconstructed spectrum data.

The processor may be further configured to generate the hyperspectral image by generating an RGB image, which is color-converted from the reconstructed spectrum data by using a color matching function.

The image sensor may include a multispectral image sensor configured to obtain spectrum data of three or more channels.

The threshold value may be determined based on a singular value of an n-th (n being a natural number) singular value, among singular values of the bases corresponding to the obtained spectrum response signals in an order from smallest to greatest.

According to an aspect of an example embodiment, according to claim <NUM>, there is provided a non-transitory computer-readable recording medium having recorded thereon a program for executing a method of processing spectrum data, the method including: obtaining spectrum response signals corresponding to channels of spectrum data of light, wherein a light source irradiates the light of a plurality of different wavelength bands toward an object, and the spectrum data being obtained from the object by an image sensor; determining a set of bases corresponding to the obtained spectrum response signals; performing, based on the determined set of bases, a change of basis on at least one basis included in the determined set of bases by using a pseudo inverse with respect to the spectrum data obtained by the image sensor, wherein the changed bases provide orthogonality for the spectrum response signal of each channel; and generating, by using the pseudo inverse, reconstructed spectrum data from the spectrum response signals on which the change of basis has been performed, wherein the pseudo inverse provides elements truncated by a truncation matrix in that for bases of which a singular value is determined to be smaller than a predetermined magnitude threshold value, the respective singular value is replaced with the threshold value or with a lowest value among the singular values greater than or equal to the threshold value.

In this regard, example embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein. Accordingly, example embodiments are merely described below, by referring to the figures, to explain aspects of the invention.

Although the terms used in the disclosure are selected from among common terms that are currently widely used in consideration of their function in the disclosure, the terms may be different according to an intention of one of ordinary skill in the art, a precedent, or the advent of new technology. Also, in particular cases, the terms are discretionally selected by the applicant of the disclosure, in which case, the meaning of those terms will be described in detail in the corresponding part of the detailed description. Therefore, the terms used in the disclosure are not merely designations of the terms, but the terms are defined based on the meaning of the terms and content throughout the disclosure.

In the description of embodiments, it will be understood that when an element is referred to as being "connected to" another element, it may be "directly connected to" the other element or be "electrically connected to" the other element through an intervening element. The singular expression also includes the plural meaning as long as it is not inconsistent with the context. When an element is referred to as "including" a component, the element may additionally include other components rather than excluding other components as long as there is no particular opposing recitation.

The terms such as "include" or "comprise" used herein should not be construed as necessarily including all various elements or operations described herein and should be understood that some of the elements or operations may be omitted or additional elements or operations may be further provided.

In addition, although the terms such as "first" or "second" may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element.

It should be understood that the scope of the example embodiments is not limited by the description of certain embodiments below and matters that can be easily derived by those of ordinary skill in the art fall within the scope of the example embodiments. Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings below.

<FIG> is a block diagram illustrating an example of an imaging device according to some example embodiments.

Referring to <FIG>, an imaging device <NUM> may be any device capable of analyzing characteristics of an object or identifying the object. For example, the imaging device <NUM> may correspond to an imaging system using a hyperspectral or multispectral imaging technique, but is not limited thereto. The object may include a person, a thing, a terrain, a plant, and the like, and may be changed without limitation depending on a field in which the imaging device <NUM> may be used.

Referring to <FIG>, the imaging device <NUM> may include at least one light source <NUM>, at least one image sensor <NUM>, and at least one processor <NUM>. However, only components related to some example embodiments are illustrated in the imaging device <NUM> of <FIG>. Therefore, it is apparent to those skilled in the art that the imaging device <NUM> may further include other general-purpose components in addition to the components illustrated in <FIG>. For example, the imaging device <NUM> may further include hardware components such as a light source, a memory, etc..

Furthermore, even an imaging device including only some of the components illustrated in <FIG> but capable of achieving the objective of the disclosure may correspond to the imaging device <NUM>. For example, the imaging device <NUM> may include only the at least one image sensor <NUM> and the at least one processor <NUM>, and the at least one light source <NUM> may be external to the imaging device <NUM>.

The memory, which may be included in the imaging device <NUM>, may be hardware for storing various pieces of data processed by the imaging device <NUM>, and for example, may store data that has been processed by and data to be processed by the imaging device <NUM>. In addition, the memory may store applications, drivers, etc. to be executed by the imaging device <NUM>.

The memory may include random-access memory (RAM) such as dynamic RAM (DRAM) or static SRAM, read-only memory (ROM), electrically erasable programmable ROM (EEPROM), a compact disc-ROM (CD-ROM), a Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid-state drive (SSD), or flash memory, and may also include other external storage devices accessible by the imaging device <NUM>.

The at least one light source <NUM> refers to a device for irradiating light toward the object. The at least one light source <NUM> irradiates light of a plurality of different wavelength bands toward the object. For example, the at least one light source <NUM> may selectively irradiate, toward the object, light of a first wavelength band (e.g., a range of <NUM> to <NUM>) and light of a second wavelength band (e.g., a range of <NUM> to <NUM>) that is different from the first wavelength band. However, the wavelength bands of light irradiated by the at least one light source <NUM> may be variously set.

The at least one light source <NUM> may be a light-emitting diode (LED) or a fluorescent lamp that irradiates broad light in a visible light band, or may be a laser diode that irradiates high-intensity, short-wavelength light. However, the disclosure is not limited thereto. The at least one light source <NUM> may emit light of a wavelength band suitable for obtaining information about the object.

In addition, the at least one light source <NUM> may be a single light source capable of selectively irradiating, toward the object, light of a plurality of different wavelength bands. However, the disclosure is not limited thereto, and the at least one light source <NUM> may include a plurality of light sources each irradiating light of one wavelength band. In addition, the at least one light source <NUM> may include a plurality of light sources capable of selectively irradiating, toward the object, light of a plurality of different wavelength bands.

The at least one image sensor <NUM> may refer to a device for obtaining spectrum data from light that is scattered, emitted, or reflected from the object, or absorbed by the object. However, the disclosure is not limited thereto, and the at least one image sensor <NUM> may obtain spectrum data from light transmitted or refracted through the object. In an example, when light generated by the light source <NUM> or natural light is irradiated toward the object, the object may absorb, scatter, emit, or reflect the light, and the image sensor <NUM> may measure spectrum data of the light absorbed, scattered, emitted, or reflected by or from the object. Because the spectrum data may be different depending on the types of materials constituting the object, the types of the materials constituting the object may be estimated by analyzing the measured spectrum data.

The spectrum data obtained by the image sensor <NUM> may include spectrum data (or a spectrum signal) of a plurality of channels corresponding to respective wavelengths. Here, the bandwidth and number of channels of each wavelength may be variously set. For example, the image sensor <NUM> may be a multispectral image sensor capable of obtaining spectrum data of at least three channels.

According to some example embodiments, the spectrum data obtained by the image sensor <NUM> may include at least one of a variety of spectra, such as a Raman spectrum, a visible spectrum, a fluorescence spectrum, a microwave spectrum, an infrared spectrum, an X-ray spectrum, etc. However, the disclosure is not limited thereto. Here, the Raman spectrum may refer to a spectrum obtained when high-intensity, short-wavelength light is irradiated toward the object by using a light source such as a laser diode, and then light scattered or emitted from the object is measured at a wavelength band different from the wavelength band of the light source.

Although <FIG> illustrates that the imaging device <NUM> includes only one image sensor <NUM>, the imaging device <NUM> may include two or more image sensors. In the case of the imaging device <NUM> including two or more image sensors, the types of spectrum data obtained by the respective image sensors may be different from each other.

The image sensor <NUM> may obtain the spectrum data by using at least one of a grating and a filter array. The grating may correspond to a device that performs spectroscopy by using refraction, reflection, or diffraction of light, and the filter array may correspond to a device that performs spectroscopy by using filters that selectively transmit or block a certain wavelength or wavelength range.

The image sensor <NUM> may obtain a visible image or a hyperspectral image of the object by measuring light in a wavelength band of, for example, <NUM> to <NUM>. However, the disclosure is not limited thereto, and the at least one image sensor <NUM> may measure light in an arbitrary wavelength band suitable for obtaining information about the object.

The image sensor <NUM> may include a photodiode array, a charge-coupled device (CCD) sensor, or a complementary metal-oxide semiconductor (CMOS) sensor, which is capable of obtaining a visible image including information about the appearance, such as the color or shape, of the object. In addition, the image sensor <NUM> may obtain a hyperspectral or multispectral image including information about fluorescence emitted from the object. Certain indicator materials of the object may emit fluorescence as light is emitted thereto from the light source <NUM>, and parameters indicating the amounts of the indicator materials may be obtained from the hyperspectral or multispectral image. Although an example of hyperspectral imaging will be described below, the disclosure is not limited thereto, and for example, the example embodiments may be implemented by using multispectral and hyperspectral imaging techniques.

The at least one processor <NUM> serves to perform an overall function for controlling the imaging device <NUM>. For example, the at least one processor <NUM> may control the operation of the at least one light source <NUM> and the at least one image sensor <NUM>. The at least one processor <NUM> may be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory storing a program executable by the microprocessor.

The at least one processor <NUM> may obtain first information about the object based on the visible image obtained by the at least one image sensor <NUM>, and obtain second information about the object based on the obtained first information and the hyperspectral image obtained by the image sensor <NUM>. As described above, the processor <NUM> may obtain the second information by comprehensively considering information obtained by analyzing the visible image rather than using only information obtained by analyzing the hyperspectral image, and thus, the accuracy of the second information about the object may be increased.

For example, the first information may include information about the appearance of the object, and the second information may include information about material parameters of the object. However, the disclosure is not limited thereto.

<FIG> is a conceptual diagram of the imaging device <NUM> according to some example embodiments. <FIG> is a conceptual diagram of the light source <NUM> of <FIG>. <FIG> is a conceptual diagram of an example of the image sensor <NUM> of <FIG>. <FIG> is a conceptual diagram illustrating an example of the image sensor <NUM> of <FIG>.

Referring to <FIG>, the imaging device <NUM> may include the light source <NUM>, the image sensor <NUM>, and the processor <NUM>, and may perform imaging of an object OBJ.

The light source <NUM> may face the object OBJ. The light source <NUM> may emit rays of inspection light ILa and ILb toward the object OBJ. As illustrated in <FIG>, the light source <NUM> may include a first light source array <NUM>, a second light source array <NUM>, and a transmission window <NUM>. The first light source array <NUM> and the second light source array <NUM> may be spaced apart from each other with the transmission window <NUM> therebetween. Each of the first light source array <NUM> and the second light source array <NUM> may include a plurality of light sources <NUM> arranged in one direction. For example, the plurality of light sources <NUM> may be arranged along the transmission window <NUM>. Although <FIG> illustrates that the plurality of light sources <NUM> of each of the first light source array <NUM> and the second light source array <NUM> are arranged in two columns, the disclosure is not limited thereto. For example, the plurality of light sources <NUM> may include LEDs.

The object OBJ exposed to the rays of inspection light ILa and ILb may emit fluorescence OL. The fluorescence OL may pass through the transmission window <NUM> and reach the inside of the imaging device <NUM>. The transmission window <NUM> may include a transparent material. For example, the transmission window <NUM> may include transparent plastic or glass. In an example, the transmission window <NUM> may include a material having high durability at low temperature. The fluorescence OL may be provided to the image sensor <NUM>. For example, the optical path of the fluorescence OL may be adjusted by an optical path adjustment element <NUM> such that the fluorescence OL is provided to the image sensor <NUM>. The image sensor <NUM> may include a hyperspectral camera.

In an example, the image sensor <NUM> may include a dispersive element <NUM> as illustrated in <FIG>. An image sensor 200a (see <FIG>) may include a slit element <NUM>, a collimating lens <NUM>, the dispersive element <NUM>, a condensing lens <NUM>, and a sensor <NUM>. The slit element <NUM> may be used to extract a required portion of the fluorescence OL. In an example, the fluorescence OL having passed through the slit element <NUM> may diverge. The collimating lens <NUM> may adjust the size of the fluorescence OL so as to convert the fluorescence OL into parallel light or convergent light. For example, the collimating lens <NUM> may include a convex lens. The dispersive element <NUM> may split the fluorescence OL provided from the collimating lens <NUM>. Although <FIG> illustrates that the dispersive element <NUM> is a grating, the disclosure is not limited thereto. In another example embodiment, the dispersive element <NUM> may be a prism. The split fluorescence OL may pass through the condensing lens <NUM> and then be provided to the sensor <NUM>. For example, the condensing lens <NUM> may include a convex lens. The split fluorescence OL may be provided to different positions of the sensor <NUM> according to wavelengths. The sensor <NUM> may measure the split fluorescence OL provided from the dispersive element <NUM>. The sensor <NUM> may generate a spectrum signal of the fluorescence OL. The sensor <NUM> may provide the spectrum signal to the processor <NUM>.

In an example, as illustrated in <FIG>, an image sensor 200b including a spectrum filter <NUM> may be provided. The image sensor 200b may include the slit element <NUM>, the collimating lens <NUM>, the spectrum filter <NUM>, the condensing lens <NUM>, and the sensor <NUM>. The slit element <NUM>, the collimating lens <NUM>, the condensing lens <NUM>, and the sensor <NUM> may be substantially the same as those described with reference to <FIG>. The spectrum filter <NUM> may be a set of filters that pass rays of light of different wavelength bands, respectively. The spectrum filter <NUM> may filter the fluorescence OL provided from the collimating lens <NUM> to have spatially different wavelengths. That is, portions of the fluorescence OL having passed through different regions of the spectrum filter <NUM> may have different wavelengths. The fluorescence OL split by the spectrum filter <NUM> may pass through the condensing lens <NUM> and then be provided to the sensor <NUM>. The sensor <NUM> may generate a spectrum signal of the fluorescence OL. The sensor <NUM> may provide the spectrum signal to the processor <NUM>.

The processor <NUM> may generate a hyperspectral image of the object OBJ based on the spectrum signal. The processor <NUM> may determine the state of the object OBJ by using the hyperspectral image of the object OBJ.

<FIG> is a schematic diagram illustrating an operation of an imaging device, according to some example embodiments.

Referring to <FIG>, the image sensor <NUM> may obtain spectrum data (or a spectrum signal) of an object by using a filter array.

The spectrum data obtained by the image sensor <NUM> may include, for example, information about the intensity of light at N+<NUM> (n being an integer equal to or greater than one) wavelengths λ<NUM>,. The processor <NUM> of the imaging device <NUM> may perform image processing <NUM> on the obtained spectrum data. Here, the image processing <NUM> may refer to processing for obtaining a hyperspectral image <NUM> of the object. Furthermore, the image processing <NUM> may include a spectrum reconstruction process for generating reconstructed spectrum data from input spectrum data.

The image sensor <NUM> may include a plurality of pixels. The plurality of pixels may correspond to a plurality of wavelength bands, respectively, or may include a plurality of subpixels corresponding to a plurality of wavelength bands, respectively. For example, first subpixels may measure light of a first wavelength band, and second subpixels may measure light of a second wavelength band different from the first wavelength band.

In order to increase the resolution of the image sensor <NUM> (e.g., spatial resolution or spectral resolution), it may be desirable to implement a small pixel size. However, as the pixel size of the image sensor <NUM> is decreased, an effect of crosstalk may occur between adjacent pixels. Therefore, for imaging at an appropriate resolution, an effect of crosstalk needs to be compensated for. In particular, the spatial resolution and the spectral resolution of the image sensor <NUM> for hyperspectral imaging are inversely proportional to each other. Accordingly, in order for the image sensor <NUM> to perform hyperspectral imaging at an appropriate spatial resolution and spectral resolution, a method for compensating for an effect of crosstalk may be used.

<FIG> is a flowchart of a method of processing spectrum data of an image sensor to generate reconstructed spectrum data, according to some example embodiments. The method of processing spectrum data of <FIG> may be performed by the imaging device <NUM> described above.

In operation <NUM>, the processor <NUM> obtains a spectrum response signal (or a spectrum response function) corresponding to each channel of spectrum data obtained by the image sensor <NUM>.

The spectrum data obtained by the image sensor <NUM> may include spectrum data (or a spectrum signal) of a plurality of channels corresponding to respective wavelengths, and the spectrum data may include spectrum response signals indicating spectrum characteristics corresponding to the respective plurality of channels.

In detail, the spectrum response signal of each channel may be represented by Equation <NUM>.

In Equation <NUM>, f(λ), I(λ), and D denote a spectrum of a filter, a spectrum of incident light, and the intensity of detected light, respectively. In addition, L denotes the number of channels of the spectrum data.

Equation <NUM> regarding the spectrum response signal of each channel may be represented by Equation <NUM> in a matrix form.

In operation <NUM>, the processor <NUM> determines a basis set corresponding to the spectrum response signals. Here, the processor <NUM> may use a principal component analysis (PCA) algorithm. In detail, the processor <NUM> may obtain a principal component with respect to the spectrum response signal of each channel by performing PCA, and determine the basis set with respect to the channels based on the obtained principal component.

The determined basis set may include the same number of bases as the number of channels of the spectrum. As another example, the determined basis set may include a smaller number of bases than the number of channels of the spectrum. As the number of bases is increased, the spectral resolution may be increased, but the image sensor <NUM> may be sensitive to noise. On the other hand, as the number of bases is decreased, the image sensor <NUM> may be relatively insensitive to noise, but the spectral resolution may be decreased. Therefore, the performance of the reconstructed spectrum data may be adjusted by optimizing the number of bases to be included in the basis set according to the characteristics of the imaging device.

In operation <NUM>, the processor <NUM> performs a change of basis on the spectrum response signals of the channels, based on the determined basis set. As the change of basis is performed on the spectrum response signal of each channel, the spectrum response signal of each channel has orthogonality. Consequently, an effect of crosstalk between the channels is reduced, and thus, errors in spectrum reconstruction using a pseudo inverse are reduced.

In operation <NUM>, the processor <NUM> generates, by using the pseudo inverse, reconstructed spectrum data from the spectrum response signals on which the change of basis has been performed. That is, the processor <NUM> may generate the reconstructed spectrum data in which the effect of crosstalk is compensated for, from input spectrum data, by performing the change of basis and using the pseudo inverse with respect to the spectrum data obtained by the image sensor <NUM>.

<FIG> is a flowchart of a method of processing spectrum data of an image sensor to generate reconstructed spectrum data, according to some example embodiments. The method of <FIG> may also be applied to operations <NUM> to <NUM> of <FIG>.

In operation <NUM>, the processor <NUM> obtains a principal component with respect to a spectrum response signal of each channel by using a PCA algorithm, and obtains bases corresponding to the spectrum response signals of the channels, based on the obtained principal component.

In operation <NUM>, the processor <NUM> may perform the following processing schemes to select only meaningful (or optimal) bases from among the obtained bases and perform a change of basis.

In detail, in order to select only the meaningful bases, a matrix F may be expressed as a singular value decomposition (SVD) as shown in Equation <NUM>.

In Equation <NUM>, ∑ denotes an N × L -dimensional diagonal matrix, U, VT denote N × N, L × L -dimensional unitary matrices, respectively, and the SVD may be possible for any matrix F.

In order to determine a unique SVD, the diagonal elements of the matrix ∑ are usually arranged in descending order of magnitude, and the values of the diagonal elements may be defined as singular values.

In the basis converted into the unitary matrices U, V , the spectrum response signal may be represented by Equation <NUM>.

Small singular values may amplify noise when their inverses are used. Accordingly, in order to exclude small singular values, it may be desirable to use only a few singular values greater than or equal to a preset magnitude. Accordingly, a truncation matrix is used for removing small singular values that may cause noise.

A conventional PCA method as shown in Equation <NUM> is a dimensionality reduction method of replacing all small singular values with <NUM> and removing singular eigenvectors corresponding to the small singular values.

However, in an example embodiment, a method of equally replacing small singular values with the previous smallest values by using a truncation matrix of Equation <NUM> modified from the conventional PCA (e.g., Equation <NUM>) is used, rather than regarding them as <NUM>'s, such that dimensionality reduction is not performed. Accordingly, compared to the conventional method (e.g., Equation <NUM>), the method of an example embodiment may minimize information lost due to dimensionality reduction.

The processor <NUM> may perform the processing method described above to determine a number of bases that have a preset threshold value or greater and singular values of the bases among the bases obtained in operation <NUM> and thus determine a basis set.

As another example, when there is a basis, the singular value of which is less than or equal to a threshold value, the processor <NUM> determines the basis set by replacing the singular value of the basis that is less than or equal to the threshold value with the threshold value or with the lowest value among the singular values greater than or equal to the threshold value.

That is, the processor <NUM> determines at least one basis having a singular value less than the threshold value among bases corresponding to the obtained spectrum response signals, and replaces the singular value of the determined at least one basis with a certain singular value. Accordingly, the basis set includes at least one basis, the singular value of which is replaced with a certain singular value. The certain singular value is the threshold value or the lowest value among the singular values of other bases having a singular value greater than or equal to the threshold value.

In operation <NUM>, the processor <NUM> performs a change of basis on the spectrum response signals of the channels, based on the determined basis set, and generates reconstructed spectrum data from the spectrum response signals on which the change of basis has been performed, by using a pseudo inverse. That is, the processor <NUM> may generate, from input spectrum data, the reconstructed spectrum data in which the effect of crosstalk is compensated for, by performing the change of basis and using the pseudo inverse with respect to the spectrum data obtained by the image sensor <NUM>.

The processor <NUM> may obtain a spectrum I(λ) of incident light by using a pseudo inverse as shown in Equation <NUM>.

A reconstructed spectrum D is obtained by using pseudo inverses Ĩ, D̃ having elements truncated by the truncation matrix described with reference to Equation <NUM>.

In operation <NUM>, the processor <NUM> may generate hyperspectral image data based on the reconstructed spectrum data. For example, the reconstructed spectrum data D may be converted into values of an XYZ color space by using Equation <NUM>, which is an XYZ color matching function of the CIE color space.

In addition, values of an XYZ color space may be converted into values of an RGB color space by using Equation <NUM>.

The processor <NUM> may obtain image pixel data constituting a hyperspectral image by converting the reconstructed spectrum data into values of a certain color space.

That is, the processor <NUM> may perform imaging processing having an improved resolution (e.g., spatial resolution and/or spectral resolution) by generating hyperspectral image data in which an effect of crosstalk is reduced, based on the reconstruction spectrum data instead of input spectrum data obtained by the image sensor <NUM>.

<FIG> is a diagram illustrating results of simulation of applying reconstructed spectrum data, according to some example embodiments.

<FIG> shows graphs for comparing simulation results with respect to reconstructed spectrum data according to example embodiments, with simulation results of spectrum data to which a reconstruction scheme according to the example embodiments is not applied.

Graphs <NUM>, <NUM>, and <NUM> represent spectrum data reconstructed from a blue patch, a green patch, and a red patch of a Macbeth color chart measured under a fluorescent lamp, according to example embodiments. The measured spectra represent signals having passed through <NUM> filters, with respect to the center wavelength of each filter.

Graphs <NUM>, <NUM>, and <NUM> represent spectrum data obtained by measuring a blue patch, a green patch, and a red patch of a Macbeth color chart under a fluorescent lamp. That is, the graphs <NUM>, <NUM>, and <NUM> are based on the spectrum data to which the reconstruction scheme according to the example embodiments is not applied. Likewise, the measured spectra represent signals having passed through <NUM> filters, with respect to the center wavelength of each filter.

Comparing the graphs <NUM>, <NUM>, and <NUM> with the graphs <NUM>, <NUM>, and <NUM>, respectively, the graphs based on the reconstructed spectrum data according to the example embodiments are measured to be similar to the reference spectra than the spectrum data without the reconstruction scheme of the example embodiments, and thus, it may be seen that the effect of crosstalk is reduced.

<FIG> is a diagram illustrating an influence of the number of bases on the performance of reconstructed spectrum data, according to some example embodiments.

Referring to <FIG>, graphs <NUM> to <NUM> represent simulation results of spectrum data reconstructed by using different numbers of bases, the numbers ascending in the order of the corresponding reference numerals. The graphs <NUM> to <NUM> shows that, as the number of bases is increased, the spectral resolution may be increased but the data may be sensitive to noise. On the other hand, the graphs <NUM> to <NUM> show that, as the number of bases is decreased, the data may be insensitive to noise but the spectral resolution may be decreased. Therefore, the performance of reconstructed spectrum data may be adjusted by optimizing the number of bases to be included in a basis set according to the characteristics of the imaging device.

<FIG> is a diagram illustrating an influence of singular values of a basis, according to some example embodiments.

Referring to <FIG>, graphs <NUM>, <NUM>, <NUM>, and <NUM> represent results of performing spectrum reconstruction by setting threshold values to singular values of bases having first, fifth, ninth, and thirteenth magnitudes, respectively, among a total of <NUM> bases (that is, among <NUM> magnitudes of bases arranged from smallest to greatest).

As the number of bases maintaining their original singular values is increased, the spectral resolution may be maintained to be high but the data may be sensitive to noise. On the other hand, as the number of bases maintaining their original singular values is decreased, the spectral resolution may be decreased, but distortion due to noise may be reduced.

<FIG> is a diagram illustrating RGB conversion of reconstructed spectrum data, according to some example embodiments.

<FIG> illustrates an original image <NUM>, an RGB image <NUM> that is color-converted from reconstructed spectrum data with respect to the original image <NUM> according to the example embodiments, and an RGB image <NUM> that is expressed with channels corresponding to R, G, and B wavelengths without reconstruction, with respect to the original image <NUM>. Comparing the RGB image <NUM> with the RGB image <NUM>, it may be seen that the RGB image <NUM> that is color-converted from the reconstructed spectrum data according to the example embodiments may be more improved in color representation than the RGB image <NUM>.

The above-described method may be provided by using a non-transitory computer-readable recording medium having recorded thereon one or more programs including instructions for executing the method. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, or magnetic tapes, optical media such as CD-ROMs or digital video discs (DVDs), magneto-optical media such as floptical disks, and hardware devices such as ROM, RAM, and flash memory, which are specially configured to store and execute program instructions. Examples of the program instructions include not only machine code, such as code made by a compiler, but also high-level language code that is executable by a computer by using an interpreter or the like.

At least one of the components, elements, modules or units (collectively "components" in this paragraph) represented by a block in the drawings may be embodied as various numbers of hardware, software and/or firmware structures that execute respective functions described above, according to an example embodiment. According to example embodiments, at least one of these components may use a direct circuit structure, such as a memory, a processor, a logic circuit, a look-up table, etc. that may execute the respective functions through controls of one or more microprocessors or other control apparatuses. Also, at least one of these components may be specifically embodied by a module, a program, or a part of code, which contains one or more executable instructions for performing specified logic functions, and executed by one or more microprocessors or other control apparatuses. Further, at least one of these components may include or may be implemented by a processor such as a central processing unit (CPU) that performs the respective functions, a microprocessor, or the like. Two or more of these components may be combined into one single component which performs all operations or functions of the combined two or more components. Also, at least part of functions of at least one of these components may be performed by another of these components. Functional aspects of the above exemplary embodiments may be implemented in algorithms that execute on one or more processors. Furthermore, the components represented by a block or processing steps may employ any number of related art techniques for electronics configuration, signal processing and/or control, data processing and the like.

Claim 1:
A method of processing spectrum data, the method comprising:
obtaining (<NUM>) spectrum response signals corresponding to channels of spectrum data of light, wherein a light source (<NUM>) irradiates the light of a plurality of different wavelength bands toward an object, and the spectrum data is obtained from the object by an image sensor (<NUM>);
determining (<NUM>) a set of bases corresponding to the obtained spectrum response signals;
performing (<NUM>), based on the determined set of bases, a change of basis on at least one basis included in the determined set of bases by using a pseudo inverse with respect to the spectrum data obtained by the image sensor (<NUM>), wherein the changed bases provide orthogonality for the spectrum response signal of each channel; and
generating (<NUM>), by using the pseudo inverse, reconstructed spectrum data from the spectrum response signals on which the change of basis has been performed,
wherein the pseudo inverse provides elements truncated by a truncation matrix in that for bases of which a singular value is determined to be smaller than a predetermined magnitude threshold value, the respective singular value is replaced with the threshold value or with a lowest value among the singular values greater than or equal to the threshold value.