Patent ID: 12231830

DETAILED DESCRIPTION

Chromophores and fluorophores, mainly hemoglobin, melanin, and residing bacteria, in the skin tissue are factors which have potential impacts on the skin assessments, both in clinical dermatology and cosmetics. Hemoglobin concentration is related to features including but not limited to skin redness, inflammations and vascular abnormalities. Melanin variation often presents in skin pigmentations, nevus and some skin cancers. Typical bacteria that reside in the skin, including but not limited toCutibacterium acnes, are usually active in sebaceous glands. The relevant metabolic activities of these bacteria trigger risk of inflammation, leading to skin disorders such as folliculitis and acne vulgaris. Another common site of bacterial reproduction is within the oral cavity, potentially causing dental plaque that may progress to tooth decay, and further to periodontal diseases. Due to different optical properties, spectroscopic analysis is sometimes used in the quantitative measurements of these chromophores and fluorophores. Hyperspectral imaging provides a strategy with both spectral analysis and snapshot visualization abilities, which has shown the attractive potential to be more widely used in skin assessments. This is also true in the assessment of ocular tissue, ear tissue, and the oral cavity. There also may have interesting applications in the open surgery where the blood perfusion and chromophore/fluorophore concentrations within the exposed organs are often desired information to plan steps of surgical procedures.

Embodiments of systems and methods to use images captured by a low-dimensional color space camera (including but not limited to an RGB camera, such as an RGB camera of a smartphone) to provide hyperspectral information are presented herein. Since an RGB image lacks enough wavebands and spectral resolutions to properly conduct hyperspectral analysis, a transformation matrix may be applied to transform RGB images into hyperspectral images. Wiener estimation may provide accurate and high-resolution hyperspectral reconstruction.

The creation of a suitable transformation matrix may be conducted by the use of high resolution spectrometers. However, this type of calibration process uses an extremely high workload and perhaps instabilities because it has to be done at each wavelength one by one. Alternatively, in some embodiments of the present disclosure, a different training strategy may be used that lowers the workload: a state-of-the-art snapshot hyperspectral camera with 16 wavebands ranging from 470 nm to 720 nm may provide spectral reflection calibration. With the snapshot hyperspectral camera, the measurements of multiple wavelengths, multiple samples, and the co-registration of sampling area in each sample may be achieved without the high workload by selecting and calculating corresponding areas of the color chart in the RGB and hyperspectral images. The calibration may be performed by taking RGB and hyperspectral images of the color chart under the same illumination, reducing the workload in the process and increasing the stability of calibration.

In some embodiments of the present disclosure, the calibration process results in the construction of a Wiener estimation matrix that is applied to transform RGB images into hyperspectral images. Since the transformation may be done pixel by pixel, the reconstructed hyperspectral images may possess the same spatial pixel resolution as the original RGB images (typically around 800 million spatial pixels), and each pixel bearing spectral information with 16 wavebands in visible range. With weighted subtractions between different wavebands, the target chromophores may be contrasted from surrounding tissues.

Coupling with above hyperspectral reconstruction and post-processing steps, the smartphone camera's abilities to visualize blood, melanin absorption maps and oxygen saturation in the facial skin were tested. Compared with conventional hyperspectral imaging systems, which mostly rely on lasers or tunable optical filters, the smartphone-based hyperspectral imaging system eliminates the internal time difference within frames, greatly improving the imaging speed and immunity to motion artifacts. Furthermore, compared with advanced and costly hyperspectral cameras, the smartphone-based hyperspectral imaging disclosed herein is superior in terms of its spatial resolution. In addition, embodiments of the system and method show flexibility in terms of different illumination conditions. Embodiments of the present disclosure do not require any modification or addition to the existing hardware of smartphones, which makes hyperspectral imaging and analysis of skin tissue possible in daily scenes outside of controlled lab environments. This may be particularly important and appealing for the inhabitant regions where the resource-settings are relatively low.

Embodiments of the present disclosure may also apply to facilitate the diagnosis and prognosis of some other dermatosis with chromophore abnormalities, like malignant melanoma. In addition, the smartphone-based operation would hold the enormous promises in cosmetic applications, where the assessments of the UV spots and the skin hyperpigmentation are often desired. Embodiments of the system and method were further noted, with the extraction and separation of chromophores, to provide an ability of smartphone-based systems to quantitatively analyze and monitor skin temporal activities.

Embodiments of the present disclosure may also apply to facilitate the detection and analysis of fluorophore-producing bacteria when illuminated with a short-wavelength illumination source to cause autofluorescence. The presence of bacteria may be used to predict or diagnose various skin disorders, and/or may be used to predict or diagnose the formation of undesirable dental plaques.

As mentioned above, a transformation matrix can be used in order to obtain high-dimensional color space information from an image that contains low-dimensional color space information. For example, a transformation matrix can be determined that can extract information from more than three color bands from image information limited to three color bands (such as red-green-blue (RGB) information). In order to use such a transformation matrix, it is first determined using a calibration process. Traditionally, a calibration process includes measuring spectral reflection from color blocks with a well-characterized spectrometer. The sampling areas in the images taken by low-dimensional color space camera and the spectral reflection measurements need to match with each other, which is termed the co-registration step. While this is a viable approach, it increases the workload and introduces additional instabilities in the calibration. Furthermore, it requires performing the reflection measurement and co-registration at each color sample for all of the color samples, which may cause even heavier workloads and more instabilities.

In some embodiments, a snapshot hyperspectral camera may be used to mitigate the tedious procedures when using spectrometer for calibration.FIG.1is a schematic drawing that illustrates some components of a non-limiting example embodiment of such a system for calibrating a device with a low-dimensional color space camera to generate pseudo-high-dimensional color space information according to various aspects of the present disclosure.

As shown, the calibration system100includes a high-dimensional color space camera102, a computing device104, and a color chart106. The color chart106is illuminated by an illumination source, a reference image of the color chart106is captured by the high-dimensional color space camera102, and a calibration image of the color chart106is captured by a low-dimensional color space camera of the computing device104.

In some embodiments, the high-dimensional color space camera102is any camera capable of capturing images in a high-dimensional color space. As used herein, the term “high-dimensional color space” refers to color information that is divided into more than three spectral channels, color bands, or wavelength bands, and the term “high-dimensional color space” may be used interchangeably with the term “hyperspectral”. One non-limiting example of a high-dimensional color space camera102is a MQ022HG-IM-SM4X4-VIS, from XIMEA, Germany, with 16 spectral channels. By using a high-dimensional color space camera102, many of the training steps that would be used if a spectrometer were being used are replaced by taking one low-dimensional color space image with the computing device104and one high-dimensional color space image of the color chart106. One non-limiting example of a device suitable for use as a high-dimensional color space camera102in the calibration system100is a MQ022HG-IM-SM4X4-VIS, made by XIMEA, Germany. This device houses a CMOS sensor with 2048×1088 pixels, where a filter array separates the sensor array into 512×272 super-pixels. Each super-pixel contains a 4×4 spectral sensitive pixel-matrix that are sensitive to 16 wavebands.

In some embodiments, the computing device104includes a low-dimensional color space camera, which is used to capture an image of the color chart106in the low-dimensional color space. As used herein, the term “low-dimensional color space” refers to color information that is divided into three spectral channels, color bands, or wavelength bands, and the term “RGB” may be used interchangeably with the term “low-dimensional.” Though a red-green-blue (RGB) color space is discussed primarily herein as the low-dimensional color space, one will note that other low-dimensional color spaces may be used without departing from the scope of the present disclosure. For example, instead of an RGB color space, some embodiments of the present disclosure may use a CMYK color space, a YIQ color space, a YPbPr color space, a YCbCr color space, an HSV color space, an HSL color space, a TSL color space, a CIEXYZ color space, an sRGB color space, an L*A*B color space, or an ICtCp color space.

In some embodiments, the computing device104includes the low-dimensional color space camera, the illumination source, and processing elements to be used for calibration as discussed below. In some such embodiments, the computing device104may be a smartphone or other device that includes all of these elements. In some embodiments, the low-dimensional color space camera, the illumination source, and/or the processing elements may be provided in separate devices. Importantly, the illumination source and low-dimensional color space camera used during calibration should match the illumination source and low-dimensional color space camera used during visualization generation or other processes. To match, in some embodiments the illumination source used during visualization generation may be the same device as the illumination source used during calibration, and the low-dimensional color space camera used during visualization may be the same device as the low-dimensional color space camera used during calibration. In some embodiments, matching devices may be different devices with matching performance characteristics. For example, the illumination source may be a flashlight of a smartphone, and matching devices may be different smartphones of the same model (which would have the same model flashlight with the same spectral characteristics). As another example, the illumination source may be a camera of a smartphone, and matching devices may again be different smartphones of the same model (which would have the same resolution and spectral performance characteristics).

In some embodiments, the color chart106includes a plurality of different colors to be imaged by the high-dimensional color space camera102and the computing device104. In some embodiments, the color chart106may include 100 different colors spaced throughout a visible spectrum using any suitable technique, including but not limited to being randomly spaced and being evenly spaced. In some embodiments, a smaller or greater number of colors may be used. In some embodiments, the color chart106may also include a portion from which spectral characteristics of the illumination source may be determined. As a non-limiting example, the color chart106may include a polymer white diffuser standard, such as a standard of 95% reflectance manufactured by SphereOptics GmbH. In some embodiments, such a standard from which spectral characteristics of the illumination source may be separate from the color chart106.

In some embodiments, the wavelength information may be obtained from the color chart106using some other technique, including but not limited to measurement by a spectrometer. In some embodiments, the color chart106may include colors that represent known values (that is, the wavelength information associated with each color block is known). In such embodiments, capturing the reference image may be skipped, and the known values for the color chart106may be used to calibrate the computing device104as descried below.

FIG.2AandFIG.2Bare schematic drawings that illustrate a non-limiting example embodiment of a case for a smartphone computing device according to various aspects of the present disclosure. InFIG.2A, an exterior portion of a smartphone case202is illustrated. The smartphone case202is configured to be attached to a smartphone computing device in order to protect the smartphone computing device from various types of physical damage. As shown, the smartphone case202includes an aperture204sized to avoid obscuring a low-dimensional color space camera and an illumination source of the smartphone computing device while the smartphone case202is attached to the smartphone computing device. The smartphone case202also includes a set of short-wavelength LEDs206. By including short-wavelength LEDs206on the smartphone case202, a smartphone computing device may be used to detect autofluorescence even if the smartphone computing device does not otherwise have a short-wavelength illumination source.

InFIG.2B, an interior portion of a smartphone case202is illustrated. The aperture204is still visible inFIG.2B, as is a color chart208. The color chart208is provided on the smartphone case202so that any smartphone computing device that fits the smartphone case202may be calibrated using the same color chart208, and the smartphone computing device may conveniently be recalibrated in different illumination conditions.

FIG.3is a block diagram that illustrates a non-limiting example embodiment of a computing device according to various aspects of the present disclosure. As mentioned above, the computing device104may be a smartphone, or could be any other type of computing device with the illustrated components, including but not limited to a laptop computing device, a tablet computing device, a desktop computing device, or a kiosk.

In some embodiments, the components illustrated as part of the computing device104may be provided by more than one computing device. For example, one computing device may provide the low-dimensional color space camera302and the illumination source304, while another computing device provides the processor(s)306and one or more of the components of the computer-readable medium308. In one such embodiment, an image may be captured using the low-dimensional color space camera302and illumination source304of a first device, and the image may then be transmitted to a second computing device that provides the other components for further processing. As another example, the components of the computer-readable medium308may be provided by separate computing devices. For instance, one computing device (or collection of computing devices) may provide an image capture engine312, a calibration engine320, and a transform data store310for use in calibration, while another computing device (or collection of computing devices) may provide another transform data store310, another image capture engine312, a subtractive information engine316, a color space generation engine314, and a visualization generation engine318for generating visualizations. In some embodiments, some of the components illustrated as being present on the computer-readable medium308may be provided by a remote computing device including but not limited to one or more computing devices of a cloud computing system accessible via a network.

As shown inFIG.3, the computing device104includes a low-dimensional color space camera302, an illumination source304, one or more processor(s)306, and a computer-readable medium308.

In some embodiments, the low-dimensional color space camera302is a camera of a smartphone, and is configured to capture information in a low-dimensional color space including but not limited to an RGB color space. Typically, the low-dimensional color space camera302is capable of capturing images in a relatively high resolution, such as 3264×2448 pixels. In other embodiments, any low-dimensional color space camera302capable of capturing images in a low-dimensional color space may be used.

In some embodiments, the illumination source304may be incorporated into the computing device104, such as the flashlight of a smartphone. In some embodiments, the illumination source304may be separate from the computing device104, such as overhead lighting or studio lighting. It has been determined that regular and consistent light sources other than a flashlight of a smartphone, such as a fluorescent lamp, may also be used as an illumination source304.

In some embodiments, the processor(s)306include one or more general-purpose processors configured to execute instructions stored on the computer-readable medium308. In some embodiments, the processor(s)306may include one or more special-purpose processors that are optimized for performing particular types of operations, including but not limited to specialized graphics processors (GPUs) and processors optimized for machine learning operations. In some embodiments, the computer-readable medium308may be any type of medium on which data and/or computer-executable instructions may be stored, including but not limited to flash memory, a magnetic disk such as a hard disk drive, an optical disk such as a CD-ROM or DVD-ROM, random access memory (RAM), or a read-only memory (ROM). In some embodiments, some aspects of the processor(s)306and/or the computer-readable medium308may be provided via specially configured hardware such as an ASIC or an FPGA.

As shown, the computer-readable medium308has stored thereon computer-executable instructions that, in response to execution by one or more of the processor(s)306, cause the computing device104to provide a transform data store310, a calibration engine320, an image capture engine312, a color space generation engine314, a subtractive information engine316, and a visualization generation engine318.

In some embodiments, the calibration engine320is configured to use images captured by the high-dimensional color space camera102and the low-dimensional color space camera302to generate a transformation matrix, and to store the generated transformation matrix in the transform data store310.

In some embodiments, the image capture engine312is configured to control the high-dimensional color space camera102and the low-dimensional color space camera302in order to collect calibration images, reference images, and input images for various purposes. In some embodiments, the image capture engine312may also be configured to control the illumination source304. In some embodiments, the image capture engine312may not control the high-dimensional color space camera102and the low-dimensional color space camera302, but may instead receive or retrieve images captured by those devices on their own.

In some embodiments, the color space generation engine314is configured to use a transformation matrix stored in the transform data store310to extract high-dimensional color space information from an image captured by the low-dimensional color space camera302that contains low-dimensional color space information. In some embodiments, the subtractive information engine316is configured to process high-dimensional color space information to emphasize the contribution of reflections from desired wavelength bands while minimizing contributions of reflections from undesired wavelength bands. In some embodiments, the visualization generation engine318is configured to generate visualizations of the emphasized wavelength bands, which may be useful for analyzing wavelength-dependent features of the target surface. Further details of each of these actions are described below.

FIG.4is a flowchart that illustrates a non-limiting example embodiment of a method of calibrating a system for capturing high-dimensional color space information according to various aspects of the present disclosure. The method400may performed at any time that a transformation matrix is needed. In some embodiments, the method400may be used to create a transformation matrix for each computing device104that will be used to generate visualizations as described below. In some embodiments, the method400may be used to create a transformation matrix for each model of computing device104that will be used to generate visualizations, with the assumption that any illumination source304and low-dimensional color space camera302in computing devices of matching models will also match, and so a transformation matrix created for one example of a model of a computing device104will be usable by any other example of the same model of the computing device104(e.g., a transformation matrix created using an iPhone 11 as the computing device104would work with any iPhone 11).

From a start block, the method400proceeds to block402, where an illumination source304is used to illuminate a color chart106. As discussed above, the illumination source304may be incorporated into the computing device104, or may be separate from the computing device104. Either way, it should be noted that the same illumination source304(or an illumination source304with matching spectral band characteristics) will be used during any method that consumes the transformation matrix determined by this method400. In some embodiments, the room or other environment in which the color chart106is situated is kept dark other than the illumination source304so as to isolate reflections of the illumination source304from other ambient light sources of differing spectral characteristics.

At block404, a low-dimensional color space camera302captures a calibration image of the color chart106as illuminated by the illumination source304. The calibration image captured by the low-dimensional color space camera302includes low-dimensional color space information, such as RGB information split between a red band, a green band, and a blue band.

At block406, a high-dimensional color space camera102captures a reference image of the color chart106as illuminated by the illumination source304. The reference image captured by the high-dimensional color space camera102includes high-dimensional color space information, such as information separated into more than the three wavebands of the low-dimensional color space information. In one example embodiment, the high-dimensional color space information may be separated into 16 wavebands (also referred to herein as “subchannels”), though in other embodiments, more or fewer wavebands may be used. Typically, the reference image may have a lower resolution than the calibration image, though in some embodiments, the images may have matching resolutions.

At block408, a calibration engine320of a computing device104determines color reference information using the reference image. The color reference information uses the wavelength band intensities detected by the high-dimensional color space camera102as the true wavelength band intensities reflected by the color chart106. In some embodiments, only pixels that represent a center portion of each color box of the color chart106may be used so that precise detection of the edges of the color boxes is not necessary.

At block410, the calibration engine320determines a transformation matrix using the color reference information and the calibration image. When a reference image is captured by the high-dimensional color space camera102and color reference information is extracted, the response of each subchannel of n subchannels is depicted as:
V′C′=∫l(λ)γ(λ)fC′(λ)s′(λ)dλ=∫m′C′(λ)γ(λ)dλ(1)

where V′c′is the response of c'th subchannel (c=1, 2, 3, . . . , n), fc′(λ) is the spectral transmittance of the filter in c'th subchannel, s′(λ) is the spectral sensitivity. m′c′(λ) is the product of l(λ), fc(λ) and s(λ), which is the spectral responsivity of each subchannel in the high-dimensional color space camera102. The matrix form of equation (1) is then expressed as:
V′=M′γ(2)

where V′ is the vector of hyperspectral camera response, M′ is the matrix of spectral responsivity in the high-dimensional color space camera102. To reconstruct high-dimensional color space information from low-dimensional color space information, we assume the reconstruction matrix is W. The process is expressed as:
{tilde over (V)}′=WV(3)

where {tilde over (V)}′ is the reconstructed image having high-dimensional color space information. To ensure the accuracy of reconstruction, the minimum square error between the reconstructed high-dimensional color space information and the original reference image should be minimized. The minimum square error is calculated as:
e=(V′−{tilde over (V)}′)t(V′−{tilde over (V)}′)=V′tV−WtVtV′−WtVtV′+WtWV′tV′(6)
When the partial derivative of e with respect to W is zero, the minimum square error is minimized, expressed as:

∂e∂W=-〈V′⁢t⁢V〉+Wt⁢〈Vt⁢V〉=0(4)

The transformation matrix is derived as:
W=V′VtVVt−1(5)

where W is an ensemble-averaging operator,V′Vtis the correlation matrix between the hyperspectral response and low-dimensional color space camera response, andVVtis the autocorrelation matrix of the low-dimensional color space camera response.

At block412, the calibration engine320stores the transformation matrix in a transform data store310of the computing device104. In some embodiments, the transform data store310may be on the same computing device104as the calibration engine320. In some embodiments, the transform data store310may be on a different computing device104, such as in embodiments wherein a first computing device104is used to generate the transformation matrix, and that transformation matrix is then provided for use by other computing devices that match the computing device104.

The method400then proceeds to an end block and terminates. One will note that though the method400describes the convenient use of a high-dimensional color space camera102to capture a reference image and extract color reference information in block406to block408, other techniques for extracting the color reference information may be used. For example, in some embodiments, a spectrometer may be used to measure the color chart106and determine the color reference information. As another example, in some embodiments, known color values may be used to create the color chart106, and color reference information may be directly generated from the specification of the color chart106.

FIG.5is a flowchart that illustrates a non-limiting example embodiment of a method of generating a visualization of wavelength-dependent surface characteristics according to various aspects of the present disclosure.

From a start block, the method500proceeds to block502, where an illumination source304is used to illuminate a target surface. In some embodiments, the target surface may be an area of skin of a subject. Some embodiments of the present disclosure may be particularly suited for skin as a target surface, since reflectance of various components of skin (such as hemoglobin and melanin) varies by wavelength, and so the method500can be used to separately visualize these components. However, these examples should not be seen as limiting, and some embodiments of method500may be used to visualize wavelength-dependent surface features on other types of target surfaces.

The illumination source304matches the illumination source304used in method400during creation of the transformation matrix. In some embodiments, the illumination source304used at block502may be the same device used as the illumination source304in method400. In some embodiments, the illumination source304used in block502may be a different device that nevertheless has spectral wavelength characteristics that match those of the illumination source304used in method400. In some embodiments, the target surface and illumination source304may be positioned in an otherwise dark environment so that an affect of ambient light from sources other than the illumination source304is minimized.

At block504, a low-dimensional color space camera302captures an input image of the target surface. In some embodiments, the low-dimensional color space camera302may be operated to capture the input image by the image capture engine312. In some embodiments, the low-dimensional color space camera302may be operated manually by a user of the computing device104to capture the input image.

At block506, an image capture engine312of a computing device104receives the input image and provides the input image to a color space generation engine314of the computing device104. At block508, the color space generation engine314retrieves a transformation matrix from a transform data store310of the computing device104. The transformation matrix is the transformation matrix that was generated for the low-dimensional color space camera302and the illumination source304using method400. At block510, the color space generation engine314uses the transformation matrix as described above to convert information from the input image in the low-dimensional color space to information in a high-dimensional color space.

At block512, a subtractive information engine316of the computing device104uses a subtractive technique to obtain wavelength-dependent surface characteristic information from the high-dimensional color space information. By using a weighted subtraction technique, absorption caused by a first wavelength-dependent characteristic can be minimized while absorption caused by a second wavelength-dependent characteristic can be emphasized, thus allowing visualizations of the second characteristic.

The weighted subtraction is expressed as:
Cr=C1−KC2=mx1l1+ny1l1−K(mx2l2+ny2l2)=m(x1l1−Kx2l2)+n(y1l1−Ky2l2)  (6)
where Cris the estimated reflection of the target surface that is assumed to be influenced by the absorption of the first characteristic and the second characteristic, C1and C2are the detected reflections at two selected wavebands. K represents the ratio of weighted subtraction. m and n are the concentrations of the first characteristic and the second characteristic in the target surface. l1and l2are the illumination intensity at two selected wavebands. x1and x2are the reflectance of the first characteristic at two selected wavebands. y1and y2are the reflectance of the second characteristic at two selected wavebands. By setting the value of K to be

y1⁢l1y2⁢l2,
the reflection of the first characteristic in the data can be extracted via the simplified equation:

m=Cr(x1⁢l1-y1⁢l1·x2⁢l2y2⁢l2)(7)

For example, values of absorption caused by hemoglobin in blood may be subtracted from values of absorption caused by melanin in order to emphasize the melanin concentrations in various areas of skin. As another example, values of absorption caused by melanin may be subtracted from values of absorption caused by hemoglobin in order to show blood perfusion, or from values of oxyhemoglobin and deoxyhemoglobin in order to visualize circulation. In one example where hemoglobin was used as the first characteristic and melanin was used as the second characteristic, several red light wavebands, including 615 nm, 625 nm, 603 nm, and 595 nm were selected and subtracted one by one from green light wavebands, including 556 nm, 544 nm, 529 nm, and 543 nm. This weighted subtraction minimized or eliminated the melanin absorption to emphasize the hemoglobin absorption, thus showing blood perfusion in the target surface. By conducting similar weighted subtraction processing between blue light wavebands (582 nm, 494 nm, 472 nm, and 465 nm) and the green light wavebands, the superficial melanin can be emphasized. Likewise, by conducting similar weighted subtraction processing between 600 nm and 560 nm, autofluorescence of porphyrin-producing bacteria can be emphasized compared to background autofluorescence.

At block514, a visualization generation engine318of the computing device104generates a visualization of the wavelength-dependent surface characteristic. In some embodiments, the visualization may be an image that includes the reflection values as updated by the weighted subtraction process. In some embodiments, the visualization may be an image with data limited to the wavelengths of interest, as updated by the weighted subtraction. Some examples of visualizations are illustrated inFIG.6and described below.

The method500then proceeds to an end block and terminates.

Method500describes emphasizing a single surface characteristic. In some embodiments, multiple characteristics may be analyzed. For example, in one real-time monitoring experiment, hemoglobin absorption information was extracted from hyperspectral images reconstructed from RGB-image sequences and quantitatively analyzed the skin hemodynamics during heartbeat cycle and vascular occlusion. Furthermore, in the monitoring to vascular occlusion, besides the blood absorption, oxygen saturation (SaO2) was also estimated. In this case, oxyhemoglobin and deoxyhemoglobin are studied independently. The reflection is expressed as:
Ci=moxyxioxy+mdeoxideo+α  (8)

where Ciis the detected reflection at the selected wavebands, moxyand mdeoare the concentrations of oxy and deoxyhemoglobin, respectively, xioxyand xideoare the corresponding reflectance coefficients of oxyhemoglobin and deoxyhemoglobin, and α is a term that represents the light intensity losses caused by other chromophores, including melanin. Bands 4 (570 nm), 5 (581 nm) and 6 (556 nm) were selected for the evaluation of SaO2. Since the sensitive wavelengths in these bands are close to each other, it may be assumed that is a constant in the processing. From equation (8), the SaO2can be evaluated as:

SaO2=moxy(moxy+mdeo)(9)

To investigate the reconstruction performance, RGB images of 100 color blocks were reconstructed from the color chart into hyperspectral images with a Wiener estimation matrix. For each color block, the average value of relative errors of 16 subchannels were between initial and reconstructed hyperspectral reflectance were calculated. The maximum, minimum and average values of the averaged relative errors of 100 color blocks are 10.950%, 0.424% and 4.933%, respectively. Relative errors of the reconstruction are higher in some color blocks that were mainly with dark cold tone, which coincides with the lower spectral intensity at the band around 500 nm. To show the reconstruction in more detail, 6 samples were randomly selected and compared regarding their initial and reconstructed reflectance in 16 wavebands. The results indicate that reconstructed hyperspectral images from RGB images match well with the initial hyperspectral images.

FIG.6illustrates results of a skin analysis performed with a high-dimensional color space camera102versus a computing device104according to various aspects of the present disclosure. To compare the skin analysis performance of the RGB-camera-based hyperspectral imaging system and snapshot hyperspectral camera, the same skin area was imaged with two cameras and conducted via the same process. Melanin absorption information was extracted and compared as an example.FIG.6(a)shows the raw image from the band 9 in the snapshot hyperspectral camera. The extracted melanin absorption map from the images captured by hyperspectral camera is shown inFIG.6(b), where the details of two melanocytic nevi (marked by square boxes) are shown in the zoomed-in images (FIG.6(c, d)). Analyses of the images captured by the smartphone resulted in melanin absorption map shown inFIG.6(f).FIG.6(g)andFIG.6(h)are the zoomed-in view of the two melanocytic nevus, marked by square boxes inFIG.6(f). Since the smartphone camera has much more pixels than the snapshot camera, the melanin absorption map inFIG.6(f)performs much better in terms of image resolution. From the zoomed-in images, we can see that two melanocytic nevus absorption spots from smartphone-based system (FIG.6(g, h)) show clear edges and sizes. However, these characteristics are not clearly depicted by the snapshot hyperspectral camera largely due to its limited spatial resolution.

In some embodiments, characteristics other than melanin and hemoglobin may be analyzed. Indeed, any characteristic for which spectral characteristics differ may be analyzed. For example, the recent developments of fluorescence imaging have presented new interests and opportunities for assessing bacteria contents in human tissue. Porphyrins, for example, as the byproducts of bacterial reproduction and metabolism accumulating in sebaceous glands in skin and dental plaque, fluoresce red light signals under black light stimulation, which can be contrasted from green autofluorescence background (originated from endogenous tissue). By tracking porphyrins-induced red fluorescence signals, the bacteria-contaminated human tissue can be effectively mapped. To detect autofluorescence, an illumination source304that includes black light LEDs (365 nm) may be used, and a fluorescence color chart106may be used, such as a paper stained with various colors of black light fluorescent dyes used for body paint, mixed to create 40 different color blocks. This illumination source304and color chart106may be used to create a transformation matrix with 15 wavebands In some embodiments, a computing device104calibrated using a visible light illumination source may be used to make measurements with a short-wavelength illumination source304in order to cause autofluorescence in the calibrated wavelength ranges.

Once the transformation matrix is created and an input image is captured, the weighted subtraction may be used to emphasize porphyrin-related autofluorescence of bacteria from the background autofluorescence in order to generate a visualization. For emphasizing porphyrin-related autofluorescence compared to background autofluorescence from skin, we selected 600 nm and 560 nm as two wavebands to conduct dual-waveband processing as described above, and used a subtraction ratio of 1.22. We found that the autofluorescence signals from the background skin tissue were greatly decreased so that porphyrins were contrasted. For emphasizing porphyrin-related autofluorescence compared to background autofluorescence from teeth, we used the same wavebands, but used a weighted subtraction ratio of 0.58. We found that by using the techniques described herein, the bacteria-related features were enhanced between 5 to 15 times, thus allowing a clear visualization of the bacteria-related features to be generated.

FIG.7shows results of using the techniques described above to extract and visualize autofluorescence information according to various aspects of the present disclosure. Images (A1), (B1), and (C1) illustrate visualization of the background information from the 600 nm waveband. Images (A2), (B2), and (C2) illustrate visualization of the autofluorescence information from the 560 nm waveband. Images (A3), (B3), and (C3) illustrate the background-eliminated, bacteria-targeted feature mapping.

It is clear from the difference between the (1) images and the (3) images that the techniques described above provide a clear visualization of the autofluorescence information with the background information minimized.FIG.8illustrates a quantitative analysis of the effectiveness of these techniques of extracting information representing bacteria-produced porphyrins from background autofluorescence. The labeled numbers on the histogram are intensity ratios between porphyrin signals and background signals. The bars for (A), (B), and (C) show the unprocessed images, and the bars for (A1) through (C3) correspond to the images inFIG.7. As shown, the intensity ratios for the processed images are remarkably greater than the other images.

The particulars shown herein are by way of example and for purposes of illustrative discussion of the preferred embodiments of the present invention only and are presented in the cause of providing what is believed to be the most useful and readily understood description of the principles and conceptual aspects of various embodiments of the invention. In this regard, no attempt is made to show structural details of the invention in more detail than is necessary for the fundamental understanding of the invention, the description taken with the drawings and/or examples making apparent to those skilled in the art how the several forms of the invention may be embodied in practice.

As used herein and unless otherwise indicated, the terms “a” and “an” are taken to mean “one”, “at least one” or “one or more”. Unless otherwise required by context, singular terms used herein shall include pluralities and plural terms shall include the singular.

Unless the context clearly requires otherwise, throughout the description and the claims, the words ‘comprise’, ‘comprising’, and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to”. Words using the singular or plural number also include the plural and singular number, respectively. Additionally, the words “herein,” “above,” and “below” and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of the application.

The description of embodiments of the disclosure is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. While the specific embodiments of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize.

All of the references cited herein are incorporated by reference. Aspects of the disclosure can be modified, if necessary, to employ the systems, functions, and concepts of the above references and application to provide yet further embodiments of the disclosure. These and other changes can be made to the disclosure in light of the detailed description.

Specific elements of any foregoing embodiments can be combined or substituted for elements in other embodiments. Moreover, the inclusion of specific elements in at least some of these embodiments may be optional, wherein further embodiments may include one or more embodiments that specifically exclude one or more of these specific elements. Furthermore, while advantages associated with certain embodiments of the disclosure have been described in the context of these embodiments, other embodiments may also exhibit such advantages, and not all embodiments need necessarily exhibit such advantages to fall within the scope of the disclosure.

Example devices, methods, and systems are described herein. It should be understood the words “example,” “exemplary,” and “illustrative” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or feature described herein as being an “example,” being “exemplary,” or being “illustrative” is not necessarily to be construed as preferred or advantageous over other embodiments or features. The example embodiments described herein are not meant to be limiting. It will be readily understood aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

Furthermore, the particular arrangements shown in the Figures should not be viewed as limiting. It should be understood other embodiments may include more or less of each element shown in a given Figure. Further, some of the illustrated elements may be combined or omitted. Yet further, an example embodiment may include elements not illustrated in the Figures. As used herein and unless stated otherwise, “about” means +/−5% of the stated amount.