Patent Description:
Hyperpigmented spots are a common concern in the cosmetic skin industry. Like other perceived cosmetic skin blemishes, hyperpigmented spots can cause emotional and psychological distress to those afflicted by the condition. The vast majority of hyperpigmented facial spots are benign, but in some rare instances, a hyperpigmented spot may be an indication of a more serious skin condition (e.g., melanoma). Although histopathology is commonly used for the diagnosis of skin spots, non-invasive measurements are generally preferred because it reduces or eliminates some of the drawbacks associated with breaking the skin's barrier (risk of infection, scarring, etc.).

Non-invasive diagnostic techniques are known, but some non-invasive diagnostic techniques may not provide the desired level of accuracy for diagnosing spot type and/or severity. For example, different types of hyperpigmented spots can be difficult differentiate using naked eye examination. Additionally, naked eye examination can introduce varying degrees of subjectivity into a skin spot diagnosis, which may result in an inconsistent skin care regimen or skin care product recommendation, especially if different people are consulted for a diagnosis (e.g., beauty consultant versus a dermatologist). Thus, it would be desirable to use a non-invasive diagnostic method that removes at least some, and ideally all, of the subjectivity associated with a naked eye examination.

In some instances, a more objective assessment of hyperpigmentation may be provided by using a colorimeter or spectral meter, but only a small area of skin can be examined at each measurement. As a result, this process requires taking multiple measurements if the number of spots involved is large. In some instances, it can be difficult to provide a desired level of repeatability using colorimeter or spectral meter because it is difficult to relocate the same exact area in each test. Accordingly, a need exists in the industry for a system for identifying and classifying hyperpigmented spots on a subject.

Prior art document <CIT> discloses a system for identifying a hyperpigmented spot, but this system does not use a cross-polarised light, it does not use a baseline image for determining changes in the hyperpigmented spot and performs only a very basic classification thereof.

Disclosed herein is a system for identifying hyperpigmented spots in accordance with claim <NUM>. The dependent claims relate to further embodiments.

In accordance with the present invention, the system herein includes a computing device that stores logic that, when executed by a processor, causes the computing device to receive a digital image of a subject, where the digital image of the subject is captured using cross-polarized lighting, receive a baseline image of the subject that was not captured using cross-polarized lighting, and identify a hyperpigmented spot in the digital image of the subject. The logic causes the computing device to provide the baseline image and an electronically annotated version the digital image of the subject to distinguish the hyperpigmented spot for display, classify the hyperpigmented spot into a predetermined class, and determine a product for treating the hyperpigmented spot according to the predetermined class. The logic also causes the computing device to provide information related to the product for use by the subject.

Also disclosed is a dispensing device as provided by claim <NUM>.

"About" means inclusively within <NUM>% of the stated value.

"Cosmetic" means a non-medical method of providing a desired visual effect on an area of the human body. The visual cosmetic effect may be temporary, semi-permanent, or permanent.

"Cosmetic agent" means any substance, as well any component thereof, intended to be rubbed, poured, sprinkled, sprayed, introduced into, or otherwise applied to a mammalian body or any part thereof to provide a cosmetic effect (e.g., cleansing, beautifying, promoting attractiveness, and/or altering the appearance).

"Cosmetic products" are products that include a cosmetic agent (e.g., skin moisturizers, lotions, perfumes, lipsticks, fingernail polishes, eye and facial makeup preparations, cleansing shampoos, hair colors, shave prep, and deodorants).

"Hyperpigmented" and "hyperpigmented spot" mean a localized portion of skin with relatively high melanin content compared to nearby portions of skin in the same general area of the body. Examples of hyperpigmented spots include, but are not limited to age spots, melasma, chloasma, freckles, post-inflammatory hyperpigmentation, sun-induced pigmented blemishes, and the like.

"Improve the appearance of" means providing a measurable, desirable change or benefit in skin appearance, which may be quantified, for example, by a reduction in the spot area fraction of a hyperpigmented spot and/or an increase in L* value of a hyperpigmented spot. Methods for determining spot area fraction and L* value and changes in these properties are known to those skilled in the art. Some non-limiting examples of these methods are described in co-pending <CIT>.

"Skin care" means regulating and/or improving a skin condition. Some nonlimiting examples include improving skin appearance and/or feel by providing a smoother, more even appearance and/or feel; increasing the thickness of one or more layers of the skin; improving the elasticity or resiliency of the skin; improving the firmness of the skin; and reducing the oily, shiny, and/or dull appearance of skin, improving the hydration status or moisturization of the skin, improving the appearance of fine lines and/or wrinkles, improving skin exfoliation or desquamation, plumping the skin, improving skin barrier properties, improve skin tone, reducing the appearance of redness or skin blotches, and/or improving the brightness, radiancy, or translucency of skin.

"Subject" refers to a person upon whom the use of methods and systems herein is for cosmetic purposes.

Disclosed herein are systems for identifying hyperpigmented spots and methods of using the system. Different types of hyperpigmented spots have different treatments and prognoses, and thus the systems and methods herein may be configured to provide correct diagnoses and consistent monitoring of hyperpigmented spots for planning management. For example, the systems and method herein may be configured to automatically classify hyperpigmented facial spots into eight different types of hyperpigmentation: solar lentigo, melasma, seborrhoeic keratosis, melanocytic nevus, freckle, actinic keratosis, post inflammatory hyperpigmentation and none of the above. Surprisingly, it has been found that classifying and annotating hyperpigmented spots, as described herein, has had a dramatic effect in proper treatment and reduction of the hyperpigmented spots. It has also been found that the process of creating a fitted ellipse around an image of the hyperpigmented spot and performing pixel analysis to determine texture of a hyperpigmented spot has greatly improved the classification, treatment, and appearance of hyperpigmented spots. In particular, the present method improves the ability of a computer to accurately predict the classification of a hyperpigmented spot.

<FIG> depicts an exemplary computing environment for identifying hyperpigmented spots. As illustrated, a network <NUM> is coupled to a user computing device 102a, a dispensing device 102b, a mobile device 102c, and a remote computing device <NUM>. The network <NUM> may include any wide area network, local network, etc. As an example, the network <NUM> may include the internet, a public switch telephone network, a cellular network (such as <NUM>, <NUM>, LTE, etc.). Similarly, the network <NUM> may include local networks, such as a local area network, Bluetooth network, Zigbee, near field communication, etc..

Coupled to the network <NUM> are the user computing device 102a, the dispensing device 102b and the mobile device 102c (individually and collectively referred to herein as "the device <NUM>"). The user computing device 102a may be configured as any computing device that may be utilized for capturing images, communicating with the remote computing device <NUM>, and/or providing one or more user interfaces to a user. As such, the user computing device 102a may be configured as a personal computer, a laptop, and the like. Additionally, while the image capture device may be integrated into the user computing device 102a (and/or the devices 102b, 102c), some embodiments of a system may include a separate image capture device (e.g., a conventional stand-alone digital camera) that captures imagery described herein and is capable of transferring that imagery (or data related to that imagery) to the appropriate device.

The dispensing device 102b may include a computer, display, input device, as well as hardware for dispensing one or more products. As such, the dispensing device 102b may include similar functionality as the user computing device 102a, except with the ability to dispense products, such as one or more cosmetic products or cosmetic agents. The mobile device 102c may also include similar hardware and functionality but may be configured as a mobile phone, tablet, personal digital assistant, and/or the like.

Regardless, the user computing device 102a, the dispensing device 102b, and/or the mobile device 102c may include an image capture device that is configured to capture digital images of a subject. As described in more detail below, some of the images may include a cross polarization light and/or filter. As such, some embodiments of the image capture device may utilize one or more lenses when capturing an image.

The remote computing device <NUM> may be configured to communicate with the user computing device 102a, the dispensing device 102b, and/or the mobile device 102c via the network <NUM>. As such, the remote computing device <NUM> may be configured as a server, personal computer, smart phone, laptop, notebook, kiosk, and the like. The remote computing device <NUM> may include a memory component <NUM> and other components depicted in <FIG>, which store identifier logic 144a and treatment logic 144b. As described in more detail below, the identifier logic 144a may be configured to analyze images to identify a hyperpigmented spot. The treatment logic 144b may be configured to determine one or more product and/or treatment regimens for treating the identified hyperpigmented spot.

It will be understood that while the identifier logic 144a and the treatment logic 144b are depicted as residing in the memory component <NUM> of the remote computing device <NUM>, this is merely an example. Some embodiments may be configured with logic for performing the described functionality in the user computing device 102a, the dispensing device 102b, and/or the mobile device 102c. Similarly, some embodiments may be configured to utilize another computing device not depicted in <FIG> for providing at least a portion of the described functionality.

It will also be understood that the systems in accordance with the present invention are utilized for a consumer in the field of cosmetics (e.g., for skin care). Embodiments related to the medical field include products for and/or methods relating to the treatment of a medical condition and are not part of the present invention. This includes products that require operation by a health care professional; products used by a health care professional in the course of a medical diagnosis; products used in the treatment of a disease or other medical condition requiring treatment by a healthcare professional; products sold with a prescription; and the activities of cosmetic/plastic surgeons, dermatologists, general medical practitioners, and pharmaceutical companies.

Additionally, it will be understood that while the remote computing device <NUM> is depicted in <FIG> as including the logic 144a, 144b, this is also an example. In some embodiments, the device <NUM> may operate independently from the remote computing device <NUM> and may only communicate with the remote computing device <NUM> for updates and other administrative data. Other embodiments may be configured such that the remote computing device <NUM> provides substantially all of the processing described herein and the user computing device 102a is simply used as a terminal. Still other embodiments may operate as hybrids of these examples and/or leverage one or more of the devices <NUM> for providing functionality for another of the devices <NUM>. As an example, a user may capture an image via the mobile device 102c and may send that image to the dispensing device 102b to analyze and provide product and treatment recommendations.

<FIG> depicts a user interface <NUM> for capturing an image of a subject and performing spot determination, according to embodiments described herein. As illustrated, the user interface <NUM> includes a captured image, a capture image option <NUM>, a capture filtered image option <NUM>, a run spot determination option <NUM>, and a manually identify spot option <NUM>.

In response to selection of the capture image option <NUM>, the device <NUM> may capture an image of the subject. As discussed above, the image may be captured by the device <NUM> or may be communicated to the device <NUM> and/or to the remote computing device <NUM>. Regardless, the image may depict one or more hyperpigmented spots on the face of the subject and may be a white light image, unfiltered image, and/or baseline image of the subject.

In response to selection of the capture filtered image option <NUM>, a cross-polarized image may be captured. Depending on the particular embodiment, the cross-polarized image may be captured using cross-polarized light and/or may be captured via a cross-polarized filter. The cross-polarized image is a digital image in some embodiments. In response to selection of the run spot determination option <NUM>, spot identification and classification may commence. In response to selection of the manually identify spot option <NUM>, the user may manually identify a hyperpigmented spot, as described in more detail below.

In response to selection of the run spot determination option <NUM> from <FIG>, the user interface <NUM> illustrated in <FIG> may be provided. Additionally, the remote computing device <NUM> (and/or the device <NUM>, depending on the embodiment) may process the image to identify and classify hyperpigmented spots on the image of the subject. The user interface <NUM> also includes an annotate spot option <NUM>, a zoom filter spot option <NUM>, a manually annotate spot option <NUM>, a zoom spot option <NUM>, and a remove spot option <NUM>.

Also provided in the user interface <NUM> is an image <NUM> of the subject, and images of the hyperpigmented spot <NUM> and <NUM>. In response to selection of the annotate spot option <NUM>, the image <NUM> may be annotated with an overlay <NUM> that highlights the identified spot. In response to selection of the zoom filter spot option <NUM>, the digital image of the subject <NUM> may be provided, which is a cross polarized and zoomed image (e.g., 2x, 3x, 4x, 5x, <NUM>×, or even up to 100x magnification) of the identified spot. In response to selection of the manually annotate spot option <NUM>, additional options may be provided for the user to select and annotate the image manually. In response to selection of the zoom spot option <NUM>, a baseline image <NUM> may be provided, which is a zoomed image (e.g., 2x, 3x, 4x, 5x, 10x, or even up to 100x magnification) of the annotated hyperpigmented spot (without filter). In some embodiments the digital image of the subject <NUM> may be compared with the baseline image <NUM> to determine at least one feature of the hyperpigmented spot. In response to selection of the remove spot option <NUM>, a previously identified spot may be removed from consideration by the user.

It should be understood that while zoomed versions of the images may be compared, as depicted in <FIG>, this is just one embodiment. Some embodiments are configured to compare a baseline image of a larger portion of a subject's skin, which may contain a plurality of hyperpigmented spots with a filtered image of the same area. Additionally, while some embodiments utilize a baseline image as an unfiltered image and the digital image as the crossfiltered image, this is also an embodiment. Some embodiments compare identical (or substantially similar) images at different points in time to compare progress of a hyperpigmented spot.

<FIG> depicts a user interface <NUM> for creating a fitted ellipse to define a spatial feature of a hyperpigmented spot, according to embodiments described herein. In response to selection of the annotate spot option <NUM> from <FIG> and/or the run spot determination option <NUM> from <FIG>, discolorations in the skin of the subject may be analyzed. As an example, for classification of each spot, <NUM> dimensional features (or up to about <NUM>) may be derived from a respective region of the cross-polarized image to characterize the hyperpigmented spot. These embodiments take into account the contrast, shape, size, texture, as well as colors in different channels (e.g., RGB color space) for each spot. One or more multiclass learning algorithms may be utilized to classify the spot, including decision tree, AdaBoosting, etc. A multiclass error correcting output code (ECOC) may be utilized as well. The ECOC algorithm is a multiclass classifier built from binary base learners and makes use of code design to distinguish among different classes (i.e., features used to characterize the spot). The ECOC assigns a set of predefined binary codes for each class and a binary base learner is trained for each bit position in the binary code. For a testing sample feature, the classifier will generate a representative binary code, which will be compared with the predefined binary codes for the existing classes. The sample will be assigned to the class having the shortest code distance. An example of features utilized for classification is provided in Table <NUM>, below.

To derive shape related parameters, a fitted ellipse <NUM> with the same (or substantially similar) normalized second central moments as the spot region is fitted to the spot boundary, as illustrated in <FIG>. The fitted ellipse <NUM> may be utilized to define and/or create a pixel neighborhood for identifying the hyperpigmented spot. Eccentricity of the fitted ellipse may be defined as: <MAT> where α is the major axis length and b is the minor axis length. Eccentricity of value <NUM> indicates a circle while eccentricity of value <NUM> indicates a line segment.

Texture features of the spot may be derived from the rotational invariant uniform local binary pattern (LBP). The LBP operator assigns a label to every pixel (or a plurality of pixels) of an image by thresholding the <NUM>×<NUM> pixel neighborhood of each pixel in the image with the central pixel value (as shown in Table <NUM>, below) and mapping the resultant binary pattern. The rotational invariant uniform LBP label is defined as <MAT> where s(gi - g<NUM>) = <NUM> if (gi - g<NUM>) ≥ <NUM>, s(gi - g<NUM>) = <NUM> if(gi - g<NUM>) ≤ <NUM>, and U(LBP<NUM>) is a uniform operator which computes the number of spatial transitions in the pattern (e.g., the bitwise change from <NUM> to <NUM> or vice versa). This leads to <NUM> different labels (<NUM>,<NUM>,<NUM>. ,<NUM>), whose occurrence is represented as a <NUM>-bin normalised histogram to describe the texture feature of the image. This may be used to measure and/or compare pixel intensity and/or pixel color of a plurality of pixels in the pixel neighborhood, such as depicted in Table <NUM>.

By using this LBP process, one can determine low level texture features of the hyperpigmented spot. Because textures are often a differentiator in the different types of hyperpigmented spots, this class may be beneficial in identifying a particular type of hyperpigmented spot.

Referring again to <FIG>, a view calculation option <NUM> may be provided for viewing the calculations described above. A reprocess option <NUM> may cause the spot to be reprocessed with the same information, different information, and/or using a different image. Once the features of the hyperpigmented spot are identified, the hyperpigmented spot is classified according to one or more of eight possible classifications: solar lentigo, melasma, seborrhoeic keratosis, melanocytic nevus, freckle, actinic keratosis, post inflammatory hyperpigmentation and none of above. It should be understood that <FIG> is depicted as an illustration of calculations and processing that may occur. As such, some embodiments may not actually provide the user interface <NUM> for display to a user, but may be internally computed for providing the resulting output described herein.

<FIG> depicts a user interface <NUM> for capturing a plurality of hyperpigmented spots on a subject, according to embodiments described herein. As illustrated, a plurality of hyperpigmented spots may be identified and classified on a subject. Additionally, one or more of the identified spots may be annotated to show a user the location and types of spots identified. Further, treatment areas may be annotated on an image of the subject to illustrate where to apply product. <FIG> also illustrates a variety of options that a user can select, including a provide treatment option <NUM>, a provide product option <NUM>, a provide spot classifications option <NUM>, and a return option <NUM>.

In response to selection of the provide treatment option <NUM> a treatment regimen may be provided, as illustrated in <FIG>. In response to selection of the provide product option <NUM>, a product may be provided to the user, as also illustrated in <FIG>. In response to selection of the provide spot classifications option <NUM>, a listing of classifications for one or more of the hyperpigmented spots may be communicated to a user, for example, via a textual list and/or a color coding (or other coding) on the image to identify a plurality of different classified spots, as illustrated in <FIG>. In response to selection of the return option <NUM>, the user may be returned to a previous user interface.

<FIG> depicts a user interface <NUM> for classifying a plurality of different hyperpigmented spots on a subject, according to embodiments described herein. In response to classification of the hyperpigmented spots, the user interface <NUM> provides color coding of those spots for the user to more easily identify the location of each type of spot, as well as identify problem areas and treatment areas. Other options that may be available to a user include a provide treatment option <NUM>, a provide product option <NUM>, and a return option <NUM>.

In response to selection of the provide treatment option <NUM>, a treatment regimen may be provided, as illustrated in <FIG>. In response to selection of the provide product option <NUM> a product recommendation may be provided, as also illustrated in <FIG>.

<FIG> depicts a user interface <NUM> for providing product and treatment recommendations, according to embodiments described herein. In response to a user selection of one or more of the options <NUM>, <NUM> (<FIG>) <NUM>, and/or <NUM> (<FIG>), the user interface <NUM> may be provided. As illustrated, the user interface <NUM> may provide recommended products, as well as purchase options <NUM>, <NUM>, <NUM> for the user to purchase the product for general skin care and/or for treatment of the types of hyperpigmented spots identified and classified. Depending on the particular embodiment, in response to selection of one or more of the purchase options <NUM>, <NUM>, <NUM>, a product may be dispensed and/or queued for order.

Additionally, a treatment regimen may be provided for one or more of the identified problem areas. The treatment regimen and the recommended products may be based on the classifications of hyperpigmented spots. As one will understand, as the subject may be unable to apply a different product to each individual spot, the product and treatment regimens contemplate that the subject will only be able to apply product to an area of the skin that covers more than one spot. As such, customized treatment regimens and products may be provided to account for this anticipated macro level application of product.

Also provided are a track progress option <NUM>, a simulate product option <NUM>, and a home option <NUM>. In response to selection of the track progress option, the user may view historical images of the subject to illustrate how the hyperpigmented spot has changed over time (either improved with the treatment regimen or regressed without using the treatment regimen). In response to selection of the simulate product option <NUM>, imagery may be provided that simulates improvement that the subject may expect if he/she follows the treatment regimen. In response to selection of the home option <NUM>, the user may be taken to a previous user interface.

<FIG> illustrates a method of identifying hyperpigmented spots, according to embodiments described herein. As illustrated in block <NUM> of the flowchart <NUM>, a digital image of a subject may be received. In block <NUM>, a hyperpigmented spot may be determined and/or identified in the digital image of the subject. In block <NUM>, the hyperpigmented spot may be classified into a predetermined class. In block <NUM>, a treatment regimen for treating the hyperpigmented spot may be determined, according to the predetermined classification. In block <NUM>, information related to the treatment regimen may be provided for use by the subject. In block <NUM>, in response to a user selection, a product may be dispensed that is part of the treatment regimen. It is to be appreciated that a more detailed description of each step of the method illustrated in <FIG> can be found in the preceding disclosure.

<FIG> illustrates a remote computing device <NUM> for identifying hyperpigmented spots, according to embodiments described herein. As illustrated, the remote computing device <NUM>, which includes a processor <NUM>, input/output hardware <NUM>, network interface hardware <NUM>, a data storage component <NUM> (which stores spot data 938a, treatment data 938b, and/or other data), and the memory component <NUM>. The memory component <NUM> may be configured as volatile and/or nonvolatile memory and as such, may include random access memory (including SRAM, DRAM, and/or other types of RAM), flash memory, secure digital (SD) memory, registers, compact discs (CD), digital versatile discs (DVD), and/or other types of non-transitory computer-readable mediums. Depending on the particular embodiment, these non-transitory computer-readable mediums may reside within the remote computing device <NUM> and/or external to the remote computing device <NUM>.

The memory component <NUM> may store operating logic <NUM>, the identifier logic 144a and the treatment logic 144b. The identifier logic 144a and the treatment logic 144b may each include a plurality of different pieces of logic, each of which may be embodied as a computer program, firmware, and/or hardware, as an example. A local interface <NUM> is also included in <FIG> and may be implemented as a bus or other communication interface to facilitate communication among the components of the remote computing device <NUM>.

The processor <NUM> may include any processing component operable to receive and execute instructions (such as from a data storage component <NUM> and/or the memory component <NUM>). The input/output hardware <NUM> may include and/or be configured to interface with microphones, speakers, a display, and/or other hardware.

The network interface hardware <NUM> may include and/or be configured for communicating with any wired or wireless networking hardware, including an antenna, a modem, LAN port, wireless fidelity (Wi-Fi) card, WiMax card, Bluetooth chip, USB card, mobile communications hardware, and/or other hardware for communicating with other networks and/or devices. From this connection, communication may be facilitated between the remote computing device <NUM> and other computing devices, such as the user computing device 102a.

The operating logic <NUM> may include an operating system and/or other software for managing components of the remote computing device <NUM>. As also discussed above, the identifier logic 144a may reside in the memory component <NUM> and may be configured to cause the processor <NUM> to identify, classify, and annotate one or more hyperpigmented spots. Similarly, the treatment logic 144b may be utilized to determine a product and treatment regimen for treating the one or more hyperpigmented spots, as described herein.

It should be understood that while the components in <FIG> are illustrated as residing within the remote computing device <NUM>, this is merely an example. In some embodiments, one or more of the components may reside external to the remote computing device <NUM>. It should also be understood that, while the remote computing device <NUM> is illustrated as a single device, this is also merely an example. In some embodiments, the identifier logic 144a and the treatment logic 144b may reside on different computing devices. As an example, one or more of the functionalities and/or components described herein may be provided by a remote computing device <NUM> and/or user computing device 102a, which may be coupled to the remote computing device <NUM> via the network <NUM>.

Additionally, while the remote computing device <NUM> is illustrated with the identifier logic 144a and the treatment logic 144b as separate logical components, this is also an example. In some embodiments, a single piece of logic may cause the remote computing device <NUM> to provide the described functionality.

Claim 1:
A system for identifying a hyperpigmented spot for cosmetic purposes, comprising: a computing device that includes a processor and a memory component, wherein the memory component stores logic that, when executed by the processor, causes the computing device to
(i) receive a digital image of a subject captured using cross-polarized light,
(ii) receive a baseline image of the subject, where the baseline image is either an unfiltered image of the subject or an image of the subject made at a different point in time,
(iii) identify a hyperpigmented spot in the image of the subject,
(iv) compare the baseline image with the image of the subject to determine changes to the hyperpigmented spot;
(v) electronically annotate the image of the subject to distinguish the hyperpigmented spot in the image,
(vi) classify the hyperpigmented spot into a predetermined class, the predetermined class includes at least two of the following: solar lentigo, melasma, seborrhoeic keratosis, melanocytic nevus, freckle, actinic keratosis, post inflammatory hyperpigmentation,
(vii) determine a cosmetic skin care product for treating the hyperpigmented spot according to the predetermined class, and
(viii) provide information related to the cosmetic skin care product for use by the subject.