System and method for image segmentation and digital analysis for clinical trial scoring in skin disease

Disclosed are systems and methods for clinical trial assessment of skin disease treatment. The disclosure includes obtaining a series of digital images over a period of time, wherein each digital image includes an affected area of the subject; identifying characteristic morphologies and lesions in the affected area of the subject in each of the digital images; classifying each of the detected and segmented morphologies and lesions into one or more identified categories for each of the digital images; assigning a global score to each of the digital images based on a count of the detected and segmented characteristic morphologies and lesions in each of the one or more identified categories; analyzing the global scores of each of the digital images; and making an assessment of the clinical trial based on the analysis of the global scores of each of the digital images.

BACKGROUND

Acne is a disease that affects around 85% of individuals during the course of life. Although it is typically seen in the teenage years, the adult form of acne may persist throughout adulthood in certain patients. Although the disease may be seen as inconsequential and a normal part of aging, in reality the disease has been shown to significantly deter the quality of life. Both suicide attempts and suicide are increased in the acne population. Costs are extensive with over $3 Billion in direct and indirect costs estimated each year attributed to acne in the US. Furthermore, in terms of life-altering affects, active acne and scarring residua may be associated with diminished job prospects throughout life.

There is no gold standard for acne evaluation in non-research clinics; it entirely depends on dermatologists' experience for acne severity evaluation. Counting lesions is the simplest and the most favored method of validating acne scoring for clinical trials. In acne diagnosis and treatment, it is important to accurately quantify or to evaluate the severity. Acne can be classified into several skin lesions including closed comedones (whiteheads), open comedones (blackheads), papules, pustules, cysts (nodules) and scars. It is difficult for a dermatologist to count and document each lesion type because of the limited time available during consultation. Furthermore, it is difficult to determine the efficacy of various treatment options as this requires repetitive visits by the patient to medical personnel over a period of time. Such visits can be time-consuming for the patient and the medical specialists, and efficacy of the treatment may be subject to subjectivity of medical personnel.

Determining the efficacy of clinical trials of treatments for other skin diseases such as rosacea also requires multiple visits to the dermatologists or other medical personnel and is subject to subjective determinations by the medical personnel on whether the skin disease is progressing, regressing or staying the same.

Therefore, systems and methods are desired that overcome challenges in the art, some of which are described above.

SUMMARY

Disclosed herein are computer-implemented systems and methods for clinical trial assessment of skin disease treatment. Digital images can be obtained and analyzed to validate the efficacy of clinical trials for treatment. Described herein are processes of region-of-interest determination using entropy-based filtering and thresholding as well as skin-disease lesion feature extraction. Feature extraction methods include using discrete wavelet frames and gray-level co-occurrence matrix in separating the skin disease into pre-identified lesion classes.

DETAILED DESCRIPTION

FIG. 1illustrates an exemplary overview apparatus for making clinical trial assessments of skin disease treatments. As shown inFIG. 1, one embodiment of the apparatus100comprises an image capture mechanism102. In one aspect, the image capture mechanism102can be a camera. The image capture mechanism102can take still and/or video images. Generally, the image capture mechanism102will be a digital camera, but can be an analog device equipped with or in communication with an appropriate analog/digital converter. The image capture mechanism102may also be a webcam, scanner, recorder, or any other device capable of capturing a still image or a video.

As shown inFIG. 1A, the image capture mechanism102is in direct communication with a computing device110(e.g., computer) through, for example, a network (wired (including fiber optic)), wireless or a combination of wired and wireless) or a direct-connect cable (e.g., using a universal serial bus (USB) connection, IEEE 1394 “Firewire” connections, and the like). In other aspects, the image capture mechanism102can be located remotely from the computing device110, but capable of capturing an image and storing it on a memory device such that the image can be downloaded or transferred to the computing device110using, for example, a portable memory device and the like. In one aspect, the computing device110and the image capture mechanism102can comprise or be a part of a device such as a smart device, smart phone, tablet, laptop computer or any other fixed or mobile computing device.

In a basic configuration, the computing device110can be comprised of a processor104and a memory108. The processor104can execute computer-readable instructions that are stored in the memory108. Moreover, images captured by the image capture device102, whether still images or video, can be stored in the memory108and processed by the processor104using computer-readable instructions stored in the memory108.

The processor104is in communication with the image capture device102and the memory108. The processor104can execute computer-readable instructions stored on the memory108to capture, using the image capture device102, an image. In one aspect, the captured image can include a series of digital images captured over a period of time of at least a portion of a subject, wherein each of the digital images includes an affected area106of a subject. The affected area106may be acne and/or rosacea.

The processor104can further execute computer-readable instructions stored on the memory108to capture, using the image capture device102, a series of digital images over a period of time of the at least a portion of a subject, wherein each of the digital images includes the same affected area of the subject. The processor of the computer can execute computer-readable instructions to identify characteristic morphologies and lesions in the affected area of the subject in each of the digital images, wherein said identification comprises detection and segmentation of the characteristic morphologies and lesions in each of the digital images. The detected and segmented characteristic morphologies and lesions can be classified into one or more identified categories for each of the digital images.

In one example, detection may comprise the use of neural network algorithms. For example, detection and/or image classification may be performed using a convolutional neural network (CNN) such as “Inception V3,” which can be trained using skin disease data (e.g., images of identified and classified skin diseases) such as acne and/or rosacea.

Consider another example where a region-of-interest is segmented particularly for closed comedones and pustules types of acne lesion, and the acne lesions are classified into six classes. Though this example focuses on acne, it can be applicable to other skin disease as well such as rosacea and the like. Acne lesion analysis is a challenging task because it has large inter-class variability within subjects and lesion types. From the perspective of region-of-interest extraction for digital image analysis, various approaches are needed to extract the meaningful acne affected areas. In this example, the regions-of-interests are extracted before individual lesions are detected. The typical appearance of the lesions and the surrounding skin areas in terms of color and textural characteristics is taken into consideration.FIG. 2is a flowchart illustrating the region-of-interest localization that generates the composite image showing regions-of-interest with regards to closed comedones and pustules types of acne lesion.

The method disclosed relative toFIG. 2involves202, converting an input image into its equivalent Hue-Saturation-Value (HSV) color format representation. HSV separates color information from the luminance of an image; hence, by ignoring the saturation and the value components of the color space uneven illumination due to lighting variations on the image can be reduced and the color information utilized for further processing. The next steps perform morphological opening204and closing206on the hue component to enhance certain structural features from the image. An iterative thresholding approach208is performed on the images separately. For every iteration, an accumulator matrix210is generated that accumulates the binary 1 pixels from the outcome of each dichotomizing step. The number of iterations determines the resolution of the accumulation values. A composite image is then produced212by taking the average intensity values of the corresponding pixels of the closed image and the opened image.

The second part of the example demonstrates the feature extraction and supervised classification of acne lesions based on manually selected regions-of-interest. The feature bank is predominantly occupied by textural based features comprised of discrete wavelet frames (DWF) and gray-level co-occurrence matrix descriptors (GLCM).

Seemingly different from other wavelet-based approaches, DWF is invariant with respect to translations of the input signal. This property is quite desirable in the context of acne lesion classification. Channels are decomposed in DWF using a similar approach as the wavelet transform, but without the subsampling process resulting in four filtered images with the same size as the input image. The number of channels for DWF is similar as in the Pyramidal Wavelet Transform (PWT). As described herein, three levels of decomposition are performed. To compute the features, the mean energy E of each channel or filtered image is used and is given as:

E=1M×N⁢∑a=0M-1⁢∑b=0N-1⁢Wk⁡(a,b)(1)
where M and N are the number of rows and columns of the channel or filtered images, and Wkis the k-th channel of the filtered images.

Gray-level co-occurrence matrix (GLCM), also known as the gray-level spatial dependence matrix is a statistical method of examining texture that considers the spatial relationship of pixels. The GLCM functions characterize the texture of an image by calculating how often pairs of pixels with specific values and in a specified spatial relationship occur. Once a GLCM is created, statistical measures can be computed from the matrix.

GLCM is a statistical method that can well describe second-order statistics of a textured image. GLCM is a two-dimensional histogram in which each entry (i, j) corresponds to the number of occurrences of the pair of levels i and j which are a distance d apart. A co-occurrence matrix is specified by the relative frequencies P(i, j, d, θ) in which two pixels, separated by d, occur in a direction specified by the angle theta, one with gray level i and another with gray level j. GLCM is therefore a function of distance r and angle θ.

In the disclosed example, GLCM is computed for fixed d (=2) and θ=0°, 45°, 90° and 135°, which results in four GLCMs. Based on the GLCMs, the following features are calculated:

In classifying the acne patterns, a fusion of features is used combining nine DWF features and eight GLCM-based features as described in Equation 2 to Equation 8, above. Using the methodology described in the example, classifying each of the detected and segmented characteristic morphologies and lesions into one or more identified categories for each of the digital images may comprise classifying each of the detected and segmented characteristic morphologies and lesions into one or more of open/closed comedones, pustules, papules, cysts, scars, and the like, or erythema (degree of redness), telangiectases or broken capillaries, plaques (raised plateau-like areas), and rhinophymatous changes (typically development of enlarged bumpy nose), and the like (which may be considered normal or abnormal morphologies).

Additional information about the results of the segmentation and classification example described above can be found in “Acne Image Analysis: Lesion Localization And Classification,” Fazly Salleh Abas; Benjamin Kaffenberger; Joseph Bikowski; Metin N. Gurcan; Proceedings Volume 9785, Medical Imaging 2016: Computer-Aided Diagnosis; 97850B (2016); doi: 10.1117/12.2216444, presented at SPIE Medical Imaging, 2016, San Diego, Calif., United States, which is incorporated by reference and made a part hereof.

Once segmented and classified, the processor104can further execute computer-readable instructions stored on the memory108to assign, a global score to each of the digital images that comprise the series of digital images based on a count of the detected and segmented characteristic morphologies and lesions in each of the one or more identified categories.

The processor104further execute computer-readable instructions stored on the memory108to analyze the global scores of each of the digital images and make an assessment of the clinical trial based on the analysis of the global scores of each of the digital images.

Generally, the subject will be undergoing treatment for at least one skin disease over the period of time. This may involve medications such as those administered orally, through injection, topically, and the like, or any other form of treatment. The assessment of the clinical trial based on the analysis of the global scores of each of the digital images considers an efficacy of the treatment. For example, is the skin disease clearing (e.g., the affected area getting smaller), progressing (e.g., the affected area getting larger or new affected areas appearing), or staying the same during the period of time of the clinical trial.

Identifying characteristic morphologies and lesions in the affected area of the subject in each of the digital images where such identification comprises detection and segmentation of the characteristic morphologies and lesions in each of the digital images, may comprise the processor104identifying and registering a lesion in a latter of the series of digital images that was in an earlier of the series of digital images. Such registration enables determining a change in the lesion in the latter of the series of digital images that was in an earlier of the series of digital images. The change may comprise a change in one or more of size, shape, intensity or image characteristics of the lesion in the latter of the series of digital images that was in an earlier of the series of digital images. The processor104may determine if the change in one or more of size, shape, intensity or image characteristics of the lesion in the latter of the series of digital images that was in an earlier of the series of digital images is consistent with a normal disease progression with a sample medicine. For example, the computer may be pre-programmed with expected results for the sample medicine, and the actual results compared with the expected results, by the computer, to determine if the change in one or more of size, shape, intensity or image characteristics of the lesion in the latter of the series of digital images that was in an earlier of the series of digital images is consistent with a normal disease progression with a sample medicine. In one aspect, the processor104can be configured such that if the change in one or more of size, shape, intensity or image characteristics of the lesion in the latter of the series of digital images that was in an earlier of the series of digital images is not consistent with a normal disease progression with a sample medicine, then alerting a medical professional. For example, a display of the computer100can alert the medical professional by use of a dashboard displayed on a screen of the computer100or another electronic device such as a smart phone or other portable electronic device.

In some aspects, the memory108may be used to store an electronic database comprising the series of images and information about any identified characteristic morphologies and lesions in the affected area of the subject in each of the digital images. The information about any identified characteristic morphologies and lesions in the affected area of the subject in each of the digital images may include a size of the identified characteristic morphologies and lesions in the affected area of the subject in each of the digital images.

As noted herein, the image capture device102does not have to be a part of or affixed to the computer100. For example, at least one image of the series of images can be captured in electronic format by the subject and transmitted to the computer100. This form of image capture and transmittal may alleviate the need for the subject to travel to a location where the medical professional is located. Such convenience may increase the subject's participation in the clinical trial. For example, the subject may capture the at least one image of the series of images using a camera of the subject's smartphone and then transmit the captured image or images to the computer100via a text message or by email.

The system and methods described above and illustrated with reference toFIGS. 1 and 2may also benefit from adaptive learning. For example, feedback can be provided by experts that evaluation the segmentation of the images and the segmentation and classification parameters can be updated based on the positive and negative feedback

FIG. 3is a flowchart that illustrates an exemplary method for clinical trial assessment of skin disease treatment comprising302, obtaining a series of digital images over a period of time, wherein at least a portion of each of the digital images comprise an affected area (i.e., exhibiting indications of a skin disease). The digital images may be obtained directly via an image capture device as described herein, retrieved from a file, and the like. For example, a subject could be prompted to take a picture of his or her affected area and email, text or otherwise transmit it to their medical professional for analysis. This could occur on a periodic basis. The period of time can be, for example, a week, a month, two months, three months, a year, two years, etc., as such period of time is defined by the medical professional conducting the clinical trial.

At304, characteristic morphologies and lesions in the affected area of the subject in each of the digital images are identified. Such identification comprises detection and segmentation of the characteristic morphologies and lesions in each of the digital images.

In one aspect, segmentation of the characteristic morphologies and lesions in each of the digital images comprises performing color space normalization on each of the series of digital images to determine one or more hue components for each digital image of the series of digital images; performing morphological opening and closing on the one or more hue components to enhance structural features for each digital image of the series of digital images and to create a morphological opening image and a morphological closing image for each digital image of the series of digital images; performing iterative thresholding on each of the morphological opening image and the morphological closing image for each digital image of the series of digital images; producing a composite image for each digital image of the series of digital images by taking an average between the morphological opening image and the morphological closing image for each digital image of the series of digital images.

At306, each of the detected and segmented characteristic morphologies and lesions are classified into one or more identified categories for each of the digital images. For example, classifying each of the detected and segmented characteristic morphologies and lesions into one or more identified categories for each of the digital images may comprise classifying each of the detected and segmented characteristic morphologies and lesions into one or more of open/closed comedones, pustules, papules, cysts, scars, erythema (degree of redness), telangiectases or broken capillaries, plaques (raised plateau-like areas), and rhinophymatous changes (typically development of enlarged bumpy nose), and the like. This can be performed by determining a plurality of textural based features of each composite image and classifying each of the detected and segmented characteristic morphologies and lesions into the one or more identified categories based on the determined textural based features. The determined textural based features may comprise one or more discrete wavelet frames (DWF) and one or more gray-level co-occurrence matrix descriptors (GLCM). For example, the determined textural based features may comprise nine DWF features and eight GLCM features.

At308, a global score is assigned to each of the digital images based on a count of the detected and segmented characteristic morphologies and lesions in each of the one or more identified categories. At310, the global scores for each of the series of digital images are analyzed to make, at312, an assessment of the clinical trial based on the analysis of the global scores of each of the digital images. Such an assessment may comprise, for example, determining that the clinical trial is successful (e.g., the affected area has been eliminated or reduce by a desired amount), unsuccessful (e.g., the affected area has not reduced in size and/or intensity, new affected areas have developed, etc.), additional testing/trials are needed, and the like.

The system has been described above as comprised of units. One skilled in the art will appreciate that this is a functional description and that the respective functions can be performed by software, hardware, or a combination of software and hardware. A unit can be software, hardware, or a combination of software and hardware. The units can comprise software for discriminating tissue of a specimen. In one exemplary aspect, the units can comprise a computing device that comprises a processor321as illustrated inFIG. 4and described below.

FIG. 4illustrates an exemplary computer that can be used for clinical trial assessment of skin disease treatment. As used herein, “computer” may include a plurality of computers. The computers may include one or more hardware components such as, for example, a processor421, a random access memory (RAM) module422, a read-only memory (ROM) module423, a storage424, a database425, one or more input/output (I/O) devices426, and an interface427. Alternatively and/or additionally, the computer may include one or more software components such as, for example, a computer-readable medium including computer executable instructions for performing a method associated with the exemplary embodiments. It is contemplated that one or more of the hardware components listed above may be implemented using software. For example, storage424may include a software partition associated with one or more other hardware components. It is understood that the components listed above are exemplary only and not intended to be limiting.

Processor421may include one or more processors, each configured to execute instructions and process data to perform one or more functions associated with a computer for making a clinical trial assessment of skin disease treatment. Processor421may be communicatively coupled to RAM422, ROM423, storage424, database425, I/O devices426, and interface427. Processor421may be configured to execute sequences of computer program instructions to perform various processes. The computer program instructions may be loaded into RAM422for execution by processor421.

RAM422and ROM423may each include one or more devices for storing information associated with operation of processor421. For example, ROM423may include a memory device configured to access and store information associated with the computer, including information for identifying, initializing, and monitoring the operation of one or more components and subsystems. RAM422may include a memory device for storing data associated with one or more operations of processor421. For example, ROM423may load instructions into RAM422for execution by processor421.

Storage424may include any type of mass storage device configured to store information that processor421may need to perform processes consistent with the disclosed embodiments. For example, storage424may include one or more magnetic and/or optical disk devices, such as hard drives, CD-ROMs, DVD-ROMs, or any other type of mass media device.

Database425may include one or more software and/or hardware components that cooperate to store, organize, sort, filter, and/or arrange data used by the computer and/or processor421. For example, database425may store digital images of an affected area, computer-executable instructions for identifying characteristic morphologies and lesions in the affected area of the subject in each of the digital images; classifying the characteristic morphologies and lesions in the affected area of the subject in each of the digital images; assigning a global score to each of the digital images based on a count of the detected and segmented characteristic morphologies and lesions in each of the one or more identified categories of each digital image; analyzing the global scores of each of the digital images that comprise the series of digital images; and making an assessment of the clinical trial based on the analysis of the global scores of each of the digital. It is contemplated that database325may store additional and/or different information than that listed above.

I/O devices426may include one or more components configured to communicate information with a user associated with computer. For example, I/O devices may include a console with an integrated keyboard and mouse to allow a user to maintain a database of digital images, results of the analysis of the digital images, metrics, and the like. I/O devices426may also include a display including a graphical user interface (GUI) for outputting information on a monitor. I/O devices426may also include peripheral devices such as, for example, a printer for printing information associated with the computer, a user-accessible disk drive (e.g., a USB port, a floppy, CD-ROM, or DVD-ROM drive, etc.) to allow a user to input data stored on a portable media device, a microphone, a speaker system, a camera, or any other suitable type of interface device.

Interface427may include one or more components configured to transmit and receive data via a communication network, such as the Internet, a local area network, a workstation peer-to-peer network, a direct link network, a wireless network, or any other suitable communication platform. For example, interface427may include one or more modulators, demodulators, multiplexers, demultiplexers, network communication devices, wireless devices, antennas, modems, and any other type of device configured to enable data communication via a communication network.

Throughout this application, various publications may be referenced. The disclosures of these publications in their entireties are hereby incorporated by reference into this application in order to more fully describe the state of the art to which the methods and systems pertain.