Patent Description:
Millions of patients around the world suffering from acute and chronic nonhealing wounds, which cause a great reduction in quality of life in those patients. These wounds require periodic examination and thorough treatment to prevent deterioration. Otherwise, they can lead to severe complications such as limb amputations and death. Monitoring the progression of the wound is critical, as it involves repeated clinical trips and lab tests over days. Chronic wounds take around a <NUM>-month period, which require continuous monitoring. Further, chronic wounds fail to progress through the phases of healing in an orderly and timely manner thereby requiring hospitalization and additional treatment that adds billions in cost for health care services annually. This healing time duration is unpredicted, as it depends on various factors such as etiology, age, nutrition, comorbidity condition of patient, medication, and environments.

The cost of healthcare services for all the wounds are estimated to be around $<NUM>. 8B in United States alone. This has led to the development of wound management system that has become an essential part of the chronic wound treatment.

Wound measurement is one of the important components in the wound management system, the accuracy of which influences the diagnosis and treatment by the healthcare professionals. Also, it is critical to the evaluation of wound healing trajectory and to determine the future treatment for the patients by the doctors. In addition, the wound area gives an effective and reliable index of later complete wound healing. The most healthcare professionals depend only on imprecise manual measurement and optical assessment of wounds, which is time-consuming and often inaccurate causing negative impact on patients such as infection risks, inaccurate measurements, and discomfort to patients. These problems can be solved by exploiting image processing and computer vision techniques paired with artificial intelligence (AI). Evaluation of wound images using image processing and AI is a challenging task due to the complexities involved in the wound capturing process such as variable lighting condition, time constraints in clinical laboratories.

<NPL> compares the segmentation performance of AHRF and CNN approaches (FCN, U-Net, DeepLabV3) using various metrics including segmentation accuracy (dice score), inference time, amount of training data required and performance on diverse wound sizes and tissue types. Improvements possible using various image pre- and post-processing techniques are also explored.

<NPL> proposes the Automatic Skin Ulcer Region Assessment ASURA framework to accurately segment the wound and automatically measure its size using an encoder/decoder deep neural network to perform the segmentation, which detects the measurement ruler/tape present in the image and estimates its pixel density.

<NPL> builds 3D models of skin wounds from color images. The method can deal with uncalibrated images acquired with a handheld digital camera with free zooming. An original iterative matching scheme is used to generate a dense estimation of the surface geometry from <NUM> widely separated views. From the therefrom reconstructed 3D model of the skin wound, accurate volumetric measurements are available.

<CIT>, Tissue Analystics Inc. describes collecting an image of a region of skin condition to obtain visual recording representing the skin condition and a reference model. Visual recording representing wound and the reference model is normalized such that an image of the reference model in the visual recording conforms to objective visual characteristics of the reference model for normalizing an image of the wound in the visual recording. Respective parameter values are compared at successive times using the image of the wound in the visual recording to be normalized.

Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems. For example, in one embodiment, a system for analyzing a wound is provided. The system includes a user interface, one or more hardware processors and a memory. The user interface receives a plurality of images of a plurality of wounds as an input. The memory is in communication with the one or more hardware processors, wherein the one or more first hardware processors are configured to execute programmed instructions stored the memory, the memory further comprises: a training module configured to train a deep learning model using the plurality of preprocessed images, wherein the deep learning model having an encoder decoder architecture, the encoder decoder architecture comprises: a pre-trained Resnet50 as the encoder, an Atrous spatial pyramid pooling (ASPP) comprising of dilated convolution, wherein the ASPP is connected to the encoder, a bilinear up sampler configured to up sample the plurality of images by a factor of four, and a channel wise attention mechanism using squeeze and excitation (SE) network, designed to improve representational power of the network by enabling to perform dynamic channel-wise feature recalibration; an image segmentation module configured to: preprocess an input image of the wound to be analyzed; generate an output image from preprocessed image using the trained deep learning model; and post-process the output image using a binary segmentation mask with a fixed threshold of a predefined number; an AI based estimation module configured to: perform morphology operations on the segmented image to remove small regions and spurious noises to fill small holes within the wound to improve a true positive rate; perform connected component analysis on the binary image to label connected regions; calculate a plurality of wound parameters of the wound by measuring the labelled connected regions; and overlay the calculated plurality of wound parameters and wound segmented boundary on the image of the wound.

In another aspect, a method for analyzing a wound is provided. The method includes a plurality of images of a plurality of wounds is received as an input. The plurality of images then preprocessed. In the next step, a deep learning model is trained using the plurality of preprocessed images, wherein the deep learning model having an encoder decoder architecture, the encoder decoder architecture comprises: a pre-trained Resnet50 as the encoder, an Atrous spatial pyramid pooling (ASPP) comprising of dilated convolution, wherein the ASPP is connected to the encoder, a bilinear up sampler configured to up sample the plurality of images by a factor of four, and a channel wise attention mechanism using a squeeze and excitation (SE) network, designed to improve representational power of the SE network by enabling to perform dynamic channel-wise feature recalibration. In the next step, an image of the wound to be analyzed is provided. The image is then preprocessed. In the next step, an output image is generated from preprocessed image using the trained deep learning model. Further, the output image is post-processed using a binary segmentation mask with a fixed threshold of a predefined number to get a segmented output image. In the next step, morphology operations are performed on the segmented output image to remove small regions and spurious noises to fill small holes within the wound to improve a true positive rate. Further, connected component analysis is performed on the morphology operated image to label connected region. In the next step, a plurality of wound parameters of the wound is calculated by measuring the labelled connected regions. And finally, the calculated plurality of wound parameters and the wound segmentation boundary or region is overlaid on the image of the wound for user/physician's diagnosis.

In yet another aspect, one or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause analyzing a wound. Initially, a plurality of images of a plurality of wounds is received as an input. The plurality of images then preprocessed. In the next step, a deep learning model is trained using the plurality of preprocessed images, wherein the deep learning model having an encoder decoder architecture, the encoder decoder architecture comprises: a pre-trained Resnet50 as the encoder, an Atrous spatial pyramid pooling (ASPP) comprising of dilated convolution, wherein the ASPP is connected to the encoder, a bilinear up sampler configured to up sample the plurality of images by a factor of four, and a channel wise attention mechanism using a squeeze and excitation (SE) network, designed to improve representational power of the SE network by enabling to perform dynamic channel-wise feature recalibration. In the next step, an image of the wound to be analyzed is provided. The image is then preprocessed. In the next step, an output image is generated from preprocessed image using the trained deep learning model. Further, the output image is post-processed using a binary segmentation mask with a fixed threshold of a predefined number to get a segmented output image. In the next step, morphology operations are performed on the segmented output image to remove small regions and spurious noises to fill small holes within the wound to improve a true positive rate. Further, connected component analysis is performed on the morphology operated image to label connected region. In the next step, a plurality of wound parameters of the wound is calculated by measuring the labelled connected regions. And finally, the calculated plurality of wound parameters and wound segmented boundary is overlaid on the image of the wound.

Monitoring the progression of a wound is critical, as it involves repeated clinical trips and lab tests over days. Chronic wounds take around a <NUM>-month period, which require continuous monitoring. This has led to the development of wound management system that has become an essential part of the chronic wound treatment.

Wound measurement is one of the important components in the wound management system, the accuracy of which influences the diagnosis and treatment by the healthcare professionals. Also, it is critical to the evaluation of wound healing trajectory and to determine the future treatment for the patients by the doctors.

In the prior art, the wound management system comprises wound segmentation. The wound segmentation has been roughly categorized into traditional image processing and deep learning-based methods. The former category focuses on combining image processing techniques with machine learning approaches by using hand-crafted features for processing wound images. Most of these methods suffer from one or more of the following limitations: (a) no guarantee of optimal result as they depend on manually curated parameters and empirically handcrafted features, (b) the handcrafted features are affected by illumination, image resolution and skin pigmentation, (c) they are not immune to severe pathologies and rare cases that are very impractical from a clinical perspective. The latter category focuses on the exploitation of deep learning (DL) networks on wound segmentation. Unlike with the traditional machine learning and image processing-based methods, which make decisions based on the handcrafted features, the DL based-methods combine both feature extraction and decision making.

The present disclosure provides an artificial intelligence (AI) based system and method for analyzing wounds on a person. The system is configured to take an image of the wound taken from a camera of a person. In addition, the AI module estimate the healing index based on the wound parameter for example using the area of wound history. This image is then provided to the physician after the analysis and physician is able to provide feedback to the person in terms of a healing index. In the analysis part, the system provides a fully automatized wound segmentation and quantify the parameters that assist wound care professionals. An AI based estimation module is provided, implemented with morphological operations, connected component analysis, and shape analysis, improving accuracy and providing the wound parameter and metrics such as area, perimeter, circle diameter, major and minor axis length of an ellipse.

According to an embodiment of the disclosure, <FIG> illustrates a block diagram a system <NUM> for analyzing a wound in the person or a patient. <FIG> shows a schematic architecture of the system <NUM>. It should be appreciated that the system <NUM> may also be referred as a wound management system <NUM>. In an embodiment, the network <NUM> may be a wireless or a wired network, or a combination thereof. In an example, the network <NUM> can be implemented as a computer network, as one of the different types of networks, such as virtual private network (VPN), intranet, local area network (LAN), wide area network (WAN), the internet, and such. The network <NUM> may either be a dedicated network or a shared network, which represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), and Wireless Application Protocol (WAP), to communicate with each other. Further, the network <NUM> may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices. The network devices within the network <NUM> may interact with the system <NUM> through communication links.

The system <NUM> may be implemented in a workstation, a mainframe computer, a server, and a network server. In an embodiment, the computing device <NUM> further comprises one or more hardware processors <NUM>, one or more memory <NUM>, hereinafter referred as a memory <NUM> and a data repository <NUM>, for example, a repository <NUM>. The memory <NUM> is in communication with the one or more hardware processors <NUM>, wherein the one or more hardware processors <NUM> are configured to execute programmed instructions stored in the memory <NUM>, to perform various functions as explained in the later part of the disclosure. The repository <NUM> may store data processed, received, and generated by the system <NUM>. The memory <NUM> further comprises a plurality of modules for performing various functions. The plurality of modules comprises a training module <NUM>, an image segmentation module <NUM>, an AI based estimation module <NUM>, and a self-learning module <NUM> as shown in the block diagram of <FIG>.

The system <NUM> supports various connectivity options such as BLUETOOTH®, USB, ZigBee and other cellular services. The network environment enables connection of various components of the system <NUM> using any communication link including Internet, WAN, MAN, and so on. In an exemplary embodiment, the system <NUM> is implemented to operate as a stand-alone device. In another embodiment, the system <NUM> may be implemented to work as a loosely coupled device to a smart computing environment. The components and functionalities of the system <NUM> are described further in detail.

An example of a software block diagram <NUM> of the system <NUM> is shown in the FIG. In general, the system <NUM> comprises a patient portal <NUM>, a physician portal <NUM> and a server <NUM> or any other place where wound analysis happens. In an example, the patient portal <NUM> is an application present in the mobile phone of the patient, where the patient can click an image of a wound to be analyzed and share it to the server for wound analysis. Normally, the patient portal <NUM> comprises following features: a camera for capturing the image to be analyzed, an interface through which patient ID, date and time stamp can be entered, converting image to byte string array, ability to upload json to server, camera calibration and a display screen displaying a healing index as feedback from the analysis. In an embodiment the date and time stamp can be taken automatically from the processor.

According to an embodiment of the disclosure, the system <NUM> is also configured to a plurality of patient parameters as input to the physician analyzing the image via the patient portal <NUM>, wherein the plurality of patient parameters comprises patient name, ID, patient medical history, and patient physical statistics.

<FIG> also shows the server such as Amazon web service (AWS) cloud gateway, the server <NUM> comprises authentication, database and feature for appending json to database. The physician portal <NUM> comprises a GUI showing patient details, input image and the analyzed image. In an example the physician portal is in communication with an AWS-Python API Server.

According to an embodiments of the disclosure, the user interface <NUM> is configured to receive a plurality of images. The plurality of images is further used for training a deep learning mode using the training module <NUM>.

According to an embodiment of the disclosure, the system <NUM> comprises the training module <NUM>. The training module <NUM> is configured to train a deep learning (DL) model using the plurality of preprocessed images, wherein the deep learning model having an encoder decoder architecture as shown in <FIG>. the encoder decoder architecture comprises: a pre-trained Resnet50 as the encoder, an Atrous spatial pyramid pooling (ASPP) comprising of dilated convolution, wherein the ASPP is connected to the encoder, a bilinear up sampler configured to up sample the plurality of images by a factor of four, and a channel wise attention mechanism using squeeze and excitation (SE) network, designed to improve representational power of the network by enabling to perform dynamic channel-wise feature recalibration.

To make the training more efficient, transfer learning was used for the deep learning model. Instead of randomly initializing the weights, the Resnet50 model, pre-trained on the Image Net dataset is loaded before the model is trained on the plurality of data. Transfer learning with the pre-trained model is beneficial to the training process in the sense that the weights converge faster and better.

In an embodiment, DeepLabV3+ architecture was used. This is an encoder decoder architecture with Atrous Spatial Pyramid Pooling (ASPP) and bilinear up sampling. The network begins with a pre-trained Resnet50 as encoder, which is followed by ASPP. The ASPP consists of dilated convolution which helps to encode multi-scale contextual information. It is followed by a bilinear up sampling by a factor of <NUM> to get the output mask. A novel Channel wise Attention Mechanism was also utilized which is using - Squeeze and Excitation (SE) Network, an architectural unit designed to improve the representational power of a network by enabling it to perform dynamic channel-wise feature recalibration.

Atrous Spatial Pyramid Pooling (ASPP) is a semantic segmentation module for resampling a given feature layer at multiple rates prior to convolution. This amounts to probing the original image with multiple filters that have complementary effective fields of view, thus capturing objects as well as useful image context at multiple scales. Rather than resampling features, the mapping is implemented using multiple parallel Atrous convolutional layers with different sampling rates.

Squeeze and Excitation Networks Architecture consists of three operations: Squeeze, Excitation, and Scaling. The squeeze operation is mainly used to extract the global information from each channel of the feature map. The feature map is basically the output of the convolution layer, which is a 4D tensor of size B × H × W × C. In the Squeeze operation the Global Average Pooling (GAP) is used to reduce the B x H x W x C feature map to B x <NUM> x <NUM> x C since GAP was performing much better than Global Max Pooling (GMP). Since the feature map is now reduced to a smaller dimension (B x <NUM> x <NUM> x C), basically for each channel of size H x W is reduced to a singular vector. For the excitation operation, a fully connected multiLayer perceptron (MLP) with a bottleneck structure is used. The MLP is used to generate the weights to scale each channel of the feature map adaptively.

The excitation operation passes the "excited" tensor of shape B x <NUM> x <NUM> x C. This tensor is then passed through a sigmoid activation function. The sigmoid activation function converts the tensor values in the range of <NUM> and <NUM>. Then an element-wise multiplication is performed between the output of the sigmoid activation function and the input feature map. If the value is close to <NUM> means the channel is less important so, the values of the feature channel would be reduced, and if the value is close to <NUM>, this means that is channel is important.

Thus, the Squeeze and Excitation Network basically scale each channel information. It reduces the non-relevant channel information and the relevant channel are not much affected. So, after the whole operation, the feature map only contains the relevant information, which increases the representational power of the entire network.

According to an embodiment of the disclosure, the system <NUM> comprises the image segmentation module <NUM>. The image segmentation module <NUM> configured to preprocess an input image of the wound to be analyzed; generate an output image from the preprocessed image using the trained deep learning model; and post-process the output image using a binary segmentation mask with a fixed threshold of a predefined number. The binary segmentation masks predicted by the trained DL model are grayscale images with pixel intensities that range from <NUM> to <NUM>. In the post processing step, binary segmentation masks are first generated from thresholding with a fixed threshold of <NUM>, which is half the max intensity.

According to an embodiment of the disclosure, the system <NUM> comprises the AI based estimation module <NUM>. The AI based estimation module <NUM> configured to perform morphology operations on the segmented image to remove the small regions and spurious noises to fill the small holes within the wound to improve a true positive rate; perform connected component analysis on the binary image to label connected regions; calculate a plurality of wound parameters of the wound by measuring the labelled connected regions; and overlay the calculated plurality of parameters and wound segmented boundary on the image of the wound.

The block diagram of the AI based estimation module <NUM> is shown in <FIG>, which includes morphological operations, connected component analysis, and shape analysis followed by wound measurements. Input to the post processing stage is the binary segmentation mask that is generated from the deep learning output by thresholding. Morphology operations are performed on the binary mask to remove the small regions/spurious noises and to fill the small holes within the wound to improve the true positive rate. Sometimes, the blood stain could be identified as wound by the deep learning network causing small false-positive region/noise in the segmented mask. This small false-positive region is detected and removed by finding small, connected components in the segmented mask. On the other hand, the abnormal tissue like fibrinous tissue inside the wound could be treated as non-wound by the network representing it as small holes inside the segmented mask. These holes are detected and filled by finding the small, connected components.

Connected component analysis is used to label the connected regions followed by measurement of those labeled connected regions. The measurements, such as, area, perimeter, circle diameter, major and minor axis length of ellipse etc. are used in conjunction with shape analysis to find the approximate shape of the wound and its measurements. Finally, the measured wound parameters and wound segmented boundary are overlaid on the wound image.

According to an embodiment of the disclosure, the system <NUM> also comprises the self-learning module. The self-learning module configured to receive annotating on the output image of the wound provided by a user, if the segmented output image is not matching with the input image based on an evaluation done by the user; and utilizing the annotation provided by the user to self-learn the training module to improve the accuracy of the system. In another embodiment, the user can provide an annotation tool that has been developed based on graph-cut method with minimum touch points of two or more that distinguish the foreground and background of the image. In yet another embodiment, the user can provide the annotation tool using a manual re-draw approach.

<FIG> illustrates a flow chart of an AI based method <NUM> for analyzing the wound in the person, in accordance with an example embodiment of the present disclosure. The method <NUM> depicted in the flow chart may be executed by a system, for example, the system <NUM> of <FIG>. In an example embodiment, the system <NUM> may be embodied in the computing device.

Operations of the flowchart, and combinations of operations in the flowchart, may be implemented by various means, such as hardware, firmware, processor, circuitry and/or other device associated with execution of software including one or more computer program instructions. For example, one or more of the procedures described in various embodiments may be embodied by computer program instructions. In an example embodiment, the computer program instructions, which embody the procedures, described in various embodiments may be stored by at least one memory device of a system and executed by at least one processor in the system. Any such computer program instructions may be loaded onto a computer or other programmable system (for example, hardware) to produce a machine, such that the resulting computer or other programmable system embody means for implementing the operations specified in the flowchart. It will be noted herein that the operations of the method <NUM> are described with help of system <NUM>. However, the operations of the method <NUM> can be described and/or practiced by using any other system.

Initially at step <NUM> of the method <NUM>, the plurality of images of a plurality of wounds is received as the input via the user interface <NUM>. At step <NUM>, the plurality of images is then preprocessed. Further at step <NUM>, the deep learning model is trained using the plurality of preprocessed images, wherein the deep learning model having an encoder decoder architecture, the encoder decoder architecture comprises: a pre-trained Resnet50 as the encoder, an Atrous spatial pyramid pooling (ASPP) comprising of dilated convolution, wherein the ASPP is connected to the encoder, a bilinear up sampler configured to up sample the plurality of images by a factor of four, and a channel wise attention mechanism using a squeeze and excitation (SE) network, designed to improve representational power of the SE network by enabling to perform dynamic channel-wise feature recalibration.

At step <NUM> of the method <NUM>, the image of the wound to be analyzed is provided. At step <NUM>, the image to be analyzed is preprocessed. In the next step <NUM>, the output image is generated from the preprocessed image using the trained deep learning model. Further at step <NUM>, the output image is preprocessed using the binary segmentation mask with the fixed threshold of the predefined number to get the segmented output image.

In the next step <NUM> of the method <NUM>, morphology operations are performed on the segmented output image to remove the small regions and spurious noises to fill the small holes within the wound to improve a true positive rate. In the next step <NUM>, connected component analysis is performed on the morphology operated image to label connected regions. Further at step <NUM>, the plurality of wound parameters of the wound is calculated by measuring the labelled connected regions. And finally, at step <NUM>, the calculated plurality of wound parameters and wound segmented boundary is overlaid on the image of the wound. This overlaid image can be displayed on the display screen which can further be analyzed by the physician.

According to an embodiment of the disclosure, the method <NUM> further comprises receiving annotation on the segmented output image of the wound provided by a user, if the segmented output image is not matching with the input image based on an evaluation done by the user; and utilizing the annotation provided by the user to self-learn the training module <NUM> to improve the accuracy of the system <NUM>.

According to an embodiment of the disclosure, the system <NUM> also comprises a 3D printer. The 3D printer is configured to provide a wound dress patch estimated using the plurality of wound parameter.

According to an embodiment of the disclosure, the system <NUM> can also be explained with the help of experimental results. In an example, preexisting wound care data have been merged segmentation dataset, which leads to <NUM> images. Further, images were resized and augmented as a process homogeneousness, extending the dataset as <NUM> images. The DeepLab V3+, and Resnet-<NUM> encoder was adapted with pre-trained weights from image net dataset, resulting in a dice score of <NUM> and IOU -<NUM>.

Later the performance was improved to a dice score of <NUM> and IOU-<NUM> for DeepLab-V3+ with squeeze & Excite model. Further, the AI module is implemented with morphological operations, connected component analysis, and shape analysis, improving accuracy and providing the wound parameter and metrics such as area, perimeter, circle diameter, major and minor axis length of an ellipse. In addendum, to ease the wound care Pro with the mobile app, a lightweight approach was attempted using the U-Net model with Mobile Net Encoder, which yields a dice score of <NUM> and IOU of <NUM>.

For homogeneousness, the images were resized to <NUM>*<NUM> and the Image Augmentation techniques which includes, Gamma, Saturation, Hue, Horizontal and Vertical Flips etc. were adapted, which made the training data set as <NUM> wound images. The augmentations were not applied to the testing and validation sets and contained <NUM> images in each set.

The deep learning model in the presented work was implemented in python with Keras and TensorFlow backend. The model was trained on Tesla P100 GPU with <NUM> GB memory. For updating the parameters in the network, the Adam optimization algorithm was employed. Dice loss was used as the loss function and also Precision, Recall and the Dice score were monitored as the evaluation matrices. The initial learning rate was set to <NUM> and each minibatch contained <NUM> images for balancing the training accuracy and efficiency. A drop out of <NUM> was specified in order to prevent any overfitting. Early stopping was used to terminate the training so that the best result was saved when there was no improvement for more than <NUM> epochs in terms of validation loss. Eventually, the deep learning model was trained for around <NUM> epochs before overfitting illustrates the training and validation losses.

The embodiments of present disclosure herein addresses unresolved problem of accurate, timely and cost effective analysis of the wounds of the person. The embodiment, thus providesan AI based method and a system for analyzing the wounds.

Claim 1:
A system (<NUM>) for analyzing a wound, the system comprising:
a user interface (<NUM>) for receiving a plurality of images of a plurality of wounds as an input,
one or more hardware processors (<NUM>); and
a memory (<NUM>) in communication with the one or more hardware processors, wherein the one or more first hardware processors are configured to execute programmed instructions stored the memory, the memory further comprises:
a training module configured to train a deep learning model using the plurality of preprocessed images, wherein the deep learning model having an encoder decoder architecture, the encoder decoder architecture comprises:
a pre-trained Resnet50 as the encoder,
an Atrous spatial pyramid pooling (ASPP) comprising of dilated convolution, wherein the ASPP is connected to the encoder,
a bilinear up sampler configured to up sample the plurality of images by a factor of four, and
a channel wise attention mechanism using squeeze and excitation (SE) network, designed to improve representational power of the network by enabling to perform dynamic channel-wise feature recalibration;
an image segmentation module configured to:
preprocess an input image of the wound to be analyzed;
generate an output image from preprocessed image using the trained deep learning model; and
post-process the output image using a binary segmentation mask with a fixed threshold of a predefined number;
an AI based estimation module configured to:
perform morphology operations on the segmented image to remove small regions and spurious noises to fill small holes within the wound to improve a true positive rate;
perform connected component analysis on the binary image to label connected regions;
calculate a plurality of wound parameters of the wound by measuring the labelled connected regions; and
overlay the calculated plurality of wound parameters on the image of the wound.