System and method of using visually-descriptive words to diagnose ear pathology

Disclosed herein are systems and methods to detect a wide range of eardrum conditions by using visually-descriptive words of a tympanic membrane of a subject.

BACKGROUND

Ear infections, specifically acute infections of the middle ear (acute otitis media), are the most commonly treated childhood disease and account for approximately 20 million annual physician visits in the U.S. alone.

Ear diseases are one of the diseases that can easily be treated when diagnosed at the right time, and when appropriate treatment methods are applied. Otherwise, ear diseases may cause hearing loss or other complications. An otoscopic examination is one of the most basic and common tools used to examine the ear canal and eardrum (tympanic membrane, TM). However, an examination by an experienced and skillful physician may not always be possible. To help physicians who lack the same otoscopy experience, computer-aided diagnosis (CAD) systems may be useful. However, most CAD systems require obtaining and analyzing an image obtained from a subject undergoing examination. US PG-Patent Publication No. US 2019/0216308 Al published Jul. 18, 2019, which is fully incorporated by reference, is an example of a content-based image retrieval (CBIR) system, which is also a good example of a CAD system designed to help physicians in making diagnostic decisions based on TM images.

However, image data can be large and difficult to transmit from remote sites to a location (and device) with processing power capable of performing the image analysis.

Therefore, systems and methods are desired that overcome challenges in the art, some of which are described above. In particular, there is a need for a timely and accurate method and system to make diagnostic decisions about TM conditions based on visually-descriptive words used to describe the condition of the TM to properly identify and classify any of a multitude of ear pathologies.

SUMMARY

Herein, creation of a database is described that utilizes a digital otoscopy video summarization and automated diagnostic label assignment model that benefits from the synergy of deep learning and natural language processing (NLP). Key visual features of TM diseases are obtained from short descriptive reports of TM images. Otoscopy records from a plurality of different TM diseases were obtained and composite images were generated for TM that exhibited visual indicators of the various disease, and normal TM. An ENT expert reviewed these composite images and wrote short reports describing the TM's visual landmarks and the disease for each ear. Based on NLP and a bag-of-words (BoW) model, a reduced set (e.g., five) most frequent words characterizing each TM diagnostic category were determined. These reduced sets of words and each set's corresponding TM condition (normal., TM disease 1, TM disease 2, . . . etc.) were then stored in a database. Once trained, the model can automatically create a reduced set of words characterizing the TM diagnostic category of additional images.

Once the model and database are created, visual descriptions of a subject's TM are then received from a healthcare provider and compared to the reduced sets of words. A diagnosis of the subject's TM condition is then made based on the best match between the visually-descriptive words received from the healthcare provider and the reduced sets of words associated with TM conditions, as stored in the database. In some instances, the disclosed systems, methods and computer program product comprise a telemedicine application in that the model can automatically make a diagnosis of the TM by analyzing its visual descriptions provided by a healthcare provider from a mobile device.

DETAILED DESCRIPTION

FIG.1illustrates an exemplary overview system100for classifying ear pathologies from visual descriptors of a tympanic membrane of a subject104. As shown inFIG.1, one embodiment of the system100comprises an image capture mechanism102. In one aspect, the image capture mechanism102can be a camera. More specifically, the image capture mechanism102may be a digital otoscope and/or a video otoscope. The image capture mechanism102can take still and/or video images of each ear. Generally, the image capture mechanism102is a digital camera, but can be an analog device equipped with or in communication with an appropriate analog/digital converter. The image capture mechanism102may also be a webcam, scanner, recorder, or any other device capable of capturing a still image or a video. In some instances, the system may comprise two image capture mechanisms102for capturing images of each ear either simultaneously or in sequence.

As shown inFIG.1, the image capture mechanism102is in direct communication with a device configured to display images captured by the image capture device102. For example, an otoscope or video otoscope may be connected to or integrated with a display106. A healthcare provider108reviews the image or images on the display106. The healthcare provider uses words to visually describe the one or more images of the tympanic membrane of the subject shown on the display106. The visually-descriptive words may be spoken into or typed into a device110, where they may be stored and are transmitted to a cloud computing architecture112. The device110may comprise or be a part of a smart device such as a smart phone, tablet, laptop computer or any other fixed or mobile computing device. In some instances, the device110may be executing a specialized application for receiving the visually descriptive words (whether spoken or typed in) and transmitting them to the cloud computing architecture112. The specialized application may have security features such that only the healthcare provider108and/or a designee of the healthcare provider108has access to the specialized application and/or the words used to describe the one or more images visually. Such security features may be those that are in compliance with government protocols for data security and privacy, such as HIPAA (in the United States). The cloud computing architecture112delivers the visually-descriptive words to a processing device114. The processing device114may be a part of the cloud computing architecture112or it may be a processing device that is in communication with the cloud computing architecture112. The visually descriptive words may be transmitted in real-time from the device110or may be stored in the device110and transmitted at a later time.

The processing device114, in a basic configuration, can be comprised of a processor116and a memory118. The processor116can execute computer-readable instructions that are stored in the memory118. The processor116can further execute computer-readable instructions stored on the memory118to determine, using the received visually-descriptive words from the healthcare provider108, and classify ear pathologies from the visually-descriptive words. Moreover, the processor116can execute computer-readable instructions to compare the visually descriptive words received from the healthcare provider to words in a database120residing in the memory118that have been selected by an artificial intelligence (AI) algorithm that are most-often used to make a diagnosis of a condition of the tympanic membrane of the ear from visually looking at images of tympanic membranes of ears with various conditions (e.g., Normal, Effusion, Retraction, Tympanosclerosis, etc.). The processor116then makes a match between the visually-descriptive words received from the healthcare provider and the closest descriptive words in the database that are used to describe a condition of the TM, such that a diagnosis of a condition of the ear can be made. The diagnosis is then transmitted back from the processing device114through the cloud computing architecture112to the device110of the healthcare provider108, where the diagnosis of the ear (or ears) of the subject104is received by the healthcare provider108.

In some instances, the visually descriptive words may be transmitted from the device110directly over a network (not shown inFIG.1) to the processing device114for analysis, thereby omitting the cloud computing architecture112. The network may comprise, for example, a wired (including fiber optic) network, wireless or a combination of wired and wireless) or a direct-connect cable (e.g., using a universal serial bus (USB) connection, IEEE 1394 “Firewire” connections, and the like). In other aspects, the visually descriptive words from the healthcare provider108may be recorded or stored on a memory device such that the visually descriptive words can be downloaded or transferred to the processing device114using, for example, a portable memory device and the like.

FIG.2illustrates an alternate exemplary overview system200for classifying ear pathologies from visual descriptors of a tympanic membrane of a subject. In the instance illustrated inFIG.2, the image capture mechanism102captures an image of the TM of the subject104, which is displayed on a display associated with device202. Device202also includes a network interface mechanism that allows it to communicate with the cloud computing architecture112directly. Device202further includes I/O mechanisms such that the healthcare provider108can speak, type or otherwise enter visually-descriptive words about the TM of the subject104into the device202and transmit these visually-descriptive words to the cloud computing architecture112where they are compared, by the processing device114, to the reduced sets of words in the database that are associated with corresponding TM conditions to make an automated diagnosis of the TM condition of the subject104. The diagnosis then goes back to the device202through the cloud computing architecture112, and provided to the healthcare provider108. As withFIG.1, in some instances, the cloud computing architecture112ofFIG.2may be replaced with a direct network condition or, in some instances, the descriptive words from the healthcare provider may be stored and physically brought to the processing device for analysis. Device202may also be configured to execute an application, as described above, to display the images and transmit visually-descriptive words of the image as provided by the healthcare provider108. As with the above, the application and/or device202may have data privacy and security protocols and may also transmit the visually-descriptive words in real-time, or store them for later transmission. In some instances, device202is a smart device such as a smart phone, tablet, laptop computer or any other fixed or mobile computing device.

Creating the Model

FIG.3illustrates a process for creating a model for determining a diagnosis of a TM of a subject based upon visually-descriptive words used to describe a condition of the TM. The model begins with composite image generation from otoscope images and/or video clips and creating reports by viewing the composite images.

In the example ofFIG.3, images for creating the database were obtained from high-resolution digital adult and pediatric videos captured at ENT clinics and primary care settings at the Ohio State University (OSU) and Nationwide Children's Hospital (NCH) in Columbus, Ohio, USA. An Institutional Review Board (IRB) approved the data collection process. A high definition (HD) video otoscope (JEDMED Horus+ HD Video Otoscope, St. Louis, MO) was used to capture and record the video data. The dataset included 173 otoscope videos, including 86 instances of otitis media with effusion (this condition will be referred to as effusion for the rest of the paper), 35 instances of a retracted TM (referred to as retraction), and 52 instances of tympanosclerosis. After generating composite images, an ENT physician provided a diagnosis for each image, delineated the lesion on the TM images, and wrote a short descriptive report describing the corresponding TM abnormality's visual features. It is to be appreciated that other datasets may be used to create the composite images and corresponding reports.

Composite Image Generation

A U-Net based semantic segmentation method was utilized to determine the meaningful video frames from otoscopy videos, though other methods are contemplated to be within the scope of this disclosure. This process is described in greater detail with reference toFIG.4, below. In this non-limiting example of the semantic segmentation task, the segmentation model was developed with 36 otoscope videos. The 764 frames were picked from those videos, and TM regions were manually identified and annotated by two ENT physicians.

The acquisition of adequate images can be a challenging task because of visual obstruction (e.g., wax, hair, etc.), poor illumination, a small field of view, black margins around the images, time/text stamps on the image, and the like. If the patient is a child, there may also be the problem of being able to capture a good still image while the patient is uncooperative. To solve these challenges, a short video (around 3-5 seconds) of each ear canal of the subject is captured. Then, software, executing the algorithm shown inFIG.4, analyzes video frames of the eardrum and creates a new mosaicked image.

For each new frame in the video sequence, the mosaic image creation algorithm as described inFIG.4determines the regions of interest which are free of obstruction (e.g., wax, hair, dark margins, text, etc.). Each of these regions is divided into subsections, and the image quality in each section is evaluated in terms of being in-focus, having adequate contrast and illumination. If the frame includes the part of the eardrum that is not included in the previous frames or includes an already included part of the eardrum but with higher quality (in terms of focus, contrast and illumination), then this frame is labeled as an “important frame” or otherwise identified. Finally, the method constructs the mosaic image by considering the regions of interest in all the “important frames” in the video sequence.

The frames may include different amounts of visual obstruction (e.g., wax, hair, glare, text, dark margins, etc.) and/or quality of illumination. As described herein, the method includes constructing composite obstruction-free images with excellent illumination. Therefore, the algorithm detects obstructions (wax, glare, hair, text, dark margins—see below) and out-of-focus regions during the composite image generation. To do that, the algorithm compares each new frame with the previous frames and updates the new image using the regions that are more in-focus and well-illuminated. To decide on in-focus and illumination quality, an image entropy is computed, and the frame with the highest entropy is selected.

Regarding wax detection, one of the typical characteristics of cerumen is its yellow color. Therefore, yellow regions are identified by using thresholding in CMYK color space. After these potential cerumen regions are detected as those regions with the highest “Y” values in the CMYK space, the mean and standard variation of the gradient magnitude of the intensities (i.e. “Y” values) of these cerumen regions are computed. These features are input to the FSG classifier to detect wax regions.

Glare is caused by the reflection of light from the otoscope on the surface of the tympanic membrane. Glare may be a problem for the calculation of some of the features (e.g., the mean color value of tympanic membrane). On the other hand, the cone of light, an important clinical diagnostic clue, can be inadvertently considered as glare by the glare detection algorithm and removed. In order to correctly extract the features, the disclosed method includes calculating the histogram of the intensity values and finds the peak corresponding to the highest intensity value in the histogram. That peak corresponds to the glare and cone of lights. To differentiate between the glare and cone of lights, area thresholding is applied (where glare(s) is larger than the cone of light(s)).

Hair detection includes detecting thin linear structures by using a line segment detector such as that described in R. G. von Gioi, J. Jakubowicz, J. -M. Morel, and G. Randall, “LSD: A fast line segment detector with a false detection control,” IEEE transactions on pattern analysis and machine intelligence, vol. 32, pp. 722-732, 2010, which is incorporated by reference. Each hair strand is represented by two lines (both edges of the hair), approximately parallel to each other and the lines are close to each other. So, each approximately parallel line pair with a short distance is considered a hair candidate. The image texture is calculated between these parallel lines, and those with small textural variation are marked as hair.

In some instances, after the regions of interest are extracted, these regions are divided into 64×64 pixel blocks. For each block, the standard deviation, gray level co-occurrence matrix, contrast, and the mean intensity value are calculated. These values are weighted to calculate the tile quality. The weights may be determined manually or automatically.

To register two frames, points of interest are automatically extracted and the feature vectors for these points are matched. To extract points of interest, the performance of three state-of-the-art approaches is compared (see H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robust features,” Computer vision—ECCV 2006, pp. 404-417, 2006; D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International journal of computer vision, vol. 60, pp. 91-110, 2004; and E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “ORB: An efficient alternative to SIFT or SURF,” in Computer Vision (ICCV), 2011 IEEE International Conference on, 2011, pp. 2564-2571, each of which is fully incorporated by reference.). In order to identify the matched points, the approach computes the distance between all possible pairs of detected features in two frames. The approach estimates the initial Homograph matrix with Random Sample Consensus (RANSAC) (see M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the ACM, vol. 24, pp. 381-395, 1981, which is also incorporated by reference).

Each frame is identified as an “important frame” or not according to two criteria: (1) If the new frame includes new regions of interest which are not covered previously by another important frame; or (2), if the region which is already covered by a previous important frame has a higher quality in this new frame. A composite image can then be created by stitching. The disclosed method uses ‘important frames” during the composite image construction. The algorithm selects the most suitable “important frames” for subparts of the eardrum and uses a multi-band blending (pyramid blending) method, which ensures smooth transitions between images despite illumination differences, while preserving high-frequency details.

A freeware image stitching engine [12] was then used to create enhanced composite images by employing selected frames.FIGS.5A-5Dshow examples of TM conditions from the described dataset along with a normal TM for reference, whereFIG.5Ais a normal TM,FIG.5Bis a TM with effusion,FIG.5Cis a TM with retraction, andFIG.5Dis a TM with tympanosclerosis.

Text Analysis and Classification

Referring back toFIG.3, several NLP tools were used in order to preprocess the TM reports of the ENT physician (i.e., expert) for analysis as follows: tokenization [13]; stop word removal [14]; word normalization [15]; and punctuation erasing [16].

A tokenized document is a document represented as a collection of words (also known as tokens). Words like “a,” “and,” and “to” (known as stop words) in English can add noise to data. Using a stop word removal functionality, these words were removed. Then, word normalization was used to reduce words to a root form (e.g., ‘Connection’→‘connect’). As the final preprocessing step, punctuation was removed from the text.

A bag-of-words (BoW) model [17] (also known as a term-frequency counter) was applied to obtain the number of times that words appear in each report, implying their frequency. To record the most important words in the BoW model for each diagnostic category, the five words with the highest word counts were specified for the training data, though more or fewer words may be used in other instances. For the multiclass classification of the preprocessed test report, a neighborhood components analysis (NCA) [18] was created using the BoW models of training data with the five most frequent words. NCA learns a linear transformation of data that maximizes k-nearest neighbor performance [19]. Moreover, generating a low-rank transformation, NCA provides dimensionality reduction, which extends the disclosure to high-dimensional feature space while integrating visual features with textual features.

For each TM condition, word clouds were generated. See, for example,FIGS.6A-6C. By analyzing the resulting word clouds, it can be seen that the keywords for effusion (FIG.6A) were mainly indicating the color features such as “amber” and “dark.” For retraction (FIG.6B), the keywords were “retracted” and “sucked,” and for tympanosclerosis (FIG.6C), a lesion was mostly described with the words “white” and “plaque.” Word clouds can be generated in this manner for any number of TM conditions, including the “normal” condition. Once trained, the model can automatically create a reduced set of words characterizing the TM diagnostic category of additional images.

Once a database of TM conditions and the associated word clouds of each condition is created, it can be used to make determinations about the TM condition of a subject when a healthcare provider visually describes the TM of the subject.FIG.7is a flowchart that illustrates an exemplary method of classifying ear pathologies from visually descriptive words describing a subject's TM. Steps702and704describe the process of creating a database of word clouds and associated TM conditions, as described above. At706, one or more words that provide a visual description of the TM of a subject are received from a healthcare provider. These words may be received in a variety of ways, as described herein. For example, they may be expressed by the voice of the healthcare provider, in real-time or recorded, or they may be provided in written format (e.g., text). The one or more words of the visual description are received electronically, and at708they are compared, using a processing device executing computer-executable instructions, to the word clouds associated with various TM conditions in the database. The closest fit is found between the one or more visual description words and a word cloud. The TM condition that corresponds with the selected word cloud is then provided as the diagnosis of the TM of the subject.

The system has been described above as comprised of units. One skilled in the art will appreciate that this is a functional description and that the respective functions can be performed by software, hardware, or a combination of software and hardware. A unit can be software, hardware, or a combination of software and hardware. The units can comprise software for making a determination of a diagnosis of a TM condition based upon words visually describing the TM of a subject. In one exemplary aspect, the units can comprise a computing device that comprises a processor821as illustrated inFIG.8and described below.

FIG.8illustrates an exemplary computer that can be used for classifying tympanic membrane pathologies from images. As used herein, “computer” may include a plurality of computers. The computers may include one or more hardware components such as, for example, a processor821, a random access memory (RAM) module822, a read-only memory (ROM) module823, a storage824, a database825, one or more input/output (I/O) devices826, and an interface827. Alternatively and/or additionally, the computer may include one or more software components such as, for example, a computer-readable medium including computer executable instructions for performing a method associated with the exemplary embodiments. It is contemplated that one or more of the hardware components listed above may be implemented using software. For example, storage824may include a software partition associated with one or more other hardware components. It is understood that the components listed above are exemplary only and not intended to be limiting.

Processor821may include one or more processors, each configured to execute instructions and process data to perform one or more functions associated with a computer for classifying pathologies of an eardrum based upon one or more images of the eardrum. Processor821may be communicatively coupled to RAM822, ROM823, storage824, database825, I/O devices826, and interface827. Processor821may be configured to execute sequences of computer program instructions to perform various processes. The computer program instructions may be loaded into RAM822for execution by processor821.

RAM822and ROM823may each include one or more devices for storing information associated with the operation of processor821. For example, ROM823may include a memory device configured to access and store information associated with the computer, including information for identifying, initializing, and monitoring the operation of one or more components and subsystems. RAM822may include a memory device for storing data associated with one or more operations of processor821. For example, ROM823may load instructions into RAM822for execution by processor821.

Storage824may include any type of mass storage device configured to store information that processor821may need to perform processes consistent with the disclosed embodiments. For example, storage824may include one or more magnetic and/or optical disk devices, such as hard drives, CD-ROMs, DVD-ROMs, or any other type of mass media device.

Database825may include one or more software and/or hardware components that cooperate to store, organize, sort, filter, and/or arrange data used by the computer and/or processor821. For example, database825may store a plurality of word clouds, and a TM condition associated with each word cloud, along with computer-executable instructions for receiving one or more words that visually describe a TM of a subject; comparing the one or more visually descriptive words to each of the plurality of word clouds, determining the closest fit between the one or more visually descriptive words and one of the word clouds; and diagnosing a condition of the TM of the subject based on the TM condition associated with the selected word cloud. It is contemplated that database825may store additional and/or different information than that listed above.

I/O devices826may include one or more components configured to communicate information with a user associated with computer. For example, I/O devices may include a console with an integrated keyboard and mouse to allow a user to maintain a database of digital images, results of the analysis of the digital images, metrics, and the like. I/O devices826may also include a display including a graphical user interface (GUI) for outputting information on a monitor. I/O devices826may also include peripheral devices such as, for example, a printer for printing information associated with the computer, a user-accessible disk drive (e.g., a USB port, a floppy, CD-ROM, or DVD-ROM drive, etc.) to allow a user to input data stored on a portable media device, a microphone, a speaker system, or any other suitable type of interface device.

Interface827may include one or more components configured to transmit and receive data via a communication network, such as the Internet, a local area network, a workstation peer-to-peer network, a direct link network, a wireless network, or any other suitable communication platform. For example, interface827may include one or more modulators, demodulators, multiplexers, demultiplexers, network communication devices, wireless devices, antennas, modems, and any other type of device configured to enable data communication via a communication network.

EXAMPLES

Experimental Setup

Three global evaluation measures, i.e., sensitivity, specificity, and F1-Score, were used to evaluate the proposed model's performance.

Sensitivity is computed as:

where TP denotes the number of true positives, TN the number of true negatives, FP the number of false positives, and FN the number of false negatives. A leave-one-patient-out cross-validation scheme was applied to validate our model.

Results and Discussion

Word clouds were created for eardrum conditions of effusion, retraction, and tympanosclerosis, as described herein.

The classifier's confusion matrix, which is based on text classification from bag-of-words using top-5 words for each TM category, is presented in Table 1.

The NCA Predictor Confusion Matrix

From 86 instances of descriptions labeled as effusion, the model predicted all of them correctly. On the other hand, 26 instances were successfully predicted as retraction, with nine misclassifications; 46 instances were predicted as tympanosclerosis with six misclassifications. Table 2 lists the sensitivity, specificity, and F1-Score of the proposed classification model based on these values.

Performance Results Obtained by the Proposed Model in Terms of Sensitivity, Specificity, and F1-Score in Percentage

As shown in Table 2, the experimental results showed that an embodiment of the disclosed model performed effectively (overall F1-Score is 90.2%) in the task of diagnostic label prediction of otoscopy records using short descriptive reports. It is worth noticing that our model uses the top-5 words in any disease category instead of all the words. Our aim in determining the top-5 words was to obtain the “key” words for each disease and explore the model's efficiency with these keywords.

It was also observed that the model predicted almost all of the misclassified retraction and tympanosclerosis instances as effusion. Effusion is one of the hardest TM conditions to identify and distinguish from others. In the misclassified instances, the ENT physician mainly described findings as suspicious for, but not clearly, effusion. For example, the true label was determined as retraction by the ENT physician instead of effusion, whereas he wrote some minor visual evidence of effusion, but he evaluated these pieces of evidence as subtle. Hence, our model estimated these instances as effusion.

Conclusions

There is a growing interest in the analysis of TM imagery. While most of these studies focus on image analysis and disease classification, it is important to develop other data sources that could be useful in better diagnosis and treatment. One such source could be physician observations and notes. This study's main contributions are to analyze the TM reports, describe different types of TM abnormalities generated by reviewing composite images of raw otoscopy videos and predict the label of the test instance from these reports using NLP and machine learning. In one example, we extracted the five most frequent words from each category using NLP techniques and created a classifier based on these five words from training data. By following a leave-one-patient-out cross-validation method, the exemplary model achieved 92.5%, 85.3%, and 92.9% F1-Score for effusion, retraction, and tympanosclerosis, respectively. Thus, it is shown that using textual features can provide satisfactory information for the computerized diagnosis of TM videos and images.

REFERENCES

Throughout this application, various publications may be referenced. The disclosures of these publications in their entireties are hereby fully incorporated by reference into this application in order to more fully describe the state of the art to which the methods and systems pertain. The below publications are each fully incorporated by reference and made a part hereof: