_Applied_  
_sciences_  
_Article_  
**OtoPair: Combining Right and Left Eardrum Otoscopy Images**  
**to Improve the Accuracy of Automated Image Analysis**  
**[PERSON] 1,*\({}^{\text{*}}\)**\({}^{\text{\textregistered}}\), [PERSON] 2, [PERSON] 3, [PERSON] 2, [PERSON] 2,  
**[PERSON] 4** and [PERSON] 1\({}^{\text{\textregistered}}\)  
Footnote 1: [[http://www.mdpi.com/journal/applsci](http://www.mdpi.com/journal/applsci)]([http://www.mdpi.com/journal/applsci](http://www.mdpi.com/journal/applsci))  
Footnote 2: [[https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci)]([https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci))  
Footnote 3: [[https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci)]([https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci))  
Footnote 4: [[https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci)]([https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci))  
Footnote 5: [[https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci)]([https://www.mdpi.com/journal/applsci](https://www.mdpi.com/journal/applsci))  
Most CAD systems use a classification method to determine whether the middle ear has an infection [4; 5; 6; 7; 8], because otitis media (OM) is one of the most common diseases for children under the age of five years [9; 10]. Other ear diseases, such as retraction, perforation, and tympanosclerosis are classified as abnormal eardrum images against normal eardrum images by the automated CAD system [11; 12; 13]. In addition to classification methods, there are also approaches to segment and classify the TM on otoscopic images [14; 15; 16]. Recently, composite images, which are created by selecting certain otoscopy video frames and stitching them together, are also used to increase the probability of detecting ear pathology [17; 18]. Our previous study using OtoMatch, a content-based image retrieval (CBIR) system, is also a good example of a CAD system that is designed to help physicians [19].  
CAD approaches for TM analysis, which are used to classify and/or segment the eardrum, can be collected under two categories: hand-crafted and deep learning-based. For a hand-crafted approach, the most commonly used features are color-based information in addition to traditional texture approaches [5; 12; 14; 15; 20; 21]. The color-based information has been common, because there are significant differences between normal and abnormal cases of eardrums. The deep learning-based approach is also used more than the texture-based approach, because it is typically more accurate [4; 22]. One study has used both a hand-crafted and deep learning-based approach to classify otoscopy images [7].  
CAD for OM abnormalities has only been applied to single TM images, to the best of our knowledge. [PERSON] et al. proposed a convolutional neural network (CNN)-based approach that detects the ear's side, but this information was not used to classify paired images (right and left ears) together [23]. However, physicians typically examine both ears during a physical exam before making a diagnosis. In this first study to classify a pair of eardrum images together, we propose a system called OtoPair that uses deep learning- and color-based features to classify a pair of TM images as 'normal-normal', 'normal-abnormal', 'abnormal-normal', or 'abnormal-abnormal'. A lookup table was created to extract deep learning-based features. The pre-processing steps for creating the lookup table are similar to our previous study, called OtoMatch [19]. The lookup table values of the paired images were analyzed according to their labels to determine the association between right and left ear values. Additionally, we investigated the contribution of color-based features to the classification accuracy.