## 2 Materials and Methods
### Forming Feature Vector
#### 2.3.1 Transfer Learning based Feature Extraction  
In our previous work [19], Inception-ResNet-V2 Convolutional Neural Network (CNN) was used. It was pre-trained and validated with 50,000 images set to classify 1000 object categories and learned rich feature representations with 825 layers. A huge number of images is required to re-train the whole network. The first 820 layers were frozen to limit the number of parameters required to retrain the network to avoid overfitting with  
Figure 2: Preprocessed image example from each category: (**a**) Normal Right, (**b**) Normal Left, (**c**) Abnormal Right (Effusion) (**d**) Abnormal Left (Effusion), (**e**) Abnormal Right (Tympanostomy tube), and (**f**) Abnormal Left (Tympanostomy tube).  
our limited dataset. The last three layers (prediction, softmax, and classification) of the pre-trained network were retrained with otoscope images in our database.  
After retraining the network, the resulting features were subjected to pooling that mapped each image into a vector of 1536 features. We relied on the fully connected layer's output, which produced \(1\times 3\) vectors for each training and test image, where three represents the number of image categories in our database: normal, effusion, and tympanostomy tube. The features formed a \(1536\times 1\) vector at the output of the average pooling layer. Therefore, the weights were a \(1536\times 3\) matrix of the fully connected layer. When the transpose of the feature vector was multiplied by the weight vector, it produced a \(1\times 3\) vector, which was established for each of the training set of images. When these vectors were turned to rows of a matrix (size of the number of training images \(\times\) 3), this constitutes the lookup table.  
This procedure was applied to a pair of eardrum images for normal/abnormal feature extraction. Inception-ResNet-V2 pre-trained CNN was used by freezing the first 820 layers. In this study, the number of categories was two (i.e., normal, abnormal), and the weights constitute a \(1536\times 2\) matrix of the fully connected layer. The generated lookup table is a vector of length \(N\times 2\), where \(N\) is the number of training images. Additionally, test images have a \(1\times 2\) vector after multiplying them by weights. For each pair of the eardrum images, these lookup values are calculated, and a new feature vector is formed using these values. The steps to create a lookup table from transfer learning can be generalized, as follows:  
* Form a feature vector \(f_{i}\) as the average pooling layer output for each image, \(i\). Its size is \((F\times 1)\) (F = 1536 in this case).
* Let \(w\), of size \((F\times C)\), be the weights of the fully connected layer, where \(C\) is the number of training classes (\(C=2\) in this case because of two categories: 'normal' and 'abnormal').
* The lookup table values \(l_{i}\) for one image, \(i\), can be calculated as \(l_{i}={f_{i}}^{T}\times w\), and its size is \((1\times C)\).
* If \(N\) is the number of images in the dataset (both for training and testing), the lookup table \(L_{T}\) is a matrix calculated as concatenation of the lookup table values \(l_{i}\) for each image, \(i\) and with a size \(N\times C\) (\(300\times 2\) for this case).  
The lookup table values of the right eardrum image are R\({}_{1}\) and R\({}_{2}\), and of the left eardrum image are L\({}_{1}\) and L\({}_{2}\). Their ratio (R\({}_{1}\)/L\({}_{1}\), R\({}_{2}\)/L\({}_{2}\)), summation (R\({}_{1}\) + L\({}_{1}\), R\({}_{2}\) + L\({}_{2}\)), and difference (R\({}_{1}\) - L\({}_{1}\), R\({}_{2}\) - L\({}_{2}\)) are also concatenated to form a feature vector. This new vector, which contains both eardrum pairs' features, enables us to classify the pair together by combining the derivative of lookup table values for a pair of eardrum images.