# A Few-Shot Learning-Based Material Recognition Scheme Using Smartphones
## 5 Performance Evaluation
### Performance Evaluation of FSMR
#### 5.2.1 Few-Shot Learning Evaluation  
Figure 8 illustrates the classification performance of CNN and few-shot learning models in a graphical format. This experiment was conducted to evaluate whether the few-shot learning model can classify effectively with small datasets, comparing its performance to that of the CNN model, which is commonly used for classification tasks. The total number of classes for classification was 10, and the performance was evaluated under two conditions: when trained with a small dataset of 15 s per class and when trained with a larger dataset of 70 s per class. \begin{table}
\begin{tabular}{c c c} \hline \hline  & \multicolumn{2}{c}{**Dataset**} \\ \hline
**Type** & **Train** & **Test** \\ \hline N & 45 & 10 \\ \hline T & 190 & 190 \\ \hline \multirow{7}{*}{Class} & bag1(leather1), bag2(leather2), cd player, chair1, clipboard, & \\  & computer, cushion, desk1, desk2, desk3(glass1), & \\  & desk4(glass2), desk5, deskpad1, deskpad2, desktop, & \\  & doormat, drawer1, drawer2, drawingtabletlet, hand(sitting), & \\  & hand(walking), hand1, laptop1, ledgel(concrete), mattress1, & \\  & mattress2, mattress3, mattresspad, monitorsand1, & \\  & monitorstand2, mousepad1, mousepad2, mousepad3, & \\  & newspaper, pillow1, pillow2, pocket1, pocket2(jacket), sofa1, & \\  & speaker, table1, table2(small), table3, table4, trivet(rubber) & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Overview of the dataset collected for performance evaluation. Figure 8: Material recognition accuracy of CNN and FSMR trained by 2 datasets that have different sizes. When trained with the larger dataset (70 s per class), the CNN model achieved 87.5% accuracy, outperforming the few-shot learning model, which achieved 72.5%. However, the practical application of the CNN model in real context-aware services remains limited due to its data requirements. Conversely, when trained with the smaller dataset (15 s per class), the few-shot learning model achieved a significantly higher accuracy of 68.5% compared to the CNN model's 34.29%. This result demonstrates that a dataset of 15 s per class is insufficient for conventional deep learning models to achieve reasonable classification accuracy. However, the few-shot learning model can achieve a certain level of classification accuracy even with minimal data, thus validating the effectiveness of the few-shot learning approach. Figure 9 presents the results of applying various training configurations to the few-shot learning model. This experiment aimed to identify the optimal N-way K-shot configuration for classification purposes. The number of classes ranged from a minimum of 3-way to a maximum of 10-way, and the number of data points used for training ranged from a minimum of 1-shot to a maximum of 15-shot. For N-way training, N random classes were selected from a total of 10 classes, and the classification accuracy for these selected classes was evaluated. This process was repeated 1000 times with random class selections, and the average accuracy was calculated to evaluate the performance of N-way classification across various class combinations. The results show that the fewer the number of classes to classify and the greater the number of data points used for training, the higher the classification accuracy. The best-performing configurations were 3-way 10-shot and 3-way 15-shot, achieving accuracies of 86.12% and 86.17%, respectively. These results align with the general pattern observed in classification models, where smaller class numbers and larger datasets lead to higher accuracy. However, it is important to note that the training data used in this experiment were only 15 s per class, and the accuracy difference between 10-shot and 15-shot was minimal. This highlights the data efficiency of few-shot learning. The worst-performing configuration was 10-way 1-shot, where the number of classes was high, and the amount of training data was minimal, resulting in an accuracy of 40.13%.