## 5 Evaluation  
To compare the proposed learning method with the SOTA learning method, the triplet loss learning method and the ArcFace learning method, a performance evaluation was conducted by using the lowest loss models of Section 4.3. This evaluation used two methods of the testing set configuration: closed-set and open-set. A closed-set is the untrained dataset of the object used for training, which means the test for the object trained at the training process, whereas an open-set is a dataset of an untrained object, which means the test for the new untrained object. The test for an open-set is more difficult than that of a closed-set and is more closely related to real-world problems. The present evaluation was carried out with the open-set because the model must be able to correctly identify new objects for the return of newly occurring lost animals. The testing set for evaluation consists of 580 images of 100 untrained dogs. The model, which trained by the proposed loss function and learning method, was evaluated in the following three methods:  
* Comparison of embedding vector distribution between the learning methods: The embedding vectors of the open set were extracted from the model trained by the base learning method, the SOTA learning method, and the proposed learning method, and their distributions were compared. The embedding vectors used values before the L2-norm layer. The distribution was compared by the mean and variance for the length of the embedding vector and the distance between the center of the object's embedding vectors and the embedding vector for each object;
* Face verification: The model discerned whether a pair of images was the same object and calculated its accuracy. Specifically, the distance (\(d\)) between the two embedding vectors extracted by the model was compared to a specific threshold (\(t\)), and when \(d<t\), the images were discerned as the same object. The accuracy of the discernment result is calculated. The receiver operating characteristic (ROC) curve and the optimum accuracy were compared between models while changing the threshold. This procedure was repeated 100 times to calculate the average of the ROC curves;
* Face identification: one image was selected for each object included in the open set to configure a set of 100 sub-training sets, and the remaining data comprised the sub-testing set. Rank 1 and rank 5 were extracted by comparing the distance between the embedding vectors of one sub-testing data and all sub-training data, their accuracy was calculated, and this procedure was repeated 1000 times for the entire sub-testing set to compare the mean, maximum, and minimum accuracy of rank 1 and rank 5.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Learning Methods** & **Loss** & **Accuracy** \\ \hline SOTA & 1.0612 & 65.00\% \\ \hline Triplet & 0.2069 & 93.33\% \\ \hline ArcFace & 0.3305 & 84.00\% \\ \hline Triplet + VL stage 1 & 0.27058 & 89.00\% \\ \hline Triplet + VL stage 2 & 0.1833 & 97.33\% \\ \hline ArcFace + VL stage 1 & 0.2809 & 84.33\% \\ \hline ArcFace + VL stage 2 & 0.2566 & 96.33\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Loss and accuracy of learning methods for the architecture described in Section 3.1. Vector length (VL) method indicates vector length loss.