# Neural Network Based Approach to Recognition of Meteor Tracks in the Mini-EUSO Telescope Data
## 4 Results  
An important question to discuss before presenting ANNs is how to evaluate their performance. It is usually advised to use balanced datasets whenever possible, both for training and testing. In this case, the Receiver Operating Characteristic Area Under the Curve (ROC AUC) is one of the popular performance metrics [29]. Recall that the ROC curve is a plot of the true positive rate against the false positive rate at various threshold settings. Given one randomly selected positive instance and one randomly selected negative instance, AUC is the probability that the classifier will be able to tell which one is which. Due to its definition, ROC AUC does not depend on the sample size.  
However, the number of meteor signals in the Mini-EUSO data is negligibly small in comparison with the number of non-meteor signals, so that using balanced datasets for testing would provide unrealistic results, while using them for training would not represent the full diversity of non-meteor signals thus resulting in lower performance and numerous false positives during tests. Thus, we unavoidably arrive at the necessity to use strongly imbalanced datasets both for training and testing. It is argued in the literature that ROC AUC does not act as a fully adequate performance metric in this case, and other metrics should be used instead, see, e.g., [30; 31; 32]. In what follows, we provide results obtained in terms of three more metrics besides ROC AUC. These are the Precision-Recall (PR) AUC, the Matthews correlation coefficient (MCC), and the \(F_{1}\) score. One more metric will be introduced below.  
Recall that the PR AUC equals an area under the plot of precision vs. recall with  
\[\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}\,,\qquad\text{Recall}= \text{TP}=\frac{\text{TP}}{\text{TP}+\text{FN}}\,,\]  
where TP, FP, and FN are the number of true positives, false positives, and false negatives as classified by the model, respectively. The Matthews correlation coefficient can be calculated from the confusion matrix as  
\[\text{MCC}=\frac{\text{TP}\cdot\text{TN}-\text{FP}\cdot\text{FN}}{\sqrt{( \text{TP}+\text{FP})(\text{TP}+\text{FN})(\text{TN}+\text{FP})(\text{TN}+ \text{FN})}}\,,\]  
where TN is the number of true negatives. Finally, the \(F_{1}\) score is the harmonic mean of the precision and recall. It can be presented as  
\[F_{1}=\frac{2\text{TP}}{2\text{TP}+\text{FP}+\text{FN}}.\]  
Notice that these metrics will be applied to three-dimensional data chunks that partially overlap due to the employment of a sliding window for preparing input datasets. This can lead to a situation when a part of chunks containing a meteor signal are classified as non-meteor ones while others are classified properly, so that the value of a performance metric might be misleading. Since we are interested in maximizing the number of recognized meteors but not meteor chunks, it can be beneficial to also introduce a metric expressed in terms of the original 1068 meteors. Probably the most straightforward way is to use \(1-\text{FN}\) (met), where FNR (met) is the false negative rate of meteor signals represented as the number of meteors lost by the classifier divided by the total number of meteors in a session used for testing.  
All these metrics are equal to 1 for a perfect model. The MCC equals \(-1\) for the worst possible model; other metrics give 0 in this case.