# Direction-Aware Lightweight Framework for Traditional Mongolian Document Layout Analysis
## 4 Experiments
### Datasets and Evaluation Metrics
#### 4.1.3 Evaluation Metrics  
We evaluate our model using the following comprehensive metrics, each defined mathematically to ensure clarity and reproducibility:  
* Detection performance:
* Average precision (AP): Calculated at IoU thresholds of 0.5 and 0.75, as well as the mean AP across IoU thresholds from 0.5 to 0.95 with a step of 0.05 (denoted as mAP): \[\text{AP}=\int_{0}^{1}p(r)dr\] (20) where \(p(r)\) is the precision at recall level \(r\). In practice, this is approximated by \[\text{AP}=\sum_{i=1}^{n}(r_{i}-r_{i-1})p(r_{i})\] (21) where \(p(r_{i})\) is the precision at the \(i\)-th recall level.
* Category-wise AP: AP calculated separately for each of the eight layout categories: \[\text{AP}_{c}=\int_{0}^{1}p_{c}(r)dr\] (22)
where \(p_{c}(r)\) is the precision at recall level \(r\) for category \(c\).
* Classification performance:
* Precision: The ratio of correctly predicted positive observations to the total predicted positives: \[\text{Precision}=\frac{TP}{TP+FP}\] (23) where \(TP\) is the number of true positives and \(FP\) is the number of false positives.
* Recall: The ratio of correctly predicted positive observations to all actual positives: \[\text{Recall}=\frac{TP}{TP+FN}\] (24) where \(FN\) is the number of false negatives.
* F1 score: The harmonic mean of precision and recall: \[\text{F1}=2\cdot\frac{\text{Precision}\cdot\text{Recall}}{\text{Precision}+ \text{Recall}}\] (25)
* Direction prediction performance:
* Direction accuracy (DA): The percentage of detected objects with predicted direction within a threshold \(\tau\) (set to \(5^{\circ}\)) of the ground truth: \[\text{DA}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}(|\theta_{i}-\theta_{gt,i}|<\tau)\] (26) where \(\mathbf{1}(\cdot)\) is the indicator function, \(\theta_{i}\) is the predicted angle, and \(\theta_{gt,i}\) is the ground truth angle for the \(i\)-th object.
* Direction mean absolute error (DMAE): The average absolute angular difference between the predicted and ground truth directions: \[\text{DMAE}=\frac{1}{N}\sum_{i=1}^{N}|\theta_{i}-\theta_{gt,i}|\] (27)
* Category-wise DMAE: Direction error analyzed by layout category: \[\text{DMAE}_{c}=\frac{1}{N_{c}}\sum_{i\in c}|\theta_{i}-\theta_{gt,i}|\] (28) where \(N_{c}\) is the number of objects in category \(c\).
* Efficiency metrics:
* Frames per second (FPS): Inference speed measured on standard hardware: \[\text{FPS}=\frac{N_{frames}}{T_{processing}}\] (29) where \(N_{frames}\) is the number of frames processed and \(T_{processing}\) is the total processing time.
* Parameters (M): Total number of trainable parameters in the model.  
For multi-class evaluation, we report both micro-averaged and macro-averaged metrics:  
\[\text{Micro-avg}=f\left(\sum_{c=1}^{C}TP_{c},\sum_{c=1}^{C}FP_{c},\sum_{c=1}^ {C}FN_{c}\right) \tag{30}\]\[\text{Macro-avg}=\frac{1}{C}\sum_{c=1}^{C}f(TP_{c},FP_{c},FN_{c}) \tag{31}\]  
where \(f\) is the metric function (precision, recall, or F1), \(C\) is the number of classes, and \(TP_{c}\), \(FP_{c}\), and \(FN_{c}\) are the true positives, false positives, and false negatives for class \(c\), respectively. This approach accounts for class imbalance in the dataset by giving equal weight to each class in the macro-average.