# Lightweight Anomaly Detection in Digit Recognition Using Federated Learning
## 7 Experimental Results and Analysis
### Evaluation of Autoencoder Performance on PC  
Before deploying autoencoder models to embedded devices, we evaluated seven architectures on a PC to understand how model complexity and batch size influence both training efficiency and AD performance. All the models were trained on the MNIST \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & **Architecture** & **Inference Memory (KB)** & **Training Memory (KB)** \\ \hline Model 1 & [196; 16; 196] & 25.33 & 103.73 \\ Model 2 & [196; 32; 196] & 49.89 & 202.11 \\ Model 3 & [196; 64; 196] & 99.02 & 398.86 \\ Model 4 & [196; 32; 32; 196] & 54.02 & 218.86 \\ Model 5 & [196; 32; 8; 32; 196] & 52.05 & 211.05 \\ Model 6 & [196; 64; 16; 64; 196] & 107.33 & 432.73 \\ Model 7 & [196; 64; 32; 32; 64; 196] & 119.52 & 481.86 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of model size and total training memory usage on the ESP32 platform. using TensorFlow, with three different batch sizes: 1, 10, and 50. The goal was to explore whether larger models offer significantly better anomaly detection quality and whether such improvements justify the associated resource overhead. The primary performance metric was the F1 score, while training loss and time were tracked as secondary indicators. As shown in Table 3, we report the number of epochs to convergence (based on a dual-phase early stopping criterion), final training loss, total training time (in seconds), inference time (in milliseconds), AD rates for all the evaluation sets, and F1 scores, which were used to select the best-performing batch size for each model. We observed that some larger models, particularly when trained with higher batch sizes, failed to converge, triggering early stopping due to a lack of improvement in training loss. Models with a final training loss above the defined acceptable loss threshold of 0.02 are considered not converged and can be identified in the results table by their loss values. This indicates that greater model complexity or larger batch sizes do not necessarily lead to better performance and may instead inhibit effective learning. On the other hand, smaller batch sizes (especially batch size 1) led to longer training times due to reduced computational efficiency, while larger batches benefited from TensorFlow's internal optimizations. To assess each model's effectiveness in AD, we evaluated their performance using the MNIST test set (in-distribution), four MNIST-C variants (fog, motion blur, dotted line, and spatter), and the EMNIST dataset (out-of-distribution). In our setup, images from the MNIST dataset were treated as normal data, while samples from the EMNIST and MNIST-C datasets represented anomalies. The goal was to optimize the model to minimize both types of misclassification: false positives (incorrectly flagging MNIST digits as anomalies) and false negatives (failing to detect EMNIST and MNIST-C as anomalies). Ideally, the model should identify anything that is not a digit as anomalous, making the F1 score a suitable metric. Since distinguishing digits (MNIST) from letters (EMNIST) is the most relevant and challenging task (due to the structural differences between letters and digits), we focused on optimizing the F1 score specifically for EMNIST. Table 3 highlights the best F1 score for each model across all the batch sizes. These configurations were selected for deployment and testing on the embedded platform, as described in the next subsection. Figure 2 further illustrates AE behavior, showing the distribution of AD rates across datasets for the first model trained with various batch sizes. An ideal behavior is characterized by low detection rates for MNIST (test set) and high for EMNIST and MNIST-C. As the figure shows, increasing the batch size leads to higher rates across all the datasets, indicating a shift toward higher recall at the cost of precision. This trade-off reinforces our use of the F1 score to select the most balanced configuration.