# FedQAS: Privacy-Aware Machine Reading Comprehension with Federated Learning
## 4 Experiment and Results
### Framework Evaluation  
For the evaluation, we used exact match (EM) and F1 score, the main metrics commonly used for question answering systems. These metrics are computed on individual (question, answer) pairs. In case of multiple correct answers for a given question, the maximum score over all possible correct answers is computed. In the EM metric, for each pair (question, answer), if the characters of the model's prediction exactly match the characters of (one of) the True Answer(s), EM = 1, otherwise EM = 0. The Accuracy represents the percentage of the questions that an MRC system accurately answers. Each question corresponds to one correct answer. For the span prediction task, the accuracy is the same as Exact Match and can be computed by the Formula (1) as follows:  
\[Accuracy=EM=\frac{Number\ of\ correct\ answers}{Number\ of\ questions} \tag{1}\]  
The precision represents the percentage of token overlap between the tokens in the correct answer and the tokens in the predicted answer, while the recall is the percentage of tokens in a correct answer that have been correctly predicted in a question. The True Positive (TP) denotes the same tokens between the predicted answer and the correct answer, the False Positive (FP) denotes the tokens which are not in the correct answer but the predicted answer, while the False Negative (FN) presents the tokens that are not in the predicted answer but the correct answer. Precision and Recall can be computed by the Formulas (2) and (3) as follows:  
\[Precision=\frac{N(TP)}{N(TP)+N(FP)} \tag{2}\] \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Rounds** & **Total Number of Clients** & **Update Size** & **Total Number of Parameters** \\ \hline
5 & 5 & 400 MB & 109.483.776 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Federated training configuration. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Hyper-Parameter** & **Range** & **Value** \\ \hline Epochs & [1; 2; 3] & 1 \\ Batch size & [8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28] & 8 \\ Learning rate & [0.001–0.004] & \(5\times 10^{-5}\) \\ Optim. method & [PERSON], SGD, RMSProp & Adam \\ \(MAX\_SEQ\_LENGTH\) & [1–1000] & 384 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Experimental model parameters. \[Recall=\frac{N(TP)}{N(TP)+N(FN)} \tag{3}\]  
The F1 score is a measure of a test's accuracy. It is the weighted average between precision and recall. The formula for this score is given in (4). In our case, it's computed over the individual words in the prediction against those in the True Answer. The number of shared words between the prediction and the truth is the basis of the F1 score. \[F1\ score=2\times\frac{(Precision\times Recall)}{(Precision+Recall)} \tag{4}\]  
To demonstrate the benefit of FedQAS, we partitioned the SQuAD dataset into 5 equal chunks, so that each client has \"20%\" of the total dataset. We then compare the federated scenario to centralized model training. Table 4 lists the available metrics for different training rounds of the global model. Our implemented model baselines show similar EM and F1 scores with the global model during the first rounds and slightly outperform the baseline with respect to data privacy and knowledge sharing across participants. Overall, the result shows that question-answering in federated learning settings performs well compared to centralized settings. This is due to the used hyper-parameters in the federated learning setting, see Figure 5 for the convergence of accuracy (EM) and Figure 6 for the convergence of F1. In terms of resources, the result proves the fact that model architecture affects client training time and combiner round time. Hence, training a large model (400 MB) in a centralized way requires more resources than the federated setting. For demonstration, we consider a FEDn network consisting of a single, high-powered combiner (8 VCPU, 32 GB RAM) with connected clients (8 VCPU, 32 GB RAM) instances in SSC (SNIC Science Cloud [37]) and measure the average round time over five global rounds.