# FedQAS: Privacy-Aware Machine Reading Comprehension with Federated Learning
## 4 Experiment and Results
### Implementation and Demo Environment  
Designing and developing question answering in a privacy-aware manner is not a trivial task. It requires design strategies to comply with data governance and privacy regulations. Several third-party frameworks have been proposed for federated learning; providing open-source building blocks that help to collaborate in training machine learning models. The present QAS application framework needs to provide scalability, large models training, and production-grade features such as robustness to failure. Based on these requirements, we chose to design and develop our proposed FedQAS system on top of private data using the FEDn framework [2]. FEDn is an open-source, modular, and model agnostic framework for federated machine learning. We developed interactive and user-friendly interfaces using the Flask framework ([[https://flask.palletsprojects.com](https://flask.palletsprojects.com)]([https://flask.palletsprojects.com](https://flask.palletsprojects.com)), accessed on 24 January 2022), which make it easy for a third party to contribute to data annotation and then participate in training global models directly from their location site. The proposed FedQAS has the following features:  
* Privacy-preserving: sharing only model parameters with a central server (cloud) and keeping data private on the client side,
* Incremental learning: improving the global models by attaching more clients and adding new data points,
* Robust: robust enough to deal with natural language tasks (e.g., question answering, chatbot, etc.) and large models in a geographically distributed manner, \begin{table}
\begin{tabular}{l} \hline \hline
**Question 2:** What program was created to carry out these projects and missions? \\ \hline
**Gold answer (human):** Apollo program \\
**Baseline model answer:** National Aeronautics and Space Administration \\
**FedQAS answer:** Apollo program \\ \hline
**Question 3:** What year did the first manned Apollo flight occur? \\ \hline
**Gold answer (human):** 1968 \\
**Baseline model answer:** 1968 \\
**FedQAS answer:** 1968 \\
**Question 4:** What President is credited with the original notion of putting Americans in space? \\ \hline
**Gold answer (human):** [PERSON] \\
**Baseline model answer:** [PERSON] \\
**FedQAS answer:** [PERSON] \\ \hline
**Question 5:** Who did the U.S. collaborate with on an Earth orbit mission in 1975? \\ \hline
**Gold answer (human):** Soviet Union \\
**Baseline model answer:** Soviet Union \\
**FedQAS answer:** Soviet Union \\ \hline
**Question 6:** How long did Project Apollo run? \\ \hline
**Gold answer (human):** 1962 to 1966 \\
**Baseline model answer:** 1961 to 1972, and was supported by the two man Gemini program which ran 1966 \\
**FedQAS answer:** 1962 to 1966 \\ \hline
**Question 7:** What program helped develop space travel techniques that Project Apollo used? \\ \hline
**Gold answer (human):** Gemini \\
**Baseline model answer:** Gemini \\
**FedQAS answer:** Gemini \\ \hline
**Question 8:** What space station supported three manned missions in 1973-1974? \\ \hline
**Gold answer (human):** Skylab \\
**Baseline model answer:** Skylab \\
**FedQAS answer:** Skylab \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Cont. * Multilingual: language agnostic, can be trained on any language,
* Standalone: multiple platforms (i.e., guarantee for low disk and memory footprint). It can be run production-grade on a standard laptop having two cores and 2 GB of RAM,
* Accuracy and F1 score: achieve competitive performance compared with centralized training and the used baseline model (see experiment and evaluation section). The proposed FedQAS is composed of three main components: FEDn for collaborative training, MongoDB ([[https://www.mongodb.com](https://www.mongodb.com)]([https://www.mongodb.com](https://www.mongodb.com)), accessed on 24 January 2022) as a NoSQL [38] database and question answering UI for prediction and local incremental learning.