# An Ontology-Driven Learning Assessment Using the Script Concordance Test
## 2 Methods and Materials
### Proposed Framework  
The architecture of the proposed ontology-driven automated script concordance test generation framework is presented in Figure 3. The framework relies on ontology mapping and code generation algorithms that leverage semantic annotations based on ontologies. First, the mapping between the medical records database and question ontology was performed. In the data preparation phase, a query that retrieves only information relevant to the procedure of SCT question generation was executed (Listing 2). Hypotheses corresponding to diagnostic possibility were selected, as well as information part of the question that corresponds to laboratory results descriptions from the health records database. The average execution time of this query was around 1 s using the data.world [45] online service. Finally, the query results were downloaded from the data.world cloud data catalog and stored in.CSV format on our server. The former query returned 833 results that were then parsed and semantically annotated with respect to mapping given in Table 2.  
However, due to the way the SCT questions were constructed [6], it was not possible to retrieve Likert-scale response scores directly from the database. Generating Likert scale scores for answers to SCT questions is a knowledge-intensive task that is traditionally performed by domain experts. In this paper, two possible strategies to cover this aspect are proposed: (1) Direct strategy is based on a direct selection of healthcare expert-approved lab results for a given hypothesis. The number of cases for each of the results is summarized, and Likert scores are assigned with respect to the number of experts that agree on the hypothesis used for obtaining these results. This type of question is simpler for code generation but is considered quite difficult for students, due to fact that precise knowledge is needed to select the most appropriate answer. (2) Indirect strategy, on the other hand, selects the lab results for a disease that is closely or distantly related to the one provided by an evaluator. The difficulty level is considered higher when the new information is derived from lab results of a closely related disease (diabetes type 1 and type 2, for example), while it is considered easier when the diseases are not much related (e.g., infraction and diabetes type 1). For this \begin{table}
\begin{tabular}{c c} \hline \hline
**Medical Record Database Column** & **Ontology Class** \\ \hline encounter.soap\_note & CaseDescription \\ \hline encounter\_dx.description & Hypothesis \\ \hline lab\_results.result\_description & NewInformation \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Mapping between medical records database and SCT ontology. Figure 3: Ontology-driven framework for automated SCT question generation: 1, 2—mapping of patient health records DB to SCT question ontology; 3—question topic and difficulty settings; 4—SPARQL queries; 5—query results; 6—code generation results; 7—rendering mobile app starting from Google Sheets data; 8—quizzes within mobile app; 9—answering the questions; 10—question responses; 11—student assessment results. purpose, a domain ontology about the hierarchy and relations between diseases, such as Disease Ontology (DO) [46], should be adopted to provide the necessary knowledge. In the direct strategy, processing and calculation of the extracted data are performed for each hypothesis that is identified among the results. First, all of the possible result descriptions (NewInformation) for a given disease description (Hypothesis) are identified. After that, the number of medical records that exist for each of the possible result descriptions for the given disease is determined by simple counting. The probability of each particular result is then determined by dividing the number of records by the total number of cases for that hypothesis.