# An Ontology-Driven Learning Assessment Using the Script Concordance Test
## 2 Methods and Materials
### Proposed Framework
In order to adapt it to the Likert scale, the resulting probabilities are classified into five ranges of 0.0-0.2, 0.2-0.4, 0.4-0.6, 0.6-0.8, and 0.8-1.0, where each range corresponds to one of the scores of \(-2\), \(-1\), 0, 1, and 2, accordingly. Finally, the Likert-scale response score is calculated based on the probability in the following way. If the probability is between 80 and 100%, answer 2 is assigned a maximum score, while the other options are assigned lesser scores:--1 will be assigned a score of 4/5, 0 will be assigned a score of 3/5, \(-1\) is assigned a score of 2/5, and \(-2\) is assigned a score of 1/5. If the probability is between 60% and 80%, then 1 is assigned the maximum score, 2 is assigned a score of 4/5, 0 is assigned a score of 3/5, etc. The pseudo-code of the Likert-scale-score assignment algorithm for the direct strategy is given in Listing 3. ```
Input: health_records_db, disease_hypothesis Output: Likert scale scores Steps:
```
1. Result_descriptions: = GetAllResultDescriptionsForGivenHypothesis(disease_hypothesis, health_records_db);
2. total_records_num: = CountRecords(disease_hypothesis);
3. For each description in result_descriptions
4. new_information_records_num: = CountRecords(disease_hypothesis, description);
5. probability: = new_information_records_num/total_records_num;
6. If(probability > 0.8)
7. LikertPlus2: = probability
8. LikertPlus1: = probability * 0.8;
9. Likert0: = probability * 0.6;
10. LikertMinus1: = probability * 0.4;
11. LikertMinus2: = probability * 0.2;
12. else if(probability > 0.6)
13. LikertPlus2: = probability * 0.8;
14. LikertPlus1: = probability;
15. Likert0: = probability * 0.6;
16. LikertMinus1: = probability * 0.4;
17. LikertMinus2: = probability * 0.2;
18....
19. Endif;
20. Endfor;
21. End. ```
**Listing 3**Pseudo-code of Likert-sale-score assignment algorithm for direct strategy. In the indirect strategy, the user first defines desired difficulty level that is used to select another disease hypothesis. If a \"hard\" difficulty level is selected, then the candidate hypothesis is taken from the same disease category. Otherwise, the candidate hypothesis is taken from a disease class at a higher semantic distance. In both cases, the Likert scale \(-2\) gives the full score value, while the other answers are multiplied by 0.75, 0.50, and 0.25, accordingly. However, for the Likert scale answer +2, the obtained score is 0, as it is considered an entirely wrong answer in that case. The full score value is determined with respect to the number of diseases that belong to the same class. With the higher number of diseases belonging to the selected class, the question is considered more difficult. The impact of this number is corrected by a factor of 0.5, to avoid zero scores in the case of 1 class. The previous score calculation criteria are implemented by the following equation:  
\[FullScore:=1-0.5*~{}1/NumberOf DiseasesFromSameClass(HypothesysDisease) \tag{1}\]  
Notably, _FullScore_ increases as the number of classes increases while decreasing if the number of classes from the same disease category decreases. In the case of \"easy\" questions, the full score is further corrected by a multiplicative factor of 0.5. The pseudo-code of the Likert-scale-score assignment algorithm for the indirect strategy is given in Listing 4. ```
Input: health_records_db, disease_hypothesis, difficulty level Output: Likert scale scores Steps:
```
1. If(difficulty level is hard)
2. new_hypothesis: = SelectRelatedDiseaseFromSameClass(disease_hypothesis);
3. full_score: = (\(1-1/\text{NumberOfDiseasesFromSameClass(new_hypothesis)}\));
4. Else
5. new_hypothesis: = SelectedDiseaseFromAnotherClass(disease_hypothesis);
6. full_score: = 0.5* (\(1-1/\text{NumberOfDiseasesFromSameClass(new_hypothesis)}\));
7. Endif;
8. LikertPlus2: = 0;
9. LikertPlus1: = full_score* 0.25;
10. Likert0: = full_score* 0.5;
11. LikertMinus1: = full_score* 0.75;
12. LikertMinus2: = full_score;
13. End. ```

**Listing 4** Pseudo-code of Likert-scale-score assignment algorithm for indirect strategy.