# A Technological Framework to Support Asthma Patient Adherence Using Pictograms
## 3 Results
### First Evaluation of Pictogram Comprehension by External Expert Judgement  
During this phase, the comprehension of pictograms by users was assessed with input from two external experts, both kinesisologists. These experts evaluated the responses of 50 users (24 from the Valparaiso region and 26 from the Biobio region) for each of the 18 pictograms, reporting their respective levels of understanding. Following the evaluation of all pictograms by these two experts, the inter-rater agreement coefficient was determined using the Kappa index. The resulting findings are presented below, in Table 16. This evaluation took place from 8 to 15 November 2021. After both experts assessed all the pictograms, we calculated the inter-evaluator agreement coefficient using the Kappa index. We determined an optimal level of comprehension as achieved when at least 67% of the participants demonstrated a clear understanding of the pictogram. This was supported by a substantial inter-judge agreement exceeding 0.60, indicating a'moderate' level according to the Kappa index. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Pictogram**} & \multirow{2}{*}{\begin{tabular}{c} **\% Comprehension** \\ **hension** \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} **Agreement** \\ **YES (N = 50)** \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} **\#AGreement** \\ **NO (N = 50)** \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} **Disagreement** \\ **(N = 50)** \\ \end{tabular} } & \multirow{2}{*}{\
\begin{tabular}{c} **Kappa** \\ **(N = 50)** \\ \end{tabular} } & \multirow{2}{*}{
\begin{tabular}{c} **\#Strength of** \\ **(N = 50)** \\ \end{tabular} } & \multirow{2}{*}{**Observations**} \\ \hline
1 & 84 & 42 & 7 & 1 & 0.92 & Almost Perfect & Accepted \\
2 & 62 & 31 & 8 & 11 & 0.45 & Moderate & Third Expert \\
3 & 52 & 26 & 21 & 3 & 0.87 & Almost Perfect & Redesign \\
4 & 88 & 44 & 2 & 4 & 0.46 & Moderate & Accepted \\
5 & 84 & 42 & 5 & 3 & 0.74 & Substantial & Accepted \\
6 & 68 & 34 & 12 & 4 & 0.8 & Substantial & Accepted \\
7 & 78 & 39 & 8 & 3 & 0.81 & Almost Perfect & Accepted \\
8 & 86 & 43 & 4 & 3 & 0.69 & Substantial & Accepted \\
9 & 96 & 48 & 2 & 0 & 1 & Almost Perfect & Accepted \\
10 & 86 & 43 & 3 & 4 & 0.56 & Moderate & Accepted \\
11 & 84 & 42 & 4 & 4 & 0.63 & Substantial & Accepted \\
12 & 52 & 26 & 23 & 1 & 0.96 & Almost Perfect & Redesign \\ \hline \hline \end{tabular}
\end{table} 
Table 16: Evaluation of pictograms by expert judgment. \begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Indication–Pictogram**} & \multirow{2}{*}{\ \begin{tabular}{c} **Frequency of Compatibility** \\ **between Pictograms and** \\ **Medical Indications in Both** \\ **Regions (N = 32)** \\ \end{tabular}  } & \multirow{2}{*}{ \begin{tabular}{c} **Average Percentage (\%) of** \\ **Compatibility between Pictograms and** \\ **and Medical Indications in Both** \\ **and Regions (N = 32)** \\ \end{tabular}  } \\ \hline
1 & 32 & 100 \\
2 & 28 & 87.5 \\
3 & 21 & 65.3 \\
4 & 30 & 93.8 \\
5 & 30 & 93.8 \\
6 & 26 & 81.3 \\
7 & 27 & 84.4 \\
8 & 28 & 87.5 \\
9 & 30 & 93.8 \\
10 & 32 & 100 \\
11 & 26 & 81.3 \\
12 & 31 & 96.9 \\
13 & 31 & 96.9 \\
14 & 23 & 71.9 \\
15 & 28 & 87.5 \\
16 &Table 16 indicates that 11 out of the 18 pictograms successfully met these criteria. Pictograms 2 and 15 showed a low level of comprehension without strong agreement among the evaluators, leading us to refer them for evaluation by a third external judge, independent of the project team. Pictogram 13 achieved an understanding rate of 64%, approaching the project's defined threshold, and demonstrated substantial agreement among evaluators. However, as it fell just short of the desired comprehension level, we decided to subject it to evaluation by a third judge. Pictograms 4, 10, and 18 showed moderate agreement strength. Despite this, they exhibited a high degree of agreement between judges, aligning with the limitation reported by the Kappa index.