## 3 Results and Discussion
### Prediction Models Evaluation Results  
The evaluation was performed for both the rule-based prediction model and the HMM prediction model using the data from the hold-out test set. In the rule-based prediction model, the collaborative behaviors were predicted by evaluating the observations using the defined rules in Section 2.2.4.  
As for the HMM prediction model, we chose the HMM that was generated with the highest accuracy of 97.77%. The collaborative behaviors were predicted in MATLAB using hmmviterbi by using the Viterbi Algorithm to generate the most likely collaborative behavior based on the sequence of observations.  
In both cases, the predicted collaborative behaviors were compared against the hand-labeled collaborative behaviors. Table 5 compares the performance of rule-based prediction model and HMM prediction model and Figure 10 illustrates the confusion matrix of both prediction models.  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **Rule-Based (\%)** & **HMM (\%)** \\ \hline Accuracy & 76.53 & 90.58 \\ Precision & 71.81 & 89.55 \\ Recall & 68.93 & 87.94 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluation results for rule-based and HMM prediction models.  
Figure 9: HMM accuracies for different \(k\)-fold values.  
Overall, the HMM provided higher accuracy, precision, and recall of the participants' collaborative behavior compared to a rule-based model. When we look at the behaviors as shown in Figure 10, both models performed the best for the _Engaged_ behavior since the conditions for the _Engaged_ behavior were quite simple and straightforward where both models could provide a reliable prediction. However, for _Waiting_ and _Struggling_ behaviors, the rule-based model performed quite poorly where the model predicted most of the _Struggling_ behavior as _Waiting_. The inflexibility of the rule-based model could have caused this. Rule-based models only allow one behavior for one set of conditions, whereas real hand-labeled data would have instances where the same condition produced different outcomes based on the context of the task (or previous sequence of events). For such cases, if we keep the rule-based model to predict participants' collaborative behavior, the feedback that the participants were to receive would not be true to their actual behavior. A participant that is _Struggling_ would not be prompted to seek assistance as the system would assume they are _Waiting_ for their partner to complete a turn. On the other hand, the HMM prediction results for _Waiting_ and _Struggling_ were reliable since the temporal information that was learned from the training was embedded within the state transition and emission probability matrices. This is consistent with the results reported by another study that implemented a semi-supervised model using the same dataset as this study [67]. The study compared the performance of the developed semi-supervised automated labeling of behaviors to supervised and unsupervised models. In this study, a fully supervised support vector machine (SVM) achieved 86.1% accuracy, while a semi-supervised self-training model with 2.5% of the data labelled achieved 84.5% accuracy. Based on this observation, the deterministic nature of HMMs would fit better in a dynamic interaction as it offers more flexibility than a rule-based model and traditional machine learning algorithms, without requiring a large amount of labeled data for training.