## 2 Materials and Methods
### Prediction Models Workflow
#### 2.2.5 HMM Design and Training
\begin{table}
\begin{tabular}{c c c} \hline
**Symbol** & **Definition** & **Values** \\ \hline \multirow{2}{*}{N} & Number of hidden states in & \multirow{2}{*}{\{\(\{\textit{Engaged, Struggling,Waiting\}\)} \\  & the model.} & \\ \hline \multirow{4}{*}{M} & Number of distinct & \multirow{2}{*}{\(\{\textit{Obs}_{1},\textit{ Obs}_{2},\textit{ Obs}_{3},\ldots,\textit{ Obs}_{M}\}\)} \\  & observations. & & \\ \cline{3-3} \cline{3-3} \multirow{4}{*}{A} & State transition probability & \multirow{2}{*}{\(\{\textit{Matrix size is }\ N\ \times\ N\), in our case \(3\ \times\ 3\). The values of the matrix are generated from} \\  & distribution—Probability & & training the model. \\  & matrix of transition from one & & \\  & state to another. & & \(\begin{bmatrix}a_{11}&a_{12}&a_{13}\\ a_{21}&a_{22}&a_{23}\\ a_{31}&a_{32}&a_{33}\end{bmatrix}\) \\ \hline \multirow{4}{*}{B} & Emission probability & \multirow{2}{*}{\(\{\textit{Matrix size is }\ N\ \times\ M\). The values of the matrix are generated from training the model.} \\  & matrix of observing a & & \\ \cline{3-3} \multirow{4}{*}{} & particular observation in the & \multirow{2}{*}{\(\left[\begin{matrix}b_{11}&\cdots&b_{1M}\\ \vdots&\ddots&\vdots\\ b_{31}&\cdots&b_{3M}\end{matrix}\right]\)} \\  & current state. & & & \\ \cline{3-3} \multirow{4}{*}{\(\pi\)} & Initial state probability & \multirow{2}{*}{\(\{\textit{Initial state probability matrix, usually equally distributed.}\)} \\  & distribution. & & \\ \hline \end{tabular}
\end{table} 
Table 4: Definition of HMM elements. In the \(k\)-fold cross-validation, due to the imbalance in the labeled collaborative behaviors, we needed to split the data points to address the imbalance by including all possible observations and states in each training instances of the \(k\)-fold cross-validation. Since the sequence of the observation datapoints is important in generating the transition and emission matrices, we could not use random selection of the datapoints or stratification of the datapoints. As such, we opted to treat the data continuously by splitting the data into certain percentages instead of fixed datapoints for each fold. We found that by splitting the fold into 80% for training and 20% for validation, we were able to generate sets with all observations and behaviors included in each split. The validation set was then shifted upwards in each instance of the \(k\)-fold until every datapoint from the training set is used for validation in the same sequence without shuffling them into random positions. This is illustrated in the top portion of Figure 8. As part of the splitting of the data, we would label each datapoint to either training or validation to ensure that the same data point would not end up in both training and validation. The function hmmestimate used the maximum likelihood estimate to generate the state transition and emission matrices based on the binary observation sequences and hand-labeled hidden states. The matrices were then optimized using the function hmmtrain, where a Baum-Welch Algorithm was used to improve the probabilities. With the optimized matrices, we validated the model with the 20% remaining datapoints using the function hmmviterbi. This function used the Viterbi Algorithm to predict the most likely collaborative behavior based on the sequence of observations and probability matrices. The  
Figure 7: HMM diagram of all elements. predicted collaborative behaviors were compared to the actual hand-labeled behavior to find the accuracies of each fold and the average accuracies for all \(k\) values from 5 through 10.