## 4 Evaluation
### Evaluation Method  
To evaluate the machine learning algorithms and features used for knee joint angle estimation in the proposed system, the accuracy of knee flexion and knee valgus angle estimations was verified through cross-validation using the stretch sensor values from  
Figure 6: Marker placement for estimating knee joint angles.  
the supporter device worn on each knee. Cross-validation is a method for assessing the generalization performance of a predictive model by partitioning the sample data, using one part as the validation data and the rest as the training data. In this evaluation, within-participant cross-validation and between-participant cross-validation were used as described below:  
* **Within-participant Cross-Validation** Three-fold cross-validation using data from the same participant.
* **Between-participant Cross-Validation** Validation using all trial data from one participant as the test data and all trial data from the remaining participants as the training data.  
The user's burden varies depending on the data used for training. If the accuracy is high with within-participant trial data, the user needs to collect their own training data beforehand, which increases the user's burden but allows high-accuracy knee joint angle estimation. Conversely, if the accuracy is high with training data from different users, the system can be used without the need for the user's own ground truth data, reducing the user's burden.  
The proposed system aimed to evaluate ACL injury risk during actual exercises without requiring special equipment. Thus, it is ideal to estimate knee joint angles from other users' training data without collecting the user's own ground truth data, making good between-participant cross-validation results desirable.  
A Random Forest Regressor (RFR) [35] and Linear SVR (LSVR) [36] were used as the machine learning algorithms. These were selected using Auto ML in a preliminary survey using collected data. The features extracted from the three stretch sensors on the supporter device for each leg were the following:  
* Instantaneous values: a single sample of data output from the sensor (total of three features)
* Instantaneous values and past 20 samples of instantaneous values obtained at 50 Hz (total of 63 features)  
These features were normalized for each leg based on the stretch sensor values during flexion, using min-max normalization. Min-max normalization is represented by the following Equation (5):  
\[X^{\prime}=\frac{X-X_{\text{min}}}{X_{\text{max}}-X_{\text{min}}} \tag{5}\]  
where \(X\) is the original value, \(X_{\text{min}}\) is the minimum value of the feature, and \(X_{\text{max}}\) is the maximum value.  
In between-participant cross-validation, the estimated knee joint angles were calibrated using Equation (4) to reduce differences in body size and device attachment among participants. The slope \(a_{2}\) and intercept \(b_{2}\) were calculated from the flexion data of the test participant, while the slope \(a_{1}\) and intercept \(b_{1}\) were obtained from the scatter plots of the flexion data of the remaining participants. Calibration was not performed in within-participant cross-validation because the supporter device was not removed during the experiments.  
For the evaluation of the estimation results, the features and machine learning algorithms used in the proposed method were determined based on the results of cross-validation. The evaluation metrics were the mean absolute error (MAE) and root mean squared error (RMSE) between the estimated knee joint angles and the actual angles obtained by the 3D motion analysis system. ACL injury risk is higher when the knee valgus angle is large at a small knee flexion angle. Therefore, it is crucial to detect high-risk athletes, even if it means occasionally misclassifying low-risk athletes. Consequently, when determining the machine learning algorithms based on the estimation results, it is necessary to ensure that the knee flexion angle is not overestimated and the knee valgus angle is not underestimated.