4 Forest LM(1) LM(2) RandomD/S/A M41L x x x x x D67N x x x x T69D x x x x K70R x x x x L74V x x x x V118I x x x M184V/I x x x L210W x x x L215Y/F x x x K219Q/E x x x (Tmwoax-w=ay4s5)interaction terms – 31zenroon- – 0 – x x x x – egression CARTLogicR Figure 1: Selected mutations for each model on the complete Jaguar data Trial. function and a 10-fold cross-validation, LM(1) was identi- ﬁed as the top learner followed by Random Forest and CART. LM(1) slightly decreases its performance from the 1st rank on 10-fold to 3th rank on 2-fold while Random Forest becomes the second learners for the remaining k-folds. Sur- prisingly, linear model with interaction terms, LM(2), pro- vided poor performance for all k-fold. The Super Learner-5 provided at least as good performance as the top learner whatever the k-fold cross-validation. R loss function drew similar ﬁndings. Although the ranks of the diﬀerent learn- ers are relatively stable, the combination of the Super Learn- er-5 provided the best performance. Inclusion of Logic Reg as additional learner in the previous set of candi- date learners led to diﬀerent ﬁndings (Table 2). Globally Logic Reg performed poorly, and only LM(2) produced worse performance than Logic Reg. Based on the SqE as loss function, including Logic Reg in the Super Learner-6 decreased its performance compared to Super Learner-5. Based on R as loss function, the performance of the Super Learner-6 was very good. We applied all learners including Super Learner-5 and Super Learner-6 on the entire dataset (Table 3). Based on SqE, R, and R2 measure estimates, Super Learner-5 and −6 provided very good performances. The use of LM(2) on the full dataset provided a high level of prediction (R2 = 0.540) while, based on k-fold cross-validated risk, this learner was the poorest candidate. Comparing cross-validation and full model results indicate the LM(2) model was over ﬁt. Figure 1 displays the mutations retained by each learner. All mutations were retained for LM(1), LM(2), and Random Forest (not surprisingly all mutations are at least selected one time in a tree). CART selected M41L, D67N, T69D, K70R, L74V, and K219Q/E mutations. Of note the D/S/A method selected only the M41L mutation which should be balanced with its poor performance. The ﬁnal goal of interpreting genotypic resistance testing is to classify patients as “sensitive” or “resistant” to a speciﬁc drug. Figure 2 displays the rates of patients being well classiﬁed for the two threshold values investigated. For AIDS Research and Treatment 100 90 80 70 60 50 40 30 20 10 0 LM(1) LM(2) Random Forest DSA CART Super Super Leaner-5 Leaner-6 −0.5 log 10 −0.6 log 10 Figure 2: Rates of patients being well classiﬁed for threshold −0.5 and −0.6log10 for all models applied on the Jaguar trial full-dataset. both threshold values LM(2), Super Learner-5 and −6 have the highest accuracy with around 80% of patients correctly classiﬁed. CART and Random Forest provided the lowest accuracy, slightly below 70% of patients correctly classiﬁed, corresponding to a 10% diﬀerence. As expected the accuracy of Random Forest model depends on the mtry values. 5. Discussion The choice of subsequent treatment in failing patients is of major importance in the management of HIV-infected pa- tients. Genotypic and phenotypic resistance tests are impor- tant tools for choosing promising combination therapy for those patients. We investigated on a small sample a frame- work both for choosing optimal learner and building an estimator among a set of candidate through two diﬀerent loss functions and k-fold cross-validation. Based on cross-validation risk, the Super Learner estima- tor was the “best” learner though the linear model with only main terms LM(1) providing similar performance to that of Super Learner-5 and -6. The use of the SqE as loss func- tion indicated that the inclusion of Logic Reg as an additional learner decreased the performance of the Super Learner esti- mator. However, prediction results based on the full dataset as well as accuracy questioned the use of SqE as loss function, although it is known that full dataset provided diﬀerent results than those based on cross-validation strategy [34, 35]. Based on cross-validation risk, the good performance of LM(1) should be compared with the poor performance of the linear model with interaction terms LM(2). Inversely, LM(2) outperforms LM(1) in the full dataset. In our small dataset, this ﬁnding is clearly due to overﬁt of the data by the LM(2) model. A researcher ignoring the Super Learner methodology using a linear model with interaction terms would obtain a good performance on the full dataset while such a learner would have not been selected from the discrete Super Learner methodology. The choice of mtry parameter for Random Forest is a real problem. However, the common mtry used in regression setting (number of covariables divided by three) appears as a good compromise. Whatever the mtry value is, all mutations were selected at least on time using Random Forest on full