2 increasing number of antiretroviral drug-resistance-asso- ciated mutations has increased the diﬃculty of the interpre- tation of those assays [5]. In both cases many HIV-1 drug resistance analysis ap- proaches have been explored, from simple linear models [6] to more sophisticated ones, such as database pattern search method [7], neural networks/machine learning [8–11], or genotype-phenotype mapping [12]. Such methods, or learn- ers, diﬀer by the mechanism used to search over the space of parameters. It appears that diﬀerent interpretation sys- tems lead to distinct results [13–15]. Current widely used genotypic interpretation systems may have no satisfactory performance on newly derived datasets. Such poor perfor- mances emphasize the need for an external validation dataset or a suﬃcient large database to create a validation set. It has been shown that the variability observed in diﬀerent rule- based algorithms was mainly due to the patients’ baseline characteristics than to the statistical methods used [16, 17]. A framework for the uniﬁed loss-based estimation sug- gested a solution to this problem in the form of a new esti- mator, called the “Super Learner” [18, 19]. Initially this methodology, called Discrete Super Learner, compared dif- ferent learners (methods) on the basis of the loss-based esti- mation theory and choose the optimal learner for a given prediction problem based on cross-validated risk (repartition between training sample and validation sample) [20]. The Super Learner methodology has been improved building now an estimator based on a linear combination of the diﬀer- ent learners investigated [19, 21, 22]. Originally, the Super Learner used both mean square of residuals (diﬀerences between observed and predicted out- comes) and R2 for evaluation and assessment. However, sta- tistical investigations showed the importance of exploring diﬀerent loss functions [23], such as ﬁrst-order coeﬃcient R. Our aim is to study the performance of the discrete and the most recent Super Learner methodology on a small sam- ple of HIV-1 data from a randomized clinical trial. Especially, based on this methodology, we investigate four diﬀerent cross-validation setting, and the use of two loss func- tions for six statistical learning methods. This methodology is applied on the Jaguar trial data [24]. 2. Methods 2.1. Datasets. For a patient i, the data consist of a vector Xi of binary variables indicating presence or absence of a mutation and Yi denotes the virologic outcome. In the regression set- ting, the objective is to predict Y using X. Then, the param- eter of interest is denoted as E(Y |X). We analyzed the data obtained from the Jaguar trial which are described elsewhere [24]. Brieﬂy the Jaguar trial was a randomized multicenter, double-blind placebo-controlled trial evaluating the eﬃcacy of adding didanosine (ddI) to an on-going antiretroviral (ARV) regimen. Patients were randomly assigned at a ratio 2 : 1 to receive ddI or a matching placebo added to their current regimen. The primary eﬃcacy end point was the magnitude of change in plasma HIV-1 RNA levels in log10 copies per mL from baseline to week 4. The na¨ıve method was used to compute viral load reduction; that is, all HIV-1 RNA AIDS Research and Treatment levels <50 copies/mL at week four were ﬁxed at 50 copies/mL. Although censored methods are preferred to compute HIV-1 RNA changes, the low percentage (11%) of patients censored provides in this case an unbiased estimate [25–27]. The med- ian changes in HIV-1 RNA at week 4 were −0.56log10 cop- ies/mL (IQR, −0.14 to −1.2) and +0.07log10 copies/mL (IQR, 0.12 to 0.21) in patients receiving ddI and placebo, res- pectively (P < .0001). HIV-1 sequences were available for all patients, but only patients in the ddI group were used in the present work. HIV-1 sequences and HIV-1 RNA reduction at week 4 were available for 102 patients. Mutations were deﬁned as amino acid diﬀerences from subtype B consensus wild-type sequence (wild-type virus HXB2). We investigate the virologic impact at week 4 of ten resistance mutations: M41L (prevalence 48%), D67N (34.3%), T69D (8.8%), K70R (26.5%), L74V (8.8%), V118I (18.6%), M184VI (92.2%), L210W (27.5%), T215Y/F (53.9%), and K219Q/E (24.5%). This set has been the starting point for building ANRS ddI rules and was potentially linked to the ddI resis- tance at the time of the study. Moreover, the choice of using a subset of mutations is driven by Soo Yon Rhee et al. study [28], in which they show that expert mutation selection is preferable than using the entire sequences. 2.2. Super Learner. The methodology has been proposed by Mark van der Laan et al. [18, 19] as a setting to choose the optimal learner (method) among a set of candidate learners, this version of the methodology was called the Discrete Super Learner. Recently, the methodology has been reﬁned and proposed a new estimator based on a weighted linear combination of candidate learners to build a Super Learner estimator [19, 21, 22]. We brieﬂy introduced the general principle and few key features of this methodology. The general strategy for loss-based estimation is driven by the choice of a loss function and relies on cross-validation for estimator selection and performance assessment. Cross- validation divides the available dataset into k mutually exclu- sive and exhaustive sets of as nearly equal size as possible. Each set and its complement play the role of the validation and training samples. Observations in the training set are used to construct (or train) the estimators, and observations in the validation set are used to assess the performance (or validate) of the estimators. For each estimator/learner the k risks over the k validation sets are averaged resulting in the so-called cross-validated risk. For example, with a 10- fold cross-validation the learning set is partitioned into 10 parts, each part in turn served as a validation set, while the other 9/10ths of the data served as the training set. Based on cross-validated risks, estimators/learners can be ranked from those identiﬁed as top learners to those providing poor performance. In the discrete version of the methodology, the optimal learner is applied to the entire dataset. In the most recent version, a new estimator (the Super Learner) is proposed based on a family of weighted combinations of the estimators/learners. The new Super Learner appears as a generalization of the discrete Super Learner. We applied all individual learners and the new estimator on full dataset (which will be called full model in the following). Learners are ranked from those identiﬁed as top