dataesr/scientific-paragraphs-categorization huggingface dataset:
com/xdevroey/jhipster-dataset. The remainder of this article is organised as follows: Section 2 provides background information on sampling techniques and motivates the case; Section 3 presents the JHipster case study, the research questions, and methodology applied in this article; Section 4 presents the human and computational cost of testing all JHipster configurations; Section 5 presents the faults and failures found during JHipster testing; Section 6 makes a ground truth comparison of the sampling strategies; Section 7 positions our approach with respect to studies comparing sampling strategies on other configurable systems; Section 8 gives the practitioners point of view on JHipster testing by presenting the results of our interview with JHipster developers; Section 9 discusses the threats to validity; and Section 10 wraps up with conclusions.
Background and Related Work Configurable systems have long been studied by the Software Product Line (SPL) engineering community  [6, 65]. They use a tree-like structure, called feature model  [40] , to represent the set of valid combinations of options: i. e. , the variants (also called products). Each option (or features 1 ) maybe decomposed into sub-features and additional constraints may be specified amongst the different features. For instance, Figure  1  presents the full feature model of JHipster. Each JHipster variant has a Generator option that may be either a Server, a Client, or an Application; may also have a Database that is SQL or Cassandra or MongoDB; etc. Additional constraints specify for instance that SocialLogin may only be selected for Monolithic applications.
Motivation of this Study Despite the number of empirical investigations (e. g. ,  [22, 66] ) and surveys (e. g. ,  [16, 20, 77] ) to compare such approaches, many focused on subsets to make the analyses tractable. Being able to execute all configurations led us to consider actual failures and collect a ground truth. It helps to gather insights for better understanding the interactions in large configuration spaces  [53, 79]. And provide a complete, open, and reusable dataset to the configurable system testing community to evaluate and compare new approaches.
Case Study JHipster is an open-source, industrially used generator for developing Web applications  [34]. Started in 2013, the JHipster project has been increasingly popular (6000+ stars on GitHub) with a strong community of users and around 300 contributors in February 2017. From a user-specified configuration, JHipster generates a complete technological stack constituted of Java and Spring Boot code (on the server side) and Angular and Bootstrap (on the front-end side). The generator supports several technologies ranging from the database used (e. g. , MySQL or MongoDB ), the authentication mechanism (e. g. , HTTP Session or Oauth2 ), the support for social log-in (via existing social networks accounts), to the use of microservices. Technically, JHipster uses npm and Bower to manage dependencies and Yeoman 2 (aka yo) tool to scaffold the application  [68].
JHipster relies on conditional compilation with EJS 3 as a variability realisation mechanism. Listing 1 presents an excerpt of class DatabaseConfiguration. java. The options sql, mongodb, h2Disk, h2Memory operate over Java annotations, fields, methods, etc. For instance, on line 8, the inclusion of mongodb in a configuration means that DatabaseConfiguration will inherit from Abstract-MongoConfiguration. JHipster is a complex configurable system with the following characteristics: (i) a variety of languages (Ja-vaScript, CSS, SQL, etc. ) and advanced technologies (Maven, Docker, etc. ) are combined to generate variants; (ii) there are 48 configuration options and a configurator guides user throughout different questions. Not all combinations of options are possible and there are 15 constraints between options; (iii) variability is scattered among numerous kinds of artefacts (pom.
Listing 2 gives an excerpt for the choice of a databaseType. Users can select no database, sql, mongodb, or cassandra options. There is a pre-condition stating that the prompt is presented only if the microservice option has been previously selected (in a previous question related to ap-plicationType). In general, there are several conditions used for basically encoding constraints between options. We modelled JHispter's variability using a feature model (e. g. ,  [40] ) to benefit from state-of-the-art reasoning techniques developed in software product line engineering  [4, 6, 12, 13, 77]. Though there is a gap with the configurator specification (see Listing 2), we can encode its configuration semantics and hierarchically organize options with a feature model. We decided to interpret the meaning of the configurator as follows: 1.
each multiple-choice question is an (abstract) feature. In case of "yes" or "no" answer, questions are encoded as optional features (e. g. , databaseType is optional in Listing 2); 2. each answer is a concrete feature (e. g. , sql, mongodb, or cassandra in Listing 2). All answers to questions are exclusive and translated as alternative groups in the feature modelling jargon. A notable exception is the selection of testing frameworks in which several answers can be both selected; we translated them as an Or-group; 3. pre-conditions of questions are translated as constraints between features. Based on an in-depth analysis of the source code and attempts with the configurator, we have manually reverse-engineered an initial feature model presented in Figure  1 : 48 identified features and 15 constraints (we only present four of them in Figure  1  for the sake of clarity).
Engineering a configurable system for testing configurations The execution of a unique and generic command for testing JHipster variants was not directly possible. For instance, the build of a JHipster application relies either on Maven or Gradle, two alternative features of our variability model. We developed variability-aware scripts to execute commands specific to a JHipster configuration. Command scripts include: starting database services, running database scripts (creation of tables, keyspaces, generation of entities, etc. ), launching test commands, starting/stopping Docker, etc. As a concrete example, the inclusion of features h2 and Maven lead to the execution of the command: "mvnw -Pdev"; the choice of Gradle (instead of Maven) and mysql (instead of h2) in production mode would lead to the execution of another command: "gradlew -Pprod".
In total, 15 features of the original feature model influence (individually or through interactions with others) the way the testing workflow is executed. The first lessons learned are that (i) a non-trivial engineering effort is needed to build a configuration-aware testing workflow -testing a configurable system like JHipster requires to develop another configurable system; (ii) the development was iterative and mainly consisted in automating all tasks originally considered as manual (e. g. , starting database services).
accept ( MediaType. A P P L I C A T I O N _ J S O N ) ). andExpect ( status (). isOk () ). andExpect ( content (). string (" test ") ) ; } conditions (i. e. , input data) under which all variants will be executed and tested. Technically, we need to populate Web applications with entities (i. e. , structured data like tables in an SQL database or documents in MongoDB for instance) to test both the server-side (in charge of accessing and storing data) and the client-side (in charge of presenting data). JHipster entities are created using a domain-specific language called JDL, close to UML class diagram formalism. We decided to reuse the entity model template given by the JHipster team 6. We created 3 entity models for MongoDB, Cassandra, and "others" because some database technologies vary in terms of JDL expressiveness they can support (e.
g. , you cannot have relationships between entities with a MongoDB database). After entities creation with JDL (Entities Generation in Figure  3 ), we run several tests: integration tests written in Java using the Spring Test Context framework (see Listing 3 for instance), user interface tests written in JavaScript using the Karma. js framework (see Listing 4 for instance), etc. , and create an executable JHipster variant (Build Maven/Gradle in Figure  3 ). The tests run at this step are automatically generated and include defaults tests common to all JHipster variants and additional tests generated by the JDL entities creation. On average, the Java line coverage is 44. 53% and the JavaScript line coverage is 32. 19%. We instantiate the generated entities (Entities Populating in Figure  3 ) using the Web user interface through Selenium scripts.
Bugs: A Quick Inventory Out of the 26,256 configurations we tested, we found that 9,376 (35. 70%) failed. This failure occurred either during the compilation of the variant (Compilation in Figure  3 ) or during its packaging as an executable Jar file (Build Maven/Gradle in Figure  3 , which includes execution of the different Java and JavaScript tests generated by JHipster), although the generation (App generation in Figure  3 ) was successful. We also found that some features were more concerned by failures as depicted in Figure  4. Regarding the application type, for instance, microservice gateways and microservice applications are proportionally more impacted than monolithic applications or UAA server with, respectively, 58. 37% of failures (4,184 failing microservice gateways 8 Complete results are available at https://github.
Each configuration in C has a unique identifier and contains a subset of the features in F and the status of its build. A rule is defined as an implication of the form: X ⇒ Y , where X, Y ⊆ F. The outputs of the method are a set of rules, each constituted by: -X the left-hand side (LHS) or antecedent of the rule; -Y the right-hand side (RHS) or consequent of the rule. For our problem, we consider that Y is a single target: the status of the build. For example, we want to understand what combinations of features lead to a failure, either during the compilation or the build process. To illustrate the method, let us take a small example (see Table  1 ). The set of features is F = { mariadb, gradle, enableSocialSignIn, websocket, failure } and in the table is shown a small database containing the configurations, where, in each entry, the value 1 means the presence of the feature in the corresponding configuration, and the value 0 represents the absence of a feature in that configuration.
We found no rule involving 1 or 3 features. We decided to examine the 200 association rules for which the LHS is of size 4. We found out a sixth association rule that incidentally corresponds to one of the first failures we encountered in the early stages of this study. Table  2  shows that there is only one rule with the RHS being Compile=KO. According to this rule, all configurations in which the database is MongoDB and social login feature is enabled (128 configurations) fail to compile. The other 5 rules are related to a build failure. Figure  5  reports on the proportion of failed configurations that include the LHS of each association rule. Such LHS can be seen as a feature interaction fault that causes failures. For example, the combination of Mari-aDB and Gradle explains 37% of failed configurations (or 13% of all configurations).
Qualitative Analysis We now characterize the 6 important faults, caused by the interactions of several features (between 2 features and 4 features). Table  2  gives the support, confidence for each association rule. We also confirm each fault by giving the GitHub issue and date of fix. MariaDB with Docker. This fault is the only one caused by the interaction of 4 features: it concerns monolithic web-applications relying on MariaDB as production database, where the search-engine (ElasticSearch) is disabled and built with Docker. These variants amount to 1,468 configurations and the root cause of this bug lies in the template file src/main/docker/_app. yml where a condition (if prodDB = MariaDB ) is missing. UAA authentication with Ehcache as Hibernate second level cache. This fourth fault concerns Microservice Gateways and Microservice applications, using a UAA authentication mechanism.
When deploying manually (i. e. , with Maven or Gradle), the web application is unable to reach the deployed UAA instance. This bug seems to be related to the selection of Hibernate cache and impacts 1,667 configurations. OAuth2 authentication with SQL database. This defect is faced 649 times, when trying to deploy a web-app, using an SQL database (MySQL, PostgreSQL or Mari-aDB) and an OAuth2 authentication, with Docker. It was reported on August 20th, 2016 but the JHipster team was unable to reproduce it on their end. Social Login with MongoDB. This sixth fault is the only one occurring at compile time. Combining MongoDB and social login leads to 128 configurations that fail. The source of this issue is a missing import in class So-cialUserConnection. java. This import is not in a conditional compilation condition in the template file while it should be.
Abal et al. report that, for the Linux bugs they have qualitatively examined, more than a half (22/43) are attributed to 2-wise interactions  [2]. Yet, for different interaction strengths, there is no common trend: we do not have 3-wise interactions while this is second most common case in Linux, we did not find any fault caused by one feature only. (RQ2. 1) How many and what kinds of failures/faults can be found in all configurations? Exhaustive testing shows that almost 36% of the configurations fail. Our analysis identifies 6 interaction faults as the root cause for this high percentage. 6 Sampling Techniques Comparison (RQ2. 2) In this section, we first discuss the sampling strategy used by the JHipster team. We then use our dataset to make a ground truth comparison of six state-of-the-art sampling techniques.
Before 2014 (i. e. , before the publication of the systematic literature review from Machado et al. [50] ), empirical evaluations of configurations sampling approaches are focused on their capability to select a sampling satisfying t-wise criteria in a reasonable amount of time or with fewer configurations  [21, 29, 32, 37, 38, 44, 47, 61, 62]. We discarded them as they do not match our selection criteria. To select relevant studies without performing a full systematic literature survey, we applied a forward and backward snowballing search  [33]. Snowballing is particularity relevant in our case, given the diversity of terms used in the research literature (product line, configurable systems, etc. ) and our goal to compare more than one sampling approach. Searching and filtering studies from literature databases would require a large amount of work with few guarantees on the quality of the result.
Continuous testing JHipster relies on a continuous integration platform (Travis) integrated into GitHub. At the time of the release 3. 6. 1, the free installation of Travis allowed to perform 5 different builds in parallel, at each commit. JHipster exploits this feature to only test 12 configurations. JHipster developers give the following explanations: "The only limit was that you can only run 5 concurrent jobs so having more options would take more time to run the CI and hence affect our turn around hence we decided on a practical limit on the number [. ] We only test the 12 combinations because we focus on most popular options and leave the less popular options out. " Julien also mentioned that his company IPPON provides some machines used to perform additional tests. We can consider that the testing budget of JHipster 3.
Sampling size Our empirical results and discussions with JHipster developers suggest that the testing budget was simply too low for JHipster 3. 6. 1, especially when popular configurations are included in the sampling. According to JHipster developers, the testing budget "has increased to 19 now with JHipster 4, and we also have additional batch jobs running daily tripling the number of combinations [. ] We settled on 19 configurations to keep build times within acceptable limits" 13. An ambitious and long-term objective is to crowdsource the testing effort with contributors. Users can 12 https://github. com/jhipster/jhipster-devbox 13 Discussions are available at https://github. com/ jhipster/generator-jhipster/issues/4301 lend their machines for testing some JHipster configurations while a subset of developers could also be involved with the help of dedicated machines.
We have used several strategies, from statistical computations to manual reviews of individual failures to mitigate this threat. Despite those limitations, we found all faults reported by the JHipster community and even new failures. For the other remaining 242 configurations that fail due to our test infrastructure (see Section 5. 3), there might be false positives. Since they only represent 0. 9% of all JHipster configurations, such false positives would have a marginal incidence on the results. In fact, this situation is likely to happen in a real (continuous integration) distributed testing environment (e. g. , as reported in  [11] ). We thus decided to keep those configurations in the dataset. Additionally, they can serve as a baseline to improve our testing infrastructure for the next versions of JHipster.
Acknowledgements We would like to thank  Prof. Arnaud Blouin  for his comments and feedback on the paper. This research was partially funded by the  EU  Project  STAMP ICT-16-10 No. 731529  and the  Dutch 4TU  project " Big Software on the Run " as well as by the  European Regional Development Fund (ERDF)  " Ideas for the future Internet" (IDEES)  project.
Listing 2 Configurator: server/prompt. js (excerpt) (. ) when : function ( response ) { return applicati o nT yp e === ' microservice '; } , type : ' list ' , name : ' databaseType ' , message : function ( response ) { return g e t N um b e r e d Q u e s t i o n ( ' Which * type * of database would you like to use ? ' , applicat i on Ty p e === ' microservice ') ;} , choices : [ { value : 'no ' , name : ' No database '} , { value : 'sql ' , name : ' SQL ( H2 , MySQL , MariaDB , PostgreSQL , Oracle ) '} , { value : ' mongodb ' , name : ' MongoDB '} , { value : ' cassandra ' , name : ' Cassandra '} ] , default : 1 (. ).
590% 1 4543 24 Nov 2016 (report and applicationType="monolith", fix for milestone 3. 12. 0) searchEngine="false", Docker="true" MariaDB using Gradle. This second fault concerns variants relying on Gradle as build tool and MariaDB as the database (3,519 configurations). It is caused by a missing dependency in template file server/template/gradle/_liquibase. gradle. UAA authentication with Docker. The third fault occurs in Microservice Gateways or Microservice applications using an UAA server as authentication mechanism (1,703 Web apps). This bug is encountered at build time, with Docker, and it is due to the absence of UAA server Docker image. It is a known issue, but it has not been corrected yet, UAA servers are still in beta versions.
Cross-validation for parameter-tuning is based on an appropriate loss function that takes into account the different contributions of censored and uncensored observations. We specify how these techniques can be easily implemented using available R packages. The Lasso-regularized Buckley-James least square method was compared to simple imputation strategies to predict the response to antiretroviral therapy measured by HIV viral load according to the HIV genotypic mutations. We used a dataset composed of several clinical trials and cohorts from the Forum for Collaborative HIV Research (HIV Med. 2008;7:27-40). The proposed methods were also assessed on simulated data mimicking the observed data. Results: Approaches accounting for left-censoring outperformed simple imputation methods in a high-dimensional setting.
Lasso on complete data The Lasso (Least Absolute Shrinkage and Selection Operator)  [53]  is one of the most popular methods in high-dimensional data analyses. It allows for simultaneous estimation and variable selection and has efficient algorithms available. It is considered here as the Gold Standard for our simulation studies. The Lasso estimator of parameters in model (  1 ) is: β(λ) GoldS = argmin β Y -Xβ 2 2 + λ β 1 (2) where 2 is the quadratic loss, β 1 = p j=1 β j is the Lasso penalty on the parameter size, and λ > 0 controls the amount of regularization. When λ is large enough (which depends on data), all coefficients are forced to be exactly zero. Inversely, λ = 0 corresponds to the unpenalized ordinary leastsquares estimate. Y -Xβ 2 2 = n i=1 (Y i -X i β) This model on complete data (no left-censored measures) is considered as a reference when comparing the other methods applied to incomplete datasets that include left-censored values.
Implementation issues All statistical analyses, comparisons and implementations were performed using the computing environment R (R Development Core Team, 2017)  [69]. We used the function cv. glmnet from package glmnet  [70]  to choose the optimal λ value of Lasso linear regression on complete data (GoldS) and Lasso linear regression with simple substitution of left-censored values by the detection limit (LOD). We implemented the Lasso nonparametric Buckley-James (NonParBJ) using the bujar package  [71]. We modified the function to support stratified K-fold cross-validation and conserve the same proportion of censoring in all the folds. Lasso Gaussian Buckley-James (Gaussian BJ) was implemented in a new function cvGaussBJ. Algorithm 1 specifies how to solve the problem. The stopping criterion is based on the difference between current and previous regression coefficient estimates, variance estimates, and imputed data.
Because of the tendency to oscillate between different parameter values of the iterative procedure, the algorithm is also stopped if the number of oscillations is high  [40]. Alternatively, we also considered the one-step algorithm, which stops at the first iteration  [36, 64]. Crossvalidation based on both imputation and loss function accounting for censored and uncensored contributions is considered. The Lasso estimation step depends on package glmnet. All these implementations and an artificial example are available at: https://github. com/psBiostat/left-censored-Lasso.
Real data The database used in this study was provided by the Standardization and Clinical Relevance of HIV Drug Resistance Testing Project for the Forum for Collaborative HIV Research  [49]. Patients included in this study were all treatment-experienced and switched to an abacavircontaining regimen. The investigated drug, abacavir, is a nucleoside reverse transcriptase inhibitor (NRTI) that blocks HIV reverse transcriptase. The sample size n = 99 was slightly smaller than the number of predictors p = 121. 54 of the 121 predictors correspond to the presence or absence of specific mutations in the reverse transcriptase gene (RTG), which were reported to be probably associated with resistance to abacavir, multi-NRTI, NRTI (other than abacavir) or nonnucleoside reverse transcriptase inhibitors (NNRTI) at the time of the study  [73, 74].
• ε is generated from a normal distribution with mean 0 and variance σ 2 chosen such that the signal-to-noise ratio is fixed at 3 : 1. Our primary goal was to compare competing methods in terms of prediction accuracy. Consequently, we simulated training and test datasets. The former were used to estimate, the latter were used to evaluate the prediction performance. We ensured that training and test datasets contained roughly the same proportions of censoring. We computed the mean squared error on test data as: MSE = 1 n test n test i=1 Y test i -X test i β( λ) 2 with β( λ) estimated on training data by stratified 5fold cross-validation using a given regression method (Gold standard, Lasso-regularized Non-parametric and Gaussian Buckley-James -with complete convergence and 1-step-and Lasso-regularized linear regression with simple imputation -by LOD and LOD/2-) and the corresponding loss function in the cross-validation criterion.