dataesr/scientific-paragraphs-categorization huggingface dataset:
Subgroups' generalization credibility In Q-Finder the final step consists in computing the credibility metrics of the top-k subgroups on the testing set, in order to assess their generalization credibility, that is subgroups consistency across databases  (Sun, Briel, Walter, et al. 2010; Dijkman et al. 2009 ). However, contrary to the candidate subgroups generation 11. Above that delta value, the increase in effect size is worth enough to justify an increase in complexity. 12. Wasserstein et al. (2019)  argue to be open in study designs and analyses: "One might say that subjectivity is not a problem; it is part of the solution. " phase previously performed, the number of tested subgroups in this phase is well-controlled (as recommended in  Sun, Briel, Walter, et al. (2010)  and  Dijkman et al.
(2009) ), as it is limited by the parameter k. This allows a better control of the type 1 error that was more difficult to achieve until then. For that purpose, Q-Finder performs a correction for multiple testing during computation of the significance metrics, to account for the number of subgroups tested on independent data (default: Benjamini-Hochberg procedure). top-k subgroups satisfying the credibility criteria on the test dataset are considered highly credible.
For most T2DM patients, this threshold is set at 7%, which is how we define glycemic control for TD2M patients. Our research question can then be formulated as follows: "What are the prognostic factors of glycemic control in TD2M patients?". We consider the following variables as confounding factors: Patient's age  (Ma et al. 2016) , Gender  (Ma et al. 2016) , BMI  (Candler et al. 2018) , Level of education  (Tshiananga et al. 2012 ) and Time since diabetes diagnosis  (Juarez et al. 2012). Considering the geographical heterogeneity in IDMPS, we added the continent where the data was collected. This experiment included 1857 patients from IDMPS wave 6 and 2330 patients from IDMPS wave 7, with 63 variables considered as candidate prognostic factors. In wave 6, 17. 7% of patients were under the 7% HbA1c threshold, versus 18.
While these four methods do cover the spectrum of SD and identification methods, both SIDES and Virtual Twins are well adapted to predictive tasks, APRIORI-SD and CN2-SD can only address prognostic tasks. Since Q-Finder can address both tasks, it is compared with the two methods that are adapted to each of the two tasks described in section 3. 2. For all the analyses, IDMPS wave 6 were used as the discovery dataset and IDMPS wave 7 as the test dataset. To allow comparison of results, only the top-10 subgroups of each algorithm are considered without any human intervention during the selection. Finally, default parameters of each algorithm were selected, except shared parameters which we kept as similar as possible.
APRIORI-SD 16  : An exhaustive search algorithm adapted from association rule learning APRIORI to SD. Compared to APRIORI it only considers subgroups that contain the target variable in the right-hand side. Like CN2-SD, it also uses the weighted covering method. The Python package pysubgroup version 0. 6. 1  (Lemmerich et al. 2018 ) is used, with the following parameters: W RAcc as the optimisation metric, maxdepth = 3 14 and result set size coverage = 10% 15. Q-Finder prognostic mode: The version used is 5. 4 with C max = 3, #Bins = 10 and #Cats = ∞ (see section 2. 2). Only left-bounded and right-bounded intervals are considered. The thresholds for credibility criteria are the default values presented in section 2. 3. 1 : minimum coverage = 10%, minimum basic pattern absolute contribution = 0.
2, maximum basic pattern contribution ratio = 5, minimum ef f ect size = 1. 2 (with or without correction for confounders), and maximum ef f ectś signif icance threshold = 0. 05 (with or without correction for confounders). Multiple testing correction is addressed using Bonferroni correction in the discovery dataset and Benjamini-Hochberg procedure in the test dataset. For the ranking steps, aggregation rules are the ones presented in section 2. 3. 2, m c being the p-value for subgroup's effect when corrected for confounders. The default top-k selection is performed with the oddsratio corrected for confounders as the second metric and δ ES = 0. 2 (see section 2. 4. 2).
Exploring predictive subgroups For each of the three approaches to identify predictive subgroups (Virtual Twins, SIDES and Q-Finder) we detail the version and main parameters. Virtual Twins 17  : Following the vignette's recommendation from the R package aVirtualTwins version 1. 0. 1, missing values were a priori imputed on the discovery dataset using rf Impute() from the random-Forest package version 4. 6. 14. For this step and each of the following, the seed was set to 42. After the imputation, Virtual Twin's first step consisted in using randomForest() from the randomForest package (version 4. 6. 14) with ntree = 500 and threshold = 0. 5 (threshold above which the treatment effect is considered significant for a patient). The second step consisted in performing a classification tree with maxdepth = 3 (maximum depth of the classification tree 14 ).
Only the leaves for which the predicted outcome was the target were considered as outputted subgroups. SIDES 18  : The version considered is 1. 14 from the SIDES R package. The parameters considered are: M = 5 (maximum number of best promising subgroups selected at each step of the algorithm), alpha = 0. 05 (overall type 1 error rate, which is compared with p-values corrected for multiple testing using a resamplingbased method to protect the overall type 1 error rate), S = 200 (minimum subgroup size desired, set at 10% of the discovery dataset 15 ), L = 3 (maximum depth of the tree 14 ), D = 0 (minimum difference between the treatment and the control arm), gamma = 1 (relative improvement parameter), num crit = 1 (splitting criterion used: maximizing the differential effect between the two child subgroups), H = 1 (i.
e. no random split of the discovery dataset), ord. bin = 10 (number of classes continuous covariates are discretized into  19  ). As SIDES is a non-deterministic algorithm, the seed was set to 42. Q-Finder predictive mode: The version used is 5. 4 with C max = 3, #Bins = 10 and #Cats = ∞ (see section 2. 2). Only left-bounded and right-bounded intervals are considered. The thresholds 20 for credibility criteria are the default values presented in section 2. 3. 1 : minimum coverage = 10%, minimum basic pattern absolute contribution = (0, 0. 2), maximum basic pattern contribution ratio = (∞, 5), minimum ef f ect size = (1, 1. 2) (with or without correction for confounders), and maximum ef f ectś signif icance threshold = (0. 05, 0. 05) (with or without correction for confounders). Multiple testing correction is addressed using Bonferroni correction in the discovery dataset and Benjamini-Hochberg procedure in the test dataset.
Prognostic factors identification Q-Finder results on the prognostic task: Q-Finder generated 203 subgroups satisfying all the credibility criteria. Among the top-10 subgroups selected while accounting for diversity, 2 are of complexity 1, none are of complexity 2 and 8 are of complexity 3. The results are presented below in Table  1  along with the main metrics of interest computed on both the discovery and the test datasets (see Table  S8  in supplementary materials for the additional metrics computed and outputted from Q-Finder). The two first-ranked subgroups S1 and S2 are both of complexity 1 and state that patients whose last postprandial glucose (PPG) level was below 172. 0 mg/dl (resp. whose last fasting blood glucose (FBG) level was below 129. 6 mg/dl) do have a better glycemic control than the others.
Both subgroups are very close to the glycemic control targets established by the American Diabetes Association (resp. 180 mg/dl for PPG and 130 mg/dl for FBG (American Diabetes Association 2017)). The coverage (or support) of the first subgroup S1 is 30% of the discovery dataset, its adjusted odds-ratio is 4. 8 ([3. 5; 6. 5]) and its p-value is 1. 81E -23 on the discovery dataset. All selected subgroups were successfully reapplied on the test dataset, with odds-ratios corrected for confounders above 1. 81 and p-values below 0. 05 when adjusted for multiple testing by Benjamini-Hochberg procedure. It is worth noticing that all the subgroups were significant using the more conservative Bonferroni correction in the discovery dataset. Results for CN2-SD and APRIORI-SD: Results for both CN2-SD and APRIORI-SD are given below.
For CN2-SD, no subgroups were outputted using the default parameters, described in 3. 3. 1. An analysis of the sensitivity is presented in the discussion of the results (see section 4. 2). For APRIORI-SD, 186 subgroups were outputted. Among the top-10 subgroups based on the WRAcc measure, 1 is of complexity 1, 2 are of complexity 2 and 7 are of complexity 3. The complexity 1 subgroup (S4 in Table  2 ) is defined by a last postprandial glucose measurement below 144 mg/dl (WRAcc on discovery dataset: 0. 0329). All complexity 2 and 3 subgroups, except S10, are also defined by this basic pattern, combined with other patterns such as Receives GLP -1 analogues = N o or Self -monitoring testing perf ormed at bed time = N o. The results are presented below in Table  2  with the WRAcc measure, both on the discovery and the test datasets:.
Predictive factors identification Q-Finder results on the predictive task: Q-Finder generated 2775 subgroups in the discovery dataset that pass all the criteria of credibility on the predictive task. Among the top-10 subgroups selected while accounting for diversity, all are of complexity 3 except one. The results are presented below in Table  3  with main criteria of interest computed on both the discovery and the test datasets (see Table  S9  and S10 in supplementary materials for the additional metrics computed and outputted from Q-Finder). Subgroup S2 states that patients who use a disposable pen, don't smoke and are not heavily treated for diabetes, have a higher risk than the others in experiencing hypoglycemia under Premixed insulin than under Basal insulin (coverage = 25%, adjusted odds-ratio for differential treatment effect = 3.
31 [2. 0 ; 5. 6], p-value = 7. 13E-6). The seven first selected subgroups were successfully reapplied on the test dataset, with adjusted oddsratios related to differential treatment effect above 1. 86. Indeed, these subgroups have a p-value below 0. 05 adjusted for multiple testing using Benjamini-Hochberg procedure, despite the fact that no subgroups were "statistically significant" after Bonferroni correction in the discovery dataset. It is worth noticing that all subgroups have adjusted odds-ratios above 1. 0 in the test dataset. Results for SIDES and Virtual Twins on the predictive task: Results for both SIDES and Virtual Twins are given below. For SIDES, no subgroups were outputted using the default parameters, described in section 3. 3. 2. An analysis of the sensitivity is presented in the discussion of the results (see section 4.
Q-Finder generates the top-k hypotheses Q-Finder has proposed 10 prognostic factors and 10 predictive factors. This is more than the set of subgroups generated by Virtual Twins and conversely to SIDES and CN2-SD that did not generate any subgroups with their default parameters This illustrates that with default parameters Q-Finder systematically gives results whose credibility are assessed. As for SIDES, the lack of results may well be explained by the strategy it uses to generate hypotheses. Indeed, SIDES filtering strategy, in which subgroups have to pass all predefined criteria in the learning phase (including the p-value corrected for multiple testing, a very conservative step), strongly limits hypotheses generation. The absence of results is therefore not uncommon with SIDES. On the contrary, the top-k selection strategy of Q-Finder favors the generation of hypotheses since the k best-ranked subgroups of the discovery dataset will be considered as hypotheses to be tested on independent data.
This both allows to assess Q-Finder's results robustness, while preserving the statistical power (as only k tests are performed in the test dataset). Therefore, conversely to SIDES, the correction for multiple testing that is performed in the discovery dataset (that both gives more credibility to the results from the learning phase and increases the subgroups discrimination in the ranking phase) does not hinder the most promising subgroups to be tested and possibly validated on an independent dataset. Q-Finder is thus aligned with the notion of "statistical thoughtfulness" 21 recently promoted by the American Statistical Association  (Wasserstein et al. 2019). For CN2-SD, the lack of results may be due to the beam search, which does not cover the entirety of the search space and may thus miss relevant subgroups.
Moreover, although the adjusted effect sizes of all APRIORI-SD subgroups appear high in both discovery and test datasets they are redundant. In fact, several subgroups sharing the same basic patterns are associated with the very same extension (as suggested by the identical results on credibility measures in the discovery dataset), which masks the fact that they are the same subgroups (e. g. S1 and S2 as well S3 and S4 in Table  5 ). The fact that an increase in complexity is not always accompanied by an increase in effect is due to the fact that APRIORI-SD does not include any parameters evaluating the contribution of basic patterns (such as the Q-Finder basic patterns contribution criteria), which leads to unnecessarily more complex subgroups. Finally, one can see that adjusted effect sizes of Virtual Twins subgroups are mostly smaller than those of the Q-Finder subgroups, and that none of the three subgroups generated by Virtual Twins have p-values below 0.
05 once confusion biases have been corrected in the test dataset (Table  6 ). Based on the credibility criteria used in clinical research, Virtual Twins has therefore generated less convincing results than Q-Finder. Moreover, whether Virtual Twins, APRIORI-SD or CN2-SD, the robustness of the results is not meant to be evaluated on independent data. Similarly, they do not seek to control and assess the risk of false positives, regardless of their presence in the results. This seriously undermines the credibility of the results. Q-Finder supports the set of subgroups with standard and recommended criteria of credibility in clinical research. Therefore, all metrics used in Q-Finder for generating hypotheses are given as outputs for transparency. As strongly recommended by the American Statistical Association  (Wasserstein et al.
2019 ): • p-values are reported in continuous. This should allow experts to better interpret them, and avoid basing any decision on a p-value threshold that would misrepresent what "worthy" and "unworthy" results are 23. • p-values can be "interpreted in lights of its context of sample size and meaningful effect size". This set of metrics is key for scientific inference of results. Q-Finder also provides confidence intervals of effect sizes, to help experts to assess results. This is to be contrasted with most packages, including the ones used in this paper to compare Q-Finder. Indeed, Virtual Twins package only gives information about size and risk ratios (not adjusted for confounding factors). The SIDES package would only output continuous p-values below an arbitrary threshold. As for CN2-SD and APRIORI-SD, we are far from the standards for the publication of prognostic factors (see Table  S5  and S6 in supplementary materials for comparison of output metrics from standard packages).
This is explained by the fact that several attributes in the dataset contain related information. Stricter pre-selection of attributes, based on both correlation analysis and clinical expertise before performing the analysis, is a classic approach to reduce this type of redundancy. 23. As mentioned by the American Statistical Association  (Wasserstein et al. 2019) , arbitrary p-value thresholds could lead to biased conclusions and published results, and are only acceptable for "automated tools" and "automated decision rule". In that respect, Q-Finder does use p-value thresholds for the automatic ranking of subgroups, but no filter on p-value thresholds is done whether to select the top-k subgroups (some of them could have p-values above 0. 05) or to report their results on both discovery and test datasets.
Limits of the IDMPS databases: surveys As the IDMPS databases are derived from surveys spread over time, they each reflect an image at a given time. As a result, treatment initiation may have occurred before data recording. In this situation, the data studied are not necessarily the baseline of the study, which gives the results a purely descriptive character. Indeed, a variable by which a subgroup is defined should not be affected by treatment response  (Dijkman et al. 2009). The most common use case of SD is rather the retrospective study of prospective data (e. g. RCT) or real world data, in which temporal information is collected, in order to only consider the information before treatment's intake (i. e. the "baseline" period).
For example, in the case of the search for treatment responders subgroups, the search may be motivated by other criteria such as "not being associated with a specific adverse effect", or "having an equally good treatment effect regardless of patient age". Regarding the experiment that was done in this article, one could have searched for subgroups predictive of low rate of hypoglycemia (outcome) while focusing on subgroups of patients with strong glycaemic control, to identify subgroups of interest associated with both higher treatment efcacy and better safety than average. Q-Finder can easily be adapted to any other research questions, including non-clinical ones, as its parameters can be set to directly target subgroups of interest in relation to any types of objective. Extracting the best hypotheses possible from a dataset, based on multiple criteria, using both statistical and business metrics is a common need in many sectors.
Conclusion Subgroup Discovery has become an important task in the field of Subgroup Analysis. Q-Finder inherits both SI-SD and KDD-SD culture, borrowing metrics and evaluation from the first one and hypothesis generation from the second. As such, Q-Finder is a SD algorithm dedicated to the identification of either prognostic or predictive factors in clinical research. The generated subgroups are driven on a set of recommended criteria in clinical studies to directly target promising and credible subgroups for their final clinical evaluation. This contrasts with most standard algorithms that rely only partially on these credibility metrics, and for which the risk of being ruled out afterwards by a clinical assessment is greater. Q-Finder also favors the hypothesis generation thanks to (1) an exhaustive dataset exploration that allows for emerging synergistic effects, optimally-defined subgroups and new insights to come out, and (2) its top-k selection strategy that selects credible and diverse subgroups to be tested on independent datasets.
As a matter of fact, beyond its fully automatic mode, the Q-Finder approach also supports selecting subgroups based on clinical expertise to both increase subgroups relevance to the research question and reduce false positives. Applied on the IDMPS database to benchmark it against state-of-the-art algorithms, Q-Finder results were best in jointly satisfying the empirical credibility of subgroups (e. g. , higher effect sizes adjusted for confounders and lower p-values adjusted for multiple testing), and their diversity. These subgroups are also those that are supported by the largest number of credibility measures. Q-Finder has already proved its value on real-life use cases by successfully addressing high-stake research questions in relation to a specific pathology and/or drug such as efficacy and safety questions and by dealing with main limits of standard algorithms (e.
g. the lack of results or the low subgroups credibility). Its high comprehensibility did favor the acceptance by clinical teams of the identified subgroups. Finally, Q-Finder could straightforwardly be extended to other research questions (including non-clinical ones), notably by tailoring the metrics used in the exploration to directly target the subgroups of interest in relation to the objective. or of treatment-subgroup interaction effects can also be checked and promoted by including clinical experts directly into the selection step of the top-k credible subgroups to be tested on independent dataset (see section 6. 9. 6). This last step also allows to both limit the number of tests and check for subgroups consistency across datasets. Tests are also adjusted for multiplicity, and interaction tests are used for assessing betweensubgroup treatment effect interactions.
Aggregation rules visualization Other credibility metrics are currently being implemented in order to further improve subgroups credibility as recommended in  Sun, Briel, Walter et al. (2010)  and  Dijkman et al. (2009) , such as assessing the treatment-subgroup interaction consistency across closely related outcomes within the study and better assessing both the comparability of prognostic or predictive factors and the significant subgroup effect independence with the other discovered subgroups. Similarly, some of the existing measures in Q-Finder could be made more powerful, such as the default corrections for multiple tests, i. e. the Benjamini-Hochberg procedure in the test dataset, and the Bonferroni correction in the discovery dataset (which makes the calculation easier given the massive number of tests performed).
The top-k "clinician-augmented" selection After the ranking of subgroups, the top-k selection consists of selecting the most promising subgroups to be tested on an independent dataset. The presented top-k algorithm includes a "diversity" parameter in order to favor the generation of subgroups defined by diverse attributes. Other strategies could be considered, like selecting the most credible subgroups (i. e. regarding the subgroup's ranking) while maximizing the coverage of dataset (i. e. obtaining a set of subgroups that covers the maximum of patients) or even maximizing the coverage of targeted patients (i. e. obtaining a set of subgroups that explains as much as possible the phenomenon of interest). This can be generalized to any clinical strategy, like selecting the top-k subgroups that covers as much as possible any patients characteristic (e.
g. obtaining a set of subgroups that covers all countries within a study, or all subtypes of a given disease,. ). A drawback in promoting diversity is that we go deeper through the subgroups ranking to select various subgroups, which therefore favors the selection of generally lowest quality results, that is with lowest sizes or/and risks-ratios, and consequently with a higher risk that both the overall subgroup's effect and the basic patterns contribution are weaker and random. For example, one can notice that several Q-Finder prognostic subgroups replicated in the test dataset have non robust basic patterns (i. e. with negative absolute contribution values, see Table  S8 ). As such, Q-Finder allows the fine-assessment of the robustness for each subgroup's basic pattern, and let the possibility to retrospectively simplify final subgroups by removing the non robust patterns that unnecessary impair subgroups.