dataesr/scientific-paragraphs-categorization huggingface dataset:
Public transcriptomics and clinical data Processed bulk RNA-Seq and clinical data were collected for 5 different cohorts. A cohort of 331 primary and metastatic advanced clear cell Renal Cell Carcinoma (ccRCC) samples from patients that progressed on 1, 2 or 3 previous therapies (at least one systemic anti-angiogenic therapy) included in clinical trials CM-009 (NCT01358721), CM-010 (NCT01354431) and CM-025 (NCT01668784) treated by anti-PD-1 antibody Nivolumab and mTOR inhibitor Everolimus (EGAC00001001519; EGAC00001001520; EGAC00001001521)  (Braun et al, 2020). A cohort of 11 primary or metastatic ccRCC samples on one of four clinical trials (NCT00441337, NCT00730639, NCT01354431, NCT01358721) processed in  Wu et al. , 2022 (Ascierto et al. , 2016). A cohort of 64 advanced melanoma samples from patients included in clinical trial CA209-038 (NCT01621490) treated with Nivolumab (GSE91061)  (Riaz et al, 2017).
A cohort of 20 metastatic melanoma samples from patients treated by anti-PD-1 (GSE168204)  (Du et al, 2021). A cohort of 19 peripheral CD8-T cells (PBMCs) samples collected from blood of metastatic NSCLC patients (GSE111414)  (Trefny et al, 2019). Samples for each cohort were divided according to prior therapies and analyzed independently. Patients with treatment outcomes of "Complete Response" (CR) or Partial Response (PR) were considered as responders, whereas patients with treatment outcome of "Progressive Disease" (PD) as non-responders.
Single-cell RNA-seq data analysis Single-cell RNA-seq data from 11 ccRCC patients was obtained from Obradovic et al. (Obradovic et al. , 2021). Raw data was downloaded from https://data. mendeley. com/datasets/nc9bc8dn4m/1. The data set consisted of adjacent normal and tumor tissue. Both were analyzed separately. Furthermore, in the original study, the tissue was prior to sequencing FACS sorted into CD45+ and CD45-subsets. These subset annotations were kept, and the data were separately re-processed in R using  Seurat (v. 4. 1. 1)   (Hao et al. , 2021)  and harmony (v. 0. 1. 0)  (Korsunsky et al. , 2019). Initially,only features expressed in at least 50 cells were included in the analysis. Further QC included removal of cells with fewer than 200 and more than 5000 features detected. As kidney cells have in general a high mitochondrial content, its threshold was set to >25%, everything above was removed.
To confirm the tumor origin of the annotated tumor cells, copy number variation (CNV) was inferred using the R package infercnv (v. 1. 12. 0). As a normal reference the adjacent normal CD45-subset was used. The gene order file (hg38_gencode_v28. txt) was downloaded from https://data. broadinstitute. org/Trinity/CTAT/cnv/. For the analysis a gene cutoff of 0. 1 was set. Furthermore, cluster_by_groups was set to TRUE and a noise filter was applied by setting sd_amplifier to 1. 5. The hidden Markov model (HMM) i6 was used for CNV prediction.
BRB-seq data processing Pair-end reads with quality score higher than 10 were kept. The first read of the pair is 16 bases long. A first part of 6 bases corresponds to a unique sample-specific barcode and a second part of 10 bp is a unique molecular identifier (UMI). The second read of the pair, containing genomic data, was aligned to the human reference transcriptome from the UCSC website (release hg38) using BWA  (Li et al, 2009) (version 0. 7. 4. 4)  with the non-default parameter "-l 24". Reads mapping to several positions in the genome were filtered out from the analysis. The pipeline is described in Draskau et al  (Draskau et al, 2021). After quality control and data pre-processing, a gene count matrix was generated by counting the number of unique UMIs associated with each gene for each sample.
Cell deconvolution algorithms Prediction of tumor micro-environment (TME) cell type proportions from bulk gene gene expression data was performed using CIBERSORTx (version 1. 0, 12/21/2019) and MuSiC (version 1. 0. 0) algorithms  (Wang et al, 2019; Newman et al, 2019). The CIBERSORTx cell fractions module was executed using the docker image after registration and access token received (https://cibersortx. stanford. edu/download. php). For both methods, reprocessed and relabeled singlecell RNA-seq data were used as reference  (Obradovic et al. , 2021). The 3 tumor predicted fractions ("Tumor cells", "Cycling Tumor cells", "Tumor cells CA9-high") were summed into one "Tumor" fraction for clustering of samples based on estimated cell fractions.
Clustering analysis Unsupervised consensus clustering methods were implemented by the R/Bioconductor package ConsensusClusterPlus (version 1. 62. 0). Clustering of cell fractions and gene expression values were performed by k-means (parameters: reps = 1000, pItem = 0. 8, pFeature = 1, clusterAlg = "km", distance = "euclidean"). The best number of clusters was assessed by the delta area plot of consensus Cumulative Density Function (CDF). To construct relevant groups of ccRCC samples in terms of treatment response we retained only cell fractions predicted by the CIBERTSORTx algorithm associated with clinical benefit (two-sided Wilcoxon rank-sum test, corrected p-value < 0. 05 or corrected p-value < 0. 20, for, respectively, metastatic tumor and others if no one fraction was selected by the first threshold).
Immunofluorescence and histology image analysis Formalin fixed paraffin embedded tissue blocks were sectioned at 5µm thickness. The tissue sections were dewaxed and rehydrated in xylene, gradual percentages of alcohols and finally in tap water. Protein epitopes of interest (CD8, particularly for CD8 positive lymphocytes, CD34 for endothelial cells equivalent to CD34, and CD45 for all leuckocytes) were retrieved in pH6 sodium citrate buffer using heat-induced epitope retrieval (HIER) method for 5 min, followed by quenching endogenous peroxidase activity and possible non-specific staining using 3% hydrogen peroxide (Sigma, H1009) and serum-free protein block (Agilent, X090930-2), respectively. Primary antibody -CD8 (Agilent, M710301-2, 1:800), and CD34 (Human Protein Atlas, HPA067440, 1:400), CD45 (Abcam, ab40763, 1:500) -was incubated for 1 hour at room temperature, followed by HRP (horseradish peroxidase) conjugated secondary antibody (Leica biosystems, DS9800) for 30 min.
Extended labeling of ccRCC tumor micro-environment cells A comprehensive literature review was performed to identify markers, mainly cell surface proteins, known to specifically label cell types present in the tumor micro-environment. This led us to construct a hierarchical decision tree summarizing the state of the art concerning associations between cell surface markers and identification of the different cell types present in the ccRCC micro-environment. Single-cell RNA-seq data generated from 11 treatment-naive resected ccRCC tumors were then collected from a previously published study  (Obradovic et al. , 2021). We completely reprocessed this dataset through pre-processing, quality control, normalization and cell clustering steps. In this study immune-and non-immune cells were separated by CD45-based fluorescence-activated cell sorting (FACS).
We then relied on our hierarchical decision tree of cell surface markers to precisely label the different cell types present in this data set. Based on our reanalysis of the cells retrieved from all patients we could define 21 distinct cell types, including 3 tumor types (Tumor cells, CA9 High Tumor cells, Cycling Tumor cells) (Fig. 1A, B ). We assessed these identifications by controlling gene expressions of all of the specific gene markers of immune and non-immune cells. In addition, the tumor cell populations were examined by the prediction of copy number variants (CNV). We observed that predicted tumor cells harbored a deletion of chromosome 3p region (VHL, PBRM1, BAP1, SET2D) and an amplification of the 5q region which is consistent with ccRCC  (Chen et al. , 2009 , Peña-Llopis et al.
Robust prediction of ccRCC tumor micro-environment by cell deconvolution Building on the work of extensive labeling of ccRCC cell populations and its cellular microenvironment, we assessed the relevance of using these single-cell RNA-seq data as reference profiles for cell deconvolution. Transcriptome-based cell deconvolution approaches make it possible to predict the respective proportions of cell types present in a tissue from its bulk transcriptome. We selected the CIBERSORTx and MuSiC methods  (Wang et al, 2019 , Newman et al, 2019) , considered by recent benchmarks to be among the most accurate (Avila Cobos et al, 2020), and evaluated their ability to predict the respective proportions of the 21 cell types referenced in our single-cell data. To perform this evaluation, we generated test datasets corresponding to 100 pseudo-bulk mixtures by sampling and mixing in controlled and known proportions the reference single-cell RNA-seq profiles.
In this way we avoid technical (i. e. different sequencing technology) or biological (i. e. same biological samples) bias between our test datasets and the reference single-cell data. We predicted the proportions of cell types for each pseudo-bulk mixture and assessed the robustness of the estimated fractions obtained by cell deconvolution by comparing them to the expected cell type fractions. We found high correlation values (Rho ranging from 0. 57 to 0. 99, two-sided Spearman correlation test p-value < 0. 0001) between predicted and expected cell type proportions associated with low RMSE (Root Mean Squared Error) dispersion values (Fig. 2A ). In addition, the analysis of the slope of the regression line by cell type between predicted and expected proportions showed good linearity often close to 1, illustrating the quantitative accuracy of the predicted cell type fractions (Fig.
Then, we used the ESTIMATE tool on these same 311 transcriptomes to predict a second tumor purity value. Finally, we collected from the supplementary data associated with the cohort a third tumor purity value generated by ABSOLUTE tool. We observed that the deconvolution-based estimated fractions of tumor cell were highly correlated to values predicted by ESTIMATE (two-sided Spearman correlation test p-value < 2. 2e-16) (Fig. 2B ) and ABSOLUTE (two-sided Spearman correlation test p-value=1. 7e-11) (Fig. 2C ). Moreover, the proportionality of the tumor purity predictions obtained by cell deconvolution and by ESTIMATE were very reproducible, illustrating the relevance of this new approach. The proportionality of tumor purities was less stringent between those obtained by cell deconvolution and those produced by ABSOLUTE.
This was however expected given the quite weak level of correlation already observed between tumor purities predicted from transcriptomic and genomic data (add reference). Finally, we extended our validation study of cell type fractions estimated by cell deconvolution using cell counts directly measured by immunofluorescence (IF) for a cohort of 19 ccRCC primary tissues. The ccRCC tissue cohort was profiled by Bulk RNA Barcoding (BRB)-seq to quantify gene expression and imaged by IF to directly enumerate populations of CD8-positive (CD8+), endothelial cells expressing CD34 (CD34+) and CD45-positive (CD45+) leukocytes in red (left panel), yellow (center panel) and green (right panel), respectively (Fig. 3A ). (Fig. 3A ). The bulk transcriptomic data of each sample was analyzed by cell deconvolution with CIBERTSORTx using the support of our 21 cell types single-cell reference data.
We therefore used it to predict by deconvolution the cell fractions for each transcriptome of the 311 tumor samples from patients included in the clinical trials CheckMate of the anti-PD-1 antibody Nivolumab or mTOR inhibitor Everolimus in advanced ccRCCs (CM-009 (NCT01358721), CM-010 (NCT01354431) and CM-025 (NCT01668784))  (Braun et al. , 2020). This cohort was selected because it is one of the few to be composed of samples obtained from the primary tumor site or from metastatic sites of patients associated with clinical metadata describing their responses to therapy (Clinical Benefit (CB), Progression Free Survival (PFS), Overall Survival (OS)). First, to control putative gender-related bias, we estimated cell fractions for primary and metastatic tissues separately and did not find a significant difference in TME composition between men and women after correction for multiple testing.
2020 and Ascierto et al. 2016  datasets obtained poor performances close to chance (AUC-ROCs ranging from 0. 5 to 0. 75) excepting for the YWHAE gene and IMPRESS score with AUC-ROCs of 0. 75 (Fig. 7A  third and fourth plots). To explain the discrepancy between the score performances for the ccRCC datasets, we assessed the correlations between both types of samples. We used an independent cohort of 15 matched primary and metastatic ccRCC samples  (Ho et al. , 2017)  and calculated the same scores based on RNA gene expression data. We observed correlations between primary-based and metastatic-based scores for only poor performers: IG, IMPRES and TLS2020 scores (two-sided Spearman correlation test p-value < 0. 05) (Fig. 7B ) suggesting that treatment response prediction based on gene expressions in ccRCC is not correlated between primary and metastasis samples from the same patient.
7C  and 7D ). Furthermore, to assess how prior therapy could affect patient response to ICB, we divided the Riaz et al. 2017 cohort into groups of patients who had previously been treated with Ipilimumab (anti-CTLA-4) (31 samples) and those naive for this treatment (33 samples). Of note, less than 2 IG genes out of 5 having been detected in melanoma and NSCLC transcriptomic datasets, they could not be used to assess the prediction of the IG signature. Interestingly, for pre-treatment melanoma samples, the TID-associated YWHAE gene expression greatly outperformed the existing ICB response scores for the Ipilimumab-naive patients (AUC-ROC of 0. 78) (Fig. 7C  first plot) or was the best performer, with the MIAS score, for the patients who progressed after Ipilimumab (AUC-ROC of 0. 89) (Fig. 7C.
, 2017; Petitprez et al. , 2020). Predicting the composition of the TME from bulk RNA-seq data by cell deconvolution approaches, especially those able to use single-cell RNA-seq data of cell types of interest as a reference, have proven to be robust and sensitive approaches  (Avila Cobos et al, 2020). Since the quality of single-cell transcriptome labeling used as a reference of these methods has a major impact on the predictive performance, we carried out a detailed bibliographic study to identify markers allowing to annotate the cell types expected in the ccRCC. We then evaluated markers and combinations of markers on single-cell RNA-seq data from ccRCC tissues and selected the most resolving ones resulting in the annotation of 21 cell types  (Obradovic et al. , 2021). This annotated single-cell dataset was then used as a reference to perform cell deconvolution by CIBERTSORTx on bulk transcriptomic data of advanced ccRCC samples from primary and metastatic sites collected from patients before ICB treatment  (Braun et al, 2020).
, 2021). Moreover, a high tumor purity was found in the High-TID subtype of bad ICB responder, suggesting that tumor fraction or purity were a key component driving ICB treatment response in metastatic ccRCC samples. Moreover, three genes were highlighted as markers of the TID subtypes: YWHAE, CXCR6 and BTF3. The YWHAE gene belongs to the 14-3-3 protein family and was previously found to be associated with advanced ovarian cancers, and poor patient prognosis mediated by the PI3K/AKT and MAPK pathways  (Li et al. , 2021). Interestingly, phenethyl isothiocyanate (PEITC) and fusicoccin molecules are found in the DrugBank database to target 14-3-3 proteins. Previous studies revealed various anti-cancer effects of PEITC molecules  (Gupta et al. , 2014) , leading to an inhibition of carcinogen metabolism in smokers with lung cancer  (Yuan et al.
Interestingly, the investigation of a variety of tumor sample types with different biopsy time-points (pre or post-treatment by anti-PD-1 antibody) from several cohorts revealed the discrepancy in the treatment outcome prediction performance of predictive scores based on gene expression data. We observed poorer performances for samples from the primary site of ccRCC compared to the samples from one metastatic site (CheckMate cohorts) with no correlation between both types of samples between scores calculated on matched primary and metastatic sites  (Ho et al. dataset). Also, the prediction performances were not equivalent between pre or post-treatment samples or samples recruited or not after the failure of one previous anti-CTLA-4 immunotherapy in melanoma samples which is consistent with a previous study on patients with melanoma treated with Nivolumab or Pembrolizumab showing that previous anti-CTLA-4 exposure was associated with performance differences for treatment outcome prediction  (Liu et al.
Acknowledgements The study was supported by the  KATY  project, which has received funding from the  European Union 's  Horizon 2020 research and innovation program  under grant agreement No  101017453  and by the  CANVAS  project, which has received funding from the  Horizon Europe twinning program  under grant agreement No  101079510. Most of the computations presented in this paper were performed using the GRICAD infrastructure (https://gricad. univ-grenoble-alpes. fr), which is supported by  Grenoble research communities.
second plot), and was among the top predictors in the Du et al. cohort (AUC-ROC of 0. 94) (Fig. 7C third plot). Regarding PBMCs samples from the Trefny et al. 2019 dataset, the expression levels of YWHAE and PDCD1 genes were the best performers (AUC-ROC of 0. 84) (Fig. 7C fourth plot). The average AUC-ROC values highlighted the YWHAE gene as the best marker of ICB treatment response based on pre-treatment biopsies (AUC-ROC of 0. 85) (Fig. 7C fifth plot). We continued our investigation of patients with melanoma or NSCLC using samples obtained during their treatment with Nivolumab. For the Riaz et al. 2017 melanoma cohort, we observed an overall improvement in the performance of the ICB response scores for the Ipilimumab-naive samples compared to those obtained from the biopsies taken before Nivolumab treatment (Fig.
7C and 7D first plots). The best predictive power was obtained by far with the YWHAE gene for Ipilimumab-naive samples from the Riaz et al. 2017 dataset (AUC-ROC of 0. 98) (Fig. 7D first plot) and for that of Du et al. (AUC-ROC of 1) (Fig. 7D third plot). Regarding the melanoma samples from Riaz et al. 2017 taken from patients who progressed after treatment with Ipilimumab, we observed, unlike the Ipilimumab-naive patients, rather an overall reduction in the quality of the predictions with a performance for YWHAE (AUC-ROC of 0. 73) slightly below that of the MIAS score (AUC-ROC of 0. 8) (Fig. 7D second plot). Finally, for Trefny et al. PBMCs samples from the blood of patients with NSCLC, the performances of the ICB response scores were closed to those obtained from the samples taken from the patients before treatment (Fig 7D fourth plot).
MILP. The MILP heuristic  [12]  presents a Mixed Integer Linear Program (MILP) to deal with the same tri-criteria problem as in this paper (in the variant which considers task-level DVFS giving rise to the RAFTM-TL linear program; the other variants being less general). Hence, this is the closest related work to our approach. However, MILP does not take communication costs into account and is limited by design to have at most two replicas per task; hence, it cannot match high reliability thresholds that require three or more replicas. We first comment on the objective function of MILP. Then we show how to modify MILP to obtain a new version, called ExtMILP, which does not have its limitations. The objective of MILP is to minimize the energy consumption. The formula used to estimate the energy consumption (Equation (10) of  [12] ) assumes that if a task has two replicas, then both are fully executed.
Acknowledgments The authors would like to thank the reviewers and editors for their comments and suggestions, which greatly helped improve the final version of the paper. This work was supported in part by the  National Key Research and Development  ( 2019YFA0706404 ), in part by the  National Nature Science Foundation of China  ( 62202168 ,  61972150 ),  Shanghai Sailing Program   21YF1411100 , and a  JoRISS grant  from  ENS de Lyon and ECNU. Corresponding authors:  Li Han (hanli@sei. ecnu. edu. cn) from the  School of Software Engineering, East China Normal University  and  Jing Liu ( jliu@sei. ecnu. edu. cn) from  Shanghai Key Lab of Trustworthy Computing, East China Normal University  and  Frédéric Vivien ( f rederic. vivien@inria. f r) from  ENS Lyon.
We develop an intuitive, Click-and-Drag interface to elicit continuous belief distributions of any shape. We test this interface against the state of the art in the experimental literature -a text-based interface and multiple sliders -and in the online forecasting industry -a distribution-manipulation interface similar to the one used at Metaculus, a crowd-forecasting website. By means of a pre-registered experiment on Amazon Mechanical Turk we collect quantitative data on the convergence speed and accuracy of reported beliefs in a series of induced-value scenarios varying by granularity, shape, and time constraints. We also collect subjective data on ease of use, frustration and understanding. Results show that the click-and-drag interface outperforms all others by accuracy and speed, and is self-reported as being more intuitive and less frustrating than other interfaces, confirming our pre-registered hypothesis.
Besides pre-registration, we report that the click-and-drag interface generates the least drop-out rate from the task, and scores best in a sentiment analysis of an open-ended general question. Further, we use the interfaces to collect homegrown beliefs on temperature in New York City in 2022 and 2042. On average, all subjects overshoot the real temperature for 2022 by about 2°F, and all anticipate further global warming in the order of 2. 3°F; these forecasts are by and large not impacted by the interface used to elicit them. We provide a free and open source, ready to use oTree and Qualtrics plugin of our click-and-drag and all other tested interfaces available at https://beliefelicitation. github. io/.
Drawing a curve is an intuitive exercise, and it effortlessly scales to any different number of bins by simply discretizing the continuous curve. To make this curve-drawing intuitive, we built on top of standard JavaScript libraries. In Click-and-Drag, subjects click to create a point, to which the curve is linearly attached. Points can be added, moved around, and deleted by simply clicking and dragging them. Figure  1  shows a screenshot of the interface in use; an interactive demo of the interface is available at https://beliefelicitation. github. io/. Under the hood, the application uses the Otree platform (Python/Django framework, Chen et al. (  2016 )). The user interface is developed using the JavaScript libraries Highcharts. js and jQuery. The interface is inspired by the Highcharts demo tools "Click to add a point" and "Basic column".
Subjects have to input a number for each of the discrete bins of the chosen support. The sum of the inputted numbers must be equal to 1 (or, equivalently, 100). This interface has been the one most largely used in earlier experiments for its technical simplicity. It is usually limited at 2 to 5 bins, and only seldom used beyond this threshold for practical concerns. In our implementation, the normalization is carried out by the software and no "sum to 100" constraint is imposed on the subjects. One example of the use of the text-based method can be found in the Survey of Consumer Expectations of New York (FRBNY) in the United States where, inspired by Manski  (Manski, 2004; Dominitz and Manski, 2004; De Bruin et al. , 2011)  the New York Federal Reserve decided to ask experts the probability that future inflation would lie between two certain percentage levels.