dataesr/scientific-paragraphs-categorization huggingface dataset:
Data availability by modern sequencing technologies represents a major challenge in oncological survival analysis, as the increasing amount of molecular data hampers the generation of models that are both accurate and interpretable. To tackle this problem, this work evaluates the introduction of graph centrality measures in classical sparse survival models such as the elastic net. We explore the use of network information as part of the regularization applied to the inverse problem, obtained both by external knowledge on the features evaluated and the data themselves. A sparse solution is obtained either promoting features that are isolated from the network or, alternatively, hubs, i. e. , features that are highly connected within the network. We show that introducing the degree information of the features when inferring survival models consistently improves the model predictive performance in breast invasive carcinoma (BRCA) transcriptomic TCGA data while enhancing model interpretability.
DegreeCox enables the incorporation of node information by favouring particularly relevant genes, e. g. extracted from available gene co-expression networks or correlation/covariance networks extracted from the data themselves. This work evaluates different network-based strategies for the generation of survival models for the breast invasive carcinoma (BRCA) transcriptomic data available from The Cancer Genome Atlas (TCGA) Data Portal. Two strategies are evaluated for the construction of the network-based penalty term: i) data-based weights attributed to each node/gene (correlation, covariance and Bayesian networks extracted from the data), with lower penalties being attributed either to hub genes, i. e. highly connected genes, or weakly connected or isolated genes, herein called orphan genes; ii) gene weights derived from external information on a protein-protein interaction (PPI) network.
This threshold was experimentally adjusted from a set of 20 different cut-off values and tested using 10-fold cross validation. The results, however, were inferior to using independent information described below. In addition to these data-dependent methods, we used the String network for the human genome, which records functional protein-protein interactions that have been observed experimentally and estimated computationally  (Szklarczyk et al. 2017 ). The String network was downloaded from the project's web page 2  , and filtered to use the Homo sapiens interactions only and to exclude text-based scores. Each edge of the networks denotes the overall score between the pair of proteins. As each protein can be mapped to one or more genes, we used the Ensembl genome database  (Ensembl 2017)  to determine this mapping and produce a gene network.
glmSparseNet R-package An R-package called glmSparseNet was developed to the methods described in this paper. This package depends on the coordinate descent implementation of the glmnet R-package  (Friedman, Hastie, and Tibshirani 2010)  to minimize the negative partial log-likelihood problem for the survival models and allows to define the weights calculated for both HubCox and OrphanCox. It also implements efficient methods to calculate the correlation and covariance networks over large data sets by paralleling the work and having a low memory footprint. This R-package implements not only Cox Proportional hazard models, but also every other distribution family supported by glmnet, namely gaussian, poisson, binomial, multinomial, cox, and mgaussian. It adds two new main functions called glmSparseNet and cv.
glmSparseNet that extend both model inference and model selection via cross-validation with network-based regularization. These functions are very flexible and allow to transform the penalty weights after the centrality metric is calculated, thus enabling to change how the regularization is affected. To make things easier for the users, we made available a function, called glmHub, that penalizes weakly connected nodes in the network, and another called glmOrphan that penalizes hubs. An additional R-package is available with the necessary scripts and functions to reproduce the results of the present work. It is available to download at github 3  under license GNU GPL version 3. This support package requires a modified glmnet R-package that is up to three times faster at calculating crossvalidation for the data sets used in this work.
Dataset The Breast Invasive Carcinoma (BRCA) data set from The Cancer Genome Atlas (TCGA) was used to test the methods described in this manuscript, in particular, we used gene expression levels and clinical data from patient followup consultations that indicated whether the patient was deceased, our event of interest. The BRCA data comprise information from 1047 individuals and 55867 genes, from which 19868 are protein coding genes that are used in the analysis. The expression levels were already processed by TCGA using Fragments Per Kilobase Million (FPKM) expression units. The brca. data R package distributes the clinical and expression levels for BRCA and is available for download at github 5. Before survival analysis, a preprocessing step was applied to the BRCA dataset, including: the exclusion of genes that did not have any variation in the expression levels and non-coding genes identified by the Ensembl genome database  (Ensembl 2017 ) and the Consensus CDS (CCDS 2017) projects; the removal of individuals with missing follow-up time for censored observations and individuals with negative follow-up time for both censored and non-censored observations; the removal of duplicate samples per individual by keeping the first sample; the retention of samples uniquely from primary solid tumours; the normalization of the FPKM values log2 and Z-score transformations.
Results and Discussion The methods described in section 3 were compared with a classical approach using elastic net and one of the largest cancer dataset available from TCGA, the BRCA. We used the String network in the penalization of HubCox and Or-phanCox as the network is independent from the data set and avoids introducing bias in the models. These numeric experiments were performed by running a 10-fold cross validation fitting the whole data set using several α parameters and (1) Elastic net (EN); (2) HubCox; (3) OrphanCox. The value of 0. 2 for α was chosen as it minimizes the two goodness-of-fit measures (Log-rank test and c-index) described below over the different parameters that were tested, and it provides a good dimensionality reduction without degrading the performance of the model.
Since different model selection strategies may lead to a distinct optimal number of selected features, we started by running a 10-fold cross-validation for each model to chose the best λ parameter for each one, by selecting the one with the minimum log-likelihood deviation. We then tuned the λ parameter to force similar model sizes in the other two methods in order to promote an unbiased comparison with similar model sizes. This results in having three λ parameters for each of EN, HubCox and OrphanCox, the optimal one calculated via cross-validation, herein called the base model, and two others that were tuned to match the same number of selected variables called complementary. Figure  4  shows the diagram of the workflow. The dataset was divided into a training and a test data set, comprising 80% and 20% of the data respectively.
Nevertheless, there are 23 genes that were commonly selected, such as OR5AU1 (protein part of the olfactory receptor proteins) and CEL (a glycoprotein secreted from the lactating mammary gland into human milk), that have degrees of 325 and 251, respectively (network where the highest degree is 10394 and median degree value is 220). Despite these genes being heavily penalized in HubCox, they are still selected as their inclusion outweighs the network penalty. By further looking into the genes selected, we used the model with the same λ, but now trained with the full BRCA dataset to understand which genes are selected by the network-based methods and not by EN. In part of this analysis we used the Cancer Hallmarks Analytics Tool  (Hanahan and Weinberg 2011) , denoted as CHAT, to query the selected genes against a database of known functions related to cancer.
The CHAT database was populated based on a text-mining analysis of 26 million PubMed abstracts. Figure  7  shows a heatmap of the hallmarks counts per gene, per model, with some genes appearing few times (in light blue) and some highlighted (in dark blue). Looking at the highlighted ones, there are some very well known genes related to cancer, such as TP53  (Walsh et al. 2006) , NRAS  (Cimino et al. 2012)  and KRAS  (Paranjape et al. 2011). However, all of these were selected by HubCox, while Orphan-Cox selected genes that seemed less studied and discussed. Ultimately, this is directly associated with the network and the hypothesis being tested with the regularizer, as known genes are bounded to have a more central role in the network, especially if they have been studied as associated with multiple cancer types.
Conclusion The initial hypothesis of this work was that the enrichment of survival and generalized linear models with a priori network information would improve the models' predictive quality, stability and interpretability. Our results show that the models' goodness-of-fit metrics is comparable with elastic net, and seems to have a better performance in predicting long term survival individuals. Moreover, we have achieved a good stability of selected variables in resampling tests, which is crucial for models' interpretability. Both the promotion of hubs or low Figure  7 : Counts of hallmarks association in CHAT database, the darker the cell, the higher the count of hallmarks for a particular gene. Only showing genes that do not intercept with EN. connected nodes have their merits, and while it would be interesting to have a definitive model for transcriptomic breast cancer data, the results obtained from both methods are suitable for different purposes.
Acknowledgments The authors thank  Niko Beerenwinkel  and  Wolfgang Huber  for fruitful discussions. work partially supported by the  European Union   Horizon 2020 research and innovation program  under grant agreement No. 633974  ( SOUND  project), and by national funds through  Fundação para a Ciência e a Tecnologia (FCT)  with reference  UID/CEC/50021/2013 (INESC-ID ), through IDMEC, under LAETA, project  UID/EMS/50022/2013 , and  PERSEIDS (PTDC/EMS-SIS/0642/2014 ),  SFRH/BD/97415/2013  and  IF/00653/2012.
This paper presents two large newly constructed datasets of moderated news comments from two highly popular online news portals in the respective countries: the Slovene RTV MCC and the Croatian 24sata. The datasets are analyzed by performing manual annotation of the types of the content which have been deleted by moderators and by investigating deletion trends among users and threads. Next, initial experiments on automatically detecting the deleted content in the datasets are presented. Both datasets are published in encrypted form, to enable others to perform experiments on detecting content to be deleted without revealing potentially inappropriate content. Finally, the baseline classification models trained on the non-encrypted datasets are disseminated as well to enable real-world use.
Introduction With the rapid rise of user-generated content, there is increased pressure to manage inappropriate online content with (semi)automated methods. The research community is by now well aware of the multiple faces of inappropriateness in on-line communication, which preclude the use of simple vocabulary-based approaches, and are therefore turning to more robust machine learning methods  (Pavlopoulos et al. , 2017). These, however, require training data. Currently available datasets of inappropriate on-line communication are primarily datasets of English, such as a Twitter dataset annotated for racist and sexist hate speech  (Waseem and Hovy, 2016)   1  , the Wikimedia Toxicity Data Set  (Wulczyn et al. , 2017)   2  , the Hate Speech Identifica-tion dataset containing tweets annotated as hate speech, offensive language, or neither  (Davidson et al.
, 2017)   3  , and the SFU Opinion and Comment Corpus consisting of online opinion articles and their comments annotated for toxicity  4. Datasets in other languages have recently also started to emerge, with a German Twitter dataset focused on the topic of refugees in Germany  (Ross et al. , 2017)   5  and a Greek Sport News Comment dataset containing moderation metadata  (Pavlopoulos et al. , 2017)   6. In this paper we present two new and large datasets of news comments, one in Slovene, and one in Croatian. Apart from the texts, they also contain various metadata, the primary being whether the comment was removed by the site administrators. Given the sensitivity of the content, we publish the datasets in full-text form, but with user metadata semi-anonymised and the comment content encrypted via a simple character replacement method using a random, undisclosed bijective mapping, similar to the encryption method applied to the Gazzetta Greek Sport News Comments dataset 7  introduced in  Pavlopoulos et al.
(2017). The two datasets presented in this paper are aimed at further enriching the landscape of datasets on inappropriate online communication overall, but especially the dimension of multilinguality and multiculturality. The contributions of this paper are the following: (1) we introduce two new datasets annotated for content inappropriateness, (2) we perform a basic analysis of the type of the deleted content, (3) we investigate whether the deleted content is more dependent on specific users, threads or locations in a thread, (4) we build baseline predictive models on these datasets and (5) we publish the full but semi-anonymised and encrypted datasets, as well as the models built on the non-encrypted data ready to be used in real-life scenarios.
Dataset description This section gives a description of the two datasets, which we obtained from two different sources from different countries and in different languages. Both datasets are comprehensive in the sense that they contain all the comments from the given time period. Their main value in the context of studying inappropriate content lies in the fact that they also contain all the comments that were deleted by the moderators of the two sites. The Slovenian MMC dataset contains comments on news articles published on the MMC RTV web portal 8  , the on-line portal of the Slovenian national radio and television. The dataset comprises all the comments from the beginning of 2010 until the end of 2017, i. e. eight years' worth of content, including deleted comments. The portal is monitored by moderators who delete comments that contain hate speech but also those that are not relevant for the thread or are spam by advertisers.
We obtained the dataset as a CSV file, where we deleted comments with formatting errors, removed illegal UTF-8 characters and remnants of formatting, and then converted the dataset into XML. Apart from the text itself, each comment contains the following metadata: comment ID; ID of the news article that is commented (note, however, that we did not receive the news articles themselves due to copyright limitations); user ID; time stamp; whether the comment was deleted or not; and the number of up-and down-votes. The Croatian STY dataset contains comments on articles from the news portal 24sata 9  which is owned by Styria Media International. They comments in the dataset span from 2007-09-12 to 2017-07-21, i. e. almost ten years of content. Until 2016 the portal was monitored by one moderator, with the last two years of content being moderated by two moderators.
Both hate speech and spam are deleted, and the respective users are banned for an amount of time depending on the frequency of their misbehavior. We received the dataset as a SQL database dump, where we, similarly to MMC, cleaned the file and converted it to a similar XML as the MMC one. Here, the metadata was somewhat different, comprising: comment ID; ID of the news article that is commented (again, we did not receive the news articles themselves); where applicable, ID of comment that is being replied to; the ID of the thread to which the comment belongs; the user ID; the user name; time stamp; whether the comment was deleted or not; and the number of replies to the comment.
The datasets in numbers Table  1  gives the sizes of the two datasets in users, texts (comments) and words, split into retained and deleted comments, and overall. As can be seen, both datasets are substantial, having together almost 25 million comments, and over 700 million words. The two datasets have a similar size per year, but given that the STY dataset has a longer time span it is also larger, with over 407 million words, against 325 million words in MMC. Interestingly, given their comparable size, the STY dataset has many more comments (17 million, as opposed to only 7. 6 million of MMC) as well as many more users (185 thousand as against 42 thousand of MMC). On the other hand, MMC users write significantly longer texts. As for the deleted comments, we first note that for a user to be classified into either of the Yes/No deleted row, it suffices that one of their comments has been deleted (or not), so the two percentages do not sum to 100%.
Types of inappropriate content To gain more insight into the nature of the deleted comments, we manually classified 100 random deleted comments from each datasets into the 9 categories proposed by  Pavlopoulos et al. (2017) : calumniation, discrimination, disrespect, hooliganism, insult, irony, swearing, threat and other. Both samples were annotated by the same annotator. Where required, multiple labels were assigned to a comment. As shown in Table  2 , there are many differences between the two datasets. In the MMC sample, the most frequently represented category is disrespect (37) while swearing is the least frequent (3). Only 5 of the 100 comments were annotated with double labels. In the STY sample, on the other hand, 17 received a double and 1 a triple label (most frequent combinations being insult and swearing).
Dataset Analysis This section presents a basic analysis of deleted vs. retained content in both datasets. We analyze (1) the distribution of deleted vs. retained content through the years, (2) the distribution of deleted content among users, (3) the distribution of deleted content in threads, and (4) the distribution of the relative positions of the deleted comments in a thread. We compare the distributions (  2 )-(  4 ) with their random counterparts to see whether there is a dependence of comment deletion on these three phenomena: user, thread, and location in a thread.
Distribution through time Figure  1  shows the distribution of the deleted and retained comments in each dataset throughout the publication years. The trends in the two datasets are quite different. While the STY dataset has an obvious increase in the number of comments throughout the years, the number of comments in the MMC dataset is rather stable. The number of the deleted comments throughout the years is even more different in the two datasets. Most of the deleted content in the STY dataset is from 2016 and 2017, which indicates an obvious change in the policy of content deletion. 10 In the MMC dataset, on the other hand, the percentage of the filtered content is rather stable throughout the years, with only a slight increase through time. The observed difference can be followed back to the type of publishers: STY is a commercial publisher, freely modifying filtering rules, whereas MMC is a national broadcasting company and is as such required to have a much more elaborate and strict, as well as more stable code of conduct.
Distribution per user Figure  2  depicts the distribution of the percentage of comments deleted from each user, taking into account only users that published 10 or more comments to ensure a proper representation of the percentages on a histogram with 10 bins. The plot shows similar trends in both languages, with most users having 10% or less of their comments deleted. In both datasets we see an increase in the percentage of users having all their comments 10 The increase in the amount of deleted content can be followed back to the internal decision of the newspaper to make the moderation model more strict, resulting in greater identification of inappropriate content, but also an increase in the amount of inappropriate content aimed at the moderators. deleted. This phenomenon can be followed back to the practice of deleting users and all their corresponding content.
We hypothesize that this distribution is significantly different from a random one, i. e. , that there are users whose comments are deleted more often than by chance. Put in simpler terms, we assume that inappropriate comments are not a blunder that happens to everyone now and then but that there are consistently "non-conforming" users whose comments are deleted more often than those of other users. We test our hypothesis by applying the Kolmogorov-Smirnov non-parametric two-tailed test  (Massey, 1951)  of the equality of two distributions, with the null hypothesis that there is no difference between the observed and the expected random distribution. We calculate the expected, random distribution by calculating the probability of a comment to be deleted in a dataset and generating a dataset with the identical distribution of comments among users, calculating whether the comment is deleted via a random function with the deletion probability as estimated on the real dataset.
On the MMC test we obtain a statistic of 0. 274 with the p-value, i. e. , the probability that we might falsely reject the null hypothesis that the two distributions are identical being close to 0. 0. For the STY dataset we obtain a statistic of 0. 371, with a p-value being close to 0. 0. These results show that in each dataset some users' comments are being deleted more often than by chance. To measure to what amount each of the distributions are different to a random one, we calculate the Wasserstein distance  (Ramdas et al. , 2017)  between the observed and the random distributions. While we obtain a distance of 0. 068 for the MMC distribution, this distance for the STY distribution is 0. 032. 11 This inspection shows that in the Slovene MMC dataset the phenomenon of deleting content of specific users is more prominent than in the Croatian STY dataset.
Distribution per thread In this subsection we repeat the calculations performed in the previous subsection on user-focused distributions of deleted content, only that this time we utilize the thread structures which are available in both datasets. For this analysis, as well as the following one presented in Section 3. 4, we take into account only threads with at least 10 comments, again to ensure a proper representation on a 10-bin histogram. In Figure  3  we plot the probability distribution of the percentage of comments deleted in each thread, obtaining a similar, long-tailed distribution as with the user-focused distribution. As with the user distributions, the MMC dataset has a larger number of threads with slightly higher percentage of deleted content (higher than 10%), indicating a less random deletion process in the MMC dataset, i.
e. , that there are threads that have more content deleted than would be expected by chance. Similar to our previous calculations, we first calculate whether the obtained distributions are different from distributions obtained by randomly deleting comments in threads by applying the Kolmogorov-Smirnov test of the equality of two distributions. We then quantify the distance of the observed distributions to the random distributions via the Wasserstein distance. When applying the Kolmogorov-Smirnov test on the MMC dataset, we obtain a statistic of 0. 292 with a p-value close to 0. 0, while on the STY dataset the obtained statistic is 0. 333 with a pvalue also close to 0. 0. Based on this we can reject the null hypothesis that the random and the observed probability distributions are the same on both datasets.