dataesr/scientific-paragraphs-categorization huggingface dataset:
We then use these as features to train a classifier on our abuse detection task. We thoroughly assess our system on a dataset of user comments originating from a French Massively Multiplayer Online Game. We identify the most appropriate network extraction parameters and discuss the discriminative power of our features, relatively to their topological and temporal nature. Our method reaches an F -measure of 83. 89 when using the full feature set, improving on existing approaches. With a selection of the most discriminative features, we dramatically cut computing time while retaining most of the performance (82. 65).
In Section IV, we present our dataset, as well as the overall experimental setup for the classification task. We then provide a discussion and a qualitative study of the performance of our approach, with a focus on the contributions of the considered features. Because some of them are computed from information that is not yet available at the instant some messages are posted, we also examine the performances of the system based only on information available at the time (i. e. as a prediction task). Finally, we summarize our contributions in Section V and present some perspectives for this work.
They extract a corpus of tweets that is divided into two classes (obscene / not obscene) based on static rules. They then perform a log odds ratio analysis to detect the words favoring documents from the obscene class. Such an approach could be very useful in an online classification setting, but inherently requires a dataset where the number of samples in the obscene class is large. Still, they show that a list of words dynamically generated using that method contains 60% of new obscene words, and the process can be iterated over. Relatively to our problem of interest, the main limitation of this article is its focus on obscene words, which are just one specific type of abuse. In  [9] , Razavi et al. focus on a wider spectrum of types of abuse than the previously cited works, which they call inflammatory comments.
The proposed system outperforms CNN and other more mainstream classifiers. However, it is worth noticing that these tasks are slightly different, as the the Greek corpus is annotated for general moderation, whereas the English one focuses on personal attacks. It is worth noting that all these ML-based approaches perform better when a large dataset is available for training. However textbased approaches are usually language-dependent, which means that models have to be trained on a dataset of the specific language. This is usually not an issue when classifying English messages because of the wealth of publicly-available data, but is problematic in our case, since our messages come from low resource language communities. Content-based text classification usually makes for a good baseline. However, such methods have severe limitations.
First, abuse can be spread over a succession of messages. Some messages can even reference a shared history between two users. Second, it is very common for users to voluntarily obfuscate message content to work around badwords detection. Indeed, abusers can bypass automatic systems by making the abusive content difficult to detect: for instance, they can intentionally modify the spelling of a forbidden word. Hosseini et al. [13]  demonstrate such an attack against the Google Perspective API  2. Adversarial attacks based on word-level obfuscation are nothing new, and approaches exist to counter them. For instance, Lee et al. [14]  experiment with spam de-obfuscation using a Hidden Markov Model that incorporates lexical information. Such an approach yields good results for de-obfuscation, but it is computationally expensive and requires a dataset of obfuscated words for training.
Original Rank i = 1 Rank i = 2 Rank i = 3 4) Extracted Networks: Figure  8  shows a real-world example of the three conversational networks obtained by applying our extraction method to an abusive comment belonging to our dataset. They are obtained based on a context period of 200 messages, a sliding window of 10 messages, and are undirected. The isolates (disconnected vertices) present in the Before and After networks correspond to users present in the context period, but active only after or before the targeted message, respectively. The red vertex corresponds to the author of the targeted message, which we call the targeted user. One can see that the users involved in the conversation, as well as the location of the targeted user in this conversation, undergo some dramatic changes after the abuse.
IV. EXPERIMENTS This section describes our experimental setup and results regarding the automatic detection of abusive messages in chat logs. In Section IV-A, we present our dataset and the general architecture of our classification system. Because we expect some of our features to be redundant, we conduct a correlation study of our feature set in Section IV-B. We present general results and the effect of our various graph extraction parameters in Section IV-C. In Section IV-D, we investigate the temporal aspects of the system -specifically what happens when we train our models based on the features extracted from only one of the three graphs (Before, After of Full), or some of their combinations. We then examine the importance of weight and directionality in Section IV-E, before investigating the potential for computational optimization through feature selection in Section IV-F.
A. Experimental Setup We have access to a database of 4, 029, 343 messages that were exchanged by the users of the browser-based multi-player game SpaceOrigin, a French-language Massively Multiplayer Online Game. In this database, 779 messages have been flagged by one or more users as being abusive, and subsequently confirmed as abusive by the human game moderators: they constitute our Abuse class. Each message belongs to a unique communication channel. A total of 226 distinct users have authored these abusive messages. We further extract 2, 000 messages at random from the messages not confirmed as abusive, to constitute the Non-abuse class. Note that all the results we discuss in this article are relative to the Abuse class. We previously experimented with this dataset in  [17] ,  [3]. However, since then we have detected certain inconsistencies in the database, preventing us from retrieving the context of certain messages.
We cannot apply our classification method to them, so we discard them for the work presented here. Note that this concerns both classes. Moreover, our tests show that removing those samples does not significantly impact our previous performances. The resulting dataset is constituted of 1, 890 messages in the Non-Abuse class and 655 messages in the Abuse class. Figure  9  shows the distribution of abuse cases by user. It suggests that most abusive users need only a few warnings before mending their ways, but it also shows that some users are exceptional in the number of abuses they commit. Because of the relatively small dataset, we set up our experiment for a 10-fold cross-validation. We split the dataset into 10 same-sized parts containing the same ratio of abusive to non-abusive messages.
We use a 70%-train / 30%-test split, which means, for each run of the cross-validation, the train set is composed of 7 of those parts while the test set is composed of the remaining 3. We use Python-iGraph  [54]  to extract the conversational networks and process the graph-based features for each message. As a classifier, we use an SVM (Support Vector Machine), implemented in the Sklearn toolkit  [55]  under the name SVC (C-Support Vector Classification). We mainly experiment with 4 different sets of features: Full, Before, After and All. For a given message, Before, After and Full correspond to all the topological measures computed for the Before, After and Full graphs, respectively. All is the union of all three sets, i. e. it includes all topological measures for all 3 graphs. In the remainder of this section, we occasionally provide computational time requirements.
This means that our assumption regarding that large context periods would bring mainly additional noise is incorrect, because, on the contrary, they convey more relevant information concerning the classification task. We manually investigate a sample of our dataset to understand this trend. From reading a number of conversation logs, it first appears that conversation boundaries are not well defined, and so there is no typical duration for a conversation: this can explain the absence of a plateau in the plot. Furthermore, we based our assumptions with regard to performance on the idea that an abusive message has a specific impact on what happens after it is posted. Specifically, the conversation would show markers of normality before the message occurs and quickly devolves after that. As it turns out, in conversations where an abusive message is found, the author of the abusive message usually has been around for a while, and the message that is actually flagged and confirmed as abusive is not his first suspicious message.
We first apply this recursive feature elimination process to the All feature set in order to identify the overall best features, then do the same with the Before, After, Full feature sets, for comparison purposes. Table II presents the performances and computational time costs measured for each of these complete feature sets, as well as for their respective Top Feature subsets (TF). It appears that, for all 4 feature sets, using the top features during the classification allows reducing the runtime by up to 4 orders of magnitude, while retaining at least 97% of the F -measure value, which is very interesting from an application perspective. It means that the longer features to process do not bring more discriminative power than the shorter ones, regarding the classification task at hand (at least for our dataset).
This means that our networks generally do not have any community structure, which explains why the related features are not discriminative (at least for this dataset). We also have identified and studied the clusters corresponding to the top features of the Before (Table  IV ), After (Table  V ), and Full (Table  VI ) feature sets. For matters of space, we do not present them in detail, though, and only discuss our most interesting observations. Certain clusters identified for the All feature set also appear for the other sets: those focusing on the considered type of graph, i. e. Clusters 110 and 118 for Before; 10, 41, 49 and 177 for After, and 9, 10, 41 and 49 for Full. Some of the missing clusters are replaced by semantically close and relatively correlated clusters. For instance, Before has a cluster containing exactly the same measure variants as Cluster 49 (Vertex Count and Betweenness), but for the Before graph.
Background This study aims to explore the differences in the management of diabetes outcomes and prevalence among black Africans, white and South Asian populations living in western countries from published evidence. This review incorporates findings from differences in diabetes management outcome among black Africans compared to white and South Asian populations. Methods A systematic search of major electronic databases with peer review publications was conducted. PubMed, CIHNAL, Medline, Web of Science, Scopus, and Science direct databases were searched from 2007-2018. Relevant journals and citations from references were searched for selection in the review. Data were analysed to understand differences in diabetes outcomes among these populations.
Study design and data sources This review was carried out based on Search, Ap-praisaL, Synthesis and Analysis (SALSA) systematic review framework  (26) (27) (28). Major databases were searched including: PubMed, CIHNAL, Medline, Web of Science, Scopus, and Science Direct were searched because they offer a good selection of publications that are related to this review question. The main search terms for this work were 'Diabetes', 'Africa', 'ethnic minority', 'management' and their alternative synonyms were entered into the databases searched (Table  1 ). These search terms were combined using Boolean operators to allow full retrieval of all articles that are related to the focus of this review. The protocol for this review has been registered on PROSPERO  (CRD 42018088311).
Data extraction Following the quality assessment of each paper, data were extracted using a data extraction tool. There are various types of data extraction tools that can be used in a review including paper, spreadsheets, email and web-based survey. The type used depends on the availability of funding, time for review and number of reviewers on the project  (33). Data was extracted using Microsoft Excel (Microsoft Inc. , Seattle WA, USA) spreadsheet by FA. Data were extracted on the aim of the study, the type of research methodology used, demographic information of the participants, key findings and conclusions from the study (Table  4 ).
Data analysis Understandings of ethnicity within this literature review were based on the original authors' concepts. The term "Whites" was used to refer to people from the European or Caucasian origin. South Asia included people originating from Indian, Pakistan, Bangladesh, Maldives and Sri Lanka, while Black population was used to refer to people originating from Africa and the Caribbean. Studies review reported establishing ethnicity based on self-reported origin, country of birth or as indicated in the hospital records in the studies reviewed. JOGHR 2019 Vol 3 • e2019020 Articles reported recruiting black populations with no further clarification on the specific black group. Further information on specific group of black population recruited for study is needed due to several groups that make up the black population.
Obtaining training data for Question Answering (QA) is time-consuming and resourceintensive, and existing QA datasets are only available for limited domains and languages. In this work, we explore to what extent high quality training data is actually required for Extractive QA, and investigate the possibility of unsupervised Extractive QA. We approach this problem by first learning to generate context, question and answer triples in an unsupervised manner, which we then use to synthesize Extractive QA training data automatically. To generate such triples, we first sample random context paragraphs from a large corpus of documents and then random noun phrases or named entity mentions from these paragraphs as answers. Next we convert answers in context to "fill-in-the-blank" cloze questions and finally translate them into natural questions.
Introduction Extractive Question Answering (EQA) is the task of answering questions given a context document under the assumption that answers are spans of tokens within the given document. There has been substantial progress in this task in English. For SQuAD  (Rajpurkar et al. , 2016) , a common EQA benchmark dataset, current models beat human Figure  1 : A schematic of our approach. The right side (dotted arrows) represents traditional EQA. We introduce unsupervised data generation (left side, solid arrows), which we use to train standard EQA models performance; For SQuAD 2. 0  (Rajpurkar et al. , 2018) , ensembles based on BERT  (Devlin et al. , 2018)  now match human performance. Even for the recently introduced Natural Questions corpus  (Kwiatkowski et al. , 2019) , human performance is already in reach.
1  2 Unsupervised Extractive QA We consider extractive QA where we are given a question q and a context paragraph c and need to provide an answer a = (b, e) with beginning b and end e character indices in c. Figure  1  (right-hand side) shows a schematic representation of this task. We propose to address unsupervised QA in a two stage approach. We first develop a generative model p(q, a, c) using no (QA) supervision, and then train a discriminative model p r (a|q, c) using p as training data generator. The generator p(q, a, c) = p(c)p(a|c)p(q|a, c) will generate data in a "reverse direction", first sampling a context via p(c), then an answer within the context via p(a|c) and finally a question for the answer and context via p(q|a, c). In the following we present variants of these components.
Question Answering Extractive Question Answering amounts to finding the best answer a given question q and context c. We have at least two ways to achieve this using our generative model: Training a separate QA system The generator is a source of training data for any QA architecture at our disposal. Whilst the data we generate is unlikely to match the quality of real QA data, we hope QA models will learn basic QA behaviours. Using Posterior Another way to extract the answer is to find a with the highest posterior p(a|c, q). Assuming uniform answer probabilities conditioned on context p(a|c), this amounts to calculating arg max a p(q|a , c) by testing how likely each possible candidate answer could have generated the question, a similar method to the supervised approach of  Lewis and Fan (2019).
Unsupervised QA Experiments For the synthetic dataset training method, we consider two QA models: finetuning BERT  (Devlin et al. , 2018)  and BiDAF + Self Attention  (Clark and Gardner, 2017). 5 For the posterior maximisation method, we extract cloze questions from both sentences and sub-clauses, and use the NMT models to estimate p(q|c, a). We evaluate using the standard Exact Match (EM) and F1 metrics. As we cannot assume access to a development dataset when training unsupervised models, the QA model training is halted when QA performance on a held-out set of synthetic QA data plateaus. We do, however, use the SQuAD development set to assess which model components are Sliding window  (Rajpurkar et al. , 2016)  13. 0 20. 0 Context-only  (Kaushik and Lipton, 2018)  10. 9 14. 8 Random  (Rajpurkar et al.
Ablation Studies and Analysis To understand the different contributions to the performance, we undertake an ablation study. All ablations are evaluated using the SQUAD development set. We ablate using BERT-Base and BiDAF+SA, and our best performing setup is then used to fine-tune a final BERT-Large model, which is the model in Table  1. All experiments with BERT-Base were repeated with 3 seeds to account for some instability encountered in training; we report mean results. Results are shown in Table  2 , and observations and aggregated trends are highlighted below. Posterior Maximisation vs. Training on generated data Comparing Posterior Maximisation with BERT-Base and BiDAF+SA columns in Table 2 shows that training QA models is more effective than maximising question likelihood. As shown later, this could partly be attributed to QA models being able to generalise answer spans, returning answers at test-time that are not always named entity mentions.
Finally, semantic drift is an issue, with generated questions being relatively coherent but often having different answers to the inputted cloze questions (example 2). We can estimate the quality and grammaticality of generated questions by using the well-formed question dataset of  Faruqui and Das (2018). This dataset consists of search engine queries annotated with whether the query is a well-formed question or not. We train a classifier on this task, and then measure how many questions are classified as "well-formed" for our question generation methods. Full details are given in Appendix A. 5. We find that 68% of questions generated by UNMT model are classified as well-formed, compared to 75. 6% for the rule-based system and 92. 3% for SQuAD questions. We also note that using language model pretraining improves the quality of questions generated by UNMT model, with 78.
Few-Shot Question Answering Finally, we consider a few-shot learning task with very limited numbers of labelled training examples. We follow the methodology of  Dhingra et al. (2018)  and  Yang et al. (2017) , training on a small number of training examples and using a development set for early stopping. We use the splits made  available by  Dhingra et al. (2018) , but switch the development and test splits, so that the test split has n-way annotated answers. We first pretrain a BERT-large QA model using our best configuration from Section 3, then fine-tune with a small amount of SQuAD training data. We compare this to our re-implementation of  Dhingra et al. (2018) , and training the QA model directly on the available data without unsupervised QA pretraining. Figure  5  shows performance for progressively larger amounts of training data.
Yadav et al. (2019)  propose an unsupervised alignment method for multiple choice question answering. Semi-supervised  QA Yang et al. (2017)  train a QA model and also generate new questions for greater data efficiency, but require labelled data. Dhingra et al. (2018)  simplify the approach and remove the supervised requirement for question generation, but do not target unsupervised QA or attempt to generate natural questions. They also make stronger assumptions about the text used for question generation and require Wikipedia summary paragraphs. Wang et al. (2018)  consider semi-supervised cloze QA,  Chen et al. (2018)  use semi-supervision to improve semantic parsing on WebQuestions  (Berant et al. , 2013) , and  Lei et al. (2016)  leverage semi-supervision for question similarity modelling.
Discussion It is worth noting that to attain our best performance, we require the use of both an NER system, indirectly using labelled data from OntoNotes 5, and a constituency parser for extracting subclauses, trained on the Penn Treebank  (Marcus et al. , 1994). 7 Moreover, a language-specific wh* heuristic was used for training the best performing NMT models. This limits the applicability and flexibility of our best-performing approach to domains and languages that already enjoy extensive linguistic resources (named entity recognition and treebank datasets), as well as requiring some human engineering to define new heuristics. Nevertheless, our approach is unsupervised from the perspective of requiring no labelled (question, answer) or (question, context) pairs, which are usually the most challenging aspects of annotating large-scale QA training datasets.
Conclusion In this work, we explore whether it is possible to to learn extractive QA behaviour without the use of labelled QA data. We find that it is indeed possible, surpassing simple supervised systems, and strongly outperforming other approaches that do not use labelled data, achieving 56. 4% F1 on the popular SQuAD dataset, and 64. 5% F1 on the subset where the answer is a named entity mention. However, we note that whilst our results are encouraging on this relatively simple QA task, further work is required to handle more challenging QA elements and to reduce our reliance on linguistic resources and heuristics. Table  6 : Fraction of questions classified as "wellformed" by a classifier trained on the dataset of  Faruqui and Das (2018)  for different question generation models. * indicates MLM pretraining was applied before UNMT training search engine queries, annotated with whether the query is a well-formed question.
A. 6 Language Model Pretraining We experimented with Masked Language Model (MLM) pretraining of the translation models, p s→t (q|q ) and p t→s (q |q). We use the XLM implementation (https://github. com/facebookresearch/XLM) and use default hyperparameters for both MLM pretraining and and unsupervised NMT fine-tuning. The UNMT encoder is initialized with the MLM model's parameters, and the decoder is randomly initialized. We find translated questions to be qualitatively more fluent and abstractive than the those from the models used in the main paper. Table  6  supports this observation, demonstrating that questions produced by models with MLM pretraining are classified as well-formed 10. 5% more often than those without pretraining, surpassing the rule-based question generator of  Heilman and Smith (2010).