dataesr/scientific-paragraphs-categorization huggingface dataset:
Introduction BIOASQ 1 challenge is a large-scale biomedical semantic indexing and question answering task  (Tsatsaronis et al. , 2015)  which has been successful for 5 years. The challenge proposes several tasks using Biomedical data. One of the tasks focuses on Biomedical question answering (Task B Phase B -we further refer it as B) where the goal is to extract answers for a given question from relevant snippets. Several teams have participated actively, and a noticeable aspect is that the results of the task B are much lower compared to open domain QA evaluations, as in SQUAD 2. Some reasons can be the low dataset size and the format of the answers provided by the organizers. Bioasq provides only certain answer forms in the gold standard data and not all the variants of the answers in the given snippets.
In this paper, we study the influence of enriching the training data by manually annotated variants of gold standard answers on the evaluation performance. We show the impact of the enriched data by experimenting on 5B and 6B training datasets. Our method outperforms the best-1 http://bioasq. org/ 2 https://rajpurkar. github. io/SQuAD-explorer/ performing systems from Bioasq 5B by 7. 3% on strict accuracy and 18% on lenient accuracy.
Document retrieval (Task B Phase A) 3. Answer extraction by answer re-ranking on the candidate answers generated in the previous phases, done in a supervised learning manner. In the open domain, deep learning models are extensively used in machine reading task. Datasets such as MS Marco by  (Nguyen et al. , 2016) , SQUAD by  (Rajpurkar et al. , 2016)  and Wikireading by  (Hewlett et al. , 2016)  have made it easier for deep learning models to perform better on machine reading task. One of the first attempts to use deep learning algorithms for the Bioasq task was reported in BIOASQ 5 by  (Wiese et al. , 2017b)  where the dataset was adapted to be used as a machine reading dataset whose goal is to extract answers from snippets. The authors use a model trained on open domain questions, and perform domain adaptation to biomedical domain using BIOASQ data.
Similar evaluations are performed in machine reading tasks like in SQUAD where top 1 accuracy and F1 scores are computed by comparing exact matching strings. One main assumption in machine reading task is that the answer strings are substrings of the snippets, which implies that answers have to be extracted from the snippets. In Bioasq, the answers are curated by human experts by analyzing the triples, concepts, and snippets (or paragraphs). Thus, the Bioasq dataset and evaluation measures are very similar to that of machine reading task, but the major difference apart from the dataset size are the answers instances provided as gold standard which does not contain all the occurrences, abbreviations, different forms of answers which are present in the snippets. In  (Wiese et al. , 2017b) , the authors transform Bioasq Phase B as a machine reading task with domain adaptation.
Gold standard answer strings and their offsets are automatically searched in the snippets for exact match and treated as answers if only they are found in the snippets, i. e. , the answer string must be a substring of the snippet. By doing so the dataset size is reduced to 65% of Bioasq 5 train set which was suitable for adaptation. Other 35% of the questions did not have matching answers in the snippets, because of different variants of answers in the snippets, missing abbreviations, or irrelevant snippets. This snippet annotation method can result in: • False positive: an answer mentioned in the snippet which does not answer the question. • False negative: a snippet answers the question but does not have the exact string compared to the gold standard string. We found that in Bioasq 6B training dataset for factoid questions, 205 out of 619 questions have false negative answers (33% of the dataset) which may result in some problems: • Less data for learning; • The model does not learn to extract all the variants; • Evaluation is done using such gold standard data which will lower the results even though the model is performing well.
Prediction: MDR-TB Gold standard: tuberculosis Example 2: Abbreviations In example 1, because of a missing word "channels", the predicted answer is marked incorrect. In example 2, MDR-TB stands for Multi-drugresistant tuberculosis, which is from a relevant snippet but since the gold standard has only tuberculosis, it is marked incorrect. Contextually both are valid answers. To overcome this problem and enrich the answer space correctly, we manually annotated 618 factoid question-answers pairs from training dataset of 6B task, by annotating the substring of the gold standard answers in the snippets, and adding answers with abbreviations, multi-word answers, synonyms, that are likely correct answers. We explain this in detail in the following section. This section presents the details of the annotations performed manually on the BIOASQ 6B training dataset and presents some statistics.
Annotations were initially done on the Bioasq 5B training set and the additional questions from 5B test sets whose answers are present in the 6B training set were annotated later on 6B data. So the changes done (if any) on 6B training set for previous year questions from 5B set are not considered. The annotation files are freely available 4  and can be used by researchers who can get the Bioasq dataset. Some statistics of the dataset are listed in Table 1 for the automatically annotated answers from gold standard data and the fully annotated data with manual annotations. The annotations are done on 618 BIOASQ 6B training dataset questions. Out of 619 factoid questions, 1 question does not have any snippets. Only 426 questions contain answers from automatic annotation. "Answers" are the count of answers present in the snippets.
Experiments The goal of our experiments is to study the impact of the data augmentation on training and evaluating a system. Henceforth, we follow the process of  (Wiese et al. , 2017b)  and use a machine reading model developed by  (Chen et al. , 2017)  that is pre-trained on SQUAD dataset  (Rajpurkar et al. , 2016)  for open domain questions and fine tuned to biomedical questions. To study the impact on the training process and the evaluations, we train the models using separately the automatically annotated data and the fully manually annotated data. We also evaluate them using both kinds of data separately.
(Wiese et al. , 2017a)  training with SQUAD dataset is that the DNN model will learn and perform better while trained on a larger training dataset. Since the target dataset is in the biomedical domain, finetuning the previously learned model will have a positive impact on the test set predictions, as shown in  (Wiese et al. , 2017a). Several embedding spaces were tested as input vectors  (Kamath et al. , 2017)  and the best performing ones which were the Glove embeddings trained on common crawl data with 840B tokens, were chosen as input to the system. Unknown words were initialized as zero vectors. As BIOASQ questions have several answering snippets, we treat each question and a snippet as a training sample which might often result in repeated questions with different snippets, i. e. for each training example, there is a question, a unique snippet and the start and end token string offsets of the answer in the snippet.
Datasets We perform fine-tuning on two datasets namely • BIOASQ 5B training set, which contains the 4B training data + the answers of the 4B test data -We term it as 5B. • BIOASQ 6B training set, which contains the 5B training data + the answers of the 5B test set -We term it as 6B. We term the automatically annotated training data as Gold, and manually annotated training data as Anno. The pre-trained model on open domain QA data is fine-tuned on the above listed Bioasq datasets separately. Evaluation is performed by K-fold cross validation because of the small scale of the data (Table  2 ), and on the official test sets of Bioasq 5B (Table  3 ), which were separated from the training data while fine-tuning. The explanation of scores reported in table 2 and 3 along with the corresponding experiments on the datasets listed above, is as follows.
On the data of Trainset mentioned in the first row, we fine-tune it with Finetune data on the second rowwhich is Gold or Anno. version of the answers. The official evaluation measures 5  using Gold or Anno. version of the answers are highlighted in the third row. The strict and lenient accuracies along with the MRR are reported. Gold version of 5B data contains 313 questions and Gold version of 6B data contains 428 questions. We consider the remaining questions with no matching answers as incorrectly answered, hence evaluating over all the questions of the datasets (5B -486 questions, 6B -618 questions). Annotated 5B data contains 483 questions and 6B data contains 618 questions. Overall results of 5B test sets presented in Table  3  are evaluated on 150 questions from the test sets of 5B challenge whose gold standard answers are present in 6B challenge train set.
MATERIALS AND METHODS Human myeloma cell lines (HMCLs) and primary MM cells of patients HMCLs (N = 40) were obtained as previously described  [29] [30] [31] [32] [33] [34]  or purchased from DSMZ and American Type Culture Collection. Microarray data are deposited in the ArrayExpress public database (accession numbers E-TABM-937 and E-TABM-1088). Patients presenting with previously untreated MM (N = 206) or monoclonal gammopathy of undetermined significance (MGUS; N = 5) at the university hospitals of Heidelberg and Montpellier, as well as 7 healthy donors have been included in the study approved by the ethics committee of Montpellier and Heidelberg after written informed consent in accordance with the Declaration of Helsinki. Clinical parameters and treatment regimens of the MM patients included in the Heidelberg-Montpellier (HM) cohort were previously described  35.
GEP of purified MMCs was assayed using Affymetrix U133 2. 0 plus microarrays (Affymetrix, Santa Clara, CA, USA) as described  36  and data normalized using the MAS5 Affymetrix algorithm. The. CEL and MAS5 files are deposited in the ArrayExpress public database (http://www. ebi. ac. uk/ arrayexpress/), under accession number E-MTAB-362. We also used publicly available MAS5 normalized GEP data (GEO, http:// www. ncbi. nlm. nih. gov/geo/, accession number GSE2658) from purified MMCs of a cohort of 345 patients treated with total therapy 2 protocol (UAMS-TT2 cohort) at the University of Arkansas for Medical Sciences (UAMS, Little Rock, USA)  37. t(4;14) translocation was evaluated using MMSET spike expression  38  and del17p13 surrogated by TP53 probe set signal  39  for UAMS-TT2 patients. We also used Affymetrix data of 152 relapsed MM patients subsequently treated with bortezomib (GSE9782) from the study of Mulligan et al.
40  Gene expression data of normal memory B cells (MB), preplasmablasts, plasmablasts, and early plasma cells  41, 42  are deposited in the ArrayExpress databases under accession numbers E-MEXP-2360 and E-MEXP-3034. Identification of genes deregulated by the HDACi/DNMTi combination HMCLs (XG-5, XG-6, XG-7, XG-12, XG-19, XG-20, OPM2, RPMI8226, and LP1) were treated with 0. 5 μmol/L decitabine (Sigma, St Louis, MO) for 7 days in RPMI 1640, 10% fetal bovine serum supplemented with interleukin (IL)-6 for IL-6-dependent HMCLs. During the last 24 h, 0. 33 μmol/L trichostatin A (TSA; Sigma) was added as described by Heller et al. 43  Whole-genome GEP was assayed with Affymetrix U133 2. 0 plus microarrays (Affymetrix). C57BL/KaLwRij mice were purchased from Harlan CPB (Horst, The Netherlands). Mice were housed according to the conditions approved by the Ethical Committee for Animal Experiments of the Vrije Universiteit Brussel (license no.
RNA was further processed and hybridized to the Mouse Genome 430 2. 0 Array (Affymetrix, Santa Clara, CA, USA) as previously described  23, 27. Microarray data are available at ArrayExpress database (Accession number: E-MTAB-3178). Sensitivity of primary myeloma cells to HDACi/DNMTi combination Primary myeloma cells of 10 patients were cultured with or without graded concentrations of decitabine and TSA. Primary myeloma cells of 15 patients were cultured with or without graded concentrations of 5-azacitidine (Sigma) and vorinostat (SAHA) (Sigma). MMC cytotoxicity was evaluated using anti-CD138-PE monoclonal antibody (mAb; Immunotech, Marseille, France) as described  27, 44. Results were analyzed using GraphPad Prism (http://www. graphpad. com/scientific-software/prism/).
Statistical analysis Gene expression data were analyzed using the SAM (Significance Analysis of Microarrays) software  45  as published  38. The statistical significance of differences in OS between groups of patients was calculated by log-rank test. Multivariate analysis was performed using the Cox proportional hazards model. Survival curves were plotted using the Kaplan-Meier method. All these analyses have been done with R. 2. 10. 1 (http://www. r-project. org/), bioconductor version 2. 5 and Genomicscape (http://www. genomicscape. com)  46. A histone acetylation/DNA methylation risk score (termed Combo score) was built using our previously published methodology with the decitabine/HDACi combination deregulated prognostic genes  27, 28. Briefly, the Combo score was constructed as the sum of the Cox model beta coefficients of each of the decitabine/HDACi combination deregulated genes with a prognostic value, weighted by ±1 according to the patient MMC signal above or below the probeset maxstat value  27, 28, 38.
Immunophenotypic analyses For immunophenotypic analysis, the Cytofix/Cytoperm Kit (BD Biosciences) was used for intracellular staining of IRF-4 (clone IRF4. 3E4 from BioLegend (San Diego, CA, USA)) or MYC (clone #12189, Cell Signaling). Flow cytometric analysis was performed with a FORTESSA cytometer using FACSDiva 6. 1 (Becton Dickinson, San Jose, CA, USA). The Kaluza software (Beckman Coulter) was used for data analysis. The fluorescence intensity of the cell populations was quantified using the staining index formula: [mean fluorescence intensity (MFI) obtained for a given mAb minus MFI obtained with a control mAb]/[2 times the standard deviation of the MFI obtained with the same control mAb]  49.
ACKNOWLEDGEMENTS We thank the  Microarray Core Facility of IRMB  (http://irb. montp. inserm. fr/en/index. php?page = Plateau&IdEquipe = 6). This work was supported by grants from  French INCA (Institut National du Cancer) Institute  ( 2012-109/087437  and  PLBIO15-256 ),  Languedoc Roussillon CRLR  ( R14026FF ),  LR-FEDER Hemodiag, Fondation de France  ( 201400047510 ),  ITMO Cancer (MM&TT) ,  SIRIC Montpellier  ( INCa-DGOS-Inserm 6045 ),  Kom Op tegen Kanker ,  Stichting tegen Kanker  and  Fonds voor Wetenschappelijk Onderzoek, Belgium , the  German Federal Ministry of Education (BMBF) "CAMPSIMM"  ( 01ES1103 ) and within the framework of the e:Med research and funding concept "CLIOMMICS" ( 01ZX1309 ), and the  Deutsche Forschungsgemeinschaft  ( SFB/TRR79 ; subproject B1). L. H. is supported by a grant from  Labex EpiGenMed.
5 embryos with KLF1 [GeneID=10661] knockout compared to those from the wild-type embryos Genes downregulated in at least one of the three multiple myeloma (MM) cell lines treated with the DNA hypomethylating agent decitabine (5-aza-2'-deoxycytidine) [PubChem=451668] Genes downregulated in multiple myeloma (MM) cell lines treated with both decitabine [PubChem=451668] TSA [PubChem=5562] Genes with high-CpG-density promoters (HCP) bearing histone H3 dimethylation at K4 (H3K4me2) and trimethylation at K27 (H3K27me3) in the brain Genes upregulated in at least one of the three multiple myeloma (MM) cell lines by TSA [PubChem=5562] Genes upregulated in PC3 cells (prostate cancer) after knockdown of EZH2 [GeneID=2146] by RNAi Genes downregulated in PC3 cells (prostate cancer) after knockdown of EZH2 [GeneID=2146] by RNAi Genes upregulated in at least one of the three multiple myeloma (MM) cell lines treated with the DNA hypomethylating agent decitabine (5-aza-2'-deoxycytidine) [PubChem=451668] Genes upregulated in confluent IMR90 cells (fibroblast) after knockdown of RB1 [GeneID=5925] by RNAi Genes downregulated in at least one of the three multiple myeloma (MM) cell lines by TSA [PubChem=5562] Genes whose promoters are bound by MYC [GeneID=4609], according to MYC Target Gene Database Genes upregulated in CD-1 compared to CD-2 cluster of multiple myeloma samples Genes whose promoters contain E-box motifs and whose expression changed in MYCN-3 cells (neuroblastoma) upon induction of MYCN [GeneID=4613] Genes downregulated in mice with skin-specific knockout of TP53 [GeneID=7157] Genes upregulated in mice with skin-specific knockout of RB1 [GeneID=5925] by Cre-lox Genes constituting the ATM-PCC network of transcripts whose expression positively correlated (Pearson correlation coefficient, PCC≥0.
Behavioural data analysis For each subject, several behavioural parameters were measured separately in each SOA condition. In the attended condition, we measured the performance in comparing the target against 5 (objective measure of conscious access) and the rate of seen trials (subjective measure of conscious access). In the unattended condition, we measured the performance in estimating which color was more frequent. Analyses of variance (ANOVAs) were conducted on each of those behavioural measures, with SOA as a within-subject factor and group (patients or controls) as a between-subject factor. Within the patient group, Pearson.
Statistical comparisons Because many of the hypotheses at stake lie on an absence of difference (e. g. preserved feedforward processing in schizophrenic patients), besides frequentist statistics (values of the statistic, e. g. ts or Fs, as well as p-values are reported), we also conducted A C C E P T E D M A N U S C R I P T 14 Bayesian statistics whenever required. Contrary to frequentist statistics, Bayesian statistics symmetrically quantify the evidence in favour of the null (H0) and the alternative (H1) hypotheses, therefore allowing to conclude in favour of an absence of difference  (Wagenmakers et al. , 2010). To do so, the BayesFactor package (http://bayesfactorpcl. rforge. r-project. org) implemented in R (https://www. r-project. org) was used. Bayes Factor were estimated using a scale factor of r = 0.
Acknowledgements This work was supported by  INSERM ,  CEA ,  Collège de France ,  Fondation Roger de Spoelberch , and an  ERC grant  " Neuroconsc " to S. D. ;  Fondation pour la Recherche Médicale  to L. B. and S. D. ; a " Frontières du Vivant" doctoral fellowship  involving the  Ministère de l'Enseignement Supérieur et de la Recherche  and  Fondation Bettencourt  to M. M. We gratefully acknowledge the hospitality of  Marion Leboyer.
Background -A massive cholera epidemic struck Haiti on October 2010. As part of the national cholera elimination plan, the Haitian government, UNICEF and other international partners launched a nationwide alert-response strategy from July 2013. This strategy established a coordinated methodology to rapidly target cholera-affected communities with WaSH (water sanitation and hygiene) response interventions conducted by field mobile teams. An innovative red-orange-green alert system was established, based on routine surveillance data, to weekly monitor the epidemic. Methodology/Principal findings -We used cholera consolidated surveillance databases, alert records and details of 31,306 response interventions notified by WaSH mobile teams to describe and assess the implementation of this approach between July 2013 and June 2017.
Department directorates compile and validate this data using a Microsoft Excel spreadsheet, which is sent to the national Directorate for epidemiology laboratory and research (DELR), normally on a daily basis, but with frequent delays. The DELR validates the departmental data and compiles them into a national Microsoft Excel spreadsheet at a daily and communal scale, before performing analyses. Recorded communes refer to the treatment place of cholera patients, not to their residence. Community cases are not recorded. Since 2010, routine bacteriological confirmation of suspected cholera cases is performed at the National Laboratory of Public Health (LNSP) in Port-au-Prince Metropolitan Area, using standard sampling, culture and phenotyping methods  [46]. Since July 2017, a private lab run by the NGO Zanmi Lasante and located in Saint-Marc, Artibonite, notifies stool culture results to LNSP.
But for organizational reasons, they actually did not include all planned criteria (Table  1 ), such as precise location of cholera cases at a sub-communal scale (criteria #3), RDT results (#4), stool cholera cultures (#5), rumors of cholera outbreak (#9). DELR then mapped identified alerts using Philcarto software  [51] , and summarized alert information in cholera bulletins, which were diffused to partners via epidemiological situation rooms, emails or through the MSPP website, usually on a weekly basis. An example of cholera weekly bulletin can be seen online (https://mspp. gouv. ht/site/downloads/Profil%20statistique%20Cholera%2044eme%20SE%202016 %20. pdf, accessed 26 January 2018).
Data collection We used anonymous information of daily reported cholera suspected cases, deaths and stool culture results, with the authorization of, and in collaboration with, the Haitian Ministry of Health authorities. The dates of alert identification by the DELR and the dates of diffusion to partners via alert bulletin email were prospectively collected. UNICEF provided the list of its expenditure for mobile teams (through international or non-governmental organizations, and MSPP with EMIRAs) and for other cholera related activities such as cholera surveillance, coordination of partners and other WaSH prevention activities. UNICEF also provided a list of response items (chlorine, soaps, buckets, oral rehydration salts…) delivered to its partners for response and prevention activities, as well as their cost.
Records and details of field activities notified by WaSH mobile teams included the date, location (i. e. commune, communal section, locality) and nature (i. e. investigation, education sessions, distribution of soap and chlorine, implementation of water-chlorination points, household decontamination…). For each response, field organizations mentioned the main institution that notified related cases, so that it was possible to link field activities with institutional epidemiological information. Response data were used with the consent of health authorities, implementing No red or orange alert criteria for at least two weeks * DELR, Haiti Directorate for Epidemiology Laboratory and Research; LNSP, National Laboratory of Public Health $ Criteria that were prospectively used by the DELR on a weekly basis (#1 #2 #6 #7 #8 #10) using often incomplete and delayed cholera surveillance reports; the prospective collection and analysis of data actually appeared to be difficult for the other criteria £ Criteria that were retrospectively used in the study to identify retrospective alerts (#1 #2 #5 #6 #7 #8 #10), using consolidated databases.
§ Rumors define any information about cholera outbreaks that is not produced by the surveillance system of suspected cases and deaths and stool cultures organizations and funders. Country and commune population estimates were provided by the Haitian Institute of Statistics and Informatics (http://www. ihsi. ht/pdf/projection/Estimat_PopTotal_18ans_Menag2015. pdf, accessed 26 January 2018). Satellite estimates of daily-accumulated rainfall (area-averaged TRMM_3B42_daily v7) were extracted from NOAA websites on the entire surface of Haiti, and on the centroid of each of the 140 communes (http://giovanni. gsfc. nasa. gov/giovanni/, accessed 26 January 2018). Corresponding databases are listed in Supplementary Table  S2.