dataesr/scientific-paragraphs-categorization huggingface dataset:
Evaluation The evaluation we performed on our solution consists of three steps, aimed at assessing the reliability of each component in our study's pipeline. First of all, we conducted a general analysis on the use of entities as document labels. Next, we evaluated the quality of our ranking. Finally, we examined the relation between topics and labels, presenting two different adaptations of the word / topic intrusion test  (Chang et al. 2009) , as mentioned when describing the evaluation platform in the previous section. Label identification. Our pipeline's first step is identifying potential topic labels via Entity Linking. Linked entities were obtained with TagMe 2 19  , which disambiguates against Wikipedia entities. Thanks to the efforts of 3 human annotators, we collected 150 documents labeled with entities (50 for each dataset).
The inter-annotator agreement on the label identification is κ = 0. 42, which thus confirms the difficulty and subjectivity of the task 20. As can be seen in Table  2 , in all three datasets annotators chose four labels on average. This possibly high number of labels often occurs since two highly related entities (e. g. Romano_Prodi and Politics_of_Italy, Employee_stock_option and Share_(finance), Iraq and Iraq_War) were both picked by the user as labels, as they reflect different aspects of the same document. Among the labels selected, in all datasets over 90% were entities that belong to the document. It is also important to remark that selecting labels for Enron documents can be very challenging, as important aspects of the emails (like people mentioned in them) @2 @3 @4 MAP P R F 1 P R F 1 P R F 1 EuroParl 0.
The final number of labels selected per document highly depends on the needs of the researcher. In our case, based on the results of our study (presented in Table  3 ) and the behaviour of the annotators (see Table  2 ), we select labels by considering the first 4 entities in each document, ordered by tf-idf. However, researchers with different goals (for example to obtain less topics but with high precision) can conduct the same evaluation and choose the value that best fits their needs. Topic-Label Relation. While Labeled LDA is able to generate a specific topic for each label, it is also important to evaluate whether the label itself is representative of the topic and vice-versa. Thanks to the effort of three human annotators, for each dataset we collected 40 annotations for the relation between labels and topic-words, 20 for each evaluation mode (i.
e. choosing among topic-word sets given a label (Label Mode), or choosing among labels given one of topic-words (Term Mode), as explained when describing the design of the evaluation platform in Section 4. 2 above). In Table  4  we report the accuracy of user choices (i. e. the number of correct matches over the total number of annotated relations) and the ratio of skipped relations. As can be seen, performance differs considerably across datasets. By examining the errors we noticed that, especially in EuroParl, the user often had to decide between highly related labels (such as Israeli-Palestinian_conflict and Gaza_Strip), and this decision has a negative impact on the final results. On the other hand, while for Enron the performance on the Label Mode relation (choosing term-sets given a label) is better than for the other datasets, the annotators also skipped 14 relations in Term Mode (choosing labels given sets of topic-terms).
e. terms that are found in Wikipedia but that are generally common nouns not clearly classified as a named entity). For instance, TagMe 2 achieves good results with both the IITB dataset  (Kulkarni et al. 2009) , where most annotations are concepts (Waitelonis, Jürges, and Sack 2016, Table  1 ), and with the AIDA/CoNLL dataset  (Hoffart et al. 2011) , where most annotations correspond to named entities (Tjong Kim Sang and De Meulder 2003, Table  2 ). In this work, we want to annotate concepts since we believe that conceptual information (rather than only named entities) is very useful to capture the content of our corpora (Europarl, Enron, Clinton). As TagMe 2 is good at tagging both terms for conceptual information and terms for named entities, we think that the tool is a good choice as a source of topic labels.
As mentioned above, while Entity Linking provides clear labels for the content of a document, it does not offer a direct notion of the proportion of the document that is related to each entity. Conversely, standard LDA's relevance scores do provide an estimate to what extent the topic is relevant for the document, but the top- ics are not expressed with clear labels. Our solution, instead, provides both clear labels, and a quantification of the extent to which the label covers the document's content. To exemplify the advantages of this solution consider the following comparison  22. In this case we performed, using the Stanford Topic Modeling Toolbox, both Standard LDA (k=300) and Labeled LDA (with 5 labels) on speech transcripts for the 125 parties at the European Parliament (1999-2004 session)  23.
Background: Multiple treatments are frequently available for a given condition, and clinicians and patients need a comprehensive, up-to-date synthesis of evidence for all competing treatments. We aimed to quantify the waste of research related to the failure of systematic reviews to provide a complete and up-to-date evidence synthesis over time. Methods: We performed a series of systematic overviews and networks of randomized trials assessing the gap between evidence covered by systematic reviews and available trials of second-line treatments for advanced non-small cell lung cancer. We searched the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, MEDLINE, EMBASE, and other resources sequentially by year from 2009 to March 2, 2015. We sequentially compared the amount of evidence missing from systematic reviews to the randomized evidence available for inclusion each year.
Search strategy We searched for reports of randomized trials in the Cochrane Central Register of Controlled Trials, MED-LINE, and EMBASE (search equations in Additional file 1: Appendix 2) with no restriction on language, status, or year of publication, and searched other resources  [12] : 1) previous systematic reviews (see below); 2) reference lists of all selected trials; 3) conference abstracts (from the American Society of Clinical Oncology Meeting, European Society of Medical Oncology Congress, and World Lung Cancer Conference); 4) non-industry trial registries and results databases (ClinicalTrials. gov and EudraCT); 5) industry trial registries and results databases; and 6) regulatory agency online databases (US Food and Drug Administration and European Medicines Agency); details in Additional file 1: Appendix 3.
Eligibility criteria Systematic reviews of randomized trials of second-line treatments for advanced NSCLC were eligible. We selected reviews that addressed at least one comparison between the treatments considered previously, whether they included a meta-analysis or not. We excluded reviews that did not report clearly stated objectives, eligibility criteria for trials, or a systematic search strategy; reviews combining first-and second-line data were ineligible; reviews focusing exclusively on patients with EGFR mutation or ALK rearrangement or on doublet chemotherapy, or comparing two different administration schemes; and reviews that did not provide the list of included trials.
Search strategy We searched the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects, MEDLINE, EMBASE (search equations in Additional file 1: Appendix 4), and other resources: conference abstracts from the American Society of Clinical Oncology Meeting, European Society of Medical Oncology Congress, and World Lung Cancer Conference; and the PROSPERO international prospective register of systematic reviews for completed or published systematic reviews (details in Additional file 1: Appendix 5). There was no restriction on language, status, or year of publication. The last search was conducted on March 2, 2015.
Selection of studies and extraction of data Two authors independently and in duplicate examined titles, abstracts, and full-text articles to determine the eligibility of randomized trials and systematic reviews. We pilot-tested the eligibility criteria on a sample of 100 records (for the selection on titles and abstracts) and 10 reports (for the selection on full-text articles) to ensure that the selection criteria were applied consistently by the two authors. Disagreements were discussed with a third author. All data were independently extracted by two authors who used a standardized form. For each trial, we extracted the dates of publication of the full-text article(s) (online publication, if any) and conference abstract(s), date of results posting on nonindustry and industry trial registries, and date of publication of reports by regulatory agencies.
g. , monochemotherapy or targeted therapy), and lumped different types of treatments together (e. g. , monochemotherapy and EGFR tyrosine kinase inhibitors considered the same intervention). Two reviewers independently assessed the methodological quality of the systematic reviews, with a formal consensus process in case of disagreement. We used AMSTAR, a measurement tool created to assess the methodological quality of systematic reviews  [13] , which has been validated  [14, 15]. We assessed the four items pertaining to duplicate study selection and data extraction, comprehensive literature search (at least two electronic sources and one supplementary strategy among reviews, experts, or reviewing the references), searching for reports regardless of their publication type, and providing a list of included and excluded trials.
Searching for trials regardless of their publication type was judged inadequate when authors did not report searching the grey literature (conference abstracts, non-industry trial registries and results databases, industry trial registries and results databases, regulatory agency online databases) or excluding reports based on language. We focused on these four specific items because the methods used for the identification and selection of studies are directly related to a potential gap between the amount of randomized evidence covered by systematic reviews and the amount of randomized evidence available for inclusion, and other domains are unrelated.
Randomized trials of second-line treatments for NSCLC We identified 77 eligible trials with results available between May 2000 and November 2014 (Fig. 1 ). The results of 62 trials were published in 69 articles; among these 62 trials, 43 had results reported in other sources (16 in conference abstracts, 10 in non-industry trial registries and results databases or regulatory agency online databases, and 17 in both). The remaining 15 trials (20 %) had unpublished results (12 conference abstracts, 2 non-industry trial registries and results databases or regulatory agency online databases, and 1 both). In all, 61 trials (79 %) were registered at ClinicalTrials. gov. The 77 trials included 28,636 patients (median [Q1-Q3] 168 [100-559] patients); 72 included two arms, two compared three different treatments, and three compared one drug to two different doses of another drug, which were lumped together.
Two reviews performed network meta-analyses but did not cover all available treatments. Only one of the 29 reviews was an update of a previously published review. Regarding the methodological quality of the 29 systematic reviews, 45 % of reviews lacked information on independent study selection and data extraction, 31 % a comprehensive literature search, and 45 % a search for reports regardless of their publication type. Of note, 17 % of reviews lacked information on a search for conference abstracts, 86 % a search for non-industry trial registries and results databases, 97 % a search for industry trial registries and results databases, and 97 % a search of regulatory agency online databases. In all, 79 % of systematic reviews did not report duplicate study selection and data extraction, comprehensive literature search, and searching for reports regardless of their publication type.
Automated processes, such as meta-search engines, could systematically cover sources such as industry and non-industry trial registries and results databases to identify posted results and improve the evidence synthesis  [49, 50]. Beyond the example at hand, the current way of conducting systematic reviews explains why they are inherently at risk of providing a fragmented, out-ofdate panorama of the evidence for all treatments. A broader scope on all evidence available for all treatments of a given condition is naturally provided by a systematic review with network meta-analysis, which allows for examining the totality of the randomized evidence using trial networks  [51, 52]. Although the production of network meta-analyses follows an exponential growth, their number is still relatively limited  [53].
Assessment of risk of bias Need to check for consistency in extracted data between multiple reports for the same trial; in cases of inconsistency, need to justify the choice of a specific source Automatic data extraction is possible depending on the source. For instance, it is possible to abstract automatically posted results from ClinicalTrials. gov [84] [85] [86]  Future automated technologies could help experts to extract data or to assess the risk of bias within trials  [49, 50, 62, 63]  5. Updating of network metaanalysis Need to develop online software for updating the network meta-analysis * Online solutions in development for conventional meta-analysis could be extended to network meta-analysis [87, 88].
Acknowledgments This study was supported by a grant  LEGS POIX 2015  ( Chancellerie des Universités de Paris). The funding source had no role in the design of this study, its execution, analyses, interpretation of the data, and decision to submit results. The authors thank  Laura Smales  ( BioMedEditing, Toronto, Canada ) for language revision of the manuscript and  Elise Diard  for the conception of the Figures.
Study Goals We first examine categorical longitudinal trajectories of anthropometric variables to test whether variation in parity and/or reproductive pace is associated with long-term decreases in nutritional status. We next employ multilevel (or mixed effects) regression models using longitudinal data to examine short-term changes in weight, body mass index (BMI), body fat percentage, hemoglobin (Hb), ESR and WBC as a function of parity and pace of reproduction among women aged 15-44 years (n = 592). Among women age 45+ (n = 277), we test whether parity and pace of reproduction are associated with the same health measures. This study reports associations between demographic, nutritional and immunological variables on the largest sample size to date of women living under conditions most similar to those of our preindustrial past.
Acknowledgements We thank  Tsimane Health  and Life History Project personnel for their tremendous efforts over the years, especially  Edhitt Cortez Linares ,  Raul Quispe Gutierrez  and  Ivan Maldonado Suarez. Funding was provided by  NIH/NIA   R01AG024119 ,  NSF   BCS-0136274 ,  BCS-0422690  and  NICHD 5T32HD007242. JS acknowledges support from the  Agence Nationale de la Recherche (ANR) -Labex IAST.
Assessment of vulnerability to flooding has been widely investigated by researchers, policy makers and practitioners. The research project initiated between the French insurance dedicated association to natural risks, MRN, and the urban planning lab LDE ¶XUED SURSRVHV WR DGGUHVV WKLV TXHVWLRQ IURP ³DFWLRQ´ By making the assumption that territorial vulnerability depends on the collective measures implemented by communities, this research project aims to experiment a national rating method of the territorial preventive effort. To investigate collective prevention, Flood Action Programmes (PAPI) have been identified as the one bottom-up public policy instrument that mobilizes all the prevention DFWLRQV ¶ OHYers. The constitution of a 3$3, ¶V DFWLRQV ¶ database, homogenized through a codification process, provides 82 codes reflecting the diversity of actions.
/ In this research work, we assume that PAPI offer an adequate window on flood risk collective coping FDSDFLW\ ,Q SDUDOOHO WKURXJK 3$3, ¶V VFRULQJ ZH LQWHQG to highlight differences between PAPIs, according to the local effort and specificities. The analysis of the codified action database provides a first glance at the differences between risk basin scales with regards to the preventive effort.
First analysis of the difference in risk coping capacities according to risk basins )URP WKH DQDO\VLV RI WKH DFWLRQ ¶V GDWDEDVH DQG WKH result of the codification process, we here investigate the question of the adaptation of PAPIs to local specificities. In this perspective, we question the diversity of PAPI and thus the diversity of risk coping capacities. We will also point out the ability of our database to point out differences in local risk coping capacity.
Indeed, ³FRPSOHWH´ 3$3, implement in average 27 type of DFWLRQV ZKLOH ³LQWHQGHG´ 3$3,V ODXQFK OHVV WKDQ W\SHV of actions in average. As a consequence, our database makes clear evidence about the higher number of actions in complete PAPIs. However, another interesting point in that the number of codes may overlap: the minimum QXPEHU RI LPSOHPHQWHG FRGHV DPRQJ ³FRPSOHWH´ 3$3,V is 10 while an intended PAPI reached 26 types of actions. Consequently, there is a great distribution of data.
Differences between PAPIs according to local specificities Further to the adaptation of PAPIs to local knowledge, we may point out the differences in risk coping capacities throughout PAPIs by the analysis of budget allocations. Such an analysis tends to show their diversity, as illustrated in Figure  3 : in the north of France, the selected actions have enhanced resistance through upstream flood retention. This strategic choice can be linked to the topography of the risk basins and to land use ± relatively gentle slopes and more rural areas. This is different from the situation in typical coastal risk basins, especially in the Atlantic coastal areas, where the emphasis is also on resistance but focused on dike building and reinforcement. This economic analysis on the codified database also demonstrates that most of the budget is allocated to UHVLVWDQFH ,QGHHG PRUH WKDQ RI 3$3LV ¶ EXGJHW LV GHGLFDWHG WR UHVLVWDQFH LQ DYHUDJH +RZHYHU WKLV GRHVQ ¶W mean that resistance is more effective.
Study design and sampling This study was part of the SPOTLIGHT project  31  and was conducted in five urban regions across Europe: Ghent and suburbs (Belgium), Paris and inner suburbs (France), Budapest and suburbs (Hungary), the Randstad (a conurbation including the cities of Amsterdam, Rotterdam, the Hague and Utrecht in the Netherlands) and greater London (United Kingdom). Sampling of neighbourhoods and recruitment of participants have been described in detail elsewhere  32. Briefly, neighbourhood sampling was based on a combination of residential density and socioeconomic status (SES) data at the neighbourhood level. This resulted in four types of pre-specified neighbourhoods: low SES/low residential density, low SES/high residential density, high SES/low residential density and high SES/high residential density.
Self-, predefined neighbourhoods, and percent overlap Since there is a potential discrepancy between self-defined and predefined neighbourhoods, the extent of overlap was determined. The respondents were asked to draw the boundary of their selfdefined neighbourhood using an online self-mapping tool developed for this purpose (or a printout when using a paper version of the questionnaire)  36. Using ArcGiS, version 10. 1, software (Environmental System Research Institute, ESRI, Redlands, California) 37 , all neighbourhood geographical coordinate points were recorded and combined to form an enclosed area (polygon boundaries) representing the self-defined neighbourhood. GIS was also used to geolocalize home addresses and to define the administrative residential neighbourhood of each participant (defined according to small scale local administrative boundaries except for Hungary (see Lakerveld et al.
Determination of the mismatch metric at neighbourhood level The mismatch between aggregated perceptions and objectively measured data on many neighbourhood environmental features was quantified through a factor analysis. Multiple correspondence analysis (MCA) can be considered to be a generalization of principal component analysis for categorical variables  43. MCA was performed on 2x59 observations (2 observations per neighbourhood: perceived and objectively measured) with 10 environmental features. The observations were then plotted in a bi-dimensional space (factor map) to measure the distances between perceived and objectively measured data for each neighbourhood. These distances reflect the similarities between the observations. A total of 59 distances (1 distance per neighbourhood) was determined.
Sensitivity analysis In order to examine the potential impact of the percent overlap between self-defined and predefined neighbourhood, the analyses were also conducted without adjustment for percent overlap in the aggregation of self-reported perceptions at neighbourhood level. Statistical analyses were performed with R (FactoMineR package  44  ), version 3. 2 (R Development Core Team, 2010)  45  and STATA statistical software (release 13. 0; Stata Corporation, College Station, TX, USA). London. The highest level of agreement for well-maintained sidewalks was observed in greater Paris.
However it is the first to assess concordance between residents' perceptions collected by questionnaire and virtual audit data using GSV in different regions. Previously, residents' ratings of neighbourhood features were usually compared with objective indicators obtained from observational field audits  8  , GIS or through publicly available information  10, 11, 13, 15, 16, [18] [19] [20]. Lower agreement was documented when distance/access to amenities and subjective aspects were examined compared with self-reported presence/absence of a given facility. For instance, matching between perceived and objective proximity to the closest park was observed with only 18% of participants (kappa=0. 01) in Ontario  16. For distance to supermarkets the discordance amounted to 31. 5% among low-income housing residents in greater Boston  12.
A potential limitation of this study was that objectively measured data were not collected at participant level (i. e. using self-defined neighbourhoods) but at predefined neighbourhood level (i. e. using administrative neighbourhoods) in each country. Results found in aggregated data at neighbourhood level cannot be extrapolated to individual level because associations may or may not be the same, according to the "ecological fallacy"  50. In a study of this scale it would be extremely time consuming to perform a virtual audit in each self-defined neighbourhood. In addition to the cross-sectional design, another limitation of this study was the slight difference in wording used to describe environmental features in the two measures (i. e. survey and virtual audit) limiting assessment of agreement to certain item-pairs only, but multiple dimensions of the obesogenic environment (food outlets, physical activity facilities, aesthetics, housing diversity) were covered.