dataesr/scientific-paragraphs-categorization huggingface dataset:
Overview of the task The goal of the PIR-CLEF 2018 Task is to investigate the use of a laboratorybased method to enable comparative evaluation of PIR methods. The collection defined within the PIR-CLEF 2017 Pilot Study and the PIR-CLEF 2018 Lab was created with the cooperation of volunteer users, and was organized into two sequential phases: -Data gathering. This phase involved the volunteer users carrying out a taskbased search session during which a set of activities performed by the user were recorded (e. g, formulated queries, bookmarked documents, etc. ). Each search session was composed of a phase of query development, refinement and modification, and associated search with each query on a specific topical domain selected by the user, followed by a relevance assessment phase where the user indicated the relevance of documents returned in response to each query and a short report writing activity based on the search activity undertaken.
-Data cleaning and preparation. This phase took place once the data gathering had been completed, and did not involve any user participation. It consisted of filtering and elaborating the information collected in the previous phase in order to prepare a dataset with various kinds of information related to the specific user's preferences. In addition, a bag-of-words representation of the participant's user profile was created to allow comparative evaluation of PIR algorithms using the same simple user model. For the Task we made available the user profile data and raw search data produced by guided search sessions undertaken by the 10 volunteer users as detailed in section 3. 1. The aim of the task was to use the provided information to improve the ranking of a search results list over a baseline ranking of documents judged relevant to the query by the user who entered the query.
Dataset For the PIR-CLEF 2018 Task we made available both user profile data and raw search data produced by guided search sessions undertaken by the 10 volunteer users. The data provided included the submitted queries, the baseline ranked lists of documents retrieved in response to each query by using a standard search system, the items clicked by the user in the result list, and the documents relevance assessments provided by the user on a 4-grade scale. Each session was performed by the user on a topic of her choice selected from a provided list of broad topics, and search carried out over a subset of the ClueWeb12 web collection. The data was extracted and stored in csv format in 7 csv files in a zip folder which was provided to participants. The details of the contents of the csv files are as follows:  username: the user who performed the session query session: id of the query session within the search was performed category: the top level search domain query text: the submitted query document id: the document on which a particular action was performed rank: the retrieval rank of the document on which a particular action is performed action type: the type of the action executed by the user (query submission, open document, close document, bookmark) time stamp: the timestamp of the action.
Session Measures Real users often begin an interaction with a search engine with a query which they need to reformulate one or more times. The ability of the PIR system to improve results after query reformulation cannot be easily assessed by the measures normally used for measuring system effectiveness, but requires new approaches taking into consideration the user behaviour triggered by the system. Three alternative approaches are proposed to simulate user behaviour through a retrieval session. Two participant papers were accepted for presentation at the PIR-CLEF session at the CLEF 2018 conference. The papers have made use of the provided data collection to examine two different tasks, and both present interesting and useful findings and suggestions on how to improve the PIR-CLEF dataset.
The paper titled ECNU at CLEF PIR 2018: Evaluation of personalised information retrieval  [13]  presents a study exploring the potential of the dataset provided by PIR CLEF. The authors report in their paper two different experiments based on two distinct baselines, i. e. "query level baseline" and "session level baseline"; in the first baseline each single query in a session is evaluated independently, while in the second one all queries in a session are summed up to define a single query. The authors then report on the experiments they made, in which they applied two methods for query expansion and an approach to define a "topic sensitive user model" based on search sessions. In the reported discussion, the following suggestions to improve the dataset are made: i) to provide more numerous relevance labels, ii) to increase the number of provided user-related information (i.
e. to have information related to more users profiles), and iii) to define richer query logs. The paper titled PIR based on explicit and implicit feedback  [14]  addresses Task 2 of the PIR-CLEF lab, i. e. user profiling. More specifically, the authors explored the use of explicit and implicit feedback to define user profiles. Concerning explicit feedback, the subjective relevance judgments provided by the searchers for a given set of documents is employed to train a text classifier, thus exploiting the PIR task as a text classification task. Concerning implicit feedback, the correlation between information that is inferred form the data and relevance judgments provided by the users has been analyzed. Several analyses and useful remarks are reported in the paper.
Literature search approach To identify relevant studies for the inclusion into the DSS, a systematic scoping review was conducted for each item in the taxonomy. The aim of this approach is to represent the body of literature in a scientific way. While the criteria applied differed between research fields, there was a schematic approach followed for each review, consisting of initial search, screening, identifying additional papers, and prioritizing papers for coding  (Martensen & Lassarre, 2018). For the initial search, several relevant literature databases were searched, e. g. , Scopus, Medline, and Google scholar based on well-defined logical strings of keywords. (See Table  3 , as an example). The keywords as well as the resulting number of studies were documented. Table  3 : An Example of search terms (for the risk factor "fatigue") Fatigue "fatigue*" OR "Sleep*" OR "Tired*" OR "drowsy" OR "drowsiness" OR "alert*" OR "monotony" OR "time on task" AND Road Safety "road safety" OR "driv*" OR "road" OR "transport" OR "crash" OR "accident" OR "incident" OR "traffic" OR "collision" OR "traffic safety" OR "risk" OR "measure OR "Road Casualties" OR "Road Fatalities" The next step consisted of screening the potentially relevant studies to assess their eligibility for further analysis.
The enormous differences between studies constitute a big challenge for the creation of a joint database. The structure is general enough to allow coding different kinds of safety-or risk effects and flexible enough to capture all important details of different types of studies. For each study, the template therefore includes general information of the sampling frame and study conditions (e. g. road user types, severity of crashes, road types included), but also allows for the inclusion of conditions that are relevant to the specific area only (e. g. the differentiation between different injury types or details of the roadway design). Furthermore, for each estimated effect the following specifications were registered: • what was compared to what • analysis method/model • measure of effect (often odds ratio but also many other less used measures of effect) • statistical results (standard error, confidence interval) • conclusion (significant effect on road-safety or not).
DSS webtool The DSS consists of the backend (relational database), the front end (website) and the way they integrate (queries). Both study-templates and synopses are stored in a relational database, which serves as the back-end of the DSS. Front-end DSS results are retrieved through queries on the back-end database (DSS search engine)  (Yannis & Papadimitriou, 2018). In order to optimize and facilitate the use of the DSS, the design of the front end system was based on the following principles: • Linked search and linked results: the user should be able to search a road safety risk alone or through the measures, search a measure alone or through the risks, search for risks and measures related to specific road user groups or accident categories, and so on. • Fine level of detail: the user should be able to refine the search and filter the results with many parameters among those found in the database (e.
g. road types, road user groups, countries etc. ) • Flexibility: the user should be able to continuously adjust the search according to the results • Transparency: the process is fully documented and the user should be able to access background information at any stage (links, etc. ) The Search Pages include a dynamic part and a static part (see Figure  1 ). The dynamic part concerns:  The Search tab allows the user to query the DSS backend database and retrieve results for risk factors or measures, through five entry points (keywords, risk factors, measures, road user groups, accident scenarios), all leading to a Results page. Moreover, links between risks and measures are implemented in the results pages. The results pages include all the DSS outputs in terms of coded studies, summaries etc.
 The Support tab: includes contact information, the guide to DSS users, possibility to send feedback or questions, and useful links to other systems. More specifically, the Search Tab (Level 1) consists of the specific search that the user wishes to carry out on the basis of the five entry points mentioned above. The users will be able to select one of these five entry points depending on the type of search that they wish to conduct. More specifically, the text search allows the users to enter a key-word (selected among the key-words of the studies included in the database), the road safety risk factors/measures entry points allows them to seek specific risk factors or measures from the SafetyCube taxonomies respectively. As a shortcut to the results concerning crash risks or measures specifically related to a road user group, one may enter via the road user groups query (7 groups e.
The results consist of: • Short introductory texts and the colour code(s), describing the risks or the effectiveness of measures. • Links to one or more available SafetyCube synopses on the issue (pdf link button(s) next to the colour code). • A table listing the available meta-analyses and other coded studies in the SafetyCube database together with their main characteristics such as design, country, and year of publication. Selecting a study from the Table will lead the user to the individual study page. • Depending on the selected topic, adaptive search filters are available on the left side of the results page. Filters include: keyword, specific risk factor (corresponding to the most detailed taxonomy level), road user group, road type, country. The keyword filter appears only when entering from the "keyword" or "road user group" entry point, and allows the user to "un-filter" the results and obtain all the studies related to the risk factor or measure (and not only those related to the keyword or road user group).
This allows users with various backgrounds to benefit from the vast knowledge contained in the system by casting scientific evidence on every risk and every measure (or groups thereof) into comprehensive synopses, reachable through different entry points. The DSS however, as with all tools, has some limitations. One might compare it to the most famous database of studies and reviews -the Cochrane Database of Systematic Reviews (CDSR, www. cochrane. org). Although the basic idea behind the DSS is the same, the approach to study quality is different. The Cochrane protocol  (Higgins & Green, 2011)  is very rigid in terms of study-inclusion criteria and focuses mainly on controlled randomized trial experiments. This is indeed the ideal methodology to be sure about the causal relation between a treatment (e.
For safety evaluation of other safety measures, the best available knowledge comes in the ideal case from 'multiple studies employing multiple methods, all leading to the same conclusions'  (Shinar, 2007). However, there are not many examples of this. All in all this means that, even after a few decades of research, the scientific knowledge base for some topics is still quite limited. For example, studies of emerging technologies, like ECall (automatic crash notification systems), containing subjective assessments of small datasets  (Sirola, Luoma, Shirokoff, Salo, & Karkola, 2009; Olin, Strandroth, & Lie, 2014; Ponte, Ryan, & Anderso, 2016)  were reviewed and coded for the DSS. While statistical analyses were limited in these reports, the estimates of the measure's effects are important to convey to road safety stakeholders.
Methods Articles written in English and published between November 11, 2013 and November 10, 2014 were selected from Medline using the following query to search title, abstract and keyword fields: ("differential item functioning" or "item bias" or "measurement invariance" or "factorial invariance" or "measurement equivalence"). The Medline database was chosen as the most appropriate to describe current practices in health research studies and the time frame restricted to one year as the number of published articles allowed the whole of known DIF detection methods to be covered. As we focused on current practices, only articles describing empirical studies in which measurement invariance of composite measurement scale(s) was evaluated were included (reviews and editorials not included, as well as studies concerning item banks and not scales).
Methods Datasets were generated using a partial credit model, a model for polytomous data that shares the statistical properties of the Rasch model and defines k item location parameters for items with k+1 response categories  [23]. Like binary items, DIF effect-size is defined as the group difference in the item location parameter. Thus, in this model, the DIF size may vary from one category to another within the same item and thereby cause non-uniform DIF; DIF is uniform if and only if its size is identical across all the categories of the same item. Factor values conforming to the conditions encountered in health research studies were selected (Table  2 ): small numbers of items in scales (4 or 8 items) with five response categories, small group size (100 or 200 with groups of equal size), and a magnitude of the mean latent trait difference between the two groups of 0 (no difference) to 0.
Three sizes of uniform DIF were studied: 0. 25, 0. 5 and 1 logit and the item was more difficult in the focal group. • Balanced non-uniform DIF: the relation between the group and the response depends on the level of the latent trait but the overall item difficulty is identical in the two groups. To generate data with balanced non-uniform DIF, the DIF was set to be unequal for the four item location parameters but the sum of the DIF sizes was zero, such that the item characteristic curves crossed at the latent trait level equal to zero (bottom of • We also studied unbalanced non-uniform DIF in which the overall item difficulty in the focal group was higher than in the reference group as the sum of the DIF sizes affecting the four location parameters was positive. Two sizes of unbalanced non-uniform DIF were used A latent regression partial credit model was applied to each of the 500 datasets generated for each scenario to estimate differences between groups without taking into account the presence of DIF.
The measurement bias due to DIF was estimated as the delta between this estimated difference and the value of the mean latent trait difference between the two groups used to generate data (the "true" difference between groups). Four multivariate models of analysis of variance (ANOVA), one for each kind of DIF studied, were used to investigate the effects on the measurement bias of each factor manipulated in the simulation model. However, the number of simulations studied by ANOVA was huge, so a benchmark was set to distinguish between negligible and non-negligible statistically significant factor effects. To do this, the same analyses were performed on the 500 datasets generated for each of the 12 scenarios resulting from the combination of the three factors not related to DIF (number of items in the scale, group size and a magnitude of the mean latent trait difference between the two groups), with all factors related to DIF set to zero; the resulting delta between the estimated and the "true" difference between the two groups was termed random error instead of measurement bias as there was no source of DIF-related bias in the simulation model.
Results In simulations without DIF, mean random error over the 500 datasets generated for each of the 12 combinations was never higher than 0. 013 logit and was not affected by any of the three factors manipulated. In the following analyses, a factor was considered to have a non-negligible effect on measurement bias if its estimated coefficient was statistically significant and higher than 0. 015 logit; it was considered to have a major effect if its estimated coefficient was higher than 0. 1 logit, i. e. half of a small effect-size as defined by Cohen  [26]. Although the estimated ANOVA coefficients of the effects of the three factors not related to DIF (number of items in the scale, group size and a magnitude of the mean latent trait difference between the two groups) were significant (p<0.
Recommendation 1: Distinguish between uniform, balanced non-uniform and unbalanced non-uniform DIF. Justification: In the simulation study, there were large effects of DIF effect-size, proportion of DIF items and their interaction on measurement bias in case of uniform or unbalanced non-uniform DIF. However, non-uniform DIF was assessed in less than a quarter of the published articles using the Rasch framework. While in most of software that allow estimation of Rasch model item parameters, DIF tests are accessible or may be programmed, only few of them, the most popular ones, include programs aimed at distinguishing uniform from nonuniform DIF: RUMM (used in 25 articles), WINSTEPS (used in 17 articles)  [28, 29]. Recommendation 2: When statistically significant DIF is found, its effect-size should be estimated as the group difference in the item location parameter(s).
No recommendation was thus formulated on this factor but future updates of these recommendations (within 10 years) will be performed based on scenarios which were not considered in our simulation study. It will also involve literature reviews to assess their application in practice and consequences on how DIF is handled by scale users, as well as any new theoretical and methodological approaches to study DIF. Finally, although some statistical software facilitate study of DIF using the Rasch model (RUMM, Stata, R), their use to characterize DIF as meaningful or not can be complicated and thus impede the application of these recommendations  [28, 34, 35]. To raise these barriers to the application of the recommendations, development of modules or packages for existing statistics software is being considered.
Acknowledgments The authors wish to thank experts who accepted to review the recommendations:  Jakob Bjørner , ( University of Copenhagen, Denmark ),  Stefan Cano  ( Modus Outcomes, UK ),  Carlos Garcia Forero (Universitat Pompeu Fabra, Spain) ,  Laura E Gibbons  ( University of Washington, USA ),  James W Griffith  ( Northwestern University, USA ),  Francis Guillemin  ( Universite de Lorraine, France ) and  Richard N Jones  ( Brown University, USA ).
and run spatially explicit agent-based models (SE-ABMs) using the R language. SE-ABMs are models that simulate the fate of entities at the individual level within a spatial context and where patterns emerge at the population level. NetLogoR follows the same framework as the NetLogo software (Wilensky 1999). Rather than a call function to use the NetLogo software, NetLogoR is a translation into the R language of the structure and functions of NetLogo. Models built with NetLogoR are written in R language and are run on the R platform; no other software or language has to be involved. NetLogoR provides new R classes to define model agent objects and functions to implement spatially explicit agent-based models in the R environment. Users of this package benefit from the fast and easy coding provided by the highly developed NetLogo framework, coupled with the versatility, power and massive resources of the R language.
SE-ABMs have been widely used by ecologists to study populations and communities with, for example, research on human-wildlife interactions  (Chion et al. 2011 , Parrott et al. 2012) , analyses of movement and habitat selection patterns for animals  (McIntire et al. 2007 , Grosman et al. 2011 , Semeniuk et al. 2012) , and models of vegetation dynamics  (Wallentin et al. 2008). Several modeling environments already exist to build agent-based models such as NetLogo  (Wilensky 1999) , SELES  (Fall and Fall 2001) , NOVA (<https://nature. berkeley. edu/getzlab/software. html>), GAMA (<http:// gama-platform. org/>), Capsis (<www. inra. fr/capsis>), Repast  (Collier and North 2013, North et al. 2013)  or MASON  (Luke et al. 2005). NetLogo  (Wilensky 1999) , in particular, offers great potential in SE-ABMs implementation, both for research and as a teaching tool  (Tisue and Wilensky 2004 ).
NetLogo has a well-defined structure to represent model agents and a high-level language composed of less than 500 words (called 'primitives') to be used to code a model. This makes the software relatively easy to learn and use, as many actions are already pre-coded into these primitives. NetLogo has many other advantages such as a built-in visual interface, several extensions to broaden the language, and it executes quickly. R (R Core Team) is one of the globally leading open source data and statistical languages (SQL and Python being the other two) and it is widely used in many disciplines, including ecology, to explore and analyze data. R has a huge open-source user and developer community that continuously builds on and improves the software with core enhancements and new extension packages, therefore increasing the software's potential and facilitating its use.
Using the R language, already familiar to many ecologists, to implement SE-ABMs instead of using another software has advantages. Users can benefit from the power and versatility of this medium level language which includes many powerful, user-generated packages to build their SE-ABMs. There are already two ways to couple NetLogo and R to benefit from both software assets for agent-based modeling. The first one is an R extension for the NetLogo software  (Thiele and Grimm 2010). This extension is used inside a NetLogo model and allows calling and using R functions inside the NetLogo code. The model is coded in the NetLogo software using the NetLogo language and a few R functions. Then, the model is run in NetLogo. The second way is with the R packages RNetLogo  (Thiele et al. 2012 , Thiele 2014)  or nlrx  (Salecker and Sciaini 2019)  that act as an interface to run NetLogo.