dataesr/scientific-paragraphs-categorization huggingface dataset:
Human Annotations, Queries, and Relevance Assessments Task 1. The task consisted of extracting ICD10 codes from the raw lines of death certificate text (the process of identifying a single ICD code per certificate as the primary cause of death was not evaluated). This task relied on the text supplied to extract ICD10 codes from the certificates, line by line. The extraction system was to generate the ICD10 codes relevant to assign to each line. Systems were encouraged to report evidence text supporting the ICD10 code recommendations in the form of an excerpt of the original text that supports the ICD code prediction. For French, two data formats were supported. The so-called raw format supplied the text of each certificate line separately from the gold standard codes that were supplied at the certificate level.
Topic-ID. 2. The title of the review, written by Cochrane experts. 3. A part of the protocol: The Objective, the Type of Study, the Participants, the Index Tests, the Target Conditions, and the Reference Standards. 4. The entire PubMED database (which was available for downloaded directly from PubMED). Participants were provided with 30 topics of Diagnostic Test Accuracy (DTA) reviews. In Task 2 Subtask 2, focusing on title and abstract screening, topics consisted of the Boolean Search from the first step of the systematic review process. Specifically, for each topic the following information was provided: The CLEF 2017 TAR 42 topics (which excludes topics that were reviewed and found unreliable) were used as training set. A new test set consisting of 30 topics of Diagnostic Test Accuracy (DTA) reviews was generated for this year's challenge.
7%, while at the content level the average percentage is 1. 5% in the training set, and 1% in the test set. References in the original systematic reviews were collected from a variety of resources, not only MEDLINE. Therefore, studies that were cited but did not appear in the results of the Boolean query were excluded from the label set for Subtask 2, but included for Subtask 1. Hence, the total number of relevant abstracts in the test set for Subtask 1 increased to 4, 656 from 3, 964 in Subtask 2, and the total number of relevant studies increased to 759 from 678. An important note here is that the additional studies are also included in the MEDLINE database, they were simply not retrieved by the Boolean query. Task 3. The CHS task, Task 3, uses a new set of 50 queries issued by the general public to the HON search services, manually labeled with search intent and translated into French, German and Czech  [3].
Evaluation Methods Task 1. After completing our data use agreement, authorized participants were able to obtain training sets from March 2018. The test data for CLEF eHealth 2018 Task 1 was released on 27 April 2018. Teams could submit up to 2 runs per dataset by 12 May 2018. Hence, the maximum was 8 runs for all four datasets. System performance was assessed by the precision, recall and F-measure for ICD code extraction at the document level for Hungarian and Italian and both at the line and document level for French. Evaluation measures were computed overall for all ICD codes. A baseline was also implemented by the organizers  [21]. Task 2. Teams could submit up to 3 runs per task. Hence a maximum of 6 runs for both subtasks. In addition, for Subtask 2, participants were also encouraged to submit ANY number of runs that result from their 2017 frozen systems.
Results The number of groups who registered their interest in CLEF eHealth tasks was 26, 42, and 46 respectively (and a total of 70 unique teams). In total, 28 teams submitted to the three shared tasks. Task 1 received considerable interest with 14 teams submitting runs, including one team from Algeria (techno), one team from Canada (TorontoCL), two teams from China (ECNU and WebIntelligentLab), three teams from France (APHP, IAM, ISPED), one team from Germany (WBI), one team from Italy (UNIPD), three teams from Spain (IxaMed, SINAI and UNED), one team from Switzerland (SIB) and one team from the United Kingdom (KCL). The training datasets were released at the beginning of March 2018 and the test datasets by 27 April 2018. The ICD-10 coding task submission on French, Hungarian and Italian death certificates were due by 12 May 2018.
For the Hungarian raw dataset, we received 9 official runs from 5 teams (Table  3 ). For the Italian raw dataset, we received 12 official runs from 7 teams (Table  4 ). For the French raw dataset, we received 18 official runs from 12 teams (Table  2 ). For the French aligned dataset, we received 16 official runs from 8 teams (Table  1 ). In addition to these official runs, unofficial runs were submitted by some participants after the test submission deadline 2. Participants relied on a diverse range approaches including classification methods (often leveraging neural networks), information retrieval techniques and dictionary matching accommodating for different levels of lexical variation. Most participants (12 teams out of 14) used the dictionaries that were supplied as part of the training data as well as other medical terminologies and ontologies (at least one team).
Task 2 attracted the interest of 7 teams submitting runs, including one team from Canada (UWA), one team from the USA (UIC/OHSU), one team from the UK (Sheffield), one team from China (ECNU), one team from Greece (AUTH), one team from Italy (UNIPD), one team from France (Limsi-CNRS). For the subtask 1, we received 12 runs from 4 teams. The results on a selected subset of metrics are shown in Table  5. For the subtask 2, we received 19 runs from 7 teams. The results on a selected subset of metrics are shown in Table  6. The 7 teams used a variety of learning methods including batch supervised learning, continuous active learning, a variety of learning algorithms including logistic regression, support vector machines, and neural networks, as well as unsupervised retrieval methods, such as TT-IDF, BM25, with or without traditional relevance feedback The training datasets were released on February 2018 and the test datasets on March 2018.
The relevance labels on the testing data (required by active learning techniques) were provided to participants on 1 May 2018, four days before the submission deadline so that participants could not tune their systems towards the actual labels. Task 3 had seven teams submitting runs: one team from Australia (QUT), one team from Botswana (UB-Botswana), one team from Czech Republic (CUNI), one team from Italy (IMS Unipd), one team from Portugal (UEvora), one team from Spain (SINAI), and one team from Tunisia (MIRACL). Participants submissions were due by June 8th 2018 and the relevance assessments are being collected at the time of writing of this paper. See the Task 3 overview paper for further details and the results of the evaluation  [10]. Run MAP R@50 R@100 R@200 R@300 R@400 R@500 R@1000 R@2000 R@k auth run1.
Test collections generated by each of the three CLEF eHealth 2018 tasks offered a specific task definition, implemented in a dataset distributed together with an implementation of relevant evaluation metrics to allow for direct comparability of the results reported by systems evaluated on the collections. The established CLEF eHealth IE and IR tasks (Task 1 and Task 3) used a traditional shared task model for evaluation in which a community-wide evaluation is executed in a controlled setting: independent training and test datasets are used and all participants gain access to the test data at the same time, following which no further updates to systems are allowed. Shortly after releasing the test data (without labels or other solutions), the participating teams are to submit their outputs from the frozen systems to the task organizers, who are to evaluate these results and report the resulting benchmarks to the community.
Acknowledgements. The CLEF eHealth 2018 evaluation lab has been supported in part by (in alphabetical order) the  ANU , the  CLEF Initiative , the Data61/ CSIRO , and the  French National Research Agency (ANR) , under grant CABeRneT  ANR-13-JS02-0009-01. We are also thankful to the people involved in the annotation, query creation, and relevance assessment exercise. Last but not least, we gratefully acknowledge the participating teams' hard work. We thank them for their submissions and interest in the lab.
The enterprise level software application that supports the strategic product-centric, lifecycle-oriented and information-driven Product Lifecycle Management business approach should enable engineers to develop and manage requirements within a Functional Digital Mock-Up. The integrated, model-based product design ENOVIA/CATIA V6 RFLP environment makes it possible to use parametric modelling among requirements, functions, logical units and physical organs. Simulation can therefore be used to verify that the design artefacts comply with the requirements. Nevertheless, when dealing with document-based specifications, the definition of the knowledge parameters for each requirement is a labour-intensive task. Indeed, analysts have no other alternative than to go through the voluminous specifications to identify the values of the performance requirements and design constraints, and to translate them into knowledge parameters.
In 1990, Gero  [1]  proposed the FBS ontology where F stands for the set of functions, B for the set of expected behaviours (Be) and the set of actual behaviours (Bs), and S for the structure. In  [2] , Christophe extends the FBS ontology to RFBS by including the R for requirements. Back in the nineties, in his theory of axiomatic design, Suh  [3]  defined four domains of activities: the customer domain, the functional domain, the physical domain and the process domain. Stepping back and looking at these product design methods, which could also be assimilated to the systems engineering process  [4] , we notice that product design relies upon an iterative process among requirements, functions, behaviours and structures. The Dassault Système's ENOVIA/CATIA V6 software solution proposes a similar integrated product design model named RFLP  [5].
The R is for ENOVIA V6 Requirements, a requirements management workbench. The F, L and P layers are used to recursively break down the complexity of the design problem according to the Functional, Logical and Physical viewpoints of the product. This design approach follows from Descartes' reductionism method that consists in understanding a complicated problem by investigating simple parts and then reassembling each part to recreate the whole. In RFLP, the functional layer (F) relies upon a Functional Flow Block Diagram to design functional architectures in which functions transform material, energy or information input flows into output flows whose consistency is ensured by the matching of input and output typed-ports. The logical layer (L) is the behavioural viewpoint of the product and is materialised by a logical architecture within which each logical unit's behaviour is equation-based modelled with the Modelica 1  language.
g. Word, PDFand 79 % of requirements are written in unrestricted natural language  [8]. For all these reasons, in a "buy approach" of a "make vs buy" decision, OEMs struggle to deliver products that comply with the legal and contractor's requirements. Indeed, when an OEM collects the specifications and the applicable documents the specification refers to, he has no other alternative than to go through the documents to identify the applicable requirements so as to, in fine, provide a product that complies with the contractor's requirements. There are four standard verification methods: inspection, analysis/simulation, demonstration, and test  [9]. In this paper, we benefit from the simulation method that the parametric modelling CATIA V6's capabilities offer in its integrated RFLP product design environment.
From Unstructured Specifications to PPBRs Before presenting the NLP pipeline that generates the PBRs, we must present the concepts of PBR and PPBR. As Micouin introduces in  [13] , a PBR is an unambiguous formal definition of a requirement as a predicate and is defined as follows: PBR : When C ! val O:P ð Þ 2 D ð1Þ This formal statement means: "When the condition C is true, the property P of object type O is actual and its value shall belong to the domain D". A relevant characteristic of the concept of PBR is that it is grammar-free, i. e. a PBR does not have any particular syntactic structure and can therefore be implemented with various modelling language such as VHDL-AMS  [14]  and Modelica. By combining the PBR theory with parametric CAD modelling, we coin the concept of Parametric PBR (PPBR).
Step 13 <PBRs Modelling>: The NLP pipeline ends up with an XML specification that lists the PBRs in a structure that complies with the PPBRs data model in ENOVIA V6 Requirements. Thus, each PBR element has a statement, a nominal value, a minimal value and a maximum value that specify a domain D that can be inferred from the nominal domain and tolerance domain, a physical dimension and a unit attribute. The XML specification can finally be imported into ENOVIA V6 Requirements so as to automatically generate the PPBRs. This pure software development part of our proposal has not been implemented yet. Once the PPBRs have been generated in ENOVIA V6 Requirements, designers can start the design synthesis thanks to the F, L and P layers of CATIA V6 RFLP.
3 Case Study 3. 1 From Unstructured Specifications to PPBRs First, we put ourselves in the shoes of a contractor who wants to acquire a mechanical ring. We only write three requirements  10  to ease the illustration of our proposition. Each requirement is in a different specification (. doc,. pdf and SysML). Then, we play the role of the OEM who receives the specifications. First, we upload the specifications and send them through the NLP processing pipeline. Once it has finished, the NLP processor generates the XML specification of PBRs. In the XML specification, the data structure of the PBRs is defined in such a way that the PBRs are interpretable by ENOVIA V6 Requirements. Therefore, we can generate the PPBRs (Fig. 1 ) from the PBRs. Nevertheless, this functionality is not integrated into ENOVIA V6 because the NLP pipeline is a single capability of a broader on-going research project that requires Java EE as programming language.
Results and Limitations We conducted experiments with both real industrial and handcrafted data sets. The initial results that we obtained after analysing handcrafted specifications are encouraging. One key factor is that we wrote the requirements by following the requirements writing best practices defined in  [19]. Regarding the analysis of industrial specifications, the results are also promising, although we cannot fully validate the proposition without including a units converter. Our solution presents a few limitations, such as the detection of sections when the headings functionality has not been used to edit. doc,. odf or. pdf. We were also challenged by original writing-style that came across while we were testing. The Stanford CoreNLP dependencies analyser relies upon statistics; consequently, it returns few false positive and false negative numerical dependencies.
Acknowledgments The authors acknowledge financial help from  Ministerio de Ciencia y Técnica, Agencia Nacional de Promoción Científica y Tecnológica, MINCyT-ANPCyT-FONCyT  through  PICT/2014-0585 , and through  FONARSEC   FSNano-008 FSNano-008. R. M. T. S. is a member of  CONICET  and F. M. F. and M. G. acknowledge  CONICET  fellowships.
IFDS: implementation As mentioned in the previous section, IFDS was implemented with SNMP and Web Services. SNMP is used by the Monitor and Monitored Hosts to exchange messages and each Monitored Host must be registered on the fdMIB, shown in Fig. 4. fdMIB consists of three groups: the Monitor Host Group, Monitored Host Group, and Application Notify Group. These three object groups are described below. Note that we propose a single MIB that maintains information about all IFDS modules. The Application Notify Group (appNotifyGroup) keeps information about QoS parameters. Each application that is supposed to receive state information about a process must give the following parameters to fdMIB: the upper bound on the detection time (TD_U), the mistake duration time (TM_U), and the lower bound on the average mistake recurrence time (TMR_L).
Experimental results: LAN The LAN experiments were executed on two hosts: the monitor host was based on an Intel Core i5 2. 50 GHz processor, with 4 GB of RAM, running Linux Ubuntu 12. 04 with kernel 3. 2. 0-58; the monitored host was an Intel Core i5 CPU 3. 20 GHz, with 4 GB RAM running Linux Ubuntu 13. 10 with kernel 3. 2. 0-58. The hosts were on a 100 Mbps Ethernet LAN. The fdMIB was implemented using Net-SNMP version 5. 4. 4. The first experiment was executed to evaluate how IFDS adapts to changes in the heartbeat frequency. How does the failure detector react and reconfigures itself? Figure  5  shows the expected arrival time (EA curve) and timeout (Timeout curve) computed by IFDS as the monitored host sends 200 heartbeat messages to the monitor host. The figure also shows the measured RTT (RTT curve).
For client-side JavaScript programs, this definition of API applies both to: (1) interfaces of services that are provided to the program by the environment in which it executes, namely the web browser (for instance, the DOM, the XMLHttpRequest, and the W3C Geolocation APIs); (2) interfaces of JavaScript libraries that are explicitly included by the programmer (for instance, jQuery, Prototype. js, and Google Maps Image API). In the context of this work, the main difference between these two types of APIs is that in the former case their semantics escapes the JavaScript semantics, whereas in the latter it does not. The methodology proposed here was designed as a generic way of extending security monitors to deal with the first type of APIs. Nevertheless, we can also apply it to the second type whenever we want to execute the library's code in the original JavaScript semantics instead of the monitored semantics.
Runtime API Register. We assume the existence of a runtime function called the runtime API register, that simulates the API Register, which we denote by $apiRegister. The function $apiRegister makes use of the domain method of each API in its range to decide whether there is an API relation associated with its inputs, in which case it outputs an object containing the corresponding IFlow Signature, otherwise it returns null. Meta-compiler. Figure  2  presents a new meta-compiler, C API , that receives as input an inlining compiler for JavaScript expressions, C e , and outputs a new inlining compiler that can handle the invocation of the APIs whose signatures are in the range of the API register simulated by $apiRegister. Since statement redexes are not intercepted, the compilation function C s is left unchanged except that it uses the new compilation function for expressions for compiling the subexpressions of the given statement.
The object µ(r 0 ) is updated by shifting by -1 all the indexes equal to or higher than i (the index of the object being removed) and by removing index n. The levels of the indexes of the right siblings of the node to remove are accordingly shifted by -1. The constraint of the rule prevents a program from removing in a high context a node that was inserted in a low context (see  Example 13). [appendChild] The API relation ⇓ app has two different behaviors depending on the fact that the node pointed to by r 2 is or is not an orphan node. If the node pointed to by r 2 is an orphan node, the behavior of ⇓ app is the following: (1) it first checks that the node to append (µ(r 2 )) is not an ancestor of the node to which it is to be appended (µ(r 0 )); (2) it creates a new property n in µ(r 0 ) and sets it to point to µ(r 2 ) (where n is the previous number of children of µ(r 0 )); (3) the level of the new index property n is set to the least upper bound on the levels of the arguments and the level of its new left sibling provided that it exists (in order to enforce the Indexes Invariant); (4) the least upper bound on the level of the arguments must be equal to or lower than the structure security level of µ(r 0 ) because adding an index to a node changes its domain;  (5)  the least upper bound on the level of the arguments must be equal to or lower than the structure security level of µ(r 2 ) (in order to enforce the Parent Node Invariant).
These steps are described herein. To illustrate the operationalization of each step, we have proposed a case study in the flight tickets e-commerce domain in a traditional airline company, showing how the approach can support the testing of Prototypes and Final UIs. This case was chosen because it is easily comprehensible and we believe it represents a common activity for most of the readers. For the study, we have selected the American Airlines (AA) case to show these concepts. The AA model has been arbitrarily chosen to conduct this work. However, the core of business models for this kind of e-commerce is the same for all companies, so any other could have been chosen instead. The online booking process of flight tickets is basically divided into 3 main sub-processes: searches of flights based on a provided set of data, the selection of the desired flight(s) in a list of flights resultant from the search, and finally providing passenger and payment data to conclude the booking.
All the findings here are even more encouraging as they were obtained with a lowresolution 64-taxel TVSS device. Behavioural data obtained in our experiments give credence to further experiments on biological substrates of sensory substitution evaluated by neuro-imagery. In addition, not only the PTD version of the TVSS, constructed specially from nonmagnetic elements, but also the miniaturised electronic TDU version (matrix of 3 cm× 3 cm with 144 electrodes explored by the tongue) can be safely used in the f MRI environment. This unlocks huge new prospects for scientific investigation.
Acknowledgments. This research was supported by the  European Commission Quality of Life  contract  QLG-3CT-2000-01797 , ' Brain Plasticity and Sensory Substitution in Human Blindness '. We gratefully thank  Laurence de Lussy-Kubisa  and  Vincent Chrétiennot  for their assistance and/or technical help, and the volunteers who gave of their time to participate in the study.
The integration time per pixel is set to 10 ms in order to increase the total number of photons recorded and therefore the accuracy of spot center determination. The center position of each SHG spot is retrieved by a Gaussian fit of a spot image profiles along its main symmetry axes. SHG image spots of round shape exhibit typically 170 nm standard deviation (340 nm FWHM), with a total number of photons of ranging between 500 to 1500 depending on the nano-star and on the incident polarization. The accuracy of the center positioning is estimated to be 10 nm, as derived from the theoretical calculation in super-resolution imaging  [35] , accounting for the measured number of photons, the background noise (1 photon) and the pixel size (60 nm) used in the experiment. Thanks to the multiple data set taken for each incident polarization, the mechanical drift of the particle in the sample plane could be identified by following the trajectory of the sliding barycenter point taken for each 0-180 • angular range of the incident polarization.
Acknowledgments The authors thank  Jérôme Wenger  and  Hervé Rigneault  for fruitful discussions. This work has been supported by the  French National Research Agency  (NLOShaping  JC07-195504 ), the  Conseil Régional de Provence Alpes Côte d'Azur and Champagne-Ardenne , the  CNRS-Weizmann NaBi European associated laboratory  and the Nano'mat facilities.