dataesr/scientific-paragraphs-categorization huggingface dataset:
Introduction Typed functional programming languages come with rich type systems guaranteeing strong safety properties for the programs. However, the restrictions imposed by types, necessary to banish wrong programs, may prevent us from generalizing over some particular programming patterns, thus leading to boilerplate code and duplicated logic. Generic programming allows us to recover the loss of flexibility by adding an extra expressive layer to the language. The purpose of this article is to describe the user interface and explain the implementation of a generic programming library 1 for the language OCAML. We illustrate its usefulness with an implementation of a type-safe deserialisation function.
A Case for Generic Programming Generic Traversals Some common operations on structured data, eg. abstract syntax trees require a lot of boilerplate code to traverse the data structure and modify it recursively, or extract some result. This boilerplate code needs to be adapted to each new datatype. When writing specific traversals over an AST using pattern matching over the constructors, it often happens that only a few cases carry the meaningful computation, others being default cases. Boilerplate removal allows us to write such function by only giving the meaningful cases. In addition to conciseness, this has the benefit of making the code robust to changes in the AST type: since the same function would treat additional constructors using the default case. Generic traversals is the focus of section 3.
let rec show ∶ type a. a ty → a → string = fun t x → match t with Int → string_of _int x String → "\"" ∧ x ∧ "\"" List a → "[" ∧ String. concat "; " (List. map (show a) x) ∧ "]" Pair (a, b) → "(" ∧ show a (fst x) ∧ ", " ∧ show b (snd x) ∧ ")" Fun (a, b) → "<fun>" Such a definition by pattern matching is suitable for a closed type universe where the cases may be given exhaustively. However we want our universe to be extensible, so that we may add new types. Consequently the type indexed functions must also be extensible: as new type witnesses are added to ty, new cases must be added to the type indexed functions. This problem of extending a datatype and a function on that datatype is known as Wadler's expression problem  [37]  and is indicative of the modularity of the language. Solutions in Haskell have been given involving type classes  [35, 4]  and cannot easily be adapted to OCAML.
In particular, pattern matching the deserialised value does not work as expected. The function reinstate fixes that: when given a deserialised value, it returns a value structurally equal to the one that was serialised: reinstate (unmarshall (marshall x)) = x In details, when an extensible value has been deserialised, its memory representation will be different from that of the value before it was serialised. This is because constructors of extensible variants are implemented as object blocks (same as OCaml objects), and they get assigned a unique identifier when created. The expression reinstate ext x replaces the constructor of x with the original constructor object that is stored in the a extensible data structure ext. Custom Custom data are defined outside of OCAML, typically in the language C.
They are considered as abstract from an OCAML perspective. The only information available is their identifier, given by the homonymous field of the C-struct custom_operations defined in <caml/custom. h>. Abstract No information is associated with an abstract type, except for its name, thus respecting the desire of the programmer to hide the concrete implementation of the type. Still, we may run generic functions over abstract types if they have a public representation. This is explained in section 2. 3. 7. type a abstract = {name ∶ string ; module_path ∶ string list} NoDesc The constructor NoDesc is used to signify that a view is not yet, or cannot be, associated with a type. For instance a function type does not have a meaningful generic view.
Open Recursion In the OCAML compiler libraries, one can find the modules Ast_mapper and Ast_iterator whose purpose is to ease the definition of traversals over the Parsetree mutually recursive data types. Both Ast_mapper and Ast_iterator implement open recursion which takes the form of a large record mapper (resp. iterator) where each field corresponds to one of the recursively defined types and is a function that takes a mapper (resp. iterator) and a value of the corresponding type, and outputs a value of the same type (resp. unit). Defining mappers (resp. iterators) is usually done by modifying default record implementing the identity (resp. a traversal of the tree without side effect). Only specific fields of interest need to be modified, the rest being taken care of by the default behaviour.
A Case Study: Safe Deserialisation Serialisation and deserialisation in OCAML are provided by the standard library module Marshal. They are generic functions defined in C that rely on the concrete structure of the runtime values of OCAML programs. There is a serious safety issue with the use of the deserialisation primitive: since it must be able to reconstruct values of any type from an input channel, it is polymorphic in its return type: val from_channel ∶ in_channel → a This may easily cause a segmentation fault if the deserialised value is used with the wrong type. This problem may be solved using the generic library, constraining the return type by its witness: val from_channel ∶ a ty → in_channel → a The low level generic view gives us the type structure to guide the deserialisation process.
Outline of the Algorithm The algorithm follows the same approach as the work of Henry et al. [11]  to check that the result of the standard deserialisation function is compatible with a given type. A substantial difference is that our implementation also deals with abstract types. This involves converting the abstract values to a public representation before serialisation, and converting back the representation to the abstract type after deserialisation. Therefore, the heart of the program is a function convert that not only deals with such conversions but also does the compatibility check. The function convert takes a direction argument (to or from the public representation), a type witness and converts directly the runtime values using the Obj module, which gives an API to the memory representation of OCAML values.
Related Work Generic programming is a very rich topic that we have barely touched in this article, the reader may consult the following tutorials for a deeper understanding  [20, 14, 13] , mostly in the context of the Haskell type system and programming language. More fundamental ideas and higher levels of generality can be obtained using dependent types, as for instance in  [3, 2]. However, we only compare here our library to those that have been developed in Haskell or ML-like languages, whose expressive power and usage is similar to OCAML. Generic libraries have blossomed in the past twenty years, and the many different approaches have been compared extensively  [34, 15, 17]. Our design with separate type witnesses and generic view was directly influenced by Hinze and Löh  [17].
end There is an inherent problem with this approach: the type representations are not unique as they must be instantiated with the module of the generic function that is called. Deriving  [40]  is an extension to OCAML implemented using the preprocessor Camlp4. Generic functions are defined over the structure of types definitions, using a special syntax. The extension is used to implement a safe deserialisation function which supports sharing and cycles, and allows the user to override the default behaviour for specific types. SYB was implemented in MetaOCaml extended with modular implicits and was shown to perform on par with manually written traversals  [41]. The implementation uses an extensible GADT for type witnesses, like in our library. There is no support for extensible type indexed functions.
We use the Stanford parser to generate trees for Europarl source English sentences. We train SkipThought vectors  (Kiros et al. , 2015)  by predicting the next sentence given the current one  (Tang et al. , 2017) , on 30M sentences from the Toronto Book Corpus, excluding those in the probing sets. Finally, following  Conneau et al. (2017) , we train sentence encoders on Natural Language Inference using the concatenation of the SNLI  (Bowman et al. , 2015)  and  MultiNLI (Bowman et al. , 2015)  data sets (about 1M sentence pairs). In this task, a sentence encoder is trained to encode two sentences, which are fed to a classifier and whose role is to distinguish whether the sentences are contradictory, neutral or entailed. Finally, as in  Conneau et al. (2017) , we also include Untrained encoders with random weights, which act as random projections of pre-trained word embeddings.
Recall that the words containing the probed features are disjoint between train and test partitions, so we are not observing a confound here, but rather the effect of the redundancies one expects in natural language data. For example, for Tense, since sentences often contain more than one verb in the same tense, NB-uni-tfidf can exploit nontarget verbs as cues: the NB features most associated to the past class are verbs in the past tense (e. g "sensed", "lied", "announced"), and similarly for present (e. g "uses", "chuckles", "frowns"). Using bigram features (NB-bi-tfidf) brings in general little or no improvement with respect to the unigram baseline, except, trivially, for the BShift 3 Similar results are obtained summing embeddings, and using GloVe embeddings  (Pennington et al. , 2014).
Untrained BiLSTM-max also performs quite well in the downstream tasks (Appendix). This architecture must encode priors that are intrinsically good for sentence representations. Untrained BiLSTM-max exploits the input fastText embeddings, and multiplying the latter by a random recurrent matrix provides a form of positional encoding. However, good performance in a task such as SOMO, where BoV fails and positional information alone should not help (the intruder is randomly distributed across the sentence), suggests that other architectural biases are at work. In-triguingly, a preliminary comparison of untrained BiLSTM-max and human subjects on the SOMO sentences evaluated by both reveals that, whereas humans have a bias towards finding sentences acceptable (62% sentences are rated as untampered with, vs.
Finally, for the challenging SOMO task, the curves are mostly flat, suggesting that what BiLSTM-max is able to capture about this task is already encoded in its architecture, and further training doesn't help much. Probing vs. downstream tasks Figure  2  reports correlation between performance on our probing tasks and the downstream tasks available in the SentEval 5  suite  (Conneau and Kiela, 2018) , which consists of classification (MR, CR, SUBJ, MPQA, SST2, SST5, TREC), natural language inference (SICK-E), semantic relatedness (SICK-R, STSB), paraphrase detection (MRPC) and semantic textual similarity (STS 2012 to 2017) tasks. Strikingly, WC is significantly positively correlated with all downstream tasks. This suggests that, at least for current models, the latter do not require extracting particularly abstract knowledge from the data.
Indeed, logical entailment requires deeper semantic analysis than modeling similarity judgments. TopConst and the number tasks negatively correlate with various similarity and sentiment data sets (SST, STS, SICK-R). This might expose biases in these tasks: SICK-R, for example, deliberately contains sentence pairs with opposite voice, that will have different constituent structure but equal meaning  (Marelli et al. , 2014). It might also mirrors genuine factors affecting similarity judgments (e. g. , two sentences differing only in object number are very similar). Remarkably, TREC question type classification is the downstream task correlating with most probing tasks. Question classification is certainly an outlier among our downstream tasks, but we must leave a full understanding of this behaviour to future work (this is exactly the sort of analysis our probing tasks should stimulate).
5 Related work  Adi et al. (2017)  introduced SentLen, WC and a word order test, focusing on a bag-of-vectors baseline, an autoencoder and skip-thought (all trained on the same data used for the probing tasks). We recast their tasks so that they only require a sentence embedding as input (two of their tasks also require word embeddings, polluting sentencelevel evaluation), we extend the evaluation to more tasks, encoders and training objectives, and we relate performance on the probing tasks with that on downstream tasks. Shi et al. (2016)  also use 3 probing tasks, including Tense and TopConst. It is not clear that they controlled for the same factors we considered (in particular, lexical overlap and   2017 ) are also interested in understanding the type of linguistic knowledge encoded in sentence and word embeddings, but their focus is on word-level morphosyntax and lexical semantics, and specifically on NMT encoders and decoders.
The community held its initial meeting in February 2017, hosted by the European Institute for Systems Biology and Medicine in Lyon, France. There we recognized a great potential in such type of exchange, especially because, despite different disease contexts, we face similar challenges, ranging from establishing proper tools and standards for knowledge encoding, through visualization of multidimensional data sets, to handling large and complex maps. We decided to meet regularly to help shape the direction where the project is heading. In October 2017, we held the 2nd DMC meeting, hosted by the Luxembourg Centre for Systems Biomedicine in Belval, Luxembourg. Here, we summarize this meeting, highlight important topics and outcomes of our discussions and propose a roadmap for the development of disease maps.
Harmonization of curation standards (see section 'Biocuration and knowledge representation standards') is also difficult, as each of the mentioned tools uses its own encoding of the content, risking an inexact translation when transferring information between sources. An important development addressing this problem is the Web-based editor of diagrams encoded in SBGN: Newt  [8]. The creators of Newt actively participate in the DMC, helping to shape and benefiting from the discussed roadmap. A milestone on the road toward mechanistic, modelingoriented curation will be enabling support for the Systems Biology Markup Language (SBML)  [12]  (see section 'Use of maps for mathematical modeling') during the curation of disease maps.
The interoperability between existing standards approved by the community, such as SBGN Markup Language (SBGN-ML), SBML 3. 0 with Layout and Render extension and de facto standards used to construct most of disease maps, like the CellDesigner proprietary SBML extension, remains a challenge. However, this aspect happened to be relatively inexpensive to improve. For instance, at the time of writing, a new fully functional bidirectional converter from CellDesigner to SBGN-ML has been developed as a collaborative effort between DMC members (https://github. com/royludo/cd2sbgnml). Such tools will allow the use of the rich computational systems biology toolkit to analyze the existing collections of disease maps.
Access to bioinformatic databases Disease maps applications that focus on knowledge exploration require easy and direct access to various data resources. MINERVA and NaviCell platforms provide such access to a number of annotation sources, like HUGO Gene Nomenclature Committee (HGNC), UniProt, Chemical Entities of Biological Interest (ChEBI), PubChem or Gene Ontology  [46] [47] [48] [49]. From our experience, users can better understand representations of particular disease mechanisms if they can cross-check descriptions of the included molecules. However, advanced data interfaces are needed, such as querying pathway databases for entire sequences of reactions from Reactome or WikiPathways  [50, 51]. Newt implements such functionality for drawing interactions. A corresponding feature for visual exploration of disease maps remains to be implemented.
Health and disease data interpretation pipelines Clinical applications of disease maps  [45, 52]  are close to the role of a Clinical Decision Support System, with an emphasis on exploration and interpretation of medically relevant data. Big health data, collected in great amounts by health-care providers and pharmaceutical companies, need to be structured and interpreted through visualization. This is a scenario where disease maps may provide a valuable context to large data sets, allowing meaningful filtering and summary of otherwise indigestible numbers. Initial steps in creating big health data pipelines to disease maps have been taken  [45] , where a disease map is used to visualize gene expression based on patients' demographic data. In the end, disease maps may be a great support to knowledge-based drug discovery using patients' data, but only after drug databases can be linked with the maps' content and supported by dedicated analytical pipelines.
For instance, disease maps may become a platform for network data-driven drug response prediction. This will require identification and assessment of disease-rewired pathways, network analysis to identify a desired intervention set (target interactions or elements in the network) and mapping this intervention set back to drug databases, looking for secondary use of existing medications (drug repositioning). The final goal of a disease map development is to become mathematically interpretable and to support clinical decisions in a given domain. Importantly, the process of refining and exploring a disease map itself provides knowledge building, even without an immediate clinical application. Although the map is created to be quantified and analyzed with data to predict a clinically relevant outcome, its qualitative interpretation can have a great value in hypothesis generation and for guiding experimental design.
Parameterization or executable mathematical models Quantitative mathematical models usually possess unknown parameters, e. g. binding affinities and degradation rates. To ensure that the models are predictive, these parameters have to be estimated from experimental data. This requires comprehensive data sets as well as computational methods for statistical inference. Data sets are available in the literature and in established databases, such as BRENDA  [57]  and SABIO-RK  [58]. However, most literature-based data sets are unstructured and difficult to assess. Furthermore, the quality of experimental data varies heavily. A milestone for any disease map project aiming at quantitative models therefore is the establishment of a database of general and disease-specific data. The databases could be created together with the disease maps, and encode essential qualitative properties as well as quantitative data.
The databases established for different projects should ideally follow common standards. To estimate the unknown parameters from the available data, an efficient computational pipeline is required. As disease maps usually possess hundreds or even thousands of state variables and parameters, the resulting computational complexity might be challenging for established toolboxes such as COmplex PAthway SImulator (COPASI)  [59] , Data2Dynamics  [60] , Parameter EStimation TOolbox (PESTO)  [61]  or PottersWheel  [62]. Moreover, such a large number of variables will require an automated procedure to check parameter identifiability. A milestone is the establishment of a scalable computational pipeline, which is applicable to the standardized models and databases established in the disease map projects.
Summary A 'disease map lifecycle', as shown in Figure  2 , starts with curation and integration of knowledge about disease mechanisms. This collected knowledge, combined with experimental data and annotations from bioinformatics databases, supports better understanding of the disease and formulation of systems-level, data-driven hypotheses. The 'disease map lifecycle' is a dynamic process, as feedback from the interpretation of such contextualized knowledge leads to the design of further, tailored data interfaces, permits better consolidation of knowledge within the repository and may, if validated experimentally, introduce new knowledge about disease mechanisms for further curation and incorporation into the map. The milestones of the community-driven roadmap (Figure  1 ) are indicated in Figure  2.
Application example: drug repositioning Signaling pathways implicated in human diseases create a complex network with redundant pathways. This complexity explains frequent failure of one-drug-one-target paradigm of treatment, resulting in drug resistance in patients. To overcome the robustness of the cellular signaling network, the treatment should be extended to a combination therapy scheme  [68]. Disease maps allow integrating patient high-throughput data together with the information about biological metabolic and signaling machinery specific to a given disease. This in turn may help deciphering molecular patterns specific to each patient and finding the best combinations of candidates for therapeutic targeting. A simple drug repositioning scenario may involve creating data overlays for tissue-specific gene and protein expression and their visual analysis for spatial and temporal patterns in signaling cascades encoded in a given map.
Thematic highlight: mathematical modeling in human diseases research The thematic highlight of the 2nd DMC meeting was mathematical modeling and disease maps. Building a computational model from a disease map is a process of transformation of a static literature-based representation into a dynamic executable format. This is important for a better understanding of how a disease progresses over time. It is also an environment where hypotheses and assumptions can be added and tested. Here, the prior knowledge (literature curation) can be integrated with newly generated data including omics data. Different types of computational models can be developed on the basis of the same pathway-based disease map. During the community meeting, we started reviewing and discussing possible approaches. N. L.
We reviewed current and upcoming interfaces to pathway databases and data analysis pipelines that will help us to curate and interpret the maps' content. This productive series of meetings will continue. The 3rd DMC meeting is scheduled for June 2018 in Paris, hosted by Institut Curie (http://disease-maps. org/events). We aim to review and update the roadmap and enlarge the community. Most importantly, we would like to maintain the atmosphere of collaboration and open exchange within the community, which is the key to improvement and further development of the Disease Maps Project. There are several tools, approaches and platforms developed by DMC members. Exposure of the participants to these resources will allow active exchange of knowhow, and parallel hands-on tutorials will be provided.
'Biocuration' feedback means introduction of new, validated hypothesis about the disease-related mechanisms. Notes: Milestones discussed for the Disease Maps Roadmap are mapped on the diagram as follows: T: Tools, T1: Modeling-oriented curation, T2: Visualization of simulation results, T3: Information exchange between maps; B: Biocuration standards, B1: Knowledge quality indicators, B2: Review of the text mining support, B3: On-thefly consistency check, B4: Connecting mechanisms and disease hallmarks; C: Complexity management, C1: Dynamic subnetwork collapsing, C2: Algorithms for layered scale complexity, C3: Methods for dynamic layouts, C4: Handling large diagrams; A: Applications, A1: Cross-linking disease maps and pathway databases, A2: Databased tissue specificity, A3: Data interpretation pipelines, A4: Quality assessment via in silico replication; M: Modeling, M1: Minimal information set for modeling, M2: Database of general and disease-specific data, M3: Scalable computational pipeline for models, M4: Model-based individualization approaches.