dataesr/scientific-paragraphs-categorization huggingface dataset:
2) with 300K additional triples 5  that we generated with SciCheck. In summary, the main contributions of our work are the following: • We propose SciCheck, a new triple classification technique that uses a variety of features to complete KGs of research concepts with a high precision. • We compare SciCheck with nine alternative KG completion methods on AIKG-1M, AIKG-500, FB13, WN11, WN18, WN18RR, and NELL, showing that it obtains excellent results. • We release two new datasets for KG completion: AIKG-1M, including 1M triples from AI-KG, and AIKG-500, including 500 manually annotated statements. • We provide a real-world use case and apply SciCheck on AI-KG and use it to generate a new version of AI-KG containing 300K additional triples. The remainder of this paper is organized as follows.
Therefore, some language models have been proposed based on different types of KG embeddings to deal specifically with this type of graphs  [48] ,  [49] ,  [50]. Some recent techniques, such as exBERT  [47]  exploit contextualized language models rather than KG embeddings. The novelty of our approach resides in not solely using KG embeddings, language models, or random paths, but on a combination of features that leverages the strengths of embeddings and deterministic path features, and does not suffer from the high hardware requirements of GNNs. Specifically, SciCheck makes use of deterministic path-based and embedding-based features to solve the problem of triple classification in general-domain knowledge graphs, and more specifically, in scholarly KGs. In addition, according to our experimental results, SciCheck is also able to outperform the other proposals in terms of precision, which is essential to complete KGs of research concepts, while still achieving a fair recall.
• CAFE + SciBERT, which extends CAFE by considering features based on the similarity of the embeddings of head and tail, using SciBERT, an alternative BERTbased text embedding model 7  specifically tailored to scientific documents. • CAFE + Ontology, which extends CAFE by considering features that identify the types of head and tail according to the domain ontology (e. g. , AI-KG ontology) and also filters triples whose entities are not consistent with the domain and range restrictions of the relation. These methods were evaluated on the following benchmarks, whose characteristics are summarized in Table  1:  • AIKG-1M, a new dataset that we created from AI-KG. We used a de-reified version of AI-KG, in order to consider only triples which involve tasks, methods, materials, metrics, and other scientific entities.
However, negative examples were not generated for the training split, as specific KG completion techniques usually have a preferred way to generate them automatically  [60]. In total, the training split comprised 860,512 positive triples and the testing split includes 430,280 triples (50% positive and 50% negative). • AIKG-500, a new dataset that we constructed by manually annotating triples in AI-KG about the Semantic Web. To construct it, we randomly selected 250 triples which had as their head one of the 24 sub-topics of the Semantic Web according to the CSO ontology  [61]  and were considered to be correct by at least 2 methods among TransE, TransD, TransH, SimplE, ComplEx, and SciCheck. Another 250 triples were randomly selected out of those deemed incorrect by at least 2 techniques. The resulting 500 triples were manually annotated by five domain experts, with an inter-reviewer agreement of 0.
61 (according to Cohen's kappa), which is typically considered a substantial agreement. A majority vote approach was used to determine that 221 triples were correct and 279 were incorrect. Since this dataset was created for the purpose of providing a small but high-quality and manually-annotated testing split, in this evaluation we used AIKG-1M for the training split. • FB13  [34] , a subset of FreeBase  [51]  that focuses on relevant people and their family relations, locations, professions, and other personal data. • WN11  [34] , a subset of WordNet centered around different semantic relations between over 38K words. • WN18  [62] , which expands WN11 with additional relations. • WN18RR  [63] , which improves WN18 by removing reciprocal relations in the test set. This makes triple classification more challenging, since otherwise the model can predict that a triple < a, hasChildren, b> is true whenever the triple < b, hasParent, a> appears in the training set.
• NELL  [39] , a subset of the NELL KG  [35]  with information and relations about many different domains, e. g. , actors which starred in movies, writers and their works, or athletes and their teams. It is well-known  [63]  that these traditional benchmarks suffer from information leakage between the training and test sets, due to the presence of reciprocal relations. For this reason, we removed all reciprocal relations in all datasets except WN18, since we also include its previously discussed sanitized version, WN18RR. To predict the correctness of a triple, we used feedforward neural networks with 3 intermediate layers containing 128, 64 and 32 neurons, respectively. The output neuron uses a sigmoid function, returning a confidence score in the interval [0, 1]. The classifier was trained throughout 100 epochs, using a binary cross-entropy loss function.
They also confirm that completing scientific KGs is indeed a challenging task that requires specialized techniques, as the general purpose embedding-based approaches yield worse results on benchmarks extracted from AI-KG in comparison to generic ones. In order to assess the scalability of our solution, Table  6  reports the seconds used by SciCheck to process the previously discussed datasets. To ensure statistical significance, we measured the runtime for each benchmark 10 times, and we report the average and the standard deviation for each one. Table  6  shows that the runtime ranges from a few seconds to over two hours according to the dataset. These differences are caused by mainly two factors. First, the amount of distinct entities corresponds directly to the number of RoBERTa embeddings that have to be computed, which are typically quite time-consuming.
Hence, a larger number of entities has a negative impact on runtime. Second, and most importantly, the specific topology of every KG affects the size of the neighborhoods of the entities, and thus also affects the time it takes to compute features on them. The case of FB13 is particularly noteworthy since, in contrast with the other datasets, it contains many entities with a very high cardinality. This causes the of the entity neighborhoods to grow exponentially in size, resulting in longer runtimes. Finally, in order to establish a fair comparison with the existing embedding-based KG completion approaches, Table  7  reports their runtime in seconds compared to that of SciCheck for the AIKG-1M dataset. Embedding-based KG completion approaches were run using 1, 000 iterations, as it is commonly done by related studies  [24] ,  [26] ,  [31] ,  [53].
To analyze the general truth value of each claim is not currently possible. Therefore, triples in AI-KG are devised to be a means for representing specific claims by researchers. For example, the entity sentiment_analysis only represents the concept or idea of sentiment analysis as it is described in the original corpus of papers, but it is not aimed to represent or include all available prototypes and implementations to predict sentiments and emotions available today. In fact, such a modeling would require to promote research entities from concepts to classes to describe specific ontological knowledge (e. g. , by defining an ontology to describe how sentiment analysis prototypes can use datasets and machine learning approaches) which is out of the scope of AI-KG. For instance, a triple < deep_model_cnn, usedBy Task, toxicity_detection> from the paper  [57]  should be interpreted in the context of the same paper  [57]  i.
e. , deep model cnn is used for toxicity detection in  [57]  and, more broadly, some deep model cnn can be used for toxicity detection. Neither an interpretation like all deep model cnn are used for toxicity detection nor deep model cnn must be used for toxicity detection are correct according to the design and use of the current implementation of AI-KG. AI-KG is adopted by several organizations for characterizing the AI domain and it has been used for supporting several research efforts, e. g. , for extracting entities from scientific publications  [67] , describing competencies  [52] , and classifying scholarly articles  [68]. AI-KG was generated by using Natural Language Processing (NLP) and Machine Learning (ML) methods for extracting entities and their relationships  [69]. More specifically, AI-KG adopts a pipeline process that is applied on natural language scientific texts to (i) detect entities using a domain-specific extractor based on transformers  [70]  and a topic classifier developed on top of the CSO ontology  [71] ; (ii) identify relationships between entities by using open-and domain-specific ML and NLP tools  [70] ,  [72] ,  [73] , and (iii) define which facts make sense according to an ontology representing the domain semantics.
In addition, to determine whether a fact makes sense, the authors adopted a support score defined as the number of research papers where the fact was extracted from. The reader can find more details about this methodology in  [2] ,  [69]. The current version of AI-KG consists of research entities belonging to one of the following classes: • Task: A research challenge or a certain work to perform. • Method: A research proposal or approach whose aim is to perform a certain task. • Material: Resources that are employed for a certain research task, e. g. , a dataset, an image, a text corpus. • Metric: Entities that can be quantified and are used to measure the quality of a certain method. • OtherEntity: A class used to group entities that cannot be classified in any of the previous ones. The relations were created by clustering frequent verbs and asking human experts to define domain and range restrictions as well as transitiveness.
Some examples of object properties are evaluatesMethod, includesMaterial, or usesMethod. The ontology of AI-KG is available online. 8  Although the extracted facts compose a large-scale KG, the mining of such knowledge from natural language is an errorprone and challenging task and, therefore, it tends to have low coverage, i. e. well-known facts might not be materialized within the KG. As a result, AI-KG is sparse and incomplete. For example, the well-known fact <neural_network, usesMaterial, rdf_graph> cannot be found in the current AI-KG resource despite the fact that RDF graphs are the input of most of the existing neural network-based link prediction and triple classification algorithms. For this reason, scientific KGs are calling for specific approaches for their completion  [47]. However, state-ofthe-art methods developed for general-domain KGs such as TransE, TransR, RotatE, and so on fail to predict triples with a good accuracy on AI-KG.
Diego Reforgiato Recupero  (This work was supported in part by the  Spanish Ministry of Science, Innovation and Universities , under Grant  PID2019-105471RB-I00 ; and in part by the  Office for Economy and Knowledge of the Andalusian Regional Government  under Grant  P18-RT-1060  and Grant  US-1380565. funding provided by ' Università degli Studi di Cagliari ' within the  CRUI CARE Agreement 125880.
Then we will discuss recent advances in benchmarking, validation, and comparison of available spike sorters. In particular, we will describe the SpikeInterface  [12]  and SpikeForest  [50]  projects, as common efforts to provide a unified framework for spike sorting and a quantitative comparison among a large number of well-established spike sorters on a large body of ground-truth datasets. Finally, we will discuss the open challenges of the spike sorting field and provide an overview on how, in our opinion, the future of spike sorting will look like.
Drift correction One of the major disadvantages of extracellular recordings over calcium imaging is that the tissue in the vicinity of the electrodes remains invisible. Thus the number, the types, and more importantly the positions of the neurons in the surrounding of the neural probe is unknown. The latest is currently the most challenging problem in the field of spike sorting, since one wants to guarantee the identities of the cells over times. While recording, as shown in figure  2 (A), it is possible for cells to drift over time and change their    [82, 83] -p1 dataset. The row shows the depth of all the detected peaks (positional raster plot, see text) as function of the time. As one can see, when the probe starts moving (red dotted line), so does the depth of the peaks, since neurons are drifting across channels.
Deep learning for spike sorting curation The last strategy to utilize DL in the spike sorting process is to build models that automatically curate a spike sorting output. Differently from all the strategies described above, in this case the DL model will take a raw output from a spike sorter as input and output a curated version of the spike sorted units (figure  4(D) ). Manual curation of spike sorting output is still a very common and required step to improve the results of automatic spike sorters before further analysis. Spike sorters, in fact, do make mistakes  [12, 50]  and an expert electrophysiologist can correct such mistakes by inspecting features of the spike sorted units. Several software tools (Phy  [70] , Neuroscope  [31] , CellExplorer  [61] , JRClust  [39] ) are used by the community to remove bad or noisy units, merge oversplit units, and split unit clusters (when overmerged or when contaminated by noise events).
The former two curation actions (removing bad units and merging) could be well suited for a supervised DL approach, since they do not modify the underlying spike trains. However, this solution is still relatively unexplored. One option could be to gather all the annotated curation steps and train a network using these community-labeled data. However, since it is well known that manual curation is prone to bias  [70, 94] , a second or complementary option could be to use ground-truth datasets to train a network to perform the best curation steps given the ground-truth spiking activity.
Paired recordings These GT data are obtained by recording the same neuron simultaneously with an extracellular probe and a second measuring modality, such as a glass micro-pipette inserted in the target neuron (intracellular)  [91] , attached to the cell membrane (juxtacellular)  [56] , or via whole-cell patch-clamp  [2, 52, 100]  (figure  5 (1)). The second modality enables to record spikes of the target neuron unequivocally and reliably, so that they can be used as ground-truth. While paired recordings are the most realistic source of ground-truth data, the acquisition of such paired dataset is arguably challenging, especially in vivo  [2, 52, 56]. A second limitation of this approach is its very low-yield: in most cases only a single ground-truth neuron is available for each dataset and this prevents a full characterization of the spike sorting performance.
Synthetic recordings A third strategy to obtain GT data is to use simulations of extracellular signals and corresponding spiking activity  [10, 14, 29, 39, 55]  (figure  5 (3)). In this case, the GT dataset allows for an exhaustive and full benchmark of spike sorting results, since all neurons making up the spiking activity are known. However, the debate on whether synthetic recordings are realistic enough to capture subtleties of experimental data is still open. Nevertheless, simulations can provide a powerful and highly controllable tool to develop and benchmark spike sorting tools. As an example, controlled simulations were used to specifically assess the performance of spike sorters against spike collisions, which are a well known challenge for accurate spike sorting  [26]. With these different ground-truth recordings, one can start approaching the task of systematically benchmarking the performance of multiple spike sorting tools.
However, this step is challenging for several reasons: first of all, in extracellular electrophysiology, there is a large variety of proprietary formats and acquisition systems; second, the community has produced several spike sorters, each coming with its own requirements in terms of input format, parameters, configuration files, and output format. SpikeInterface is an open-source software tool developed to unify existing technologies into a single framework  [12]. Providing a standardized access to the multiple acquisition formats and spike sorters, SpikeInterface allows one to run and assess the performance of many sorters on the same recordings. Strikingly, when six modern spike sorters were run on the same Neuropixels dataset  [80]  (figure  5(B. 1) shows a snippet of the traces) there was a large disagreement among them (figure  5(B.
With the ever increasing effort of gathering ground-truth and curated datasets  [50] , it is of paramount importance for spike sorting developers and end-users to collaborate in order to identify and converge on common curation strategies to reduce the burden and bias of manual curation. A third but computationally expensive option to avoid manual curation could be running several spike sorters on the same recording and aggregating the results in an ensemble fashion  [12]. While this ensemble strategy produced curated outputs which largely overlapped with manually curated ones when applied to a neuropixels recording  [12] , its validation is still limited and more testing is required. Reproducibility can also be affected by the intrinsic life cycle of spike sorting software tools, which can undergo many modifications and releases as bugs are fixed, methods are improved, and new features are added.
While version controlled systems (VCS) can keep track of all these changes, from a user perspective it can be confusing and even complicated to know which exact version of a spike sorter was used on specific datasets and to report correctly. A possible solution to the version problem is to encapsulate spike sorters of different versions in containers, i. e. pieces of software that package all codes and dependencies for an application to run independent of the environment. Containers can be run through platforms such as Docker  [54]  and Singularity  [42]  and ensure reproducibility on the software side. The SpikeForest project first introduced containerized and version-tagged containers of 11 different spike sorters which we recently collected and distributed through SpikeInterface 5. In addition to improving reproducibility of the results, containers can also ease the process of installing and setting up different spike sorters, which can be difficult as the number and complexity of dependencies increases.
A possible solution forward is to move both data storage and spike sorting fully remotely, on cloud computing services  [1, 67]. A remote and centralized solution for spike sorting could solve several issues that we have introduced so far. As new probes with even more electrodes are developed and more and more recordings are generated by the neuroscience community, cloud-based solutions (e. g. the DANDI archive https://gui. dandiarchive. org/#/ and the EBRAINS platform https://ebrains. eu/) could provide on-demand storage as required and would prevent individual institutions to constantly upgrade their servers and storage capabilities. In addition, when uploading data onto a remote server, standardized file formats could be enforced, e. g. NWB  [72, 85] , NIX  [53] , and BIDS  [27] , and as a by-product several standardized tools for data processing and visualization can be developed.
Acknowledgments This work was supported by the  ETH Zurich  Postdoctoral Fellowship 19-2 FEL-17 (A P B). In addition, we would like to thank  Julien Boussard  for fruitful conversations on the spike localization and drift correction topic. We are thankful to  Nick Steinmetz  for acquiring and sharing the Neuropixels dataset with imposed drift movements that we used to generate figures 2 and 3. Finally, we wish to thank the spike sorting community for sharing and discussing exciting updates over the past year:  Jeremy Magland ,  Liam Paninski ,  Matthias Hennig ,  Jeff Soules ,  Fernando Chaure ,  Hernan Rey ,  Julien Boussard ,  Erdem Varol ,  Cole Hurwitz ,  Ben Dichter , and  Olivier Winter.
Tiling representation in COSMO The main coupling-related development lies in the implementation of tiling in COSMO to represent heterogeneous ocean surface cells, containing both open ocean and sea ice. Such methods have first been developed in land models (e. g. ,  Koster and Suarez, 1992) , then generalized to mixed surfaces (e. g. ,  Best et al. , 2004)  and subsequently implemented in state-of-the-art surface modules (e. g. ,  Voldoire et al. , 2017). The standard COSMO configuration requires a single set of surface values over each cell, which is then used to compute vertical diffusion. The general strategy is to have COSMO compute a set of two tile-specific surface fluxes, from two distinct sets of surface properties corresponding to the open ocean and the sea ice. This is performed on all surface cells, regardless of the sea-ice concentration.
f. ETISh configuration The f. ETISh model is run on a regular polar stereographic grid of 701×701 grid points centered around the South Pole, with a horizontal resolution of 8 km. The f. ETISh time step is 1/60 yr. The domain encompasses the continental shelf and shelf break, which limits the maximum possible extension of the ice sheet. Bedrock topography, bathymetry, observed ice thickness and grounding-line position are taken from Bed-Machine Antarctica  (Morlighem et al. , 2020). This dataset, originally at a 500 m spatial resolution, has been resampled to the f. ETISh 8 km grid using a low-pass filter. Data were subsequently corrected to ensure grounded ice being actually grounded and floating ice actually floating. The initial ice-sheet geometry for the PARASO setting is derived from the initialization and relaxation described in Sect.
4. 3. For the PARASO configuration, iceberg calving has been omitted (but they are part of the NEMO forcing; see Sect. 4. 1. 2). The land-ice extent is kept constant as the NEMO-COSMO interface has not been designed to deal with an evolving land-sea mask. This is one clear limitation which prevents PARASO from being relevant at multidecadal and longer timescales: in that case, the rapidly melting ice-shelf cavities would be carved from below and reach an unrealistically thin shape. Growing ice shelves are cut off at the calving front derived from BedMachine Antarctica  (Morlighem et al. , 2020) , but the equivalent iceberg melt flux is not injected into NEMO (an external iceberg melt dataset is used; see Sect. 4. 2. 2). Moreover, in accordance with the constraints listed in Table 1, shrinking ice shelves cannot become thinner than 11 m (≈ 10 m draft, which is considerably smaller than typical draft values).
9 in  Flather, 1994)  for 2-D dynamics (sea surface height and barotropic velocities) and a flow relaxation scheme for the baroclinic velocities, temperature and salinity. A constant 86. 4×10 -3 W m -2 geothermal flux is imposed at the bottom of the ocean  (Stein and Stein, 1992). River runoff is adapted from  Dai and Trenberth (2002)  and prescribed as a climatological dataset  (Bourdallé-Badie and Treguier, 2006). While icebergs are not dynamically simulated, their freshwater melt input is injected as additional runoff from an interannual dataset obtained from a previous iceberg-including NEMO simulation covering 1979 to 2015  (Marsh et al. , 2015; Merino et al. , 2016; Jourdain et al. , 2019). Buoyant plume mixing resulting from runoff is parameterized with enhanced mixing in the shallowest 10 m, over the river deltas and on a stripe between 20 and 200 km from the Antarctic coast (where most of the iceberg melting occurs).