dataesr/scientific-paragraphs-categorization huggingface dataset:
, 2019). Here we propose to build on the previous DIS-RPT best system and exploit the availability of multiple corpora for the same language, or the same family of languages (romance, germanic) to augment training of dedicated models. Combining this approach with a few adjustments to the base model, we manage to improve on many datasets compare to the previous best DISRPT systems, with a mean difference in F1 score of 0. 46% and 1. 24% on segment boundary detection for sentence and document level respectively, and 3. 07% on connective detection for sentences (we didn't improve results at document level for discourse connectives). Within the Disrpt 21 campaign the system ranks 3rd overall, very close to the 2nd system, but with a significant gap with respect to the best system, which uses a rich set of additional features.
Related work Discourse segmentation appeared as an NLP task with the creation of the first annotated RST documents in English, and was primarily rule-based  (Marcu, 2000). Since then the literature on discourse parsing generally assumed that elementary discourse units (discourse segments) were given, with only a handful of exceptions  (Soricut and Marcu, 2003; Fisher and Roark, 2007; Tofiloski et al. , 2009; Hernault et al. , 2010; Joty et al. , 2015) , until more recent neural-based work  (Wang et al. , 2018; Lukasik et al. , 2020) , still at the sentence level, and always on English or Mandarin. Only the work of  (Braud et al. , 2017b,c)  have considered more varied languages, after the creation of a few different datasets in the past ten years (see the Data section below). The DISRPT 2019 workshop introduced a more general evaluation framework for discourse segmentation with a shared task considering multilingual data, and segmentation both at the sentence and document level  (Zeldes et al.
Data The 2021 shared task provides 16 corpora annotated either with discourse boundaries (13) or discourse connectives in the case of PDTB corpora (3), with the RST Farsi corpus as a surprise dataset. This covers 11 different languages, mostly indoeuropean languages, and with a majority of european languages: 3 romance (Spanish, French, Portuguese), 3 germanic (English, German, Dutch), Russian, the only non indo-european being Turkish, Basque, and Mandarin. Some of the datasets depend on licences for the underlying text corpus, and are not freely available. We had licences for all of them except the Mandarin corpus (zho. pdtb. cdtb), provided by the organizers for the evaluation of the task. Except Farsi, all the datasets were present in the DISRPT 2019 shared task, but the russian Table  1 : Descriptions of all corpora, according to the underlying theoretical framework.
The tasks consist in finding connectives in the PDTB datasets, or the Elementary Discourse Units (segments) in RST and SDRT datasets. For each corpora are listed the number of documents in each split, the number of sentences and annotated units (connective tokens or segment boundaries in the training set, and whether the gold sentences were manually annotated or given by a parser. dataset has been extended since, and some corpora without gold syntax annotations have been reparsed with Stanza or Spacy to provide morpho-syntaxic information for the sentence-internal subtasks. All tasks are considered as sequence tagging, and annotated as such: for segmentation, each token is marked as being a segment boudary or not, and for connectives, which can span multiple tokens, the annotation follows the BIO convention with three labels Begin/Inside/Out for each token.
Datasets are not homogenous, as they were annotated along different principles based on three competing theoretical frameworks: • Rhetorical Structure Theory  (Mann and Thompson, 1988) , which assumes a linear segmentation of documents in discourse units (no overlaps), which are then related in constituant tree structures. This is followed in the majority of corpora (11). • Segmented Discrouse Representation Theory  (Asher and Lascarides, 2003) , which allows for embedded segments, and were linearized here for homogeneity of the task: a segment embedded in another one was re-annotated as forming 3 three segments. This is the case of two corpora. • Penn Discourse Treebank  (Prasad et al. , 2008) , which annotates discourse connectives and their arguments in a discourse relation. This gives rise to a different annotation scheme as noted above, as the task is only to locate the connective.
Base architecture We started from the architecture that showed the best results on almost all languages and configurations at DISRPT 2019, namely  (Muller et al. , 2019) , which is built around BERT  (Devlin et al. , 2019) , a contextual language model that is easy to fine-tune on sequence tagging problems. The original architecture combined BERT contextual embeddings to the output of CNN filters over characters of each word piece, that were then fed to single-layer BiL-STM layer for the final prediction. The model is initialized with the multilingual BERT model, then fine-tune on all corpora separately as sequence tagging tasks. The original implementation used the AllenNLP library  (Gardner et al. , 2017) , and so does our implementation. Since BERT has a limitation on the number of word pieces it can take as input, a preprocessing step must be taken for document-level segmentation.
In  (Muller et al. , 2019) , the core-nlp library was used to predict sentence boundaries, and use this information, while we used the more recent Stanza library by the same team for that purpose  (Qi et al. , 2020). We explored potential improvements for that architecture, swapping the multi-lingual pretrained language model XLM  (Conneau and Lample, 2019) , or adding another layer to the BiLSTM stage. The final configuration was chosen based on preliminary experiments on some of the datasets, evaluated on their respective development sets. They showed that XLM didn't help, but the extra layer of LSTM could. Changes to other hyperparameters didn't improve these preliminary results so we kept them as in the original model. Details of the parameters can be found in the declarative config file of the model, also added as supplementary material.
Dataset grouping Since the shared task involves multiple datasets with the same language (2 for Spanish RST, 2 for English RST), we assumed it would be beneficial to combine them for training. Datasets in the same language are not necessarily consistent in their annotation, but we hypothetize that they have enough commonalities to help training. We also took inspi-ration from work on multi-lingual syntactic parsing where a lot of corpora follow the same formalism, and where past work has tried to use commonalities between different languages, particularly the approach of  (Dehouck and Denis, 2019).
Base model The modification of ToNy, the best system from DISRPT 2019, gives us our base system on which we will build with multi-corpora training in a second stage. We report the results of these systems on segmentation in Table  2 , and on discourse connectives identification in Table  3 , with precision, recall and F1 score on the detection of segment boundary tokens, and discourse connectives. A first comparison with respect to each best subsystem from 2019 for the 4 subtasks is given in Table  6. That means ToNy for segmentation (both intra-sentential and plain), discourse connectives (plain), and Gumdrop for discourse connectives (Conll input). We can see that on average on all datasets, the base system gains +0. 5, mostly due to its improvements on the plain document segmentation (connective detection only involves 3 datasets).
We left out the surprise dataset for 2021 from that evaluation, since we do not have a comparison point. Note that results on this new dataset are good and consistent with the other corpora. Lower results are obtained on smaller datasets for obvious reasons (Spanish sctb, Chinese Mandarin sctb, and to a lesser extent Basque and French). Models trained without sentence information perform a little worse, as expected, with -1. 6% on average, again with wider gaps for small corpora. We do not show the breakup by dataset for the comparison with DISRPT 2019, but there is a lot of variances in results, with differences ranging from -4. 5 (French Annodis) to +6. 85 (Mandarin SCTB), and not necessarily only on small corpora. 4 : Intra sentential results on romance development datasets, with different training setups: SPA means the grouping of both spanish datasets for training, SPO the grouping of spanish and portuguese data, ROM the addition of French to the group.
Multi-dataset training As shown above, a lot of smaller datasets have lower results than larger ones, which is to be ex-pected. We present here the result when applying the strategy described in Section 4. 2. We tried it on two groups of languages: romance languages, and germanic languages. For romance languages, since there are two spanish RST corpora, we grouped them to trained a "spanish" more generic model, then we trained a model with all Spanish and Portuguese data, then a generic romance model by including also French. We then used those models for predictions on the datasets respective development part. We did something similar with germanic languages, grouping all English datasets into one, joining the Dutch and German datasets into another one, and finally training a generic germanic model on all of them.
Following a procedure similar to what was done in  (Dehouck and Denis, 2019) , we also fine-tuned some of these models on the individual datasets before using them for prediction. Lack of time during the campaign prevented us from trying all combinations and all datasets like this, but we tested this on all Spanish and English datasets, respectively fine-tuning the global Spanish-Portuguese on Spanish and Portuguese datasets (since it showed a good compromise on the dev sets) and the global English model on all English datasets. Due to time constraints, we tested this only on one type of input, the sentence level (conll files). Results are presented in Tables  4  and 5. For the final evaluation, we kept for each dataset the model that performed better on the dataset development test.
Final evaluation For the final evaluation of the campaign, every team provided their code and instructions for reproducing the experiments, and one member of the organization team reproduced entirely the experiments of one model they were not involved with (two of the four teams involved organization members, including the present system). We reported the official scores on the test sets in Tables 7 for segmentation and 8 for connective detection. Overall our system is ranked 3rd out of 4, with results very close to the 2nd-ranked system (Segformers), with only 0. 15% difference on average for treebanked data segmentation, and 0. 45% on plain data segmentation. The gap with the first ranked system (DiscoDisco) is 0. 7% on average on treebanked segmentation and 1. 28%  on plain documents. We have the best scores on these languages, but note that cross-training cannot really explain the good results on German, since results are surprisingly similar between treebanked and plain data.
Conclusion We presented an approach for discourse segmentation and discourse connective identification, both at the sentence and document level, in a multi-lingual and multi-formalism context. Building on the successful architecture from the 2019 DISRPT shared task, we leverage datasets in the same or similar languages to augment training data and improve on the best systems from the previous campaign, on 3 of the 4 sub-tasks. While below the best system which uses a rich set of features over a similar architecture, we still manage to have the best scores on some of the languages where we experimented with cross-lingual training: German and Spanish for sentence-internal segmentation. Due to time constraints, we could not fully explore all potentially useful language combinations and fine-tuning on specific datasets, that could help improve on the tasks, and give insights on how different languages help each other addressing the discourse segmentation problem.
Acknowledgments This work was partially supported by the  ANR  ( ANR-19-PI3A-0004 ) through the  AI Interdisci-plinary Institute, ANITI, as a part of France's "Investing for the Future -PIA3" program , and through the project  SLANT  ( ANR-19-CE23-0022 ) and  Quantum  ( ANR-19-CE23-0025 ).
Note that the resolution of this optimization problem can be easily accomplished while using the semidefinite programming (SDP). We opted for the CVX package  [44] , implemented in Matlab, as an advanced convex programming solver, in order to solve  (11)  and obtain X. In this equation, the matrix S ∈ IR N×N relatively reflects our knowledge regarding the spatial structure inherent in the data, since it is computed based on the learning data matrix X l p ∈ IR N×T l p. This spatial matrix expresses the similarities between the sensor nodes' readings. Suitably, we use the Euclidean distance as a distance function in the data domain of the sensor nodes to model the similarity between the rows of X l p , whereby the smaller the distance between two rows, the closer they are. Below are the steps to obtain S: 1-We initiate these steps with an all-zeros matrix S.
Multiple Linear Regression of Simulated Results In the Nordmann method, Equations (  1 ) and (3) above show that TAC is a linear function of the first equivalent volume (i. e. , A in mL). Equations (  2 ) and (  4 ) show that FOS is a linear function of the difference of the two equivalent volumes (i. e. , B in mL). Since FOS and TAC interfere in the titration of each other, two new 3-parameter models were proposed in the present work to calibrate the determination of TAC and FOS. This was a multiple linear regression assuming that TAC and FOS depended on both A and B. With the simulated data to be validated by experiments, and by using the Scilab solver for multiple linear regression, it was possible to identify the model coefficients to calculate the bicarbonate (or equivalent TAC) and acetic acid concentrations: [NaHCO 3 ] new = α 1 ×A+α 2 ×B+α 3 (6) [HAc] new = FOS new = β 1 ×A+β 2 ×B+β 3  (7)  where By applying Equation (  5 ), [NaHCO 3 ] could be converted to the equivalent alkalinity (TAC new ).
Proposing New Formulas for FOS and TAC Estimation Since the determination of both [NaHCO 3 ] and [HAc] depends on each other, it was found that the formulas according to the FOS/TAC method had a lower accuracy when the bicarbonate and/or VFAs concentration was in the average conditions low or high). Two new formulas (Equations (  6 ) and (  7 )) were proposed to recalculate FOS and TAC in a more accurate way. Using the equivalent volumes at pH = 5. 0 and 4. 4 obtained by the simulation at different conditions (12 × 12 = 144 datasets of [NaHCO 3 ] and [HAc] ranging from 0 to 40 g•L -1 ), a multiple linear regression with Scilab was performed in search of their relationships with the true values of [NaHCO 3 ] and  [HAc]. The two new formulas with the identified model parameters are shown in Equations (  8 ) and  (9).
There was a nonsignificant trend toward improved diagnostic estimates of PRECISE recommendations. These findings highlight the need to further define the optimal triggers and timing of biopsy during AS, as well as the need for optimizing the quality, interpretation, and reporting of serial prostate MRI. databases on January 11, 2021, to identify studies addressing the predictive role of serial (at least two scans) MRI for PCa progression during AS. The MRI scans had to been assessed at AS entry (either at diagnostic or at confirmatory biopsy). The search terms included the following: "prostate cancer", "MRI", "magnetic resonance imaging", and "active surveillance". Two investigators performed an independent initial screening based on the titles and abstracts, and noted the cause of exclusion of ineligible reports.
Statistical analysis All statistical analyses were performed using R version 4. 0 (2020; R Foundation for Statistical Computing, Vienna, Austria) and Cochrane Collaboration Review Manager software (RevMan v. 5. 4; Cochrane Collaboration, Oxford, UK). Statistical significance was set at p < 0. 05. Pooled sensitivity, specificity, positive predictive value (PPV), NPV, and diagnostics odds ratio (DOR) were calculated with "mada" and "meta" packages. Forest plots with 95% confidence interval (CI) were calculated and depicted. Cochrane Q test and the I 2 test were used to evaluate the heterogeneity. Significant heterogeneity was indicated by p < 0. 05 in the Cochrane Q tests and I 2 >50%. We developed a hierarchical summary receiver operating curve (SROC) and calculated the area under the curve (AUC) to examine the diagnostic accuracy of MRI progression tested overall and defined using PRECISE criteria or institutional definitions.
63, NPV: 0. 70, 95% CI 0. 34-0. 93) for significant PCa prediction and exclusion, and combination of MRI with clinicopathological variables led to the improvement of diagnostic estimates (AUC: 0. 91, NPV: 0. 96, 95% CI 0. 77-0. 99). Of note, in a study of Giganti et al  [20] , the kinetics of PSAD was significantly associated with PRECISE !4. There is also a growing interest in dedicated MRI reading software programs and radiomics, which may further improve diagnostic estimates of serial MRI  [43, 44]. On the contrary, our results imply that presently serial MRI is a robust stand-alone tool for GG !3 cancer exclusion, which parallels higher primary detection of GG !3 cancers by MRI and lower rates of PCa progression to GG !3  [12]. Furthermore, in a recent study of Chu et al  [45] , published after our formal literature search, both consistently visible and increasingly suspicious lesions were associated with GG !2 detection and definitive treatment.
Furthermore, recently Giganti et al  [49]  introduced the Prostate Imaging Quality (PI-QUAL) system, which scores MRI quality, and revealed that only 60% of all MRI scans in the PRECISION trial had at least good quality. These emphasize the role of MRI quality itself and underline the need for a better quality program assessment for MRI reading. Despite a trend toward improved diagnostic estimates of PRECISE recommendations, our findings indicate that at present, there are some technical boundaries of prostate MRI and its assessment, which do not allow achievement of sufficient diagnostic performance to determine PCa progression during AS  [20, 21]. Standardization of MRI reports in AS follow-up is a key element  [21]. It is crucial to follow PRECISE recommendations and fulfill case reports to minimalize unstandardized comparisons.
Sahni et al  [12]  used half of a data set for comparisons with a baseline model and achieved a 2% to 3% higher accuracy than the baseline model. Camacho-Collados et al  [13]  proposed a simple, distant, supervision-based postprocessing step for using noisy, user-generated text to learn cross-lingual embeddings. They used English data to train classifiers for other languages. Purver and Battersby  [10]  used tag symbols and topic tags as weak tag training models. Without human intervention, the models were used to detect multiple types of emotions in Twitter messages. Mohammed et al  [14]  proposed a novel distant supervision technique for automatically gathering noisy topic category annotations from topically focused streams. Most related research is based on the same annotation standard and eliminates the differences between different data through distance supervision.
In 2018, Devlin et al  [18]  proposed a BERT-based, pretrained language model for dynamically obtaining encoded word vectors according to context. The BERT-based, pretrained language model uses a bidirectional transformer  [19]  as a feature extractor. Compared to traditional word embedding methods, such as Word2Vec  [20] [21] [22]  and Glove  [23] , the BERT method is better for capturing the representation of word and sentence levels. The BERT model is one of the most state-of-the-art pretraining models. Our final model was modified based on BERT, integrated the psychology features (Textbox 1) of users, and provided a more comprehensive suicide risk detection system. Textbox 1. Psychology features and their definitions.
Knowledge Graph and Reasoning Rules We created a knowledge graph that was used to construct a free annotation data set. The Tree Hole knowledge graph contains four independent ontologies-the suicide ontology, time ontology, space ontology, and wish ontology. The suicide ontology consists of two major categories-suicide methods (eg, cutting the wrist and burning charcoal) and suicide plans (eg, buying drugs and meeting with suicide partners). The time ontology covers absolute time concepts, such as calendar days and holidays, and relative time concepts, such as the present, future, and past. The space ontology describes related concepts of spatial geography. The wish ontology was mainly used to analyze the subjective suicidal wishes of a specific group of people and to exclude them from people without subjective suicidal wishes.
An excerpt of the Tree Hole knowledge graph can be found in Figure  2. Once the knowledge graph was created, we constructed prolog rules based on the definite clause grammar (DCG) and DCG transformation rules. To take into account the domain knowledge for reasoning and determining the risk level, the DCG rules integrate the relevant conceptual information from the Tree Hole knowledge graph. For example, the definition for suicide risk level 8 is that the suicide plan has been determined and the suicide date has been roughly determined. This can be formally described by the logic program rules in Textbox 3, which use two predicates-rdfsSubClassOf and uninterestedText. The predicate rdfsSubClassOf was used to decide if a textual fragment mentions some sort of suicide method (or time) concept of suicideOntology (or timeOntology) in the Tree Hole knowledge graph, and uninterestedText was used to verify that such a mention was not negated by a negative expression in the text.
BERT-Based Model We used the data set generated by Tree Hole knowledge graph method to build the BERT-based model. We used BERT to obtain sentence vectors from free annotation data. The size of each sentence vector was 768 dimensions. We added a dropout function to this sentence vector to avoid overfitting. Afterward, we added a fully connected layer to classify comments that were indicative of high and low suicide probabilities. We used the sigmoid function as the activation function of the output layer. The parameters of the BERT layer and the fully connected layer participated in the training at the same time. The architecture of model 1 is shown in Figure  3.
Afterward, we invited people with different psychology knowledge levels to provide a small number of annotations. Then, based on domain knowledge, we extracted users' multidimensional psychological features and integrated them into our final model (the psychology+ model). Our operating steps greatly improved the efficiency of our model. Only 1458 professional labels were required to train a model that could analyze real situations. It would have been impossible to train a reliable model with just over 1000 data points if we did not conduct the previous steps. In future work, we will combine actual work experiences and cooperate with psychologists to propose more suitable suicide classification standards and provide immediate warnings for upcoming emergencies. Although our model could meet actual standard requirements, its run time was relatively slow (274 comments/minute) due to the large number of model parameters.