dataesr/scientific-paragraphs-categorization huggingface dataset:
About the Selection Process In order to retrieve all papers published in 2018 in the field of Natural Language Processing, we queried two databases: Medline, specifically dedicated to the biomedical domain, and the ACL (Association for Computational Linguistics) anthology, a database that brings together the major NLP conferences (ACL, International Conference on Computational Linguistics (COLING), European Chapter of the ACL (EACL), Empirical Methods in NLP (EMNLP), International Joint Conference on NLP (IJCNLP), Language Resources and Evaluation (LREC), North American Chapter of the ACL (NAACL), etc. ) and journals (CL, Transactions of the ACL). We applied a basic query on Medline (Figure  1 ) to target all journal papers published in English in 2018, having abstract, and composed of sequences "clinical language processing" or "medical language processing" or "natural language processing".
As of 2019, February 1 st , we collected 435 entries. We applied a similar query on the ACL anthology database and collected 130 entries. In order to process those 565 papers, since some of these papers are not related to NLP despite the use of one of the three previously defined sequences, we considered positive and negative filters. A first set of filters was applied to names of journal, based on both full name and concepts found in the name: a positive score was given to main journals where biomedical NLP work is published (Biomedical informatics insights, IJMI, JAMIA, JBI, BMJ bioinformatics) while a negative score was given to journals that do not concern NLP but are mainly related to cognitive studies or communication disorders (Human brain mapping, Frontiers in neuroscience, communication disorders, etc.
On a manually annotated dataset with 540 reports, the evaluation shows that the deep learning-based method achieved the best F1-score of 0. 904, when compared with rulebased and CRF-based approaches (0. 848 and 0. 881, respectively). Similar issue was addressed for the drug safety surveillance in electronic health records in English through the use of classical learning (Support Vector Machines (SVMs)) and deep learning  [14]. On the expert-annotated corpus with 791 Electronic Health Record (EHR) notes, the SVM model achieved the best average F1score of 89. 1% on test data, outperforming the long short-term memory (LSTM) model with attention (F1-score of 65. 72%) as well as the rule induction baseline system (F1score of 7. 47%). From another point of view, the researchers studied the problem of cross-domain and cross-language validity of methods and models.
The evaluation is performed on an independent test set containing manually curated hospital database, which stores most of the information written in reports. The proposed approach shows above 90% precision for most considered clinical events. In another work, NLP tools and resources are exploited as early, non-invasive biomarkers for the identification of "pre-clinical" Alzheimer's disease and other dementias  [16]. Indeed, recent studies suggested that speech alterations might be one of the earliest signs of cognitive decline. Since traditional neuropsychological language tests provide ambiguous results, the authors propose to analyse the spoken language productions using NLP techniques and, in this way, pinpoint language modifications of patients. Ninety-six participants were enrolled (48 healthy controls and 48 cognitively impaired participants).
Ten physicians evaluated the user interface and the content quality of the system. The results indicate that the system is easy to use, has good visual display, has satisfactory system speed, and proposes adequate lay definitions. Another issue addresses the generation of misspellings for a better mining of noisy health-related texts such as those provided by social media  [18]. Health-related terms are indeed often misspelled there, resulting in the exclusion of relevant data for studies. The authors propose a system that automatically A Year of Papers Using Biomedical Texts: Findings from the Section on Natural Language Processing of the IMIA Yearbook generates common misspellings for complex health-related terms. The spelling variant generator relies on a dense vector model learned from large, unlabeled text, which is used to find semantically close terms to the original keyword, followed by the filtering of terms that are lexically dissimilar beyond a given threshold.
Concluding Remarks An analysis of 2018 publications related with medical NLP points out a rich and outstanding research year. Research works addressed indeed important topics related to the robustness and reproducibility of methods and results, and they provided several datasets which can be usable in works to come. Some NLP methods prove to be mature and useful for other applications (information retrieval, diagnosis of speech pathologies and dementias, generation of imaging reports, etc. ). As the research advances, we expect that other application domains may become concerned. In 2018, researchers addressed some novel issues and used original approaches. However, more classical issues and approaches cover a wide range of research questions and provide interesting and exploitable results.
This paper presents a system for the automatic generation of medical imaging reports, which may help physicians create high-quality reports, reduce time, and assist less experi-enced physicians. To address these issues, the authors propose to focus on the detection of multiple heterogeneous forms of information including findings and tags, and the detection of abnormal regions in medical images. To cope with these challenges, the authors (i) build a multi-task learning framework which jointly performs the prediction of tags and the generation of paragraphs; (ii) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them; and (iii) develop a hierarchical LSTM (Long short-term memory) model to generate long paragraphs. They demonstrate the effectiveness of the proposed methods on two publicly available datasets.
Our proposed decision trees using local support vector regression models (tSVR, rtSVR) aim to efficiently handle the regression task for large datasets. The learning algorithm tSVR of regression models is done by two main steps. The first one is to construct a decision tree regressor for partitioning the full training dataset into k terminal-nodes (subsets), followed which the second one is to learn the SVR model from each terminal-node to predict the data locally in a parallel way on multi-core computers. The algorithm rtSVR learns the random forest of decision trees with local SVR models for improving the prediction correctness against the tSVR model alone. The performance analysis shows that our algorithms tSVR, rtSVR are efficient in terms of the algorithmic complexity and the generalization ability compared to the classical SVR.
The experimental results on five large datasets from UCI repository showed that proposed tSVR and rtSVR algorithms are faster than the standard SVR in training the non-linear regression model from large datasets while achieving the high correctness in the prediction. Typically, the average training time of tSVR and rtSVR are 1282. 66 and 482. 29 times faster than the standard SVR; Furthermore, tSVR and rtSVR improve 59. 43%, 63. 70% of the relative prediction correctness compared to the standard SVR.
Such a huge amount of data brings out challenges in data analysis because current state-of-the-art techniques are not well suited for that data scale. Therefore, it is a high priority to scale-up learning algorithms for addressing massive datasets. Our research focuses on creating new algorithms of decision trees using local support vector regression models to efficiently handle the non-linear regression task for large datasets. Support vector machines (SVM) proposed by  Vapnik (1995)  and kernel-based methods have shown the state-of-the-art relevant to many data mining problems, including the classification, the regression and the novelty detection  (Guyon, 1999). The SVM algorithm produces the most accurate models in the practice. Nevertheless, the learning problem requires the solution of a quadratic programming (QP), so that the computational cost of an SVM approach is at least square of the number of training datapoints making SVM impractical to deal with large datasets.
There is a need to scale-up SVM learning algorithms to handle massive datasets. This paper presents the extension of our works in Tran-Nguyen,  Bui, Kim, Do (2018). We propose new learning algorithms of decision tree using local support vector regression (called tSVR and rtSVR) to efficiently deal with the non-linear regression task of large datasets. Instead of training a global SVR model, as done by the classical SVR algorithm is very hard to handle large datasets, our tSVR algorithm is to learn in a parallel way an ensemble of local ones that are easily trained by the standard SVR algorithms. The tSVR trains regression models via two main steps. The first one is to use the decision tree algorithm  (Breiman, Friedman, Olshen, & Stone, 1984)  to partition the large training dataset into k terminal-nodes (subsets).
The idea is to reduce the data size for training local non-linear SVR models at the second step. And then, the tSVR learns k non-linear SVR models in a parallel way on multi-core computers in which an SVR model is trained in each terminal-node to predict the data locally. The extension of tSVR is the algorithm rtSVR which trains the random forest of decision trees with local prediction SVR models for improving the prediction correctness versus the tSVR model alone. The performance analysis shows that our algorithms tSVR, rtSVR are efficient in terms of the algorithmic complexity and the generalization ability compared to the classical SVR. The experimental results on five large datasets from UCI repository  (Lichman, 2013)  showed that our proposed tSVR, rtSVR are faster than the standard SVR in the nonlinear regression of large datasets while achieving the high prediction correctness.
tSVR algorithm for dealing with large datasets Platt studied the algorithmic complexity of the SVM approach in  Platt (1999). His analysis illustrates that the training complexity of the SVR solution in (1) is at least square of the number of training datapoints (i. e. O(m 2 )), making standard SVR intractable for large datasets. It means that learning the model from the full massive dataset using the standard SVR algorithm in the usual way is a challenge due to a very high computational cost.
tSVR for learning local SVR models Our proposed tSVR algorithm learns an ensemble of local SVR models that are easily trained by the standard SVR algorithm. As illustrated in Figure  2 , the tSVR handles the regression task with two main steps. The first one of the algorithm tSVR is to learn the decision tree regressor model (using the decision tree training algorithm  Breiman et al. , 1984)  for partitioning the full training dataset into k terminal-nodes (subsets). Thus, the second step of the algorithm tSVR is to learn k non-linear SVR models in a parallel way on multi-core computers in which the SVR model is trained in each terminal-node to predict the data locally. We consider a simple regression task given a response variable y and a predictor (variable) x. Figure  3  shows the comparison between a global SVR model (left part) and 4 local SVR models obtained by tSVR (right part) for this regression task, using a non-linear RBF kernel function with g = 10, a positive constant C = 10 5 (i.
e. the hyper-parameters u = {g, C}) and a tolerance e = 0. 05. Since the terminal-node size is smaller than the size of the full training dataset, the standard SVR can easily perform the training task from the terminal-node, due to the square reduction of the complexity compared to the learning from the full dataset. In addition, k local models in the training task of tSVR are independently learnt from k terminalnodes. This is a nice condition to train k local models in a parallel way to take into account the benefits of high-performance computing, e. g. multi-core computers or grids. We propose to develop the parallel tSVR algorithm in a simple way based on the shared memory multiprocessing programming model OpenMP  (OpenMP Architecture Review Board, 2008)  for multi-core computers. The parallel training of tSVR is described in Algorithm 1.
Performance analysis Let's consider the performance of the learning algorithm tSVR in terms of the algorithmic complexity and the generalization capacity. We starts with the algorithmic complexity of tSVR for building k local SVR models in the parallel way. Suppose that tSVR uses the decision tree regressor to partition the full dataset (with m datapoints) into k balanced terminal-nodes and then the terminal-node size is about m/k; in other words the minimum number of datapoints for a split to be tried minob j = m/k. The training complexity of a local SVR model for a terminal-node is O(minob j 2 ) = O m 2 k 2 (3) Therefore, the algorithmic complexity of tSVR for parallel training k local SVR models on a P-core processor is O k P m 2 k 2 = O m 2 k. P = O minob j. m P (4) This complexity analysis illustrates that parallel learning k local SVR models in the tSVR algorithm is k.
P times faster than building a global SVR model by the standard SVR algorithm (the complexity at least O(m 2 )). It remarks that the complexity analysis of the tSVR excepts the decision tree regressor learnt to split the full dataset. However this training the decision tree regressor has a very low computational cost compared with the quadratic programming solution required by the SVR learning algorithm. We will state how to assess the generalization capacity of tSVR models learnt by the tSVR algorithm via the margin size of the binary support vector classification (SVC). The regression task of the SVR is considered as a classification problem of the SVC as illustrated in an intuitive geometric approach  (Bi & Bennett, 2003). Let us consider x, y be the predictor variable and the response variable, respectively.
It means that the generalization ability of the maximal margin hyperplane is justified in terms of min 1 m R 2 D (6) For the training task, the tSVR algorithm trains the decision tree regressor to split the full dataset having m datapoints into k terminal-nodes (the terminal-node size m k is about minob j = m/k). And then the generalization capacity of local SVR models trained by the tSVR algorithm is assessed in terms of min 1 m k R 2 k D k (7) The justification of the generalization ability is based on the comparison between Equation (  6 ) of the global SVR model trained for the full dataset and Equation (  7 ) of the local SVR model trained from a terminal-node. According to Theorem 1 in  Do and Poulet (2016b)  and Theorem 2, 3 in  Do and Poulet (2017) , the margin size D X k of the local SVR model is greater than the margin size D X of the global one.
Learning algorithm of r random tSVR models for dealing with large datasets The performance analysis can state that there is a trade-off between the algorithmic complexity and the generalization capacity in the learning algorithm tSVR. If the parameter minob j is set to small then the tSVR algorithm significantly speed up the training time but also making the low prediction correctness, compared to the learning global SVR algorithm. To address this problem, our proposed algorithm rtSVR learns the random forest of r decision trees using local prediction SVR models to improve the generalization capacity of the tSVR alone. According to the Bias-Variance framework proposed by  Breiman (1996 Breiman ( , 2001)) , the performance of a learning model can be understood through Bias term and Variance term.
Learning r random tSVR models The rtSVR algorithm trains the ensemble of r random tSVR models using the tSVR algorithm (described in Algorithm 1 and Figure  3 ). The tSVR algorithm learning the ith random tSVR model from a ith bootstrap sample (sampling with replacement from the original dataset). For each non-terminal node in the training phase of the tree regressor, randomly choose n ′ dimensions from n original ones are used to calculate the best split among these n ′ dimensions. The tree regressor is grown with the early stopped minob j = m/k and without pruning. Finally, the algorithm tSVR learns k non-linear SVR models for k terminal-nodes. As described in Algorithm 2 and Figure  5 , the rtSVR constructs independently r random local tSVR models. This allows parallelizing the learning task with OpenMP on multi-core computers.
Software programs Our algorithms tSVR krSVR are implemented in Python programming language, using Scikit-learn library (Machine Learning in Python  Pedregosa et al. , 2011) , pymp package (OpenMP-like functionality for Python  Lassner, 2017). The highly efficient standard library SVM, LibSVM  (Chang & Lin, 2011)  with OpenMP (OpenMP Architecture Review Board, 2008) is also used to train global SVR models. Our evaluation of the performance is reported in terms of training time and prediction correctness. We are interested in the comparison the regression results obtained by our proposed tSVR, rtSVR for local SVR models with LibSVM for global SVR models. All experiments are run on machine Linux Fedora 20, Intel(R) Core i7-4790 CPU, 3. 6 GHz, 4 cores and 32 GB main memory.
Tuning parameters We propose to use RBF kernel type in tSVR, rtSVR and LibSVM from training SVR models because it is general and efficient  (Lin, 2003). The cross-validation protocol ( twofold) is used to tune the regression tolerance ϵ, the hyper-parameter γ of RBF kernel (RBF kernel of two individuals x i , x j , K[i, j] = exp ( -g x ix j 2 )) and the cost C (a trade-off between the margin size and the errors) to obtain a good correctness. For two largest datasets (Buzz in social media Twitter, YearPredictionMSD), we used a subset randomly sampling about 5% training dataset for tuning hyper-parameters due to the expensive computational cost. Our tSVR uses the parameter minob j (the minimum number of datapoints for a split to be tried by the decision tree regression). We propose to set minob j = 200 (according to the advise of  Vapnik & Bottou, 1993).
Comparison in training time The training time in Table  3  show that our decision tree algorithms using local SVR models (tSVR and rtSVR) outperform LibSVM in terms of training time. The average of tSVR, rtSVR and LibSVM training time are 1. 19, 3. 22 and 1551. 15 min, respectively. It means that:. training time of tSVR is 1303. 49 times faster than LibSVM;. rtSVR is 481. 72 times faster than LibSVM;. training time of rtSVR is about 2. 71 times longer than tSVR. The comparison of training time for 3 first small datasets (Appliances energy prediction, Facebook comment volume, BlogFeedback) illustrates that the training speed improvements of tSVR and rtSVR against LibSVM are 346. 67 and 127. 64 times, respectively. For two very large datasets (Buzz in social media --Twitter and YearPredictionMSD), the learning time improvements of tSVR and rtSVR compared to LibSVM is more significant.
Comparison in prediction correctness Prediction correctness (mean absolute error -MAE) presented in Table  3  show that the error average made by tSVR, rtSVR and LibSVM are 25. 16, 22. 51 and 62. 01, respectively. The comparison of prediction correctness, dataset by dataset, shows that tSVR and rtSVR are beaten only once (with BlogFeedback dataset) by LibSVM; tSVR and rtSVR has 4 wins and 1 defeat against LibSVM. rtSVR is more accurate than tSVR with 5/5 wins. Prediction results demonstrate that our tSVR, rtSVR are more efficient than LibSVM. These numerical test results demonstrate that our tSVR, rtSVR improve not only the training time, but also the prediction correctness when dealing with large datasets.
Comparison with decision tree algorithms We are interested in studying the impact of using local SVR rules against the average prediction ones of classical decision trees. Therefore, we would like to compare the performance of tSVR, rtSVR with the decision tree regressor (DT) and the random forest regressor (RF). The regression results of tSVR, rtSVR compared to DT, RF are presented in  improvement of the prediction correctness being 1. 21. In comparison to RF, rtSVR also reduces 1. 40 of the prediction error but requires 3. 26 times of the training time. The experimental results allow to believe that decision tree algorithms tSVR and rtSVR using local SVR models are efficient for dealing with such large datasets.
Discussion on related works Our proposed tSVR is in machine learning techniques related to local SVM algorithms. A theoretical analysis studied in  Bottou and Vapnik (1992)  and  Vapnik and Bottou (1993)  illustrates that there is the trade-off between the generalization capacity (the prediction correctness) and locality of local learning algorithms. The approaches of local SVMs can be categorized into two kinds of strategies (the hierarchical strategy and the nearest neighbours strategy). The kind of local algorithms in the hierarchical strategy performs the training task with two main stages (the partition of the training set and learning local models). The first stage of learning task is to split the full training dataset into partitions (subsets) and then the second stage is to train the local supervised models from subsets.
The learning algorithm proposed in  Jacobs, Jordan, Nowlan, and Hinton (1991)  uses the expectation-  maximization (EM) clustering algorithm  (Dempster, Laird, & Rubin, 1977)  to split the full training dataset into k joint clusters (the EM clustering algorithm uses the posterior probabilities to make a soft cluster assignment  Bishop, 2006) ; And then the algorithm learns neural network (NN) models from clusters to locally classify the individuals in clusters. Instead of building local NN models as done by  Jacobs et al. (1991) , the parallel mixture of SVMs algorithm in  Collobert, Bengio, and Bengio (2002)  proposes to learn local SVM models to predict the individuals in clusters. More recent, the Latent-lSVM  (Do & Poulet, 2016a , 2017)  uses Latent Dirichlet Allocation (LDA  Blei, Ng, & Jordan, 2003)  to perform the clustering task for sparse data representation.
CSVM  (Gu & Han, 2013) , kSVR  (Bui, Tran-Nguyen, Kim, & Do, 2017) , and kSVM  (Do, 2015)  propose to uses kmeans algorithm  (MacQueen, 1967)  to split the full training dataset into k disjoint clusters; And then kSVR and kSVM learn local non-linear SVM models in a parallel way, instead of training weighted local linear SVMs from clusters as done in CSVM. krSVM  (Do & Poulet, 2015)  is to learn the random ensemble of kSVM models. DTSVM  (Chang, Guo, Lin, & Lu, 2010; Chang & Liu, 2012)  and tSVM  (Do & Poulet, 2016b)  use decision tree algorithms  (Breiman, Friedman, Olshen, & Stone, 1984; Quinlan, 1993)  to split the full training dataset into t terminal-nodes (tree leaves); follow which the tSVM algorithm builds local SVM models for classifying impurity terminal-nodes (with a mixture of labels) while DTSVM learns local SVM models from all tree leaves.
These algorithms are shown to reduce the computational cost for dealing with large datasets while maintaining the prediction correctness. The kind of local algorithms in the nearest neighbours strategy tries to find k nearest neighbours (kNN) of a new testing individual from the training dataset and then it only learns local supervised model from these kNN to classify the new testing individual. First local learning algorithm proposed in  Bottou and Vapnik (1992)  and  Vapnik and Bottou (1993)  is to train the neural network model from k neighbourhoods to predict the label of the testing individual. The investigation of  Vincent and Bengio (2001)  is to train k-local hyperplane and convex distance nearest neighbour (large margin nearest neighbours). Recent local SVM algorithms, including SVM-kNN  (Zhang, Berg, Maire, & Malik, 2006) , ALH  (Yang & Kecman, 2008) , FaLK-SVM  (Segata & Blanzieri, 2010)  find kNN of the testing individual with different techniques.