dataesr/scientific-paragraphs-categorization huggingface dataset:
The recognition result is then obtained using a best path search decoding algorithm on the lattice of hypotheses incorporating additional knowledge sources such as language models. This system relies on numerous pre-processing, segmentation, and feature extraction heuristics which are no longer present in our new system. The new system reduces the amount of customization required, and consists of a simple stack of bidirectional LSTMs (BLSTMs), a single Logits layer, Amharic, Tigrinya Languages with distinct scripts: Armenian, Burmese, Georgian, Greek, Gujarati, Hebrew, Japanese, Kannada, Khmer, Korean, Lao, Malayalam, Odia, Punjabi, Sinhala, Tamil, Telugu, Thai, Tibetan, Vietnamese  4 and the CTC loss  [15]  (Sec. 2). We train a separate model for each script (Sec. 3). To support potentially many languages per script (see Table  1 ), language-specific language models and feature functions are used during decoding (Sec.
2. 5). E. g. we have a single recognition model for Arabic script which is combined with specific language models and feature functions for our Arabic, Persian, and Urdu language recognizers. Table  1  shows the full list of scripts and languages that we currently support. The new models are more accurate (Sec. 4), smaller, and faster (Table  9 ) than our previous segment-anddecode models and eliminate the need for a large number of engineered features and heuristics. We present an extensive comparison of the differences in recognition accuracy for eight languages (Sec. 5) and compare the accuracy of models trained on publicly available datasets where available (Sec. 4). In addition, we propose a new standard experimental protocol for the IBM-UB-1 dataset  [43]  to enable easier comparison between approaches in the future (Sec.
4. 2). The main contributions of our paper are as follows: -We describe in detail our recurrent neural networkbased recognition stack and provide a description of how we tuned the model. We also provide a detailed experimental comparison with the previous segment-and-decode based stack  [25]  on the supported languages. -We describe a novel input representation based on Bézier curve interpolation, which produces shorter input sequences, which results in faster recognitions. -Our system achieves a new state-of-the-art on the IAM-OnDB dataset, both for open and closed training sets. -We introduce an evaluation protocol for the less commonly-used English IBM-UB-1 query dataset. We provide experimental results that quantify the structural difference between IBM-UB-1, IAM-OnDB, and our internal dataset.
-We perform ablation studies and report results on numerous experiments highlighting the contributions of the individual components of the new recognition stack on our internal datasets. 2 End-to-end Model Architecture Our handwriting recognition model draws its inspiration from research aimed at building end-to-end transcription models in the context of handwriting recognition  [15] , optical character recognition  [8] , and acoustic modeling in speech recognition  [40]. The model architecture is constructed from common neural network blocks, i. e. bidirectional LSTMs and fully-connected layers (Figure  2 ). It is trained in an end-to-end manner using the CTC loss  [15]. Our architecture is similar to what is often used in the context of acoustic modeling for speech recognition  [41] , in which it is referred to as a CLDNN (Convolutions, LSTMs, and DNNs), yet we differ from it in four points.
Firstly, we do not use convolution layers, which in our own experience do not add value for large networks trained on large datasets of relatively short (compared to speech input) sequences typically seen in handwriting recognition. Secondly, we use bidirectional LSTMs, which due to latency constraints is not feasible in speech recognition systems. Thirdly, our architecture does not make use of additional fully-connected layers before and after the bidirectional LSTM layers. And finally, we train our system using the CTC loss, as opposed to the HMMs used in  [41]. This structure makes many components of our previous system  [25]  unnecessary, e. g. feature extraction and segmentation. The heuristics that were hard-coded into our previous system, e. g. stroke-reordering and character hypothesis building, are now implicitly learned from the training data.
Input Representation In our earlier paper  [25]  we presented results on our datasets with a model similar to the one proposed in  [15]. In that model we used 23 per-point features (similarly to  [22] ) as described in our segment-and-decode system to represent the input. In further experimentation we found that in substantially deeper and wider models, engineered features are unnecessary and their removal leads to better results. This confirms the observation that learned representations often outperform handcrafted features in scenarios in which sufficient training data is available, e. g. in computer vision  [28]  and in speech recognition  [46]. In the experiments presented here, we use two representations:.
Bidirectional Long-Short-Term-Memory Recurrent Neural Networks LSTMs  [19]  have become one of the most commonly used RNN cells because they are easy to train and give good results  [24]. In all experiments we use bidirectional LSTMs  [6, 12] , i. e. we process the input sequence forward and backward and merge the output states of each layer before feeding them to the next layer. The exact number of layers and nodes is determined empirically for each script. We give an overview of the impact of the number of nodes and layers in section 4. We also list the configurations for several scripts in our production system, as of this writing.
Decoding The output of the softmax layer is a sequence of T time steps of (C + 1) classes that we decode using CTC decoding  [12]. The logits from the softmax layer are combined with language-specific prior knowledge (cp. Sec. 2. 5). For each of these additional knowledge sources we learn a weight (called "decoder weight" in the following) and combine them linearly (cp. Sec. 3). The learned combination is used as described in  [13]  to guide the beam search during decoding. 6  This combination of different knowledge sources allows us to train one recognition model per script (e. g. Latin script, or Cyrillic script) and then use it to serve multiple languages (see Table  1 ). 6 We implement this as a BaseBeamScorer (https: //github. com/tensorflow/tensorflow/blob/master/ tensorflow/core/util/ctc/ctc_beam_scorer.
Feature Functions: Language Models and Character Classes Similarly to our previous work  [25] , we define several scoring functions, which we refer to as feature functions. The goal of these feature functions is to introduce prior knowledge about the underlying language into the system. The introduction of recurrent neural networks has reduced the need for many of them and we now use only the following three: -Character Language Models: For each language we support, we build a 7-gram language model over Unicode codepoints from a large web-mined text corpus using Stupid back-off  [3]. The final files are pruned to 10 million 7-grams each. Compared to our previous system  [25] , we found that language model size has a smaller impact on the recognition accuracy, which is likely due to the capability of recurrent neural networks to capture dependencies between consecutive characters.
Training The training of our system happens in two stages, on two different datasets: 1. End-to-end training of the neural network model using the CTC loss using a large training dataset 2. Tuning of the decoder weights using Bayesian optimization through Gaussian Processes in Vizier  [11] , using a much smaller and distinct dataset. Using separate datasets is important because the neural network learns the local appearance as well as an implicit language model from the training data. It will be overconfident on its training data and thus learning the decoder weights on the same dataset could result in weights biased towards the neural network model.
Bayesian Optimization for Tuning Decoder Weights To optimize the decoder weights, we rely on the Google Vizier service and its default algorithm, specifically batched Gaussian process bandits, and expected improvement as the acquisition function  [11]. For each recognizer training we start 7 Vizier studies, each performing 500 individual trials, and then we pick the configuration that performed best across all of these trials. We experimentally found that using 7 separate studies with different random initializations regularly leads to better results than running a single study once. We found that using more than 500 trials per study does not lead to any additional improvement. For each script we train these weights on a subset of the languages for which we have sufficient data, and transfer the weights to all the other languages.
Experimental Evaluation In the following, where possible, we present results for public datasets in a closed data scenario, i. e. training and testing models on the public dataset using a standard protocol. In addition we present evaluation results for public datasets in an open data scenario against our production setup, i. e. in which the model is trained on our own data. Finally, we show experimental results for some of the major languages on our internal datasets. Whenever possible we compare these results to the state of the art and to our previous system  [25].
IAM-OnDB The IAM-OnDB dataset  [31]  is probably the most used evaluation dataset for online handwriting recognition. It consists of 298 523 characters in 86 272 word instances from a dictionary of 11 059 words written by 221 writers. We use the standard IAM-OnDB dataset separation: one training set, two validations sets and a test set containing 5 363, 1 438, 1 518 and 3 859 written lines, respectively. We tune the decoder weights using the validation set with 1 438 items and report error rates on the test set. We perform a more extensive study of the number of layers and nodes per layer for both the raw and curve input formats to determine the optimal size of the bidirectional LSTM network (see Figure  4 , Table  2 ). We first run experiments without additional feature functions (Figure  4 , solid lines), then re-compute the results with tuned weights for language models and character classes (Figure  4 , dashed lines).
We observe that for both input formats, using 3 or 5 layers outperforms more shallow networks, and using more layers gives hardly any improvement. Furthermore, using 64 nodes per layer is sufficient, as wider networks give only small improvements, if at all. We see no significant difference in accuracy between the raw and the curve representation. Finally, we show a comparison of our old and new systems with the literature on the IAM-OnDB dataset in Table  3. Our method establishes a new state of the art result when relying on closed data using IAM-OnDB, as well as when relying on our in-house data that we use for our production system, which was not tuned for the IAM-OnDB data and for which none of the IAM-OnDB data was used for training. To better understand where the improvements come from, we discuss the differences between the previous state-of-the-art system (Graves et al.
IBM-UB-1 Another publicly-accessible English-language dataset is the IBM-UB-1 dataset  [43]. From the available datasets therein, we use the English query dataset, which consists of 63 268 handwritten English words. As this dataset has not been used often in the academic literature, we  We perform a similar set of experiments as we did for IAM-OnDB to determine the right depth and width of our neural network architecture. The results of these experiments are shown in Figure  5. The conclusion for this dataset is similar to the conclusions we drew for the IAM-OnDB: using networks with 5 layers of bidirectional LSTMs with 64 cells each is sufficient for good accuracy. Less deep and less wide networks perform substantially worse, but larger networks only give small improvements. This is true regardless of the input processing method chosen and again, we do not see a significant difference in accuracy between the raw and curve representation in accuracy.
We give some exemplary results and a comparison with our current production system as well as results for our previous system in Table  4. We note that our current system is about 38% and 32% better (relative) in CER and WER, respectively, when compared to the previous segment-and-decode approach. The lack of improvement in error rate when evaluating on our production system is due to the fact that our datasets contain spaces while the same setup trained solely on IBM-UB-1 does not. 7 Information about the exact experimental protocol is available at https://arxiv. org/src/1902. 10525v1/anc.
Chinese Isolated Characters (ICDAR 2013 competition) The ICDAR-2013 Competition for Online Handwriting Chinese Character Recognition  [50]  introduced a dataset for classifying the most common Chinese characters. We report the error rates in comparison to published results from the competition and more recent work done by others in Table  5. We evaluate our live production system on this dataset. Our system was not tuned to the task at hand and was trained as a multi-character recognizer, thus it is not even aware that each sample only contains a single character. Further, our system supports 12 363 different characters while the competition data only contains 3 755 characters. Note that our system did not have access to the training data for this task at all. Whenever our system returns more than one character for a sample, we count this as an error (this happened twice on the entire test set of 224 590 samples).
Despite supporting almost four times as many characters than needed for the CASIA data and not having been tuned to the task, the accuracy of our system is still competitive with systems that were tuned for this data specifically. In the ICFHR2018 Competition on Vietnamese Online Handwritten Text Recognition using VNOnDB  [34] , our production system was evaluated against other systems. The system used in the competition is the one reported and described in this paper. Due to licensing restrictions we were unable to do any experiments on the competition training data, or specific tuning for the competition, which was not the case for the other systems mentioned here. We participated in the two tasks that best suited the purpose of our system, specifically the "Word" (ref. In contrast to us, the other teams used the training and validation sets to tune their systems: -The IVTOV team's system is very similar to our system.
It makes use of bidirectional LSTM layers trained end-to-end with the CTC loss. The inputs used are delta x and y coordinates, together with pen-up strokes (boolean feature quantifying whether a stroke has ended or not). They report using a two-layer network of 100 cells each and additional preprocessing for better handling the dataset. -The MyScript team submitted two systems. The first system has an explicit segmentation component along with a feed-forward network for recognizing character hypotheses, similar in formulation to our previous system  [25]. In addition, they also make use of a bidirectional LSTM system trained end-to-end with the CTC loss. They do not provide additional details on which system is which. We note that the modeling stacks of the systems outperforming ours in this competition are not fundamentally different (to the best of our knowledge, according to released descriptions).
Tuning neural network parameters on our internal data Our in-house datasets consist of various types of training data, the amount of which varies by script. Sources of training data include data collected through prompting, commercially available data, artificially inflated data, and labeled/self-labeled anonymized recognition requests (see  [25]  for a more detailed description). This leads to more heterogeneous datasets than academic datasets such as IBM-UB-1 or IAM-OnDB which were collected under standardized conditions. The number of training samples varies from tens of thousands to several million per script, depending on the complexity and usage. We provide more information about the size of our internal training and tests datasets in table  9. The best configuration for our system was identified by running multiple experiments over a range of layer depths and widths on our internal datasets.
For the Latin script experiments shown in Figure  6 , the training set we used was a mixture of data from all the Latinscript languages we support and evaluation is done on an English validation dataset, also used for the English evaluation in Table  9. Similarly to experiments depicted in Figures  4  and 5 , increasing the depth and width of the network architecture brings diminishing returns fairly quickly. However, overfitting is less pronounced probably because our datasets are substantially larger than the publicly available datasets. For the experiments with our production datasets we are using the Bézier curve inputs which perform slightly better in terms of accuracy than the raw input encoding but are much faster to train and evaluate because of the shorter sequence lengths.
System Performance and Discussion The setup described throughout this paper that obtained the best results relies on input processing with Bézier spline interpolation (Sec. 2. 1. 2), followed by 4-5 layers of varying width bidirectional LSTMs, followed by a final softmax layer. For each script, we experimentally determined the best configuration through multiple training runs. We performed an ablation study with the best configurations for each of the six most important scripts 8  by number of users and compare the results with our previous work  [25]  (Table  9 ). The largest relative improvement comes from the overall network architecture stack, followed by the use of the character language model and the other feature functions. In addition, we show the relative improvement in error rates on the languages for which we have evaluation datasets of more than 2 000 items (Figure  7 ).
Differences Between IAM-OnDB, IBM-UB-1 and our internal datasets To understand how the different datasets relate to each other, we performed a set of experiments and evaluations with the goal of better characterizing the differences between the datasets. We trained a recognizer on each of the three training sets separately, then evaluated each system on all three test sets (Table  8 ). The neural network architecture is the same as the one we determined earlier (5 layers bidirectional LSTMs of 64 cells each) with the same feature functions, with weights tuned on the corresponding tuning dataset. The inputs are processed using Bézier curves. To better understand the source of discrepancy when training on IAM-OnDB and evaluating on IBM-UB-1, we note the different characteristics of the datasets: -IBM-UB-1 has predominantly cursive writing, while IAM-OnDB has mostly printed writing -IBM-UB-1 contains single words, while IAM-OnDB has lines of space-separated words This results in models trained on the IBM-UB-1 dataset not being able to predict spaces as they are not present in the dataset's alphabet.
In addition, the printed writing style of IAM-OnDB makes recognition harder when evaluating cursive writing from IBM-UB-1. It is likely that the lack of structure through wordsonly data makes recognizing IAM-OnDB on a system trained on IBM-UB-1 harder than vice-versa. Systems trained on IBM-UB-1 or IAM-OnDB alone perform significantly worse on our internal datasets, as our data distribution covers a wide range of use-cases not necessarily relevant to, or present, in the two academic datasets: sloppy handwriting, overlapping characters for handling writing on small input surfaces, nonuniform sampling rates, and partially rotated inputs. The network trained on the internal dataset performs well on all three datasets. It performs better on Fig. 6  CER of models trained on our internal datasets evaluated on our English-language validation set with different numbers of LSTM layers and LSTM nodes using raw (left) and curve (right) inputs.
Conclusion We describe the online handwriting recognition system that we currently use at Google for 102 languages in 26 scripts. The system is based on an end-to-end trained neural network and replaces our old Segmentand-Decode system. Recognition accuracy of the new system improves by 20% to 40% relative depending on the language while using smaller and faster models. We encode the touch inputs using a Bézier curve representation which performs at least as well as raw touch inputs but which also allows for a faster recognition because the input sequence representation is shorter. We further compare the performance of our system to the state of the art on publicly available datasets such as IAM-OnDB, IBM-UB-1, and CASIA and improve over the previous best published result on IAM-OnDB. Table  9  Character error rates on the validation data using successively more of the system components described above for English (en), Spanish (es), German (de), Arabic (ar), Korean (ko), Thai (th), Hindi (hi), and Chinese (zh) along with the respective number of items and characters in the test and training datasets.
Acknowledgements We would like to thank the following contributors for fruitful discussions, ideas, and support:  Ashok Popat ,  Yasuhisa Fujii ,  Dmitriy Genzel ,  Jake Walker ,  David Rybach ,  Daan van Esch , and  Eugene Brevdo. We thank  Google 's OCR team for the numerous collaborations throughout the years that have made this work easier, as well as the speech recognition and machine translation teams at  Google  for tools and support for some of the components we use in this paper.
CER of models trained on the IAM-OnDB dataset with different numbers of LSTM layers and LSTM nodes using raw (left) and curve (right) inputs. Solid lines indicate results without any language models or feature functions in decoding, dashed lines indicate results with the fully-tuned system. 20 30 1 layer 3 layers 5 layers 7 layers 9 layers 20 30 1 layer 3 layers 5 layers 7 layers 9 layers CER [%] 8 10 CER [%] 8 10 6 6 4 4 16 32 64 128 256 16 32 64 128 256 LSTM nodes LSTM nodes Fig. 4.
Background: Oral premalignant lesions (OPLs) represent the most common oral precancerous conditions. One of the major challenges in this field is the identification of OPLs at higher risk for oral squamous cell cancer (OSCC) development, by discovering molecular pathways deregulated in the early steps of malignant transformation. Analysis of deregulated levels of single genes and pathways has been successfully applied to head and neck squamous cell cancers (HNSCC) and OSCC with prognostic/predictive implications. Exploiting the availability of gene expression profile and clinical follow-up information of a well-characterized cohort of OPL patients, we aim to dissect tissue OPL gene expression to identify molecular clusters/signatures associated with oral cancer free survival (OCFS). Materials and methods: The gene expression data of 86 OPL patients were challenged with: an HNSCC specific 6 molecular subtypes model (Immune related: HPV related, Defense Response and Immunoreactive; Mesenchymal, Hypoxia and Classical); one OSCC-specific signature (13 genes); two metabolism-related signatures (3 genes and signatures raised from 6 metabolic pathways associated with prognosis in HNSCC and OSCC, respectively); a hypoxia gene signature.