Venue: NAACL 2019

# TOI-CNN: A Solution of Information Extraction on Chinese Insurance Policy

Lin Sun

Zhejiang University City College

50 Huzhou Street, Hangzhou, China sunl@zucc.edu.cn &Kai Zhang1

Zhejiang University City College

50 Huzhou Street, Hangzhou, China 31601102@stu.zucc.edu.cn &Fule Ji

Zhejiang University City College

50 Huzhou Street, Hangzhou, China 31601087@stu.zucc.edu.cn &Zhenhua Yang

Shanghai Fortune FIS Co., Ltd.

1687 Changyang Road, Shanghai, China yangzhenhua@fcl8.com.cn Insurance contract corpus

###### Abstract

Contract analysis can significantly ease the work for humans using AI techniques. This paper shows a problem of Element Tagging on Insurance Policy (ETIP). A novel Text-Of-Interest Convolutional Neural Network (TOI-CNN) is proposed for the ETIP solution. We introduce a TOI pooling layer to replace traditional pooling layer for processing the nested phrasal or clausal elements in insurance policies. The advantage of TOI pooling layer is that the nested elements from one sentence could share computation and context in the forward and backward passes. The computation of backpropagation through TOI pooling is also demonstrated in the paper. We have collected a large Chinese insurance contract dataset and labeled the critical elements of seven categories to test the performance of the proposed method. The results show the promising performance of our method in the ETIP problem.

+
Footnote †: Online] Research Article: [https://www.facebook.org/](https://www.facebook.org/)

+
Footnote †: Online] Research Article: [https://www.facebook.org/](https://www.facebook.org/)

+
Footnote †: Online] Research Article: [https://www.facebook.org/](https://www.facebook.org/)

+
Footnote †: Online] Research Article: [https://www.facebook.org/](https://www.facebook.org/)

+
Footnote †: Online] Research Article: [https://www.facebook.org/](https://www.facebook.org/)

+
Footnote †: Online] Research Article: [https://www.facebook.org/](https://www.facebook.org/)

+
Footnote †: Online] Research Article: [https://www.facebook.org/](https://www.facebook.org/)

+
Footnote †: Online] Research Article: [https://www.facebook.org/](https://www.facebook.org/)

## 1 Introduction

Automatic contract analysis can gain immediate insight into the content of specific contractual documents in legal or financial areas Moens et al. (2000). Compared to the traditional method of manually reviewing hundreds of contracts, it is helpful not only manage and access contracts but also significantly free knowledge workers from menial, laborious and often error-prone tasks. The insurance policy is a legal contract that outlines the rights and obligations of the insured and insurcer. It consists of a wide variety of different types of insurance coverages to meet specific needs, although most insurance policies are somewhat standardized. Understanding the various types of insurance coverage is time-consuming and error-prone. This paper shows a problem of Element Tagging on Insurance Policy (ETIP). It can automatically convert a massive amount of insurance policies into structural archives for management-t and comparison. Due to the vital information highlighted by ETIP, it can also timely provide insurance staff valuable insight into policies, quickly locate requested information and speed up claim processing.

The processing architecture for ETIP is shown in Fig. 1. We have built a large Chinese insurance contract corpus. There are two usages of the corpus. One is for learning word embeddings. In Sec. 5.3, we show the advantage of the insurance-specific corpus over other general language corpora for the training of word embeddings. Another usage of the corpus is to create insurance knowli

Figure 1: The processing architecture for ETIP.



edge base (KB). Insurance KB, which consists of seven categories of the elements manually labeled by the insurance employees, provides the training data for TOI-CNN model. Specifically, the contributions of this paper can be summarized as follows:

* To our best knowledge, this is the first work on semantic-specific tagging on insurance contracts. Compared to nested NER, not only the type of the elements varies from a short phrase to a long sentence, but also a phrase or clause element could be embedded in other elements.
* We propose a novel TOI-CNN model for the ETIP solution. The advantage of TOI pooling layer is that the elements from the same sentence could share computation and context in the forward and backward passes.
* We have collected 500 Chinese insurance contracts of 46 insurance companies and published the dataset. The experimental results show that the overall performance of TOI-CNN is promising for practical application.

## 2 Related Work

The work of contract analysis is typically divided into two categories, segmentation and information extraction (IE). Segmentation Hasan et al. (2008); Loza Mencia (2009) aims to outline the structure of a conventional text format by annotating title, section, subsection, and so on. Information extraction Cohen and McCallum (2004); Piskorski and Yangarber (2013) focuses on the classification of words, phrases or sentences. Recent works of contract information extraction have addressed recognition of some essential elements Curtotti and McCreath (2010); Indukuri and Krishna (2010). Chalkidis et al. (2017) extracted the contract element, types of which are contract title, contracting parties, date, contract period, legislation refs and so on. The extraction method was based on Logistic Regression, SVM Chalkidis et al. (2017) and BILSTM Chalkidis and Androutsopoulos (2017) with POS tag embeddings and hand-crafted features. Garcia-Constantino et al. (2017) presented the system called CIELE for extracting information from commercial law documents. CLIEL identified five element categories similar to the literature mentioned in Chalkidis et al. (2017) by rule-based layout detection. Azzopardi et al. (2016) developed a mixture extraction method of regular expressions and named entity to extract information from contract clauses, and provided an intelligent contract editing tool to lawyers. Previous works of contract information extraction always focused on title, date, layout, contracting party, etc.. They are not directly related to the semantics of contracts, and could not provide deep insight into contract understanding. The insurance policies are formal legal documents and usually have general elemental compositions, e.g., coverage, payment, and period. In this paper, we investigate how to interpret insurance clauses. Some examples of ETIP are shown in Sec. 3.

The tasks of information extraction could be Named Entity Recognition (NER) Nadeau and Sekine (2007); Ritter et al. (2011), Information Extraction by Text Segmentation (IETS) Cortez and Da Silva (2013); Hu et al. (2017), etc.. NER typically recognizes persons, organizations, location-s, dates, amounts, etc.. IETS identifies attributes from semi-structured records in the form of continuous text, e.g., product description and ads. The previous IE works on contracts Azzopardi et al. (2016); Chalkidis et al. (2017) are similar to NER.

Recently researchers pushed the field of NER towards nested representations of named entities. Muis and Lu (2017) incorporated mention separators to capture how mentions overlap with one another. Both of two works relied on hand-crafted features. Ju et al. (2018) designed a sequential s-stack of flat NER layers that detects nested entities. One bidirectional LSTM layer represented word sequences and CRF layer on top of the LSTM layer decoded label sequences globally. Katiyan and Cardie (2018) presented a standard LSTM-based sequence labeling model to learn the nested entity hypergraph structure for an input sentence. Our ETIP problem is a variant of nested NER, called lengthy nested NER. The type of nested entities varies from phrase to clause. However, in the previous nested NER datesets Kim et al. (2003); Doddington et al. (2004), the type of nested entities only contains short phrase and the average length is approximately three words.

## 3 ETIP Problem Statement

In this section, we first give the definition of elements tagging on insurance policy (ETIP) problem. Given an insurance coverage 

\((s_{1},s_{2},...,s_{n})\), where \(s_{i}\) is the \(i\)th sentence in \(C\). \(s_{i}=(w_{i,1},w_{i,2},...,w_{i,m})\), where \(w_{i,j}\) is the \(j\)th word in sentence \(s_{i}\). An element \(e\) in the coverage \(C\) is continuous words in one sentence, denoted as \(e=\{(w_{i,s},w_{i+1},...,w_{i,t}),l\}\) , where \(l\) is the category label of the element \(e\). The goal of ETIP is to find the element \(e\) of category \(l\) in the coverage \(C\). We define seven categories of insurance clauses listed as follows,

* Cover (C)
* Period of Coverage (PC)
* Condition for Payment (CP)
* Termination (T).

Here we give a coverage example in ETIP and translate it for English reading paper. One category is represented by one kind of font color.

The extractable elements in the example are listed as follows,

* **C**: _dies due to disease_
* **PC**: _within 90 days from the commencement date_
* **CP**: _The insured dies due to disease within 90 days from the commencement date of the contract_
* **IA**: _the amount of the benefit is the sum of the premium which has been paid in this contract and the premium which has been paid in additional critical disease insurance_
* **T**: _This contract terminates._

In this example, the sentence of CP in red contains the other two elements which are C in purple and PC in green respectively. It is the challenge of ETIP, a general element tagging problem, which allows that the elements of various length could be overlapped. We demonstrate other examples in ETIP along with English translation as follows, where \([~{}]_{tag}\) is a category tag labeling the range of an element.

To illustrate phrasal level of an element, we define a metric, called Element Length Ratio (ELR),

\[ELR=\frac{element\_length}{sentence\_length}. \tag{1}\]

For example, \(ELR(C)=4/16=0.25\), \(ELR(PC)=7/16=0.44\), \(ELR(CP)=1\) in the previous example. Tab. 1 in experiment section will list the statistics of ELR.

## 4 TOI-CNN Architecture

Fig. 2 illustrates the TOI-CNN architecture. TOI-CNN takes as input an entire sentence and a set of elements. The network first processes the w-hole sentence with one convolutional layer (Conv+Relu in Fig. 2) to yield 36 feature maps. Then for each element,the TOI pooling layer extracts a fixed-length feature vector from the feature map. Each feature vector is fed into a sequence of fully connected (fc1) layer that finally connects the output layer, which produces softmax probability estimates over \(K\) element classes plus a non-element class.

### The Convolutional Layer

We use the CNN model Kim (2014) with pretrained word embedding Mikolov et al. (2013) for the convolutional layer. \(w_{i}\) is \(i\)-th word in the sentence and is represented as the \(k\)-dimensional word embedding vector. The dimension of the input layer is \(n\times k\) ( padding zeros when the length of the sentence is less than \(n\) ). Our neural network 

consists of one convolution layer with ReLU activation and one TOI pooling layer, which replaces the max pooling layer of CNN in general. The convolution layer has a set of filters of size \(h\times k\) and produces \(p\) feature maps of size \((n-h+1)\times 1\).

### The TOI Pooling Layer

The TOI pooling layer uses max pooling to convert the features inside any valid region of an interesting window into a small feature map with a fixed length of \(L\) ( e.g., \(L=2\) in Fig. 2 ). Fig. 2 describes the TOI pooling in detail using red lines and rectangles. The TOI pooling layer extracts text region of the elements from the feature maps of the convolutional layer. The TOI region is shown as a red rectangle in the input sentence, shown in Fig. 2. The corresponding TOI window in the feature maps is connected by red curved lines. The length of the TOI window becomes shorter because of the narrow convolution.

TOI max pooling works by dividing the TOI window of length \(rl\) into \(L\) sub-windows of size \(\lfloor{rl/L}\rfloor\) and then max-pooling the values in each sub-window into the corresponding cell of TOI pooling layer. For example, \(rl=6\) in Fig. 2 and hence the sub-window of size \(6/2=3\) produce a cell of the pooling layer. Pooling operator is applied independently in each feature map channel. The pooling results of all feature maps are sequentially arranged into a vector, which is followed by the fully connected layer (\(fc1\)).

### Training Samples of TOI-CNN

In TOI-CNN, training samples include two categories: 1) TOI ground truths, 2) negative sliding windows. A sliding window is defined as negatives, or called non-element class, if IoU ( Intersection over Union ) with all ground truths of a sentence is less than a threshold \(th_{s}\). Function \(IoU(a,b)\), measuring how much overlap occurs between two text strings \(a\) and \(b\), is defined as,

\[IoU(a,b)=\frac{length(a\cap b)}{length(a\cup b)}. \tag{2}\]

### Backpropagation through TOI Pooling Layer

The network is modified to take two data inputs: a set of sentences and a list of training samples in those sentences. Each training sample is given as a one-hot encoding label \(p=(0,...,p_{j}=1,...,0)\) with a class \(j\) . The cross entropy loss \(L\) is,

\[L=-log~{}p^{*}_{j}, \tag{3}\]

where \(p^{*}\) is the output of the softmax layer. Then, we present the derivative rules in backpropagation through the TOI pooling layer.

Let \(x_{i}\) be the \(i\)-th activation input into the TOI pooling layer and let \(y_{s,j}\) be the layer's \(j\)-th output from the \(s\)-th training sample. The TOI pooling layer computes \(y_{s,j}=max(x_{i}),x_{i}\in\mathcal{W}_{s,j}\), where \(\mathcal{W}_{s,j}\) is the \(j\)-th input sub-window over which max-pooling outputs \(y_{s,j}\). Due to overlaps between training samples, a single \(x_{i}\) may be assigned to several different outputs \(y_{s,j}\). Let \(\mathcal{M}(x_{i})\) be the set of \(y_{s,j}\) that \(x_{i}\) activates in the TOI pooling layer.

Finally, the TOI pooling layer's backwards function computes partial derivative of the loss function with respect to input variable \(x_{i}\) as follows,

\[\frac{\partial L}{\partial x_{i}}=\sum_{s}\sum_{j}\frac{\partial L}{\partial y _{s,j}},~{}~{}y_{s,j}\in\mathcal{M}(x_{i}). \tag{4}\]

The partial derivative \(\partial L/\partial y_{s,j}\) is accumulated if \(y_{s,j}\) is activated by \(x_{i}\) in TOI max-pooling. In backpropagation, the partial derivatives \(\partial L/\partial y_{s,j}\) are already computed by the backwards function of the layer on top of the TOI pooling layer.

## 5 Experiments

### ETIP Dataset

We collected 500 Chinese insurance contracts, which include life, disability, health, property,

Figure 2: TOI-CNN architecture.

 

home, and auto insurance, where 350 contract-s are regarded as the corpus for training word embeddings (Mikolov et al., 2013) and the other 150 contracts are manually labeled for element tagging testing. The maximum nested level is three in ETIP. The dataset is available online ([https://github.com/ETIP-team/ETIP-Project/](https://github.com/ETIP-team/ETIP-Project/)) without author information. This project cooperated with an information solution provider of China Pacific Insurance Co., Ltd. (CPIC). Tab. 1 shows the number (N), average length (\(\overline{L}\)) and average element length ratio (\(\overline{ELR}\)) of seven categories in ETIP dataset. CP and IA are the two largest categories in the dataset. \(\overline{ELR}\) of C, PC and E are 0.12, 0.63 and 0.76 respectively, which means that they are usually a phrase or clause embedded in a sentence and C is a 2-3 word phrase. \(\overline{ELR}\) of CP, IA, and T are nearly 1.0, which denotes that they are always sentences.

### Experimental Settings

Chinese texts are tokenized with Jieba (Jieba, 2017) or NLPIR (NLPIR, 2018). 300-dimensional word vectors are trained on our insurance corpus. The size of the input layer in the CNN model is \(60\times 300\), and zeros are padded if the length of the training sample is less than 60. The kernel size of the convolution layer is \(5\times 300\), and the size of the feature maps is 36. the fixed length of TOI pooling layer output is \(72=2\times 36\).

The 150 labeled contracts are split into five e-qual folds, and we use the evaluation procedure in 5-fold cross-validation. Dealing with imbalanced data, the small categories, e.g., WP, E, and PC, are oversampled. The size of mini-batches is \(4\) sentences, randomly sampling up to \(48\) negative sliding windows from each sentence. We implement t TOI-CNN using PyTorch and run Adam (Kingma and Ba, 2014) optimizer for \(50\) training epochs with the learning rate of 0.0001.

In the detection, we apply a greedy non-maximum suppression within and between classes simultaneously if two sliding windows positionally intersect but they have no inclusion relation. In within-class suppression, a sliding window is rejected if its length is shorter than the other one. In between-class suppression, a sliding window is rejected if its softmax score is lower than the other one. In performance evaluation, a sliding window is recognized as true positive if IoU over a ground truth is larger than \(th_{p}\) and the predicted label is the same as the ground truth.

### Word Embedding Comparison

350 contracts in ETIP Dataset are regarded as the corpus for training word embeddings (Mikolov et al., 2013). The augmented word2vec model trained by our insurance contract corpus can improve the similarities of the insurance synonyms compared to the models trained by other corpora, e.g., Baidu Encyclopedia (Baidu, 2018), Wikipedia\(\_\)zh (Wikipedia, 2018), People's Daily News (People's_Daily, 2018). Cosine similarity between word vectors of insurance synonyms is shown in Tab. 2. The Chinese words are translated into English by Google Translate. Tab. 2 shows that the insurance corpus can greatly improve the word embedding similarity between insurance synonyms compared with other corpora.

### Performance of TOI-CNN on ETIP

Tab. 3 shows the confusion matrix computed by TOI-CNN with Jieba word segmentation, where \(th_{s}=0.5\) and \(th_{p}=0.8\) The confusion matrix has eight categories, where seven of them are the categories shown in Tab. 1 and the eighth one is negative. Each row of the matrix corresponds to an actual class, and each column of the matrix corresponds to a predicted class. The neg. in the rightmost column denote the ground truths which have been removed from the final candidates, i.e., false negatives. The neg. in the bottom row denote those final candidates who are not the real elements of seven categories, i.e., false positives. PC is more susceptible to negative sliding windows than other categories because PC is always a kind of time description and easily disturbed by other time descriptions in the insurance contracts. Condition for Payment (CP) and Insured Amount (I-A) could be confused with each other, because CP sometimes includes coverage amount descriptions like IA.

Tab. 4 shows the results of precision (P), recall (R) and F1 score on seven categories when

\begin{table}
\begin{tabular}{l l l l} \hline Category ID & N & \(\overline{L}\) & \(ELR\) \\ \hline Cover (C) & 618 & 2.6 & 0.12 \\ Waiting Period (WP) & 21 & 17.6 & 0.91 \\ Period of Coverage (PC) & 186 & 20.0 & 0.63 \\ Condition for Payment (CP ) & 1295 & 25.5 & 0.98 \\ Inserted Amount (IA) & 1068 & 27.3 & 0.99 \\ Exclusion (E) & 25 & 12.9 & 0.76 \\ Termination (T) & 398 & 9.2 & 0.97 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of seven categories in ETIP.

 

\(0.8\) and word segmentation={Jieba, NLPIR}. The overall performance of Jieba and NLPIR are approximately equal in TOI-CNN model. In TOI-CNN training, we create negative sliding windows using IoU threshold \(th_{s}\) (see Sec. 4.3). Fig. 3 shows F1 score comparison of negative samples generated with varied \(th_{s}\) when \(th_{p}=0.8\). The performance of \(th_{s}=0.5\) is better than that of others and close to that of \(th_{s}=0.6\) .

Tab. 5 shows the comparison of F1 scores with nested NER models, where \(th_{p}=0.8\). We use Jieba for word segmentation and POS tagging. We compare with two public nested NER models, Muis and Lu (2017) and Ju et al. (2018). We have tuned the hyper-parameters of the baselines for the best performance. We chose the best hyperparameters via Bayesian optimization provided by (Ju et al., 2018). We set the l2-regularization parameter (\(\lambda=0.01\)) and the number of Brown clusters (\(n=140\)) in (Muis and Lu, 2017). Our TOI-CNN outperforms other models in C, CP, E, and T, and get an excellent result in the overall performance.

## 6 Conclusion

This paper has presented a way of how to tag insurance policies. Seven critical elemental categories in the insurance policy are identified. We collected a large Chinese insurance contract corpus, labeled the samples with seven categories and published the dataset. The proposed TOI-CNN method can effectively solve the overlapping elements extraction problem. The overall performance of TOI-CNN is better than that of the probabilistic graphical models, especially for the overlapped phrases and clauses. Our method can accurately extract t the vast majority of Chinese insurance policies according to the pre-defined categories.

## 7 Acknowledgments

This work was supported by Zhejiang Provincial Natural Science Foundation of China (No.

Figure 3: Comparison of negative samples generated with varied \(th_{s}\).

\begin{table}
\begin{tabular}{c|c c c|c|c c|c c} \hline  & \multicolumn{3}{c|}{Jieba} & \multicolumn{3}{c}{NLPIR} \\ \cline{2-9}  & P & R & F1 & P & R & F1 \\ \hline C & 0.811 & 0.834 & 0.823 & 0.30 & 0.807 & 0.819 \\ WP & 1.00 & 0.524 & 0.688 & 1.00 & 0.476 & 0.645 \\ PC & 0.667 & 0.581 & 0.621 & 0.659 & 0.586 & 0.620 \\ CP & 0.908 & 0.902 & 0.905 & 0.910 & 0.901 & 0.905 \\ IA & 0.955 & 0.920 & 0.937 & 0.951 & 0.924 & 0.937 \\ E & 0.800 & 0.640 & 0.711 & 0.762 & 0.640 & 0.696 \\ T & 0.945 & 0.954 & 0.950 & 0.941 & 0.930 & 0.936 \\ \hline Avg. & 0.887 & 0.878 & **0.883** & 0.890 & 0.866 & **0.878** \\ \hline \end{tabular}
\end{table}
Table 4: Evaluation results of TOI-CNN on seven categories.

 



## References

* Azzopardi et al. (2016) Shaun Azzopardi, Albert Gatt, and Gordon J Pace. 2016. Integrating natural language and formal analysis for legal documents. In _10th Conference on Language Technologies and Digital Humanities_, volume 2016.
* Baidu (2018) Baidu. 2018. Baidu encyclopedia. [https://baike.baidu.com/](https://baike.baidu.com/).
* Chalkidis and Androutsopoulos (2017) Ilias Chalkidis and Ion Androutsopoulos. 2017. A deep learning approach to contract element extraction. In _JURIX_, pages 155-164.
* Chalkidis et al. (2017) Ilias Chalkidis, Ion Androutsopoulos, and Achilleas Michos. 2017. Extracting contract elements. In _Proc. of the 16th Int. Conf. on Artificial Intelligence and Law_, pages 19-28.
* Cohen and McCallum (2004) William Cohen and Andrew McCallum. 2004. Information extraction and integration: an overview. In _SIGKDD Conference_.
* Cortez and Da Silva (2013) Eli Cortez and Altigran S Da Silva. 2013. _Unsupervised information extraction by text segmentation_. Springer.
* Curtotti and McCreath (2010) Michael Curtotti and Eric McCreath. 2010. Corpus based classification of text in australian contracts.
* Doddington et al. (2004) George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In _LREC_, volume 2, page 1.
* Garcia-Constantino et al. (2017) Matias Garcia-Constantino, Katie Atkinson, Danushka Bollegala, Karl Chapman, Frans Coenen, Claire Roberts, and Katy Robson. 2017. Ciel: context-based information extraction from commercial law documents. In _Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law_, pages 79-87. ACM.
* Hasan et al. (2008) Ismael Hasan, Javier Parapar, and Roi Blanco. 2008. Segmentation of legislative documents using a domain-specific lexicon. In _Database and Expert Systems Application, 2008. DEXA'08. 19th International Workshop on_, pages 665-669. IEEE.
* Hu et al. (2017) Meng Hu, Zhixu Li, Yongxin Shen, An Liu, Guanfeng Liu, Kai Zheng, and Lei Zhao. 2017. cnn-jets: A cnn-based probabilistic approach for information extraction by text segmentation. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_, pages 1159-1168. ACM.
* Indukuri and Krishna (2010) Kishore Varma Indukuri and P Radha Krishna. 2010. Mining e-contract documents to classify clauses. In _Proceedings of the Third Annual ACM Bangalore Conference_, page 7. ACM.
* Jioba (2017) Jieba. 2017. [https://github.com/fxsjy/jieba](https://github.com/fxsjy/jieba). Accessed: 2018-05-28.
* Ju et al. (2018) Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. A neural layered model for nested named entity recognition. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, volume 1, pages 1446-1459.
* Katiyan and Cardie (2018) Arzoo Katiyan and Claire Cardie. 2018. Nested named entity recognition revisited. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, volume 1, pages 861-871.
* Kim et al. (2003) J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun'ichi T-sujiri. 2003. Genia corpus--a semantically annotated corpus for bio-textmining. _Bioinformatics_, 19(suppl_1):i180-i182.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. _arXiv preprint arXiv:1408.5882_.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Maccia (2009) Eneldo Loza Mencia. 2009. Segmentation of legal documents. In _Proceedings of the 12th International Conference on Artificial Intelligence and Law_, pages 88-97. ACM.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_.
* Moens et al. (2000) Marie-Francine Moens, Caroline Uyttenadele, and Jos Dumortier. 2000. Intelligent information extraction from legal texts. _Information & Communications Technology Law_, 9(1):17-26.
* Muis and Lu (2017) Aldrian Obaja Muis and Wei Lu. 2017. Labeling gaps between muis2017labelingwords: Recognizing overlapping mentions with mention separators.

\begin{table}
\begin{tabular}{l r r r r r r r r}  & C & WP & PC & CP & IA & E & T & Avg. \\ \hline Muis and Lu (2017) & 0.612 & 0.645 & 0.759 & 0.805 & **0.947** & **0.711** & 0.933 & 0.832 \\ Ju et al. (2018) & 0.808 & **0.757** & **0.788** & 0.886 & 0.783 & 0.664 & 0.906 & 0.854 \\ TOI-CNN & **0.823** & 0.688 & 0.621 & **0.905** & 0.937 & **0.711** & **0.950** & **0.883** \\ \hline \end{tabular}
\end{table}
Table 5: F1 score comparison with other models.



In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2608-2618.
* Nadeau and Sekine (2007) David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. _Lingvisticae Investigationes_, 30(1):3-26.
* NLPIR (2018) NLP PIR. 2018. [https://github.com/NLPIR-team/NLPIR](https://github.com/NLPIR-team/NLPIR). Accessed: 2018-05-28.
* People's Daily (2018) People's.Daily. 2018. News data from people's daily. [http://data.people.com.cn/](http://data.people.com.cn/).
* Piskorski and Yangarber (2013) Jakub Piskorski and Roman Yangarber. 2013. Information extraction: Past, present and future. In _Multi-source, multilingual information extraction and summarization_, pages 23-49. Springer.
* Ritter et al. (2011) Alan Ritter, Sam Clark, Oren Etzioni, et al. 2011. Named entity recognition in tweets: an experimental study. In _Proceedings of the conference on empirical methods in natural language processing_, pages 1524-1534. Association for Computational Linguistics.
* Wikipedia (2018) Wikipedia. 2018. Chinese wikipedia. [https://dumps.wikimedia.org/](https://dumps.wikimedia.org/).