Segmenting Natural Language Sentences via Lexical Unit Analysis

1 INTRODUCTION . Sequence segmentation is essentially the process of partitioning a sequence of fine-grained lexical units into a sequence of coarse-grained ones . In some scenarios , each composed unit is assigned a categorical label . For example , Chinese word segmentation splits a character sequence into a word sequence ( Xue , 2003 ) . Syntactic chunking segments a word sequence into a sequence of labeled groups of words ( i.e. , constituents ) ( Sang & Buchholz , 2000 ) . There are currently two mainstream approaches to sequence segmentation . The most common is to regard it as a sequence labeling problem by using IOB tagging scheme ( Mesnil et al. , 2014 ; Ma & Hovy , 2016 ; Liu et al. , 2019b ; Chen et al. , 2019a ; Luo et al. , 2020 ) . A representative work is Bidirectional LSTM-CRF ( Huang et al. , 2015 ) , which adopts LSTM ( Hochreiter & Schmidhuber , 1997 ) to read an input sentence and CRF ( Lafferty et al. , 2001 ) to decode the label sequence . This type of method is very effective , providing tons of state-of-the-art performances . However , it is vulnerable to producing invalid labels , for instance , “ O , I-tag , I-tag ” . This problem is very severe in low resource settings ( Peng et al. , 2017 ) . In experiments ( see section 4.6 ) , we also find that it performs poorly in recognizing long-length segments . Recently , there is a growing interest in span-based models ( Zhai et al. , 2017 ; Li et al. , 2019 ; Yu et al. , 2020 ) . They treat a span rather than a token as the basic unit for labeling . Li et al . ( 2019 ) cast named entity recognition ( NER ) to a machine reading comprehension ( MRC ) task , where entities are extracted as retrieving answer spans . Yu et al . ( 2020 ) rank all the spans in terms of the scores predicted by a bi-affine model ( Dozat & Manning , 2016 ) . In NER , span-based models have significantly outperformed their sequence labeling based counterparts . While these methods circumvent the use of IOB tagging scheme , they still rely on post-processing rules to guarantee the extracted span set to be valid . Moreover , since span-based models are locally normalized at span level , they potentially suffer from the label bias problem ( Lafferty et al. , 2001 ) . This paper seeks to provide a new framework which infers the segmentation of a unit sequence by directly selecting from all valid segmentation candidates , instead of manipulating tokens or spans . To this end , we propose Lexical Unit Analysis ( LUA ) in this paper . LUA assigns a score to every valid segmentation candidate and leverages dynamic programming ( DP ) ( Bellman , 1966 ) to search for the maximum scoring one . The score of a segmentation is computed by using the scores of its all segments . Besides , we adopt neural networks to score every segment of the input sentence . The purpose of using DP is to solve the intractability of extracting the maximum scoring segmentation candidate by brute-force search . The time complexity of LUA is quadratic time , yet it can be optimized to linear time in practice by performing parallel matrix computations . For training criterion , we incur a hinge loss between the ground truth and the predictions . We also extend LUA to unlabeled segmentation and capturing label correlations . Figure 1 illustrates the comparison between previous methods and the proposed LUA . Prior models at token level and span level are vulnerable to generating invalid predictions , and hence rely on heuristic rules to fix them . For example , in the middle part of Figure 1 , the spans of two inferred named entities , [ Word Cup ] MISC and [ Cup ] MISC , conflicts , which is mitigated by comparing the predicted scores . LUA scores all possible segmentation candidates and uses DP to extract the maximum scoring one . In this way , our models guarantee the predictions to be valid . Moreover , the globality of DP addresses the label bias problem . Extensive experiments are conducted on syntactic chunking , NER , slot filling , Chinese word segmentation , and Chinese part-of-speech ( POS ) tagging across 15 tasks . We have obtained new stateof-the-art results on 13 of them and performed competitively on the others . In particular , we observe that LUA is expert at identifying long-length segments . 2 METHODOLOGY . We denote an input sequence ( i.e. , fine-grained lexical units ) as x = [ x1 , x2 , · · · , xn ] , where n is the sequence length . An output sequence ( i.e. , coarse-grained lexical units ) is represented as the segmentation y = [ y1 , y2 , · · · , ym ] with each segment yk being a triple ( ik , jk , tk ) . m denotes its length . ( ik , jk ) specifies a span that corresponds to the phrase xik , jk = [ xik , xik+1 , · · · , xjk ] . tk is a label from the label space L. We define a valid segmentation candidate as its segments are non-overlapping and fully cover the input sequence . A case extracted from CoNLL-2003 dataset ( Sang & De Meulder , 2003 ) : x = [ [ SOS ] , Sangthai , Glory , 22/11/96 , 3000 , Singapore ] y = [ ( 1 , 1 , O ) , ( 2 , 3 , MISC ) , ( 4 , 4 , O ) , ( 5 , 5 , O ) , ( 6 , 6 , LOC ) ] . Start-of-sentence symbol [ SOS ] is added in the pre-processing stage . 2.1 MODEL : SCORING SEGMENTATION CANDIDATES . We denote Y as the universal set that contains all valid segmentation candidates . Given one of its members y ∈ Y , we compute the score f ( y ) as f ( y ) = ∑ ( i , j , t ) ∈y ( sci , j + s l i , j , t ) , ( 1 ) Algorithm 1 : Inference via Dynamic Programming ( DP ) Input : Composition score sci , j and label score sli , j , t for every possible segment ( i , j , t ) . Output : The maximum segmentation scoring candidate ŷ and its score f ( ŷ ) . 1 Set two n× n shaped matrices , cL and bc , for computing maximum scoring labels . 2 Set two n-length vectors , g and bg , for computing maximum scoring segmentation . 3 for 1 ≤ i ≤ j ≤ n do 4 Compute the maximum label score for each span ( i , j ) : sLi , j = maxt∈L s l i , j , t . 5 Record the backtracking index : bci , j = argmaxt∈L s l i , j , t . 6 Initialize the value of the base case x1,1 : g1 = sc1,1 + s L 1,1 . 7 for i ∈ [ 2 , 3 , · · · , n ] do 8 Compute the value of the prefix x1 , i : gi = max1≤j≤i−1 ( gi−j + ( s c i−j+1 , i + s L i−j+1 , i ) ) . 9 Record the backtracking index : bgi = argmax1≤j≤i−1 ( gi−j + ( s c i−j+1 , i + s L i−j+1 , i ) ) . 10 Get the maximum scoring candidate ŷ by back tracing the tables bg and bc . 11 Get the maximum segmentation score : f ( ŷ ) = gn . where sci , j is the composition score to estimate the feasibility of merging several fine-grained units [ xi , xi+1 , · · · , xj ] into a coarse-grained unit and sli , j , t is the label score to measure how likely the label of this segment is t. Both scores are obtained by a scoring model . Scoring Model . a scoring model scores all possible segments ( i , j , t ) for an input sentence x. Firstly , we get the representation for each fine-grained unit . Following prior works ( Li et al. , 2019 ; Luo et al. , 2020 ; Yu et al. , 2020 ) , we adopt BERT ( Devlin et al. , 2018 ) , a powerful pre-trained language model , as the sentence encoder . Specifically , we have [ hw1 , h w 2 · · · , hwn ] = BERT ( x ) , ( 2 ) Then , we compute the representation for a coarse-grained unit xi , j , 1 ≤ i ≤ j ≤ n as hpi , j = h w i ⊕ hwj ⊕ ( hwi − hwj ) ⊕ ( hwi hwj ) , ( 3 ) where ⊕ is vector concatenation and is element-wise product . Eventually , we employ two non-linear feedforward networks to score a segment ( i , j , t ) : sci , j = ( vc ) T tanh ( Wchpi , j ) , s l i , j , t = ( vlt ) T tanh ( Wlhpi , j ) , ( 4 ) where vc , Wc , vlt , t ∈ L , and Wl are all learnable parameters . Besides , the scoring model used here can be flexibly replaced by any regression method . 2.2 INFERENCE VIA DYNAMIC PROGRAMMING . The prediction of the maximum scoring segmentation candidate can be formulated as ŷ = argmax y∈Y f ( y ) . ( 5 ) Because the size of search space |Y| increases exponentially with respect to the sequence length n , brute-force search to solve Equation 5 is computationally infeasible . LUA uses DP to address this issue , which is facilitated by the decomposable nature of Equation 1 . DP is a well-known optimization method which solves a complicated problem by breaking it down into simpler sub-problems in a recursive manner . The relation between the value of the larger problem and the values of its sub-problems is called the Bellman equation . Sub-problem . In the context of LUA , the sub-problem of segmenting an input unit sequence x is segmenting its prefixes x1 , i , 1 ≤ i ≤ n. We define gi as the maximum segmentation score of the prefix x1 , i . Under this scheme , we have maxy∈Y f ( y ) = gn . The Bellman Equation . The relatinship between segmenting a sequence x1 , i , i > 1 and segmenting its prefixes x1 , i−j , 1 ≤ j ≤ i− 1 is built by the last segments ( i− j + 1 , i , t ) : gi = max 1≤j≤i−1 ( gi−j + ( s c i−j+1 , i +max t∈L sli−j+1 , i , t ) ) . ( 6 ) In practice , to reduce the time complexity of above equation , the last term is computed beforehand as sLi , j = maxt∈L s l i , j , t , 1 ≤ i ≤ j ≤ n. Hence , Equation 6 is reformulated as gi = max 1≤j≤i−1 ( gi−j + ( s c i−j+1 , i + s L i−j+1 , i ) ) . ( 7 ) The base case is the first token x1,1 = [ [ SOS ] ] . We get its score g1 as sc1,1 + s L 1,1 . Algorithm 1 shows how DP is applied in inference . Firstly , we set two matrices and two vectors to store the solutions to the sub-problems ( 1-st to 2-nd lines ) . Secondly , we get the maximum label scores for all the spans ( 3-rd to 5-th lines ) . Then , we initialize the trivial case g1 and recursively calculate the values for prefixes x1 , i , i > 1 ( 6-th to 9-th lines ) . Finally , we get the predicted segmentation ŷ and its score f ( ŷ ) ( 10-th to 11-th lines ) . The time complexity of Algorithm 1 is O ( n2 ) . By performing the max operation of Equation 7 in parallel on GPU , it can be optimized to only O ( n ) , which is highly efficient . Besides , DP , as the backbone of the proposed model , is non-parametric . The trainable parameters only exist in the scoring model part . These show LUA is a very light-weight algorithm .

Summary: The paper is well-written, easy to follow and clear. However, the novelty and main contribution of the paper is not clear. The authors used a scoring model to score the composition of each segment, as well as the probability of having a specific label for the segment. The BERT language model is used in the paper to encode the input sequence. The training part is a more like a supervised training and a dynamic programming (DP) approach is used for inference. It is not clear how DP contributes to the success of the model, as the scores for segments are derived during the training (which seems most of the success is coming from the labeled data (i.e. supervised training) and BERT encoding). One other thing about formatting and citing references, some of the references are published in conference proceedings, not sure why authors cited their arxiv version.