Venue: NAACL 2016

# Weak Semi-Markov CRFs for NP Chunking in Informal Text

Aldrian Obaja Muis

Wei Lu

Singapore University of Technology and Design

{aldrian_muis,luwei}@sutd.edu.sg

###### Abstract

This paper introduces a new annotated corpus based on an existing informal text corpus: the NUS SMS Corpus Chen and Kan (2013). The new corpus includes 76,490 noun phrases from 26,500 SMS messages, annotated by university students. We then explored several graphical models, including a novel variant of the semi-Markov conditional random fields (semi-CRF) for the task of noun phrase chunking. We demonstrated through empirical evaluations on the new dataset that the new variant yielded similar accuracy but ran in significantly lower running time compared to the conventional semi-CRF.

## 1 Introduction

Processing user-generated text data is getting more popular recently as a way to gather information, such as collecting facts about certain events Ritter et al. (2015), gathering and identifying user profiles Layton et al. (2010); Li et al. (2014); Spitters et al. (2015), or extracting information in open domain Ritter et al. (2012); Mitchell et al. (2015).

Most recent work focus on the texts generated through Twitter, which, due to the design of Twitter, contain a lot of announcement-like messages mostly intended for general public. In contrast, SMS was designed as a way to communicate short personal messages to a known person, and hence SMS messages tend to be more conversational and more informal compared to tweets.

As conversational texts, SMS data often contains references to named entities such as people and locations relevant to certain events. Recognizing those references will be useful for further NLP tasks. One way to recognize those named entities is to first create a list of candidates, which can be further filtered to get the desired named entities. Nadeau Nadeau and Sekine (2007) lists several methods that work upon candidates for NER. As all named entities are nouns, recognizing noun phrases (NP) is therefore a task that can be potentially useful for further steps in the NLP pipeline to build upon. Figure 1 shows an example SMS message within which noun phrases are highlighted. As can be seen from this example, recognizing the NP information on such a dataset presents some additional challenges over conventional NP recognition tasks. Specifically, the texts are highly informal and noisy, with misspelling errors and without grammatical structures. The correct casing and punctuation information is often missing. The lack of spaces between adjacent words makes the detection of NP boundaries more challenging.

Furthermore, the lack of available annotated data for such informal datasets prevents researchers from understanding what effective models can be used to resolve the above issues. In this work, we focus on tackling these issues while making the following two main contributions:

* We build a new corpus of SMS data that is fully annotated with noun phrase information.
* We propose and build a new variant of semi-Markov CRF Sarawagi and Cohen (2004) for

Figure 1: Sample SMS, with NPs underlined

the task of NP chunking on our corpus, which is faster and yields a performance similar to the conventional semi-Markov CRF models.

## 2 NP-annotated SMS Corpus

Our text corpus comes from the NUS SMS Corpus Chen and Kan (2013), containing 55,835 SMS messages from university students, mostly in English. We used the 2011 version of the corpus, containing 45,718 messages, as it is more relevant to modern phone models using full keyboard layout.

We note that there are a small portion of the messages written in non-English language, such as Tamil and Chinese. As we are focusing on English, we excluded messages written by non-native English speakers based on the metadata (21.3% of all messages). We also excluded messages which contain only one word (6.1%) and we remove duplicate messages (8.1%). 1

Footnote 1: We also manually excluded some messages (ID 1017-4016) which are mostly not written in English (4.0% of all messages).

We assigned the remaining 27,700 messages to 64 university students who conduct annotations, each annotating 500 with 100 messages co-annotated by two other annotators. After manual verification we excluded annotations with low quality from 3 students. We used the resulting 26,500 messages as our dataset. The students were asked to annotate the top-level noun phrases found in each message using the BRAT rapid annotation tool2, where they were instructed to highlight character spans to be marked as noun phrases. The number of noun phrases per message can be found in Table 1.

Footnote 2: [http://brat.nlplab.org](http://brat.nlplab.org)

Due to the noisy nature of SMS messages, there may not be proper capitalization or punctuation, and in some cases there might be missing spaces between words. Figure 1 shows a sample SMS message taken from the corpus. We can see that "Dr teh" is not properly capitalized and "she" in "butshe's" is missing spaces around it. NPs which do not have clear boundaries (_improper_ NPs) constitutes 4.0% of all NPs.

We then use this dataset to evaluate some models on base NP chunking task, where, given a text, the system should return a list of character spans denoting the noun phrases found in the text.

## 3 Models

In this paper, we will build our models based on a class of discriminative graphical models, namely conditional random fields (CRFs) Lafferty et al. (2001), for extracting NPs. The edges in the graph represents the dependencies between states and the features are defined over each edge in the graph. Though CRFs are undirected graphical models, we can use directed acyclic graphs with a root, a leaf, and some inner nodes to represent label sequences3. A path in the graph from the root to the leaf represents one possible label assignment to the input. In the labeled instance, there will be only one single path from the root to the leaf, while for the unlabeled instance, the graph will compactly encode all possible label assignments. The learning procedure is essentially the process that tries to tune the feature weights such that the true structures get assigned higher weights as compared to all other alternative structures in the graph.

Footnote 3: Extension to directed hypergraphs is possible. See Lu and Roth (2015).

In general, a CRF tries to maximize the following objective function:

\[\mathcal{L}(\mathcal{T})=\\ \sum_{(\mathbf{x},\mathbf{y})\in\mathcal{T}}\!\left[\!\sum_{e\in \mathcal{E}(\mathbf{x},\mathbf{y})}\!\!\!\mathbf{w}^{T}\mathbf{f}(e)-\log Z_{ \mathbf{w}}(\mathbf{x})\right]\!\!-\lambda||\mathbf{w}||^{2} \tag{1}\]

where \(\mathcal{T}\) is the training set, \((\mathbf{x},\mathbf{y})\) is a training instance consisting of the sentence \(\mathbf{x}\) and the label sequence \(\mathbf{y}\in\mathcal{Y}^{n}\) for a label set \(\mathcal{Y}\), \(\mathbf{w}\) is the feature weight vector, \(\mathcal{E}(\mathbf{x},\mathbf{y})\) is the set of edges which defines the input \(\mathbf{x}\) labeled with the label sequence \(\mathbf{y}\), \(\mathbf{f}(e)\) is the feature vector of the edge \(e\), \(Z_{\mathbf{w}}(\mathbf{x})\) is the normalization term which sums over all possible paths from the root to the leaf node, and \(\lambda\) is the regularization parameter.

\begin{table}
\begin{tabular}{c|c c|c c}  & \#SMS & \#NPs & \#improper & \#tokens \\ \hline total & 26,500 & 76,490 & 3,066 (4.0) & 359,009 \\ train & 21,200 & 61,212 & 2,406 (3.9) & 287,590 \\ dev & 2,650 & 7,617 & 338 (4.4) & 35,470 \\ test & 2,650 & 7,661 & 322 (4.2) & 35,949 \\ \end{tabular}
\end{table}
Table 1: Number of messages, NPs, number of _improper_ NPs (as percentage in brackets), which are NPs that do not have clear boundaries, and number of tokens.

 

The set of edges and features defined in each model affects the feature expectation and the normalization term. Computation of the normalization term, being the highest in time complexity, will determine the overall complexity of training the model. The set of edges and the normalization term in each model will be described in the following sections.

### Linear CRF

A linear-chain CRF, or linear CRF is a standard version of CRF which was introduced in Lafferty et al. (2001), where each word in the sentence is given a set of nodes representing the possible labels, and edges are present between any two nodes from adjacent words, forming a trellis graph. Here we consider only the first-order linear CRF.

The normalization term \(Z_{\mathbf{w}}(\mathbf{x})\) is calculated as:

\[Z_{\mathbf{w}}(\mathbf{x})=\sum_{\mathbf{y}}\exp_{(y^{\prime},y,i)\in\mathcal{ E}(\mathbf{x},\mathbf{y})}\hskip-56.905512pt\mathbf{w}^{T}\mathbf{f}_{\mathbf{ x}}(y^{\prime},y,i) \tag{2}\]

where \(\mathbf{f}_{\mathbf{x}}(y^{\prime},y,i)\) represents the feature vector on the edge connecting state \(y^{\prime}\) at position \(i-1\) to state \(y\) at position \(i\). The time complexity of the inference procedure for this model is \(O(n\left|\mathcal{Y}\right|^{2})\).

### Semi-CRF

In semi-CRF Sarawagi and Cohen (2004), in addition to the edges defined in linear CRF, there are additional edges from a node to all nodes up to \(L\) next words away, representing a segment within which the words will be labeled with a single label.

The normalization term \(Z_{\mathbf{w}}(\mathbf{x})\) is calculated as:

\[Z_{\mathbf{w}}(\mathbf{x})=\sum_{\mathbf{y}\in\mathcal{Y}^{n}} \exp_{(y^{\prime},y,i-k,i)\in\mathcal{E}(\mathbf{x},\mathbf{y})}\hskip-85.35827pt \mathbf{w}^{T}\mathbf{g}_{\mathbf{x}}(y^{\prime},y,i-k,i) \tag{3}\]

where \(\mathbf{g}_{\mathbf{x}}(y^{\prime},y,i-k,i)\) represents the feature vector on the edge connecting state \(y^{\prime}\) at position \(i-k\) to state \(y\) at position \(i\). The time complexity for this model is \(O(nL\left|\mathcal{Y}\right|^{2})\).

### Weak Semi-CRF

Note that in semi-CRF, each node is connected to \(L\times\left|\mathcal{Y}\right|\) next nodes. Intuitively, the model tries to decide the next segment length and type at the same time. We now propose a weaker variant that makes the two decisions separately by restricting each node to connect to either only the nodes of the same label up to \(L\) next words away, or to all the nodes only in the next word. We call this variant _Weak Semi-CRF_.

To implement this, we need to split the original nodes into Begin and End nodes, representing the start and end of a segment. The End nodes connect only to the very next Begin nodes of any label, while the Begin nodes connect only to the End nodes of same label up to next \(L\) words. We denote the set of the earlier edges as \(\mathcal{E}_{A}(\mathbf{x},\mathbf{y})\) and the latter edges as \(\mathcal{E}_{J}(\mathbf{x},\mathbf{y})\). The normalization term \(Z_{\mathbf{w}}(\mathbf{x})\) is then:

\[Z_{\mathbf{w}}(\mathbf{x})=\sum_{\mathbf{y}\in\mathcal{Y}^{n}} \exp\bigg{(}\sum_{(y^{\prime},y,i)\in\mathcal{E}_{A}(\mathbf{x}, \mathbf{y})}\hskip-85.35827pt\mathbf{w}^{T}\mathbf{f}_{\mathbf{x}}(y^{\prime}, y,i)\\ +\sum_{(y,i-k,i)\in\mathcal{E}_{J}(\mathbf{x},\mathbf{y})}\hskip-85.35827 pt\mathbf{g}_{\mathbf{x}}(y,i-k,i)\bigg{)} \tag{4}\]

where \(\mathbf{g}_{\mathbf{x}}(y,i-k,i)\) represents the feature vector on the edge connecting the Begin node with state \(y\) at position \(i-k\) to the End node with the same state \(y\) at position \(i\). Note that, different from the \(\mathbf{g}_{\mathbf{x}}\) function defined in Equation (3), this new \(\mathbf{g}_{\mathbf{x}}\) function is defined over a single (current) \(y\) label only, making the time complexity \(O(n\left|\mathcal{Y}\right|^{2}+nL\left|\mathcal{Y}\right|)\). Theoretically this model is slightly more efficient than the conventional semi-CRF model.

Unlike conventional (first-order) semi-Markov CRF, this new model does not allow us to capture the dependencies between one segment and its adjacent segment's label information. We argue that, however, such dependencies can be less crucial for our task. We will empirically assess this aspect through experiments. Figure 2 illustrates the differences among the three models.

## 4 Features

In linear CRF, the baseline feature set considers the previous word, current word, and the tag transition.

In semi-CRF, following Sarawagi and Cohen (2004) we put each word which is not part of a noun phrase in its own segment, and put each noun phrase in one segment, possibly spanning over multiple words. Here we set \(L=6\) and ignored NPs with more than six words during training, which is less than 0.5% of all NPs. For each segment, we defined the following features as the baseline: (1) 

indexed words inside current segment, running from the start and from the end of the segment, (2) the word before and after current segment, and (3) the labels of previous segment and current segment.

In weak semi-CRF we use the same feature set as semi-CRF, adjusting the features accordingly where segment-specific features (1) are defined only in the Begin-End edges, and transition features (3) are defined only in the End-Begin edges.

For each model we then add the character prefixes and suffixes up to length \(3\) for each word (+a), Brown cluster Brown et al. (1992) for current word (+D), and word shapes (+s). For Brown cluster features we used \(100\) clusters trained on the whole NUS SMS Corpus. The cluster information is then used directly as a feature.

Word shapes can be considered a generic representation of words that retains only the "shape" information, such as whether it starts with capital letter or whether it contains digits. The Brown clusters and word shapes features are applied to each of the word features described in each model.

## 5 Experiments

All models were built by us using Java, and were optimized with L-BFGS. Models are all tuned in the development set for optimal \(\lambda\). The optimal \(\lambda\) values are noted in Table 2.

Since the models that we consider are all word-based 4, we tokenize the corpus using a regex-based tokenizer similar to the wordpunct_tokenize function in Python NLTK package. We also included some rules to consider special anonymization tokens in the SMS dataset Chen and Kan (2013).

Footnote 4: We experimented with character-based models, but they do not perform well. We leave them for future investigations.

The gold character spans are converted into word labels in BIO format, reducing or extending the character spans as necessary to the closest word boundaries. The converted annotations are regarded as gold word spans. Note that this conversion is lossy due to the presence of _improper_ NPs, which makes it impossible for the converted format to represent the original gold standard.

We evaluated the models in the original character-level spans and also in the converted word-level spans, to see the impact of the lossy conversion on the scores. In character-level evaluation, the system output is converted back into character boundaries and compared with the original gold standard, while in the word-level evaluation, the system output is compared directly with the gold word spans. For this reason, we anticipate that the scores in word-level evaluation will be higher than in the character-level evaluation. The results are shown in Table 3. The scores for "Gold" in the character-level evaluation mark the upperbound of word-based models due to the presence of _improper_ NPs.

The average time per training iteration on the base models is \(1.311\)s, \(2.072\)s, and \(1.811\)s respectively for Linear CRF, Semi-CRF, and Weak Semi-CRF.

Figure 2: Graphical illustrations of the differences between three models. The bold arrows represent the path in each model to label “Dr Teh” as a noun phrase. For Linear CRF, this is a simplified diagram; in the implementation we used the “BIO” approach to represent text chunks. The underlined nodes in Weak Semi-CRF are the Begin nodes.

 



### Discussion

First, we see that the two variants of semi-CRF models perform better compared to the baseline linear CRF model, showing the benefit of using segment features over only single word features.

It is also interesting that, while being a weaker version of the semi-CRF, the weak semi-CRF can actually perform in the same level within 95% confidence interval as the conventional semi-CRF. This shows that some of the dependencies in the conventional semi-CRF do not really contribute to the strength of semi-CRF over standard linear CRF. As noted in Section 3.3, weak semi-CRF makes the decision on the segment type and length separately. This means there is enough information in the local features to decide the segment type and length separately, and so we can remove some combined features while retaining the same performance.

This result, coupled with the fact that the weak semi-CRF requires 12.5% less time than the conventional semi-CRF (1.811s vs 2.072s), shows the potentials of using this weak semi-CRF as an alternative of the conventional semi-CRF. With more label types (here only two), the difference will be larger, since the weak semi-CRF is linear in number of label types, while conventional semi-CRF is quadratic.

## 6 Related Work

Ritter et al. Ritter:2011:EMNLP previously showed that off-the-shelf NP-chunker performs worse on informal text. Then they trained a linear-CRF model on additional in-domain data, reducing the error up to 22%. However no results on semi-CRF was given.

Semi-CRF has proven effective in chunking tasks. Other variants of semi-CRF models also exist. Nguyen:2014:NICAS explored the use of higher-order dependencies to improve the performance of semi-CRF models on synthetic data and on handwriting recognition. They exploited the sparsity of label sequence in order to make the training efficient.

It is also known that feature selection is an important aspect when trying to use semi-CRF models to improve on the linear CRF. Andrew Andrew reported an error reduction of up to 25% when using features that are best exploited by semi-CRF.

## 7 Conclusion and Future Work

In this paper we present a new NP-annotated SMS corpus, together with a novel variant of the semi-CRF model, which runs in significantly lower time while maintaining similar accuracy on the NP chunking task on the new dataset. Future work includes the application of the weak semi-CRF model to other structured prediction problems, as well as performing investigations on handling other types of informal or noisy texts such as speech transcripts. We make the code and data available for download at [http://statnlp.org/research/ie/](http://statnlp.org/research/ie/).

## Acknowledgements

We would like to thank Alexander Binder, Jie Yang, Dinh Quang Thinh as well as the 64 undergraduate students who helped us with annotations. We would also like to thank the three anonymous reviewers for their helpful comments. This work is supported by SUTD grant SRG ISTD 2013 064 and MOE Tier 1 grant SUTDT12015008. We thank Razvan Bunescu for pointing out an error in Equations 2, 3, and 4 in an earlier version of this paper.





## References

* [1]G. Andrew (2006) A hybrid markov/semi-markov conditional random field for sequence segmentation. In Proc. EMNLP'06, Cited by: SS3.4.
* P. F. Brown, P. V. DeSouza, R. L. Mercer, C. J. Della Pietra, and J. C. Lai (1992) Class-Based n-gram Models of Natural Language. Computational Linguistics18 (4), pp. 467-479. External Links: ISSN 0025-921X, Link, Document Cited by: SS1, SS2.
* T. Chen and M. Kan (2013) Creating a live, public short message service corpus: the NUS SMS corpus. In Language Resources and Evaluation, Vol. 47, pp. 299-335. External Links: ISBN 978-3-319-1450-7, Link, Document Cited by: SS1, SS2.
* J. Lafferty, A. McCallum, and F. Pereira (2001) Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In International Conference on Machine Learning (ICML), pp. 282-289. Cited by: SS1, SS2.
* R. Layton, P. Watters, and R. Dazeley (2010) Authorship Attribution for Twitter in 140 Characters or Less. In 2010 Second Cybercrime and Trustworthy Computing Workshop, pp. 1-8. Cited by: SS1, SS2.
* J. Li, A. Ritter, and E. Hovy (2014) Weakly Supervised User Profile Extraction from Twitter. In Association for Computational Linguistics, pp. 165-174. Cited by: SS1, SS2.
* W. Lu and D. Roth (2015) Joint mention extraction and classification with mention hypergraphs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, September, pp. 857-867. External Links: ISBN 978-3-319-1450-7, Link, Document Cited by: SS1, SS2.
* T. M. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, J. Betteridge, A. Carlson, B. Dalvi Mishra, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Antonios Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and J. Welling (2015) Never-Ending Learning. In AAAI Conference on Artificial Intelligence, pp. 2302-2310. Cited by: SS1, SS2.
* D. Nadeau and S. Sekine (2007) A survey of named entity recognition and classification. Lingvisticae Investigationes30 (1), pp. 3-26. Cited by: SS1, SS2.
* V. Cuong Nguyen, N. Ye, W. S. Lee, and H. Leong Chieu (2014) Conditional Random Field with High-order Dependencies for Sequence Labeling and Segmentation. Journal of Machine Learning Research2014, pp. 15:981-1009. Cited by: SS1, SS2.
* A. Ritter, S. Clark, and O. Etzioni (2011) Named entity recognition in tweets: an experimental study. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, pp. 1524-1534. External Links: ISBN 978-3-319-1450-7, Link, Document Cited by: SS1, SS2.
* A. Ritter, O. Etzioni, and S. Clark (2012) Open Domain Event Extraction from Twitter. In Proceedings Workshop on Text Mining, ACM International Conference on Knowledge Discovery and Data Mining 2012 (KDD'12), Cited by: SS1, SS2.
* A. Ritter, E. Wright, W. Casey, and T. M. Mitchell (2015) Weakly Supervised Extraction of Computer Security Events from Twitter. In Proceedings of the 24th International Conference on World Wide Web, Vol. i, pp. 896-905. Cited by: SS1, SS2.
* S. Sarawagi and W. W. Cohen (2004) Semi-markov conditional random fields for information extraction. In Advances in Neural Information Processing Systems 17, pp. 1185-1192. Cited by: SS1, SS2.
* M. Spitters, F. Klaver, G. Koot, and M. van Staalduinen (2015) Authorship Analysis on Dark Marketplace Forums. In Proceedings of the IEEE European Intelligence & Security Informatics Conference 2015 (EISIC 2015), Cited by: SS1, SS2.