# A Compression-Based Toolkit for Modelling and Processing Natural Language Text
## 4 Sample Applications Provided by the Toolkit
### Word Segmentation  
The method WordSegmentFile in Algorithm 8 performs word segmentation on the file \(SourceFile\). Specifically, it will insert spaces into the file which usually contains no spaces to indicate where the words are. Word segmentation is an important NLP application for languages that are not naturally word segmented such as Chinese. The method shown in Algorithm 8 is based on the PPM-based word segmentation method [9] that has been used as a baseline model for an international evaluation on Chinese word segmentation [38]. The method uses a transform object \(Transform\) to perform the segmentation. It is created on line 3 using TTM_create_transform which specifies that the Viterbi search algorithm be used to find the best segmentation (as measured by the sequence of text with spaces inserted which has the lowest compression codelength). The method then specifies using TTM_add_transform that two transformations be used for the search (lines 4 to 8). This defines the cost of the transformation in codelength (second argument to TTM_add_transform), the symbol sequence that needs to be matched (third argument) specified using a printf-like string syntax borrowed from the C programming language, and the transformed sequence (fourth argument) also specified using printf-like string syntax. The first transformation added to the transform model in the method on line 4 specifies that whatever the current symbol is (using the wildcard \"%w\" for the third argument to indicate that any symbol should be matched), it should always be included unchanged as one pathway in the search tree. This is because the wildcard \"%w\" is also being used for the fourth argument meaning that whatever symbol matches it should be transformed unchanged. Line 4 is typically found in most transform-based applications since usually the original text symbols need to form one search pathway rather than be totally discarded and/or transformed into something else. ```
1 method WordSegmentFile (\(SourceFile\), Model, Alphanumeric?) 2\(SourceText\leftarrow\)TXT_load_text (\(SourceFile\));
3\(Transform\leftarrow\)TTM_create_transform (TTM_Viterbi);
4 TTM_add_transform (\(Transform\), 0.0, \"%w\", \"%w\");
5 ifnotAlphanumeric?then
6 TTM_add_transform (\(Transform\), 0.0, \"%w\", \"%w\");
7 else
8 TTM_add_transform (\(Transform\), 0.0, \"%f\", \"%f\", TXT_is_aplhanumeric);
9 TTM_start_transform (\(Transform\), \(Transform\)Type, \(SourceText\), Model);
10\(TransformText\leftarrow\)TTM_perform_transform (\(Transform\), \(SourceText\));
11 TTM_release_text (\(SourceText\));
12 TTM_release_transform (\(Transform\));
13 return\(TransformText\);
14
15 end if
```  
**Algorithm 8**A compression-based method for word segmentation  
The method uses the argument _Alphanumeric?_ to indicate which of the second transformations (line 6 or line 8) should be applied to the search. If this argument is not true, then a space symbol is inserted prior to each symbol (there is a space included at the beginning of the transformed sequence \" %w\" for the fourth argument on line 6). This transformation would typically be used for word segmentation of Chinese text, for example, since in this case, we would wish to search through all the possible insertions of spaces after each of the symbols in the text. If the argument _Alphanumeric?_ is true, on the other hand, the transformation on line 8 specifies that spaces only be inserted after alphanumeric characters as a possible search pathway (using the \" %f\" option for the matching and transformed sequences to specify that the function TXT_is_alphanumeric be used to match the symbol). This transformation is a useful optimisation if English text without spaces is beingsegmented (by finding where to put the spaces back) since it will substantially reduce the search as non-alphanumeric characters will remain unchanged and not cause an extra pathway to be added to the search tree. The effect of having two transformations used in this method is that two search pathways will be generated for every symbol being transformed.