# A Compression-Based Toolkit for Modelling and Processing Natural Language Text
## 4 Sample Applications Provided by the Toolkit
### Applications Implemented by the Toolkit
\begin{table}
\begin{tabular}{l l} \hline
1 & encode -a 256 -O 5 \textless{Brown.txt \textgreater{}Brown.encoded} \\  & bytes input 5998525 bytes output 1613517 2.152 bpc \\
2 & decode -a 256 -O 5 \textless{Brown.encoded} \textgreater{}Brown.decoded \\
3 & diff Brown.decoded Brown.txt \\
4 & train -S -a 256 -O 5 -e D -T \textasci{\"Brown.txt \textgreater{}Brown.model \textasci{}} \\  & Trained on 5998525 symbols \\
5 & encode -m Brown.model \textless{LOB.txt \textgreater{}LOB.encoded} \\  & bytes input 5877271 bytes output 1609902 2.191 bpc \\
6 & encode -a 256 -O 5 \textless{LOB.txt \textgreater{}/dev/null} \\  & bytes input 5877271 bytes output 1565250 2.131 bpc \\
7 & codelength -e -m models.txt \textless{}Brown.txt \\  & English & 3.514 \\  & French & 5.421 \\  & German & 5.618 \\  & Italian & 5.664 \\  & Latin & 5.400 \\  & Maori & 5.820 \\  & Spanish & 5.602 \\  & Minimum cross-entropy for English = 3.514 \\
8 & segment -m Brown.model -V -i woods\_unsegm.txt -o woods\_segm.txt \\
9 & diff woods\_segm.txt woods\_correct.txt \\  & \textless{} Of easy wind and down yflake \\  & \textgreater{} Of easy wind and downy flake \\
10 & markup -m models.txt \textless{}e\_and\_m.txt -V -F \\  & \textless{Maori\textgreater{}TE} \textless{}Maori\textgreater{}TE \textless{}/Maori\textgreater{}E\textless{}/Maori\textgreater{}E\textless{} English\textgreater{}AND MAORI INTELLIGENER \\  & LET THE PAKERHA AND THE MAORI BE UNITED. \\  & Salutations to you friends the Maori people of New Zealand! Chiefs and \\  & commoners, great and small, old and young salutations to you all! \\  & It is customary among the Maories to welcome the advent of a stranger \\  & With cries of \textless{}/English\textgreater{}Maori\textgreater{}Haeremai! Haeremai! \textless{}/Maori\textgreater{}E\textless{}English\textgreater{} \\  & The welcome is commenced with the approach \\  & of the visitor, and is prolonged until he has fairly entered the Kainga, \\  & Now therefore, the \textless{}/English\textgreater{}\textless{}Maori\textgreater{}[PERSON] \textless{}/Maori\textgreater{}E\textless{}English\textgreater{} \\  & awaits the welcome of the Maori people. \\ \hline \end{tabular}
\end{table} 
Table 4: Sample commands and output using applications implemented by the toolkit. than the static approach used in Line 5 despite the dynamic model being empty at the start. This is a typical result for PPM models and reflects the effectiveness of these models at adapting quickly to the specific nature of the text. Also, the earlier language in the source document (British English in this case from the LOB corpus) is often a better predictor of subsequent text in the same document rather than using some other source to prime the encoding model in advance (American English from the Brown corpus). Line 7 provides an example of the use of the **codelength** tool. This application computes the compression codelength without the need for performing any physical coding which is much slower since it requires performing I/O operations. In the example, a list of models trained on Bibles in various languages (English, French, German, Italian, Latin, Maori and Spanish) is specified in the text file (models.txt). These models are loaded and then the compression cross-entropies for each of these models is calculated for the source text (the Brown corpus). The output clearly shows that the English Bible model is a better predictor of American English with a compression ratio of 3.514 bcc compared to the compression ratios produced by models for the other languages (all above 5.4 bcc). The command on Line 8 uses the **segment** tool to perform word segmentation using the Viterbi algorithm on a sample text (woods_unsegm.txt) which contains the text for [PERSON]'s poem _Stopping by Woods on a Snowy Evening_ with spaces removed. The Brown corpus model is used as the training model. The output text file produced by the tool is then compared to the correct text using the diff command on Line 9.