# A Compression-Based Toolkit for Modelling and Processing Natural Language Text
## 4 Sample Applications Provided by the Toolkit
### Applications Implemented by the Toolkit  
The algorithms detailed above have all been implemented using the Tawa toolkit's libraries in the C programming language. Table 3 summarises these applications that come bundled with the toolkit. The table lists for each application its name (in bold font), a short description (in bold and italic fonts), the algorithms detailed above they are based on and selected arguments. The **encode**, **decode** and **train** applications for PPM compression, decompression and model building, for example, use similar arguments which specify amongst other things the argument size, order of the model, and whether the model performs full exclusions and update exclusions. The **train** application has additional arguments for specifying the title of the model, and whether the model should be written out as a static model. The **markup** and **segment** applications use arguments for specifying whether Viterbi search should be adopted, and the stack depth if the stack algorithm should be used instead. These application arguments provide a wide range of possible settings that have proven to be very effective in many NLP experiments as summarised in Section 5. Many further possible uses still exist for these applications which have the advantage that they can be used \"off-the-shelf\" without the need to perform any further development using the toolkit's libraries. Some examples of shell commands and output using these applications are shown in Table 4. The purpose of including these here is to demonstrate typical use and also some of the possibilities offered by the commands. Line 1 provides an example of typical use for the **encode** application. In this case the raw text from the million-word Brown corpus of American English [39] in the file Brown.txt is being encoded (compressed) to the file Brown.encoded using the order 5 PPMD algorithm (which is the default) with an alphabet of 256. Some summary data is output when the program completes such as the number of input bytes, the number of output bytes and the compression ratio of 2.152 bpc (bits per character). Line 2 provides the command that can be used to decode (decompress) the encoded file, and the diff command on Line 3 confirms that the decoded file is the same as the original file since no differences are produced as output. Line 4 provides an example of how the **train** command can be used to build a model. In this case, the Brown corpus is being used to train a static order 5 PPMD model which is written out to the file Brown.model when the command completes. Line 5 loads the model directly into memory (without the need for processing the original training text which is much slower), and then uses it to encode another text, the raw text from the million-word LOB corpus of British English [40] in the file LOB.txt with a compression ratio of 2.191 bpc for the encoded output file LOB.encoded. Line 6 encodes the LOB corpus directly without loading a model, therefore will use a dynamic model that adapts to the text being encoded (i.e., at the beginning, the model is empty but is updated as the text is processed sequentially). The compression ratio of 2.131 for this adaptive approach is slightly better