# A Compression-Based Toolkit for Modelling and Processing Natural Language Text
## 2 The Toolkit's Design
Figure 1: A diagrammatic overview of the Tawa Toolkit and its design. Following the workflow in the figure, the sender first starts from a known source text and transforms it in some way (as shown by the dark gray box labelled 'Transform') into the target text. This is the text that is the desired output for some application (such as compression, text mining, spelling correction, machine translation and so on). In order to illustrate the variety of possible transformations which includes but goes beyond those traditionally covered by the noisy channel model, some example transformations are listed below (The first two examples below are transformations possible using a related tag insertion approach [12] that was similar to what was implemented in an earlier version of the toolkit. However, the subsequent examples above go beyond the tag insertion approach and are used to illustrate what is possible in the updated toolkit. ):  
* inserting markup tags into the text (called tag insertion), such as to indicate where different languages are in the text for language segmentation [7; 13];
* inserting extra characters, such as spaces for word segmentation [9];
* changing the alphabet, such as replacing common character bigraphs or word bigrams with new symbols to aid compression [6; 14];
* transforming the text into a grammar, such as for grammar-based compression [15];
* encoding extra information such as POS tags for POS tagging and tag-based compression [6; 16];
* switching between models used to encode each symbol in the text [7; 17]. Figure 2: Two architectures for NLP: **(a)** the standard decoding-based ‘Noisy Channel Model’ versus **(b)** the encoding-based ‘Noiseless Channel Model’ adopted by the Tawa Toolkit. The transformations are done in such a way that they are _reversible_ (as shown by the line labelled 'Reverse transform' in the figure). If tags or spaces are added, for example, then these can simply be removed to recover the original source text without any loss of information. If the alphabet is altered, then there is a one-to-one mapping between the old and new alphabets so that the original source text is easily recoverable. This property of reversibility ensures that information is communicated losslessly. The target text is then encoded to produce the compressed target text (as depicted by the dark gray box labelled 'Encode'). A search process shown in the figure by the rightmost arrow is used to find the most compressed text by repeatedly trying out different variations of the transformation (for example, by inserting or not inserting spaces after each character if performing word segmentation). This search can be thought of as the transformation & encoding phase whose purpose is to generate a suitable target text. This search may also require extra transformations in order to ensure that the encoding is fully reversible (for example, reversible word segmentation requires replacing any existing spaces with another unique character so that the original text can be unambiguously recovered). The remaining parts of the workflow shown in the figure is the verification & decoding phase. This step is not necessarily needed depending on the application since just producing the target text will usually be sufficient in order to complete the task(s) required. However, as argued below, verification is of fundamental importance in ensuring the output is correct. This is done by sending the compressed target text down a noise-free communication channel, and then checking that the decoded target text matches the original target text. [PERSON] et al. [4] pointed out that the noiseless channel model could be viewed simply as \"the noisy channel model in another guise\". This is because the search processes (e.g., the Viterbi algorithm [18]) produce a similar result with the best target text being found in both cases. However, as [PERSON] et al. have argued, there are important differences.