# A Compression-Based Toolkit for Modelling and Processing Natural Language Text
## 4 Sample Applications Provided by the Toolkit
### Context-Based Encoding and Decoding  
Algorithms 2 and 3 provides pseudo-code for context-based encoding and decoding of a file respectively. They assume that a _Model_ has already been created or loaded and a _Coder_ has been created prior to these methods being called with these being passed as arguments. The encoder requires the input file (\(SourceFile\)) as an argument, and the decoder requires the compressed file (\(CompressedFile\)) as an argument.  
```
1 methodEncodeFile(\(\mathit{SourceFile}\), \(\mathit{Model}\), \(\mathit{Coder}\))
2\(\mathit{Context}\leftarrow\) TLM_create_context(\(\mathit{Model}\));
3 foreachSymbolinSourceFiledo
4 TLM_encode_symbol(\(\mathit{Model}\), \(\mathit{Context}\), \(\mathit{Symbol}\), \(\mathit{Coder}\))
5
6 TLM_encode_symbol(\(\mathit{Model}\), \(\mathit{Context}\), \(\mathit{Coder}\), \(\mathit{TXT_sentinel\_symbol}\));
7 TLM_release_context(\(\mathit{Context}\));
8
9 end
```  
**Algorithm 2**A context-based method for encoding  
Both methods require that a \(\mathit{Context}\) variable be created first using the TLM_create_context method (lines 2). Then each symbol is processed sequentially, either encoded using the TLM_encode_symbol method (line 4, Algorithm 2) or decoded using the TLM_decode_symbol method (line 3, Algorithm 3). (These methods also update the \(\mathit{Context}\) at the same time for the current symbol. The \(\mathit{Context}\) is released just before the end of both methods).  
Both methods process symbols sequentially which are specified using the variable \(\mathit{Symbol}\). The way a file is preprocessed (during encoding) and postprocessed (during decoding) is user-defined. Symbols for English text are usually mapped directly to the ASCII byte values of the characters in the file. For Unicode (e.g., UTF-8) encoded files, on the other hand, a different mapping is required. Further preprocessing/postprocessing of the symbol stream may also be performed such as replacement of common bigraphs [6; 14; 15] (also see Section 5 below) as this can often lead to improved compression. However, these steps are not detailed in Algorithms 2 and 3.  
The special sentinel symbol (returned by \(\mathtt{TXT\_sentinel\_symbol}\)) has been used to signify the end of the encoding stream and is encoded once all the other symbols in the file have been encoded (line 6, Algorithm 2). It is a unique character which effectively expands a model's alphabet by one that is encoded at the end of an encoding stream and forces the model to back off to the default order -1 context. The decoder simply decodes symbols until the sentinel symbol is decoded (line 3, Algorithm 3). An alternative way of encoding the end of the stream which the toolkit implements is to encode the length of the stream at the beginning using (for example) a start-stop-step encoder [28]. However, this requires a first pass through the stream being encoded to find its length, and using a unique extra symbol such as \(\mathtt{TXT\_sentinel\_symbol}\) does not incur much overhead except for the cost of the escapes down to the order \(-1\) context and the cost of encoding this symbol in the lowest order context which is ameliorated by PPM's full exclusions mechanism.  
```
1 methodDecodeFile(\(\mathit{CompressedFile}\), \(\mathit{Model}\), \(\mathit{Coder}\))
2\(\mathit{Context}\leftarrow\) TLM_create_context(\(\mathit{Model}\));
3 while\(\mathit{Symbol}\leftarrow\) TLM_decode_symbol(\(\mathit{Model}\), \(\mathit{Context}\), \(\mathit{Coder}\))!= TXT_sentinel_symbol(\(\mathit{Model}\))do
4 Output Symbol
5 end
```  
**Algorithm 3**A context-based method for decoding