# A Compression-Based Toolkit for Modelling and Processing Natural Language Text
###### Abstract  
A novel compression-based toolkit for modelling and processing natural language text is described. The design of the toolkit adopts an encoding perspective--applications are considered to be problems in searching for the best encoding of different transformations of the source text into the target text. This paper describes a two phase 'noiseless channel model' architecture that underpins the toolkit which models the text processing as a lossless communication down a noise-free channel. The transformation and encoding that is performed in the first phase must be both lossless and reversible. The role of the verification and decoding second phase is to verify the correctness of the communication of the target text that is produced by the application. This paper argues that this encoding approach has several advantages over the decoding approach of the standard noisy channel model. The concepts abstracted by the toolkit's design are explained together with details of the library calls. The pseudo-code for a number of algorithms is also described for the applications that the toolkit implements including encoding, decoding, classification, training (model building), parallel sentence alignment, word segmentation and language segmentation. Some experimental results, implementation details, memory usage and execution speeds are also discussed for these applications. text compression; text processing; encoding; decoding +
Footnote †: journal: Information  
described in this paper (Tawa is a common broadleaf tree found in New Zealand. The acronym stands for **T**ext **A**nalyser from **W**aikato since it was first developed at Waikato University, and subsequently improved over two decades at Lund, Robert Gordon and Bangor Universities). The toolkit is now available for download ([[https://bangoroffice365-my.sharepoint.com/:f:/g/personal/eesa0e_bangor_ac_uk/Ej-84](https://bangoroffice365-my.sharepoint.com/:f:/g/personal/eesa0e_bangor_ac_uk/Ej-84) IJuUNOuSVz44 daZCIBt_4 NQyanF_5 qH8 FLSJalQ?e=rFNW3P]([https://bangoroffice365-my.sharepoint.com/:f:/g/personal/eesa0e_bangor_ac_uk/Ej-84](https://bangoroffice365-my.sharepoint.com/:f:/g/personal/eesa0e_bangor_ac_uk/Ej-84) IJuUNOuSVz44 daZCIBt_4 NQyanF_5 qH8 FLSJalQ?e=rFNW3P)) and is a comprehensive extension of the original Cleary & Teahan API and an updated version of the TMT toolkit [4] that was based on the API. The toolkit provides a powerful method for classifying, mining and transforming text. The purpose of this paper is to describe the design and implementation behind the toolkit including its architecture, libraries, and sample applications. The contributions of this paper are as follows:  
* The compression-based architecture which underpins the toolkit, and how it differs from the standard architecture for NLP, the noisy channel model, is fully described for the first time. This paper explains the encoding approach taken by the toolkit's architecture as opposed to the decoding approach of the noisy channel model and how it provides a wider range of processing possibilities. * This paper is the first to discuss in detail the toolkit's libraries, including the methods and objects used. The toolkit provides an extensive update of previous versions including an overhaul of the transformation libraries that simplifies and consolidates the processes previously used. * The algorithms behind each of the toolkit's main applications are also described for the first time. This paper is organised as follows. In the next section, we detail the design of the toolkit and its underlying architecture. The toolkit's libraries are detailed in Section 3 and sample methods discussed in Section 3.1. The pseudo-code for nine applications built using the libraries that demonstrate how they work are detailed in Section 4. This section also discusses sample use, implementation details, memory usage, execution speeds and search algorithms used.