# A Compression-Based Toolkit for Modelling and Processing Natural Language Text
## 4 Sample Applications Provided by the Toolkit
### Verifying Sentence Alignment of a Parallel Corpus  
The method VerifyParallelFiles in Algorithm 7 checks the quality of sentence alignment between two files (_File1_ and _File2_) in a parallel corpus. It assumes that the corpus has already been aligned and the purpose of this method is to verify its quality prior to being used to build language models for statistical machine translation. The method also assumes that the sentences in the two different languages are on corresponding separate lines in the two files. Therefore, the TXT_readline_file (line 5) is being used to read each line of the two files simultaneously into two text objects _Text1_ and _Text2_.  
The method CodelengthText from Algorithm 4 is used to calculate the compression codelengths of the two sentences in _Text1_ and _Text2_ according to the models _Model1_ and _Model2_ respectively. In order for this method to be effective, it requires the two models to be separately trained on a large amount of text that is representative of the respective languages. Recent research [36; 37] has shown that comparing compression codelength in this manner is an effective way for verifying that corresponding sentences are aligned, as well as measuring the quality of translation.  
The method specifically computes the codelength ratio (\(CR\)) measure on lines 9 to 12 which is the maximum of the two ratios \(\mathit{Codelength1}/\mathit{Codelength2}\) and \(\mathit{Codelength2}/\mathit{Codelength1}\). This ensures that \(CR\) is always greater than or equal to 1.0. A value of 1.0 indicates that the two codelengths are the same value and values close to or equal to 1.0 provide strong evidence that the sentences are probably aligned. This is based on the assumption that the information (from an information theoretical sense) between corresponding sentences that are co-translations should be the same or similar, and compression codelength provides an excellent means for measuring the information [37].  
```
1 methodVerifyParallelFiles (File1,File2, Model1, Model2, \(\mathit{CRTThreshold}\))
2 Text1\(\leftarrow\)TXT_create_text ();
3 Text2\(\leftarrow\)TXT_create_text ();
4 LineNo\(\leftarrow\) 0;
5 whileTXT_readline_file (File1, Text1)andTXT_readline_text (File2, Text2)do
6 LineNo\(\leftarrow\)LineNo\(+\) 1;
7 Codelength1\(\leftarrow\)CodelengthText (Text1,Model1);
8 Codelength2\(\leftarrow\)CodelengthText (Text2,Model2);
9 ifCodelength1\(>\)Codelength1 then
10 CR\(\leftarrow\)Codelength1/Codelength2
11 else
12 CR\(\leftarrow\)Codelength2/Codelength1
13 ifCR\(>\)CRTthresholdthen
14 Output \"Check alignment for Line\"LineNo
15
16
17 XT_release_text (\(\mathit{Text1}\));
18 TXT_release_text (\(\mathit{Text2}\));
19
20 end if
```  
**Algorithm 7**A compression-based method for verifying parallel sentence alignment  
The method on line 13 compares \(CR\) to a threshold \(\mathit{CRTThreshold}\) which is passed as an argument. If \(CR\) exceeds this threshold, then a message is output with the line number of the suspicious sentence (line 14). Values of 1.8 and above for \(\mathit{CRTThreshold}\) have been found effective at locating unsatisfactory translations in experiments with Chinese/English [37] and Arabic/English [36] parallel corpora. These experiments also show that codelength-based measurements are more effective than sentence length-based measurements for verifying sentence alignment.