When a corpus is represented as a frequency list, much information is lost, but the tradeoff is an object that is susceptible to statistical processing. Word frequency lists are easy to generate, so measuring corpus similarity based on them will be viable in many circumstances where a more extensive analysis of the two corpora is not possible. A judgement of similarity runs the risk of meaninglessness if a homogeneous corpus is compared with a heterogeneous one. We propose a method which can be used initially to measure corpus homogeneity, and subsequently to measure the similarity between two corpora. 46or the corpus-similarity case, the only modifications are that one subcorpus is taken from the first corpus and the other from the second, and the similarity value is interpreted by reference to the homogeneity measure for each corpus.
High scores correspond to heterogeneous corpora and dissimilar corpora.
The last two lines in the table point to the differences between general corpora and specific corpora. General corpora which embrace a number of language varieties will score highly for heterogeneity. Corpus similarity between such corpora will depend on whether all the same language varieties are represented in each corpus, and in what proportions. Low heterogeneity scores will typically apply to corpora of a single language variety, so here, similarity scores will be interpreted as a measure of the distance between the two language varieties. The c2 test.