Unfortunately, this ideal behavior does not seem to occur. It turns out that it is quite difficult to identify real words, many of which are not likely to be found in a simple English dictionary, and to distinguish them from gibberish. For one thing, documents are written in other human languages. In files containing data other than text, semi-binary encodings like base64 and uuencoding, and even binary encodings, accidentally produce spurious words. TextSplitter uses a few heuristics to eliminate quite a bit of this gibberish, but an improved splitter class would probably bring indexing much closer to the desired asymptotic behavior.
Suggestions are welcome.
This column has only scratched the surface of the indexer module and the broader topic of full-text indexing. As the module improves with time, and as readers and users contribute suggestions, later columns will revisit the module and more of the theory behind it. Another Python indexing tool, Ransacker , has a somewhat different focus. Perlfect Search is a versatile search engine written in Perl. About the author No one, David Mertz supposes, could wish this column were longer.