Any particular document detection system uses own set of modules which may vary slightly from the ones described below. A corpus is comprised of the source documents from which the user will select the document sub-set. The content of the corpus may have significant the performance in some applications. Its documents may be part of a vary narrowly defined subject domain, or may pertain to a broad range of concepts covering several subject domains. Thus, different detection systems may have varying performance, depending upon the content of the corpus and the Detection Need.
Most systems use some method of stemming.
Stemming is the reduction of a word to its root. Most systems also use a list of Stop words. A Stop word is a word which is usually ignored, e.g., 'a', 'an', 'the'. Stop words lack significance to the determination of the subject of a document at the rather general level at which document detection works. Research is progressing at identifying phrases and multi-term items such as dates and personal names so that these can be indexed as single terms. This function is frequently very system specific because it is a key place in the detection system for optimizing run-time performance.