Furthermore, the use of colloquial language may lead to poor clustering with some algorithms but not others, so you may need to identify clustering methods for individual data sources. In addition, a process called feature extraction will identify statistically significant vocabulary items such as words or phrases that provide the basis for clustering documents. In the former, the location of uncommon words within documents is tracked. Alternatively, you can use a predefined list of commonly used words to identify low-information carrying words. This approach works best with general collections, while generated lists are useful for domain-specific document collections.
Thematic indexing provides a more generalized version of keyword indexing.
As in keyword indexing, you can make the warehouse predefine or generate the reference taxonomy based on the document collection. Generating a taxonomy requires a training phase in which a training set of documents and their assigned-to categories are presented to a tool, such as the IBM Intelligent Miner for Text categorization tool, which then performs linguistic analysis. Just as numeric data warehouses rely on aggregations to provide high-level assessments, document data warehouses require summaries of documents to provide core information from the collection without unnecessary detail. The size of these summaries depends on the size of the original document and the amount of information that document contains. The larger the summary, the greater the information content, but a practical upper limit on size can exist.