Patent ID: 7395256

Claim:
A method for statistically extracting terms from a set of documents comprising the steps of: determining for each document in the set of documents an importance vector based on importance values for words in each document; forming a binary document classification tree by clustering the documents into clusters of similar documents based on the importance vector for each of the documents; building an infrastructure for the set of documents by generalizing the binary document classification tree; analyzing distribution of the importance vectors to determine a cutting threshold; cutting the generalized tree of the infrastructure according to the cutting threshold to further cluster the documents; extracting statistically significant individual key words from the further clusters of similar documents; and extracting terms using the key words as seeds, wherein the cutting threshold is defined by S 1 *, that satisfies the equation: DEF_SCORE ⁢ ( S 1 * , S - S 1 * ) = MAX S 1 ⁢ in ⁢ ⁢ S ⁡ ( DEF_SCORE ⁢ ( S 1 , S - S 1 ) ) where, S is the binary document classification tree, S 1 is a sub-tree of S with the same root as S, S-S 1 is a forest in S with S 1 removed, DEF_SCORE(S 1 , S-S 1 ) is the difference between average score of the nodes in S 1 and the nodes in S-S 1 and is defined by: ∑ d ⁢ ⁢ ⁢ in ⁢ ⁢ S 1 ⁢ ⁢ score ⁡ ( d )  S 1  - ∑ d ⁢ ⁢ in ⁢ ⁢ S - S 1 ⁢ ⁢ score ⁡ ( d )  S - S 1  where score(d) is score of the node d, |S 1 | is the number of nodes in S 1 , and |S-S 1 | is the number of the nodes in S-S 1 .