Patent ID: 6505151
Filing Date: 2003-01-07
Classification: G06F

Abstract:
A method for automating the process of dividing sentences into phrases, comprising the following steps:(a) dividing a sentence into sub-sentences using statistical analysis, FE&it;(Ci)=-&Sum;Cj&it;PF&it;(Cj|Ci)&it;â€ƒ&it;log&it;â€ƒ&it;PF&it;(Cj|Ci)â€ƒincluding the following substeps: (a.1) for each pair of adjacent words in the sentence calculating a metric which represents a strength of disconnection between the adjacent words, and (a.2) breaking the sentence into sub-sentences at locations in the sentence where the metric exceeds a first threshold; and, (b) dividing the sub-sentences into phrases, using statistical analysis; wherein in substep (a.1), the metric is a cutability measure that is calculated as a sum of backward entropy, forward entropy and mutual information; and, wherein in substep (a.1) forward entropy (FE) of a character CI, which immediately proceeds a character Cj in a sentence is calculated using the following equation: where PF(Cj|Ci) is the probability of Cj following Cj.