Patent ID: 8341159

Claim:
A method comprising generating sets of features representing a plurality of categories from a taxonomy of categories, said generating comprising steps of: using an automated categorizer for: selecting a set of categories which are minimally overlapping with one another and conform to cultural norms about how information should be organized and what is important; selecting a set of training documents for each particular category from among the selected set of categories, wherein said set of training documents both typify documents for said particular category and are only representative of said particular category; wherein the step of selecting the set of the training documents comprises implementing a first deletion criterion by winnowing the training documents by eliminating, for each particular category, those training documents that are statistical outliers when compared to other training documents in said particular category; wherein winnowing comprises: calculating a measure of central tendency for the training documents in the particular category; computing a deviation measure for each training document in the particular category from the measure of central tendency, wherein a lower deviation measure indicates a higher similarity; and discarding each training document which has a deviation measure higher than a specified deviation measure from the measure of central tendency, thus producing a reduced set of training documents; iteratively performing: calculating a pseudo-centroid as the measure of central tendency for the reduced set by using vectors from all of the training documents in said reduced set; calculating a cosine between each of the documents in the category and the pseudo-centroid; determining if the document with a lowest cosine from the pseudo-centroid has a lower cosine than a pre-set threshold, wherein a lower cosine has a higher deviation measure from the pseudo-centroid; discarding the document with the lowest cosine from the pseudo-centroid when it is determined to have a lower cosine than the pre-set threshold; and stopping the iteration when the document with the lowest cosine from the pseudo-centroid does not have a lower cosine than the pre-set threshold; implementing a second deletion criterion by selecting documents that are highly similar to one another and located on a same server; discarding from among those documents on the same server the document with the lowest cosine to the pseudo-centroid; determining a list of differentiating features for the reduced set of the training documents; determining from the list of differentiating features the features that occur more frequently than a predetermined upper frequency threshold in a plurality of the categories; and deleting the features that occur more frequently than the upper frequency threshold in the plurality of the categories.