Patent ID: 7017113

Claim:
A software program comprising instructions, stored on computer-readable media, wherein said instructions, when executed by a computer, perform the necessary steps for removing redundant information from digital documents, comprising: organizing text into sentences and paragraphs; analyzing said sentences and said paragraphs; comparing said sentences and paragraphs with other documents; and identifying redundancies between said documents; wherein said step of analyzing further comprises the steps of: extracting statistical features selected from the group consisting of: size of a paragraph in characters; character histograms; number of words in each sentence; word histograms; starting word of each sentence; and ending word of a paragraph; determining whether similar said statistical features exist; IF similar statistical features exist, THEN deciding paragraphs are similar, removing redundant paragraph, and proceeding to said step of comparing said sentences and paragraphs with other documents OTHERWISE, postponing removal of paragraph; analyzing corresponding image and data parts of said paragraph; determining whether said paragraphs are placed in a different order; IF said paragraphs are placed in a different order, THEN analyzing the starting word of each sentence, analyzing the length of each said sentence; and proceeding to said step of comparing said sentences and paragraphs with other documents OTHERWISE, proceeding to said step of comparing said sentences and paragraphs with other documents.