Patent ID: 8073865

Claim:
A computer-implemented method for extracting content from unstructured sources, the method comprising the steps of: a) crawling web pages of a website to extract product data and one or more images associated with the product data, and storing the product data and the images on an electronic storage medium; b) parsing the stored product data to generate a standard document object model (SDOM); c) converting the SDOM into a modified document object model (MDOM) by adding structural nodes that contain web page location data and content nodes that include contiguous segments of web page text data; d) pruning nodes by removing certain nodes from the MDOM by determining leaf nodes that are not associated with a child node in the MDOM and removing those leaf nodes, calculating frequency of occurrence across the web pages of the leaf nodes, and cleaning the MDOM to construct an aggregated document object model (ADOM) by identifying duplicate phrases across web pages and removing the duplicate phrases; and cleaning the MDOM by removing certain web page data to construct an aggregated document object model (ADOM) by identifying duplicate phrases across web pages and removing the duplicate phrases; e) analyzing the attributes of the images and providing each of said images with an aggregate image score; f) determining product entities that include properties that define a product, providing each of said product entities with an aggregate quality score, and extracting the product entities from the ADOM; g) generating a product list that includes the product entities selected from the ADOM with an aggregate quality score above a pre-determined threshold; h) detecting duplicate product data in the product list and post-processing the product list to eliminate the duplicate product data; i) performing site link analysis of the product list to determine a location of the product data on a website; and j) verifying the accuracy of the product data in the product list by comparing the product data to the web pages.