Patent ID: 9449114
Filing Date: 2016-09-20
CPC Classification: G06F

Claim Text:
1. A method comprising: accessing a web page, the web page represented by a hierarchical mark-up language; parsing the web page into a tree structure based on a hierarchy associated with the web page; for each node from the tree structure, determining a text density score value, the text density score value calculated as a sum of a word count associated with a node and weighted word counts associated with adjacent nodes from the tree structure, an adjacent node from the adjacent nodes comprising a sibling of the node; identifying those nodes from the tree structure for which a text density value is above the threshold value as content nodes; extracting text from the content nodes associated with the web page and ignoring nodes that were not identified as content nodes; breaking up the extracted text and further text into a plurality of sentences; hashing each sentence from the plurality of sentences; calculating frequency of each sentence in the extracted text and the further text; and identifying sentences from one or more sentences having a frequency value above a frequency threshold value as indicative of boilerplate language.