Patent ID: 8099415

Claim:
A method of identifying semantically duplicate job listings that describe a same job, where the duplicate job listings are retrieved from a plurality of online sources, the method comprising the machine-implemented steps of: preprocessing each job listing, comprising: automatically parsing each retrieved job listing to thereby identify in the parsed job listing, a corresponding job title, a corresponding employer name and a corresponding job location; automatically normalizing the identified employer name and job location to canonical forms; automatically enriching at least one of the normalized employer name and normalized job location by adding additional data regarding the normalized data item; and automatically scrubbing descriptive body text of each retrieved and parsed job listing by eliminating rare words and stop words and expressions that do not begin with a letter; automatically determining sameness-indicating signatures for the respective job listings, where the signatures are shorter in length than the job listings they were derived from and where substantially identical signatures indicate likelihood of job listings that describe a same job; automatically testing the signatures for collisions and among the job listings that have signature collisions, identifying job listings as duplicates of each other if they share at least a predetermined amount of identical text; and automatically segregating job listings that have been identified as duplicates so that one serves as a unique master job listing for storage in a production database and for being searched through by job seeking queries of job seekers.