Source: http://www.google.com/patents/US20080306943?dq=5,832,511
Timestamp: 2014-12-25 23:11:34
Document Index: 93681497

Matched Legal Cases: ['Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No. 10', 'Application No. 10']

Patent US20080306943 - Phrase-based detection of duplicate documents in an information retrieval system - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inAdvanced Patent SearchPatentsAn information retrieval system uses phrases to index, retrieve, organize and describe documents. Phrases are identified that predict the presence of other phrases in documents. Documents are the indexed according to their included phrases. Related phrases and phrase extensions are also identified. Phrases...http://www.google.com/patents/US20080306943?utm_source=gb-gplus-sharePatent US20080306943 - Phrase-based detection of duplicate documents in an information retrieval systemAdvanced Patent SearchPublication numberUS20080306943 A1Publication typeApplicationApplication numberUS 10/900,012Publication dateDec 11, 2008Filing dateJul 26, 2004Priority dateJul 26, 2004Also published asUS7711679, US8108412, US8489628, US20100161625, US20120310902, US20140156647Publication number10900012, 900012, US 2008/0306943 A1, US 2008/306943 A1, US 20080306943 A1, US 20080306943A1, US 2008306943 A1, US 2008306943A1, US-A1-20080306943, US-A1-2008306943, US2008/0306943A1, US2008/306943A1, US20080306943 A1, US20080306943A1, US2008306943 A1, US2008306943A1InventorsAnna Lynn PattersonOriginal AssigneeAnna Lynn PattersonExport CitationBiBTeX, EndNote, RefManReferenced by (10), Classifications (7), Legal Events (2) External Links: USPTO, USPTO Assignment, EspacenetPhrase-based detection of duplicate documents in an information retrieval systemUS 20080306943 A1Abstract An information retrieval system uses phrases to index, retrieve, organize and describe documents. Phrases are identified that predict the presence of other phrases in documents. Documents are the indexed according to their included phrases. Related phrases and phrase extensions are also identified. Phrases in a query are identified and used to retrieve and rank documents. Phrases are also used to cluster documents in the search results, create document descriptions, and eliminate duplicate documents from the search results, and from the index.
3. A method of detecting a duplicate document, the method comprising:
selecting a first document and a second document from a set of documents; comparing a document description of the first document with a document description of the second document, wherein the document description of each document comprises selected sentences of the document that are ordered in the document description as a function of a number of phrases in each sentence; and responsive to the document description of the first document matching the document description of the second document, discarding at least one of the first document or the second document from the set of documents. 4. The method of claim 3, further comprising:
receiving a query comprising at least one phrase; retrieving a plurality of documents responsive to the query to form the set of documents as a search result set including the first document and the second document; and wherein discarding at least one of the first document or the second document comprises discarding at least one of the first document or the second document from the search result set. 5. The method of claim 3, wherein comparing the document description of the first document with a document description of the second document further comprises:
for each of the first and second documents, generating a document description by selecting sentences of the document, and ordering in the document description as a function of a number of phrases in each sentence. 6. The method of claim 3, wherein comparing the document description of the first document with a document description of the second document further comprises:
retrieving for each of the first document and the second document a stored document description comprising selected sentences of the document, wherein the selected sentences are ordered in the document description as a function of a number of phrases in each sentence. 7. The method of claim 3, wherein comparing the document description of the first document with a document description of the second document further comprises:
generating for the first document a document description by selecting sentences of the first document, and ordering in the document description as a function of a number of phrases in each sentence; and retrieving for the second document a stored document description comprising selected sentences of the second document, wherein the selected sentences are ordered in the document description as a function of a number of phrases in each sentence. 8. The method of claim 3, wherein selecting a first document and a second document from a set of documents further comprises:
selecting the first document and second document during indexing of the first document. 9. The method of claim 3, wherein the phrases as a function of which the sentences of the first document description are ordered are phrases related to the first document, and the phrases as a function of which the sentences of the second document description are ordered are phrases related to the second document.
10. The method of claim 3, wherein a document description is stored in association with the document to which the document description corresponds.
12. The method of claim 3, wherein the document description of the first document matches the document description of the second document when a hash value of the first document description equals a hash value of the second document description.
13. The method of claim 3, wherein the document discarded has a lower document significance measure.
14. The method of claim 13, wherein the document significance measure is page rank.
15. The method of claim 13, wherein discarding at least one of the first document or the second document from the set of documents comprises removing the first document or the second document from an index.
16. A tangible computer readable storage medium storing a computer program executable by a processor for detecting a duplicate document, the operations of the computer program comprising:
selecting a first document and a second document from a set of documents; comparing a document description of the first document with a document description of the second document, wherein the document description of each document comprises selected sentences of the document that are ordered in the document description as a function of a number of phrases in each sentence; and responsive to the document description of the first document matching the document description of the second document, discarding at least one of the first document or the second document from the set of documents. 17. A system for detecting a duplicate document, comprising:
a document description system, executed by a processor, and configured to associate a set of documents with a set of corresponding document descriptions and store the associations in a memory, wherein the corresponding document description of each document comprises selected sentences of the document that are ordered in the document description as a function of a number of phrases in each sentence; and a duplicate detection system, executed by a processor and configured to:
responsive to the document description corresponding to the first document matching the document description corresponding to the second document, disassociate at least one of the first document or the second document from the set of documents. Description
Phrase Identification in an Information Retrieval System, Application No. 10/______, filed on Jul. 26, 2004;
Phrase-Based Indexing in an Information Retrieval System, Application No. 10/______, filed on Jul. 26, 2004;
Phrase-Based Searching in an Information Retrieval System, Application No. 10/______, filed on Jul. 26, 2004;
Phrase-Based Personalization of Searches in an Information Retrieval System, Application No. 10/______, filed on Jul. 26, 2004;
Automatic Taxonomy Generation in Search Results Using Phrases, Application No. 10/______, filed on Jul. 26, 2004; and
Phrase-Based Generation of Document Descriptions, Application No. 10/______, filed on Jul. 26, 2004; all of which are co-owned, and incorporated by reference herein.
The present invention has further embodiments in system and software architectures, computer program products and computer implemented methods, and computer. generated user interfaces and presentations.
If the candidate phrase is in the good phrase list 208, as entry gj, then the index 150 entry for phrase gj is updated to include the document (e.g., its URL or other document identifier), to indicate that this candidate phrase gj appears in the current document. An entry in the index 150 for a phrase gj (or a term) is referred to as the posting list of the phrase gj. The posting list include sa list of documents d (by their document identifiers, e.g. a document number, or alternatively a URL) in which the phrase occurs.
Referring to the example of FIG. 3, assume that the �stock dogs� is on the good phrase list 208, as well as the phrases �Australian Shepherd� and �Australian Shepard Club of America�. Both of these latter phrases appear within the secondary window 304 around the current phrase �stock dogs�. However, the phrase �Australian Shepherd Club of America� appears as anchor text for a hyperlink (indicated by the underline) to website. Thus the raw co-occurrence count for the pair (�stock dogs�, �Australian Shepherd�) is incremented, and the raw occurrence count and the disjunctive interesting count for {�stock dogs�, �Australian Shepherd Club of America�} are both incremented because the latter appears as distinguished text.
i) compute the expected value E(gk). The expected co-occurrence rate E(j,k) of gj and gk, if they were unrelated phrases is then E(gj)*E(gk); ii) compute the actual co-occurrence rate A(j,k) of gj and gk. This is the raw co-occurrence count R(j, k) divided by T, the total number of documents;
Starting with the highest phrase number as the first candidate phrase, the search system 120 determines if there is another candidate phrase within a fixed numerical distance within the sorted list, i.e., the difference between the phrase numbers is within a threshold amount, e.g. 20,000. If so, then the phrase that is leftmost in the query is selected as a valid query phrase Qp. This query phrase and all of its sub-Case phrases is removed from the list of candidates, and the list is resorted and the process repeated. The result of this process is a set of valid query phrases Qp.
Referenced byCiting PatentFiling datePublication dateApplicantTitleUS7567959Jan 25, 2005Jul 28, 2009Google Inc.Multiple index based information retrieval systemUS7676463 *Nov 15, 2005Mar 9, 2010Kroll Ontrack, Inc.Information exploration systems and methodUS7698317 *Apr 20, 2007Apr 13, 2010Yahoo! Inc.Techniques for detecting duplicate web pagesUS7962462 *May 31, 2005Jun 14, 2011Google Inc.Deriving and using document and site quality signals from search query streamsUS8037053 *Oct 31, 2008Oct 11, 2011Yahoo! Inc.System and method for generating an online summary of a collection of documentsUS8606627 *Jun 12, 2008Dec 10, 2013Microsoft CorporationSponsored search data structureUS8655648 *Sep 1, 2010Feb 18, 2014Microsoft CorporationIdentifying topically-related phrases in a browsing sequenceUS8818982Apr 25, 2012Aug 26, 2014Google Inc.Deriving and using document and site quality signals from search query streamsUS20090313115 *Jun 12, 2008Dec 17, 2009Microsoft CorporationSponsored search data structureUS20100228743 *Mar 3, 2009Sep 9, 2010Microsoft CorporationDomain-based ranking in document search* Cited by examinerClassifications U.S. Classification1/1, 707/999.006International ClassificationG06F7/00Cooperative ClassificationG06Q10/10, G06F17/30864European ClassificationG06Q10/10, G06F17/30W1Legal EventsDateCodeEventDescriptionNov 4, 2013FPAYFee paymentYear of fee payment: 4Dec 6, 2004ASAssignmentOwner name: GOOGLE, INC., CALIFORNIAFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PATTERSON, ANNA LYNN;REEL/FRAME:015429/0638Effective date: 20041104Owner name: GOOGLE, INC.,CALIFORNIAFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PATTERSON, ANNA LYNN;US-ASSIGNMENT DATABASE UPDATED:20100504;REEL/FRAME:15429/638RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services©2012 Google