Source: https://patents.google.com/patent/US9367606
Timestamp: 2018-02-20 23:30:48
Document Index: 468758917

Matched Legal Cases: ['§119', 'Application No. 200680017270', 'Application No. 2008', 'Application No. 200680013556', 'Application No. 2007', 'Application No. 200', 'Application No. 200680017270', 'Application No. 2008']

US9367606B1 - Search engine that applies feedback from users to improve search results - Google Patents
US9367606B1
US9367606B1 US13474628 US201213474628A US9367606B1 US 9367606 B1 US9367606 B1 US 9367606B1 US 13474628 US13474628 US 13474628 US 201213474628 A US201213474628 A US 201213474628A US 9367606 B1 US9367606 B1 US 9367606B1
US13474628
SEARCH ENGINE TECHNOLOGIES, LLC
This patent application is a Continuation application which claims priority under 35 U.S.C. 120 of the U.S. patent application Ser. No. 11/378,756, filed Mar. 17, 2006 entitled “SEARCH ENGINE THAT APPLIES FEEDBACK FROM USERS TO IMPROVE SEARCH RESULTS” now U.S. Pat. No. 8,185,523 which, in turn, claims priority under 35 U.S.C. §119(e) of the U.S. provisional patent application Ser. No. 60/663,361, filed on Mar. 18, 2005, and entitled “Search Engine That Applies Feedback From Users to Improve Search Results”, both of which are hereby incorporated by reference in their entirety.
Embodiments of the present invention, unlike traditional search engines, make use of user feedback to provide more relevant information to users searching the Internet. For example, in accordance with the present invention, a first user performing a search is able to rate the results of that search. The first user can rate the results based on the relevance of the results to his search, the abundance of information contained in a specific Web page returned in a results list, or any other criteria. A second user performing a similar or related search is then able to view a results list based on or influenced by the rating of the first user, thereby displaying search results that are more likely most relevant to him. This second user is also able to rate search results. A community of users is thus able to provide feedback that helps users to receive search results that are as relevant to the information they are seeking as possible. Systems for and methods of using user feedback are described in U.S. patent application Ser. No. 11/364,617, titled “Methods of and Systems for Searching by Incorporating User-Entered Information,” and filed Feb. 27, 2006, issued as U.S. Pat. No. 9,092,523, on Jul. 28, 2015, which is incorporated by reference.
After observing the results of the query on the engine the user is allowed to give feedback on the document for the query in some ordinal form where the lowest rank (e.g. one star) means completely spurious and the highest (e.g. five stars) indicates perfect or near perfect relevance. The feedback is then logged as a new query-document feature (q, d, u)=s where s is the number of stars assigned by user u. At any given point this value is the user u's current rating for document d and query q. Importantly, this feature does not depend on how many times the user performs the query but only stores his or her later rating. By collecting all the user feedback for a given query-document pair, a new set of sparse indicator variables U(q, d)u,s=1 is obtained if user u assigned 5 stars for document d given query q: The feature set is then augmented with those in U and the complete model is parameterized with the vector θ. Prior to collecting user feedback the number of stars is unknown or clear indicating that the user has not evaluated the relevance of the document explicitly. This is important as datasets will be formed purely of the documents that have been rated by the user.
p(c|q)=p(q|c)p(c)/[ΣcεC p(q|c)p(c)]
using Baye's rule with C the set of user clusters. Then the relevance score p(R|X) would be equal to the weighted sum of each cluster membership probability times relevance score or probability for a given cluster membership pc (R|X)
p(R|X)=ΣcεC p(c|q)p c(R|X) Equation (2)
max 1(θ)=Σωi(y i log(μ(θ+ψu(i))i)+(1−y i)log(1−μ(θ+ψu(i))i) Equation (3)
μ(X;θ,γ c)=1/[1+e −(θ+γc+ψu)*X]
1. A method of presenting documents to a user, comprising:
a. assigning relevance scores to a plurality of the documents, by a computer programmed to compute relevance scores based on a relevance model using one or more user input events, wherein at least one of the one or more user input events is an active selection of a user input feature about at least one document and is received outside a context of a presentation of search results of a search, the one or more user input events comprising input about the at least one document from at least one user; and
b. presenting the documents to the user according to the relevance scores.
2. The method of claim 1, wherein presenting the documents is in response to a search query submitted by the user.
3. The method of claim 2, wherein the search query is submitted through a link that has one or more search terms embedded therein.
4. The method of claim 1, further comprising, after presenting the documents to the user, receiving user input about at least one of the documents presented to the user, the user input comprising a user input event.
5. The method of claim 4, further comprises storing the user input event in a database.
6. The method of claim 1, wherein the one or more user input events comprise one of: voting for a document, rating a document, tagging a document, bookmarking a document, saving a document, liking a document, setting a document as a favorite, expressing interest in a document, voting against a document, and blocking a document.
7. The method of claim 1, wherein presenting the documents comprises displaying user input information based on the input about at least one document by the at least one user.
8. The method of claim 7, wherein displaying user input information comprises displaying a count of user input events.
9. The method of claim 8, wherein the count of user input events comprises one of the number of votes “for” and “likes” associated with at least one document.
10. The method of claim 9, wherein displaying user input information comprises displaying information about the user associated with the user input event.
11. The method of claim 10, wherein displaying user input information comprises one of: displaying an image associated with the user and displaying a text description associated with the user.
12. The method of claim 7, wherein displaying user input information comprises displaying a term associated with the document.
13. The method of claim 1, wherein the one or more user input events comprise one of: the relevance of the document to a query, the bias of the user regarding the document, and the relevance of the document to a subject area.
14. The method of claim 13, wherein the one or more user input events comprise one of: expressing interest in the document, indicating that the document is relevant, and indicating that the document contains abundant information.
15. The method of claim 14, wherein the one or more user input events comprise rating the document using a rating scale comprising a plurality of values.
16. The method of claim 15, further comprising storing only a latest rating of the document from a user.
17. The method of claim 13, wherein the one or more user input events comprise saving the document for personal reference by the user.
18. The method of claim 17, wherein presenting the documents comprises displaying documents for which the user has given user input comprising a user input event, on a personal Web page of the user.
19. The method of claim 1, wherein the one or more user input events comprise one of: indicating that the document lacks relevance, voting against the document, and blocking the document from being presented in the future.
20. The method of claim 1, wherein the one or more user input events are provided by a first type of user, the first type of user being associated with one of: a user login, a user record, and cookie.
21. The method of claim 20, wherein the one or more user input events from the first type of user are used to influence the relevance score of documents to present to a second type of user that has not provided at least one user input event.
22. The method of claim 21, wherein a plurality of the first type of user is treated by the relevance model as a group.
23. The method of claim 22, wherein the user input events from the members of the group are used to generate parameters for the relevance model to assign scores to documents for presenting to the second type of user.
24. The method of claim 22, wherein the user input events from the group are used to generate parameters for the relevance model to detect input from a user attempting to modify the relevance score of a document to have a bias toward the document that is inconsistent with user input events from the group.
25. The method of claim 1, wherein the relevance score is calculated based on a number of the one or more user input events.
26. The method of claim 1, wherein the relevance model comprises a plurality of features including at least one document-specific feature and at least one query-specific feature.
27. The method of claim 26, wherein the plurality of features comprises one of: frequency of a term within the document, frequency of a term in a hyperlink to the document, frequency of a term in a summary of the document, location of a term within the document, the structure of the document, the length of the document, the type of the document, the date of creation of the document, number of hyperlinks to the document, quality of hyperlinks to the document, terms with which the document has been tagged, or ratings for the document.
28. The method of claim 26, wherein the plurality of features comprises one of: a user representation, a position of the document in a search results list, a number of times the document has been accessed from a search results list, a time of a user input, a user blocking the document, a user identifier for the document, a user saving the document, a user bookmarking the document, a user tagging the document, and a user rating of the document or otherwise expressing a positive or negative view of the document.
29. The method of claim 26, wherein the relevance model comprises one of a user model and a group model, wherein the user model is for determining a relevance score of a document and a search query for a user and the group model is for determining a relevance score of a document and a search query for a group of users.
30. The method of claim 26, further comprising modifying the relevance model, wherein modifying the relevance model comprises modifying a parameter for at least one feature in the relevance model using at least one user input event.
31. The method of claim 30, wherein modifying the relevance model comprises applying a machine learning technique.
32. The method of claim 31, wherein a machine learning technique comprises a statistical classification.
33. The method of claim 32, wherein a statistical classification comprises one of a logistic regression analysis, a support vector machine, neural networks, boosted trees, random forests, naive Bayes, and graphical models.
34. The method of claim 31, a machine learning technique includes conjugate gradient descent.
35. The method of claim 1, wherein a document is a sponsored link.
36. The method of claim 1, wherein a document comprises one of: a video, a music file, a picture, and a textual document.
37. The method of claim 1, wherein at least one of the one or more user input events is read from one of a database and a file.
38. A system for incorporating input to improve the relevance of documents presented to a user, the system comprising:
a computing device comprising a processor coupled to a computer-readable storage medium having processor-executable instructions that configure the computing device to:
a. receive input about at least one document by at least one user, wherein the input is received outside a context of a presentation of a search results of a search;
b. assign a relevance score to each of the documents according to a relevance model, using the input; and
c. present the documents and the input to the user according to the relevance scores of the documents, wherein the documents and the input about the at least one document are different.
39. The system of claim 38, further comprising a database coupled to the computing device, wherein the input is stored in the database and receiving the input comprises receiving the input from the database.
40. The system of claim 38, wherein the processor-executable instructions further configure the computing device to receive input from the user about at least one document in the documents presented to the user.
41. The system of claim 38, wherein the user input is used to modify the relevance model to improve the relevance of the documents presented to the user.
42. The system of claim 38, wherein the processor-executable instructions further configure the computing device to present the documents to the user in response to a search query.
43. The system of claim 38, wherein the processor-executable instructions further configure the computing device to receive the search query from the user.
44. The method of claim 1, wherein at least one of the user input events is received within the context of a search.
US13474628 2005-03-18 2012-05-17 Search engine that applies feedback from users to improve search results Active US9367606B1 (en)
US66336105 true 2005-03-18 2005-03-18
US11378756 US8185523B2 (en) 2005-03-18 2006-03-17 Search engine that applies feedback from users to improve search results
US13474628 US9367606B1 (en) 2005-03-18 2012-05-17 Search engine that applies feedback from users to improve search results
US15162444 US20160299981A1 (en) 2005-03-18 2016-05-23 Search engine that applies feedback from users to improve search results
US11378756 Continuation US8185523B2 (en) 2005-03-18 2006-03-17 Search engine that applies feedback from users to improve search results
US15162444 Continuation US20160299981A1 (en) 2005-03-18 2016-05-23 Search engine that applies feedback from users to improve search results
US9367606B1 true US9367606B1 (en) 2016-06-14
US11378756 Active 2026-06-13 US8185523B2 (en) 2005-03-18 2006-03-17 Search engine that applies feedback from users to improve search results
US13474628 Active US9367606B1 (en) 2005-03-18 2012-05-17 Search engine that applies feedback from users to improve search results
US15162444 Pending US20160299981A1 (en) 2005-03-18 2016-05-23 Search engine that applies feedback from users to improve search results
US (3) US8185523B2 (en)
KR (2) KR101374651B1 (en)
WO (1) WO2006102122A3 (en)
EP2377011A4 (en) 2008-12-12 2017-12-13 Atigeo Corp Providing recommendations using information determined for domains of interest
CN103646092B (en) * 2013-12-18 2017-07-04 孙燕群 Search engine ranking method based on user participation
CN103714174B (en) * 2014-01-02 2017-01-11 武汉大学 Information collection method and system for the Internet to accelerate the accumulation of experience and knowledge
CN103902705B (en) * 2014-03-31 2017-05-10 华中科技大学 Drive mechanism across the digital content based on metadata integration system and method
CN104598611A (en) * 2015-01-29 2015-05-06 百度在线网络技术（北京）有限公司 Method and system for sequencing search entries
EP3155533A1 (en) * 2015-06-29 2017-04-19 Nowfloats Technologies Pvt. Ltd. System and method for optimizing and enhancing visibility of the website
JP6174099B2 (en) * 2015-10-28 2017-08-02 ヤフー株式会社 Search device, and search method
China Office Action mailed Dec. 19, 2011, China Application No. 200680017270.7, 8 pages.
English-language translation of Office Action, Japanese Patent Application No. 2008-502111, mailed Oct. 8, 2013, pp. 1-5.
Eske,Vladimir, User Profile Management in a Web Search Engine, Computer Science Department Saarland University, Jun. 20014 https://domino.mpi-inf.mpg.de/intranet/ag5/ag5publ.nsf/a07d7d3a286130d0c12565dd006fb8e2/983fc84653920a97c1256ede00394b65/$FILE/Eske.pdf.
Foreign Office Action, China Patent Office, dated Jul. 6, 2011, China Patent Application No. 200680013556.8, Filed Feb. 28, 2006.
Foreign Office Action, Japan Patent Office, dated May 11, 2011, Japan Patent Application No. 2007-558198, Filed Aug. 28, 2007.
Foreign Office Action, Japan Patent Office, dated May 12, 2011, Japan Patent Application No. 200-502111, Filed Sep. 18, 2007.
Foreign Office Action; China State Intellectual Property Office, dated Sep. 27, 2010, Chinese Patent Application No. 200680017270.7, Filed Mar. 17, 2006.
Japan Office Action mailed Feb. 7, 2012, Japan Application No. 2008-502111, 3 pages.
Notice of Allowance mailed Jan. 20, 2012, U.S. Appl. No. 11/378,756, filed Mar. 17, 2006, Yunshan Lu et al.
tripadvisor.com Web site, searching for keywords "Paris", Aug. 18, 2004 from http://web.archive.org/web/20040818035209/http://www.tripadvisor.com/Tourism-g187147-Paris-lle-de-France-Vacations.html (Aug. 18, 2004) (pp. 1-3).
WO2006102122A2 (en) 2006-09-28 application
KR20080033137A (en) 2008-04-16 application
US8185523B2 (en) 2012-05-22 grant
CN101523338A (en) 2009-09-02 application
EP1866738A2 (en) 2007-12-19 application
CA2601768C (en) 2016-08-23 grant
KR20130119981A (en) 2013-11-01 application
US20160299981A1 (en) 2016-10-13 application
JP5632124B2 (en) 2014-11-26 grant
US20070106659A1 (en) 2007-05-10 application
JP2012212455A (en) 2012-11-01 application
WO2006102122A3 (en) 2009-04-16 application
EP1866738A4 (en) 2010-09-15 application
JP5638031B2 (en) 2014-12-10 grant
KR101532715B1 (en) 2015-07-02 grant
CN101523338B (en) 2016-10-19 grant
CA2601768A1 (en) 2006-09-28 application
JP2008538149A (en) 2008-10-09 application
KR101374651B1 (en) 2014-03-17 grant
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LU, YUNSHAN;RIOS, GORDON P.;TANNE, MICHAEL;REEL/FRAME:028229/0778
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WINK TECHNOLOGIES, INC.;REEL/FRAME:042976/0064
Free format text: CHANGE OF NAME;ASSIGNOR:MACHINE PHASE SYSTEMS LIMITED;REEL/FRAME:043153/0008