Source: https://patents.google.com/patent/US10409851B2/en
Timestamp: 2019-12-07 04:03:59
Document Index: 672988358

Matched Legal Cases: ['Application No. 11872811', 'Application No. 201180073610', 'Application No. 201180073610', 'Application No. 201180073610', 'Application No. 201180073610', 'application No. 13755014', 'application No. 11866113', 'Application No. 13755014', 'application No. 11866113']

US10409851B2 - Gesture-based search - Google Patents
US10409851B2
US10409851B2 US13/017,335 US201113017335A US10409851B2 US 10409851 B2 US10409851 B2 US 10409851B2 US 201113017335 A US201113017335 A US 201113017335A US 10409851 B2 US10409851 B2 US 10409851B2
US20120197857A1 (en
detecting a gesture which defines a region of displayed content, wherein the region of displayed content corresponds to a first content format;
formulating multiple search queries corresponding to multiple content formats including the first content format, the multiple search queries based at least in part on the region of displayed content defined by the gesture and context information providing signals usable to improve relevance of one or more search results;
presenting the multiple search queries available to conduct a search, in response to detecting the gesture;
receiving an indication of a user selection of a search query from among the multiple search queries, the search query to perform the search corresponding to the respective content format using content from the region of displayed content; and
causing the search to be performed using the search query in response to receiving the indication of the user selection of the search query from among the multiple search queries.
2. The method of claim 1, the gesture comprising an eye movement and wherein detecting the gesture comprises detecting the eye movement using an image capture device.
3. The method of claim 1, wherein the gesture further comprises a substantially circular motion that bounds or substantially bounds the region of displayed content.
4. The method of claim 1, further comprising displaying a representation of the region of displayed content defined by the gesture on the displayed content.
5. The method of claim 4, wherein the representation of the region of displayed content comprises a line enclosing the region of displayed content, a highlighting view of the region of displayed content, and/or a magnifying lens view of the region of displayed content.
6. The method of claim 4, further comprising receiving a subsequent gesture in relation to the representation of the region of displayed content, and altering a scope of the search in response to receiving the subsequent gesture.
7. The method of claim 4, further comprising receiving a resizing gesture to resize the representation of the region of displayed content, and reformulating the search query based on the resized region of displayed content.
8. The method of claim 4, further comprising receiving a relocation gesture to relocate the representation of the region of displayed content to include other content, and reformulating the search query based the relocated region of content.
9. The method of claim 1, further comprising receiving a voice input, and refining the search query based on the voice input.
10. The method of claim 1, further comprising presenting one or more search results of the search by overlaying the one or more search results on the displayed content.
11. The method of claim 10, further comprising expanding a size of the one or more search results in response to receiving an expand gesture.
12. The method of claim 11, further comprising collapsing or hiding the one or more search results in response to a collapse gesture.
14. The method of claim 1; wherein the context information comprises at least one of:
content proximate to the region of displayed content;
a paragraph having a portion thereof within the region of displayed content;
a sentence having a portion thereof within the region of displayed content;
a title of a document having a portion of the document within the region of displayed content;
an application identifier of an application used to display the displayed content; or
15. The method of claim 1, wherein detecting the gesture comprises:
determining based on characteristics of the gesture that the gesture is a gesture usable to initiate the search and is not a gesture usable to pan or scroll the displayed content.
16. The method of claim 1, further comprising prior to detecting the gesture, receiving a command indicating that the gesture is imminent, wherein the command comprises a touch and hold of a screen of a device that displays the displayed content or a touch input in a specified location of the screen of the device.
17. The method of claim 1, wherein causing the search to be performed comprises sending the search query to a remote search engine and/or performing the search locally.
18. The method of claim 1; further comprising discriminating the gesture from other types of gestures based on a curvature of the gesture, a speed at which the gesture is entered or context information.
19. The method of claim 1, wherein the presenting of the multiple search queries accompanies visual feedback that indicates a size and shape of the defined region of displayed content.
20. The method of claim 1, the instructions further including:
receiving an updated indication of a user selection of an update to the search query, to refine the search query.
21. One or more computer storage media storing instructions that, when executed by a processor, configure the processor to perform acts comprising:
determining that the detected gesture is a gesture related to a search for information associated with a region of displayed content defined by the gesture, the determining comprising discriminating the gesture from other types of gestures based on a curvature of the detected gesture or a speed at which the detected gesture is entered, wherein the region of displayed content corresponds to a first content format;
formulating multiple search queries corresponding to multiple content formats including the first content format, the multiple search queries based at least on the region of displayed content defined by the gesture and context information providing signals usable to improve relevance of one or more search results;
presenting the multiple search queries as available to conduct the search, in response to detecting the gesture;
22. The one or more computer storage media of claim 21, the acts further comprising displaying a representation of the region of displayed content defined by the gesture on the displayed content, the representation of the region of displayed content comprising a line enclosing the region of displayed content, a highlighting view of the region of displayed content, and/or a magnifying lens view of the region of displayed content.
23. The one or more computer storage media of claim 22, the acts further comprising receiving a subsequent gesture in relation to the representation of the region of displayed content, and altering a scope of the search in response to receiving the subsequent gesture.
24. The one or more computer storage media of claim 21, the acts further comprising receiving a voice input and/or eye tracking data, and refining the search query based on the voice input and/or the eye tracking data.
25. The one or more computer storage media of claim 21, the context information comprising at least one of:
a gesture detection module stored in the memory and executable by the one or more processors to detect a search gesture which defines a region of displayed content that is used for performing a search, wherein the region of displayed content corresponds to a first content format; and
a content analysis module stored in the memory and executable by the one or more processors to:
identify subject matter of the displayed content;
identify context information providing signals as to a context n which the identified subject matter is presented;
formulate multiple search queries corresponding to multiple content formats including the first content format based on the subject matter and context information;
present the multiple search queries as available to conduct a search, in response to detecting the search gesture;
receive an indication of a user selection of a search query from among the multiple search queries, the search query to perform the search corresponding to the respective content format using content from the region of displayed content; and
cause the search to be performed based at least in part on the identified subject matter and the identified context information, in response to receiving the indication of the user selection of the search query from among the multiple search queries.
27. The system of claim 26, wherein the context information comprises:
28. The system of claim 26, wherein the content analysis module causes the search to be performed by sending a search query to a remote search engine and/or performing the search locally.
29. The system of claim 26, the search gesture comprising an eye movement and wherein to detect the search gesture comprises detecting the eye movement using an image capture device.
30. The system of claim 29, further comprising prior to detecting the eye movement, the gesture detection module further receives a command indicating that the eye movement is imminent, wherein the command comprises a touch and hold of a screen of a device that displays the displayed content or a touch input in a specified location of the screen of the device that displays the displayed content.
31. The system of claim 30, wherein the specified location comprises one of bottom corners of the screen of the device.
32. The system of claim 30, wherein the touch and hold of the screen of the device is maintained for a predetermined period of time.
a gesture detection module stored in the memory and executable by the one or more processors to:
detect a predefined motion of the client device;
receive a search gesture that defines a region of interest within content that is consumed by the client device, wherein the region of interest corresponds to a first content format; and
formulate multiple search queries corresponding to multiple content formats including the first content format, the multiple search queries based at least in part on data included in the defined region of interest; and
a content analysis module stored in the memory and executable by the one or more processors to cause a search for information of at least a part of content that is presented by the display of the client device in response to the detecting of the predefined motion of the client device, the content analysis module further executable by the one or more processors to:
present the multiple search queries as available to conduct a search, in response to detection of the predefined motion of the client device;
receive an indication of a user selection of a search query from among the multiple search queries, the search query to perform the search corresponding to the respective content format using the at least part of the content that is presented by the display of the client device; and
cause the search to be performed, in response to receiving the indication of the user selection of the search query from among the multiple search queries.
34. The client device of claim 33, wherein the display further displays search results for presentation to a user of the client device.
35. The client device of claim 34, wherein the gesture detection module further detects a gaze of a user at one or more entities presented on the display of the client device; and the content analysis module further generates eye tracking data based at least in part on the detected gaze and refines the search results based at least in part on the generated eye tracking data.
US20120197857A1 US20120197857A1 (en) 2012-08-02
US10409851B2 true US10409851B2 (en) 2019-09-10
CN103563391A (en) * 2011-05-27 2014-02-05 汤姆逊许可公司 Method, apparatus and system for multiple screen media experience
CN104246680B (en) * 2012-07-24 2018-04-10 慧与发展有限责任合伙企业 Initiate to help feature
CN103455590B (en) * 2013-08-29 2017-05-31 百度在线网络技术（北京）有限公司 The method and apparatus retrieved in touch-screen equipment
US20170286552A1 (en) * 2016-03-30 2017-10-05 Microsoft Technology Licensing, Llc Using Gesture Selection to Obtain Contextually Relevant Information
WO2008002638A1 (en) 2006-06-28 2008-01-03 Microsoft Corporation Visual and multi-dimensional search
US20090167705A1 (en) 2007-12-31 2009-07-02 High Tech Computer, Corp. Method for operating software input panel
US20090259643A1 (en) 2008-04-15 2009-10-15 Yahoo! Inc. Normalizing query words in web search
US20100042935A1 (en) 2008-08-14 2010-02-18 Yield Software, Inc. Method and System for Visual Landing Page Optimization Configuration and Implementation
US20100083155A1 (en) 2006-02-14 2010-04-01 Microsoft Corporation Object search ui and dragging object results
WO2010065195A1 (en) 2008-12-05 2010-06-10 Yahoo!I Nc. System and method for context based query augmentation
WO2011024585A1 (en) 2009-08-25 2011-03-03 楽天株式会社 Information acquisition device, information acquisition program, recording medium, information acquisition method, and information acquisition system
CN102024064A (en) 2011-01-11 2011-04-20 宇龙计算机通信科技(深圳)有限公司 Rapid searching method and mobile communication terminal
US20110161852A1 (en) 2009-12-31 2011-06-30 Nokia Corporation Method and apparatus for fluid graphical user interface
JP2008500372A (en) * 2004-05-26 2008-01-10 カリフォルニア インスティテュート オブ テクノロジー Small molecule stimulators of neuronal growth
JP4952113B2 (en) * 2006-07-26 2012-06-13 ブラザー工業株式会社 Image forming apparatus
US20120131019A1 (en) 2009-08-25 2012-05-24 Rakuten, Inc. Information acquiring apparatus, information acquiring program, recording medium, information acquiring method and information acquiring system
"European Search Report Issued in European Patent Application No. 11872811.2", dated Aug. 19, 2015, 7 Pages.
"Final Office Action Issued in U.S. Appl. No. 13/609,729", dated Apr. 19, 2017, 27 Pages.
"Final Office Action Issued in U.S. Appl. No. 13/609,729", dated Apr. 23, 2018, 23 Pages.
"Final Office Action Issued in U.S. Appl. No. 13/609,729", dated Jun. 25, 2015, 19 Pages.
"Final Office Action Issued in U.S. Appl. No. 13/609,729", dated Mar. 1, 2019, 23 Pages.
"First Office Action and Search Report Issued in Chinese Patent Application No. 201180073610.9", dated Jul. 4, 2016, 19 Pages.
"International Search Report and Written Opinion Issued in PCT Application No. PCT/CN2011/79926", dated May 31, 2012, 10 Pages.
"Non Final Office Action Issued in U.S. Appl. 13/609,729", dated Aug. 24, 2018, 35 Pages.
"Non Final Office Action Issued in U.S. Appl. 13/609,729", dated Sep. 28, 2017, 20 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/609,729", dated May 19, 2016, 23 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/609,729", dated Nov. 14, 2014, 16 Pages.
"Office Action Issued in Chinese Patent Application No. 201180073610.9", dated Jul. 5, 2017, 16 Pages.
"Second Office Action Issued in Chinese Patent Application No. 201180073610.9", dated Mar. 21, 2017, 17 Pages.
"Third Office Action Issued in Chinese Patent Application No. 201180073610.9", dated Dec. 25, 2017, 6 Pages.
Aula, "Query Formulation in Web Information Search," «http://www.iadis.net/dl/final_uploads/200302L052.pdf», IADIS International Conference WWW/Internet 2003, vol. I, 8 pages.
Bounding box, radius, and polygon search-Retrieved Date: Oct. 25, 2010 at «http://msdn.microsoft.com/en-us/library/cc451895.aspx», 11 pages.
Bounding box, radius, and polygon search—Retrieved Date: Oct. 25, 2010 at «http://msdn.microsoft.com/en-us/library/cc451895.aspx», 11 pages.
European Office Action dated Jan. 22, 2015 for European patent application No. 13755014.1, a counterpart foreign application of U.S. Appl. No. 13/408,853, 5 pages.
European Office Action dated Oct. 21, 2014 for European patent application No. 11866113.1, a counterpart foreign application of U.S. Appl. No. 13/201,823, 5 pages.
Google's gesture search available for android 1.6-Published Date: Mar. 17, 2010, retrieved at «http://blog.itechtalk.com/2010/googles-gesture-search-available-for-android-1-6», 5 pages.
Google's gesture search available for android 1.6—Published Date: Mar. 17, 2010, retrieved at «http://blog.itechtalk.com/2010/googles-gesture-search-available-for-android-1-6», 5 pages.
HotPads Daily, "Launching 2011 with Hand-Drawn Housing Search," retrieved on Sep. 6, 2011 at «http://daily.hotpads.com/hotpads_daily/2011/01/launching-2011-with-hand-d . . . », Jan. 3, 2011, 3 pages.
International Search Report dated May 15, 2013 for PCT/US2013/024247, 10 pages.
Internet Explorer 8: Features, "Accelerators", retrieved on Mar. 7, 2011 at «http://www.microsoft.com/windows/internet-explorer/features/faster.aspx», 3 pages.
Lasso Search-HotPads.com, retrieved on Sep. 6, 2011, at «http://hotpads.com/pages/features/lassoSearch.htm», 3 pages.
Lasso Search—HotPads.com, retrieved on Sep. 6, 2011, at «http://hotpads.com/pages/features/lassoSearch.htm», 3 pages.
Li, Yang, "Gesture Search: A Tool for Fast Mobile Data Access", In Proceedings of the 23rd Annual ACM symposium on User interface software and technology, Oct. 3, 2010, pp. 87-96.
McArthur et al., "The Ranking of Query Refinements in Interactive Web-based Retrieval," «http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.33.1493&rep=rep1&type=pdf», Information Doors Workshop, 2000, 4 pages.
Moustakas, et al., "Master-Piece: A Multimodal (Gesture+Speech) Interface for 3D Model Search and Retrieval Integrated in a Virtual Assembly Application", Published Date: Aug. 2005, 14 pages.
Nudelman, "Design Patterns for Mobile Faceted Search: Part I", Published Date: Apr. 5, 2010, retrieved at «http://www.uxmatters.com/mt/archives/2010/04/design-patterns-for-mobile-faceted-search-part-i.php», 9 pages.
Office Action for U.S. Appl. No. 13/201,823, dated Jan. 9, 2015, Xuedong Huang, "Invisible Control," 41 pages.
Office Action for U.S. Appl. No. 13/201,823, dated Jul. 17, 2014, Xuedong Huang, "Invisible Control," 35 pages.
Office action for U.S. Appl. No. 13/201,823, dated May 21, 2015, Huang et al., "Invisible Control", 40 pages.
Office action for U.S. Appl. No. 13/201,823, dated Oct. 27, 2015, Huang et al., "Invisible Control," 42 pages.
Office action for U.S. Appl. No. 13/408,853, dated Feb. 6, 2013, Bai et al., "Context-based Search Query Formation," 19 pages.
Office action for U.S. Appl. No. 13/408,853, dated Jul. 31, 2013, Bai et al., "Context-based Search Query Formation," 21 pages.
Reimagining mobile search: introducing Yahoo!(R) Sketch-a-Search (TM) and yahoo! search for the iPhone(R) Published Date: Mar. 23, 2010, retrieved at «http://ymobileblog.com/blog/2010/03/23/reimagining-mobile-search-introducing-yahoo%C2%AE-sketch-a-search%E2%84%A2-and-yahoo-search-for-the-iphone%C2%AE-2/», 6 pages.
Schaal, "Is HotPads Lasso Search an alternative to Google Hotel Finder shape drawing?" retrieved on Sep. 6, 2011 at «http://www.tnooz.com/2011/08/08/news/is-hotpads-lasso-search-an-alternativ . . . », Aug. 8, 2011, 6 pages.
Supplementary European Search Report dated Jan. 14, 2015 for European Patent Application No. 13755014.1, 3 pages.
Supplementary European Search Report dated Oct. 6, 2014 for European patent application No. 11866113.1, 3 pages.
Yahoo search circle allows users to easily expand & narrow results on map-Published Date: Apr. 24, 2008, retrieved at «http://searchengineland.com/yahoo-search-circle-allows-users-to-easily-expand-narrow-results-on-map-13854», 12 pages.
Yahoo search circle allows users to easily expand & narrow results on map—Published Date: Apr. 24, 2008, retrieved at «http://searchengineland.com/yahoo-search-circle-allows-users-to-easily-expand-narrow-results-on-map-13854», 12 pages.
Yahoo! Sketch-a-Search, retrieved on Mar. 7, 2011 at «http://itunes.apple.com/us/app/yahoo-sketch-a-search/id361077521?mt=8», 3 pages.
Yang et al., "A Web Interface for Visualizing Web Search Engine Results," 3rd International Conference on Parallel and Distributed Computing, Applications, and Techniques (PDCAT 2002), 6 pages.
US20120197857A1 (en) 2012-08-02