Source: http://patents.com/us-9972317.html
Timestamp: 2018-05-26 08:35:58
Document Index: 467762591

Matched Legal Cases: ['Application No. 2345', 'Application No. 200510116339', 'Application No. 200510116339', 'Application No, 2005', 'Application No. 05106352', 'Application No. 05109794', 'art 1', 'art 2', 'art 3', 'Application No. 05107831', 'Application No. 07751459', 'Application No. 05107831', 'Application No. 2005101036571', 'art 1', 'art 2', 'art 1', 'art 2', 'art 1', 'art 2', 'Application No. 05109794', 'Application No. 05109794', 'Application No. 10', 'Application No. 200510116339', 'Application No. 200510116339', 'Application No. 200510116339', 'Application No. 2005']

US Patent # 9,972,317. Centralized method and system for clarifying voice commands - Patents.com
United States Patent 9,972,317
Mowatt , et al. May 15, 2018
Mowatt; David (Dublin, IE), Chambers; Robert L. (Issaquah, WA), Andrew; Felix G. T. I. (Seattle, WA)
Family ID: 1000003293519
14/563,255
US 20150095030 A1 Apr 2, 2015
10990345 Nov 16, 2004 8942985
Current CPC Class: G10L 15/22 (20130101); G10L 15/18 (20130101); G06F 3/167 (20130101); G10L 2015/223 (20130101)
Current International Class: G10L 15/00 (20130101); G10L 15/04 (20130101); G10L 15/22 (20060101); G06F 3/16 (20060101); G10L 15/18 (20130101)
5041967 August 1991 Ephrath et al.
5231691 July 1993 Yasuda
5748973 May 1998 Palmer et al.
5850627 December 1998 Gould
5852801 December 1998 Hon et al.
5890122 March 1999 Van Kleeck
6073102 June 2000 Block
6125347 September 2000 Cote
6199061 March 2001 Blewett et al.
6289312 September 2001 Raman
6308157 October 2001 Vanbuskirk
6330577 December 2001 Kim
6374226 April 2002 Hunt
6446135 September 2002 Koppolu et al.
6535854 March 2003 Buchner et al.
6615177 September 2003 Rapp et al.
6693651 February 2004 Biebesheimer et al.
6728700 April 2004 Richards et al.
6744451 June 2004 Anderson et al.
6778193 August 2004 Biebesheimer et al.
6839896 January 2005 Coffman et al.
6948133 September 2005 Haley
6948135 September 2005 Ruthfield et al.
7043700 May 2006 Bertram et al.
7062711 June 2006 Kethireddy
7073126 July 2006 Khandekar
7113950 September 2006 Brill et al.
7124129 October 2006 Bowman et al.
7133950 November 2006 Olukotun
7137071 November 2006 Fuller et al.
7149550 December 2006 Kraft et al.
7231642 June 2007 Araki et al.
7254539 August 2007 Carberry et al.
7254784 August 2007 Chang
7440941 October 2008 Borkovsky et al.
7505910 March 2009 Kujirai
7539939 May 2009 Schomer
7596754 September 2009 Wessling et al.
7703037 April 2010 McGlinchey et al.
7925975 April 2011 Bala et al.
8370743 February 2013 Bala
8942985 January 2015 Mowatt et al.
9632650 April 2017 Bala
2001/0044726 November 2001 Li et al.
2002/0048350 April 2002 Phillips et al.
2002/0052870 May 2002 Charlesworth et al.
2002/0080157 June 2002 Chickles et al.
2002/0118220 August 2002 Lui et al.
2002/0133354 September 2002 Ross
2002/0156629 October 2002 Carberry
2002/0161584 October 2002 Lewis et al.
2002/0188612 December 2002 Yu et al.
2003/0004941 January 2003 Yamada et al.
2003/0014260 January 2003 Coffman
2003/0016238 January 2003 Sullivan et al.
2003/0171928 September 2003 Falcon
2003/0177013 September 2003 Falcon et al.
2003/0200254 October 2003 Wei
2003/0234818 December 2003 Schmid et al.
2004/0073540 April 2004 Wang et al.
2004/0095263 May 2004 Hwang et al.
2005/0060138 March 2005 Wang et al.
2005/0071777 March 2005 Roessler et al.
2005/0081152 April 2005 Commarford et al.
2005/0108026 May 2005 Brierre et al.
2005/0114319 May 2005 Brent et al.
2005/0131691 June 2005 Fischer
2005/0131701 June 2005 Cross et al.
2005/0185773 August 2005 Burger
2005/0283473 December 2005 Rousso et al.
2006/0036438 February 2006 Chang
2006/0048060 March 2006 Mohr et al.
2006/0053384 March 2006 La Fetra et al.
2006/0136195 June 2006 Agapi et al.
2006/0150112 July 2006 Marchev
2006/0155687 July 2006 Chou
2006/0167851 July 2006 Ivanov
2006/0168522 July 2006 Bala
2006/0184899 August 2006 Klassen et al.
2006/0190256 August 2006 Stephanick et al.
2006/0190429 August 2006 Sidlosky et al.
2006/0241995 October 2006 McGlinchey et al.
2006/0259479 November 2006 Dai
2007/0033172 February 2007 Williams et al.
2007/0143704 June 2007 Laird-McConnell
2007/0189724 August 2007 Wan et al.
2013/0205241 August 2013 Bala
2014/0317578 October 2014 Chaudhri
2017/0032786 February 2017 Mowatt et al.
0503944 Sep 1992 EP
0317479 Jan 1997 EP
911808 Apr 1999 EP
1045374 Oct 2000 EP
1215657 Jun 2002 EP
1246430 Oct 2002 EP
1603031 Dec 2005 EP
2001-034289 Sep 2001 JP
2002-182688 Jun 2002 JP
2003-084794 Mar 2003 JP
2004-110613 Apr 2004 JP
2004-234273 Aug 2004 JP
9525326 Sep 1995 WO
WO 9735253 Sep 1997 WO
WO 0144932 Jun 2001 WO
WO 0175579 Oct 2001 WO
WO 0212982 Feb 2002 WO
2002075538 Sep 2002 WO
McGee, et al., "Confirmation in Multimodal Systems", In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, vol. 2, Aug. 10, 1998, 7 Pages. cited by applicant .
Mankoff, Jennifer, "Proposal of a Model Architecture Handling Ambiguity in Recognition-Based Input", Dissertation Submitted in Partial Fulfillment of the Requirement for the Degree of Doctor of Philosophy in College Computing, 1999, 55 Pages. cited by applicant .
Mankoff, et al., "Interaction Techniques for Ambiguity Resolution in Recognition-Based Interfaces", In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology, 2000, 10 Pages. cited by applicant .
Libuda, Lars: "Improving clarification dialogs in speech command systems with the help of user modeling: A conceptualization for an in-car user interface", Proceedings of the GI-Workshop, 2001, 5 pages. cited by applicant .
Issue Notification for U.S. Appl. No. 10/990,345 dated Jan. 7, 2015, 1 page. cited by applicant .
Office Action from India Patent Application No. 2345/DEL/2005 dated Nov. 25, 2014. 2 pages. cited by applicant .
Third Chinese Office Action for Chinese Application No. 200510116339.9 dated May 3, 2012, 6 pages. cited by applicant .
Second Chinese Office Action for Chinese Application No. 200510116339.9 dispatch dated Feb. 3, 2012, 6 pages. cited by applicant .
Notice of Rejection for Japanese Patent Application No, 2005-299982 dated May 24, 2011 with English Translation, 6 pages. cited by applicant .
Reithinger, Norbert, et al. "SmartKom: adaptive and flexible multimodal access to multiple applications." Proceedings of the 5th international conference on Multimodal interfaces. ACM, 2003. 8 pages. cited by applicant .
European Search Report dated Nov. 8. 2005 for Application No. 05106352.7, 5 pages. cited by applicant .
Potamitis, Ilyas, et al. "An integrated system for smart-home control of appliances based on remote speech interaction." INTERSPEECH. 2003. 5 pages. cited by applicant .
European Search Report dated Mar. 21, 2006 or Application No. 05109794.7, 7 pages. cited by applicant .
Chang, Eric, "Efficient multimodal method to provide input to a computing device." U.S. Appl. No. 10/889,822, 34 pages. cited by applicant .
Chang, Eric, et al. "A system for spoken query information retrieval on mobile devices." Speech and Audio Processing, IEEE Transactions on 10.8 (2002): 531-541. cited by applicant .
Bostwick et al., "Flexi-modal and multi-machine user interfaces." Proceedings of the 4th IEEE International Conference on Multimodal Interfaces. IEEE Computer Society, 2002. 6 pages. cited by applicant .
Kvale, Knut, N. Warakagoda, and Jan Eikeset Knudsen. "Speech centric multimodal interfaces for mobile communication systems." Telektronikk 99.2 (2003): 104-117. cited by applicant .
Seide, Frank, et al. "Vocabulary-independent search in spontaneous speech." Acoustics, Speech, and Signal Processing, 2004. Proceedings.(ICASSP'04). IEEE International Conference on. vol. 1. IEEE, 2004. 4 pages. cited by applicant .
Manaris, Bill Z., Valanne MacGyvers, and Michail G. Lagoudakis. "Universal Access to Mobile Computing Devices through Speech Input." Flairs Conference. 1999. 7 pages. cited by applicant .
Chang, Eric, et al. "Efficient web search on mobile devices with multi-modal input and intelligent text summarization." The 11th Int. WWW Conference. 2002. 4 pages. cited by applicant .
Gu, Huixiang, et al. "Spoken Query for Web Search and Navigation." WWW Posters. 2001. 2 pages. cited by applicant .
Wasinger, Rainer, Christoph Stahl, and Antonio Krueger. "Robust speech interaction in a mobile environment through the use of multiple and different media input types." The 8th European Conference on Speech Communication and Technology (Eurospeech). 2003. 4 pages. cited by applicant .
Iftikhar, Ahmad, et al. "Query by Image Content using Nokia 9210 Communicator." Proc. of the Workshop on Image Analysis for Multimedia Interactive Services, WIAMIS. vol. 1. 2001. 5 pages. cited by applicant .
Prosecution History for U.S. Appl. No. 10/990,345 including: Notice of Allowance dated Sep. 10, 2014, Decision on Appeal dated Jun. 11, 2014, Appeal Docketing Notice dated Nov. 10, 2011, Interview Summary dated Sep. 29, 2011, Reply Brief dated Jul. 15, 2011, Examiner's Answer to Appeal Brief dated May 16, 2011 Appeal Brief dated Feb. 14. 2011, Notice of Appeal dated Dec. 6, 2010; Final Office Action dated Aug. 5, 2010, Amendment dated Apr. 29, 2010. Non-Final Office Action dated Feb. 4, 2010, Part 1 of 3. cited by applicant .
Prosecution History for U.S. Appl. No. 10/990,345 including: Amendment dated Oct. 21, 2009, Non-Final Office Action dated Jul. 31, 2009, Appeal Brief dated Apr. 16, 2009, Notice of Appeal dated Mar. 2, 2009, Advisory Action dated Jan. 13, 2009, Response After Final dated Dec. 29, 2008, Interview Summary dated Dec. 19, 2008, Final Office Action dated Oct. 28, 2008, Interview Summary dated Jun. 19, 2008, Amendment dated Jun. 17, 2008, Part 2 of 3. cited by applicant .
Prosecution History for U.S. Appl. No. 10/990,345 including: Non-Final Office Action dated Mar. 17, 2008, Preliminary Amendment dated Jan. 24, 2008, and Application and Drawings filed Nov. 16, 2004. Part 3 of 3, 401 pages. cited by applicant .
Create Rich Client Apps with the Dom. Padilla, pp. 1-7 (Feb. 14, 2004). cited by applicant .
Mahesh, K. "Advanced Search Capabilities in Oracle8i intermedia text," Oracle Corporation, Online! Nov. 1999, pp. 1-6. cited by applicant .
Davidson, J. Running Mac OS X Panther, Dec. 2003. 1st Edition, Ch. 2. cited by applicant .
Bass & Bunker, L. & R.: A Generalized User Interface for Applications Programs,' Communications of the ACM, V. 24. No. 12, Dec. 1981, pp. 798-800. cited by applicant .
Shneiderman, B.; "Clarifying Search: A User-Interface Framework for Text Searches." D-Lib Magazine, Jan. 1997, pp. 1-18 cited by applicant .
First Chinese Office Action for Patent No. 200510116339.9, dated Jul. 25, 2008, 5 pages. cited by applicant .
Prosecution History for U.S. Appl. No. 13/732,520 including: Notice of Allowanced dated Nov. 2, 2016, Amendment dated Aug. 19, 2016, Terminal Disclaimer dated Aug. 19, 2016, Terminal Disclaimer Review Decision dated Aug. 19, 2016, Non-Final Office Action dated May 12, 2016, Amendment dated Feb. 22, 2016, Non-Final Office Action dated Oct. 22, 2015, Amendment dated Oct. 7, 2015, and Non-Final Office Action dated Jun. 25, 2015, 102 pages. cited by applicant .
European Application No. 05107831.9-2211 Article 94(3) EPC dated Nov. 17, 2009, 6 pages. cited by applicant .
Extended European Search Report for Application No. 07751459.4-2211, dated Apr. 26, 2010, 8 pages. cited by applicant .
XAML--A Business Perspective, <<http://www.xaml.net>>, Jan. 4, 2006, 2 pgs. cited by applicant .
Raman, T.V.:, "Speech-Enablement of Eclipse IDE and Eclipse Rich Client Applications Using AspectJ", IBM Search, Accessed Feb. 10, 2011, 13 pages. cited by applicant .
PCT Search Report, PCT/US2007/004699, Filed Feb. 23, 2007, 3 pages. cited by applicant .
European Search Report for Application No. 05107831.9, dated Mar. 15, 2006, 3 pages. cited by applicant .
Chinese First official Action for Application No. 2005101036571, dated Apr. 10, 2009, 13 pages. cited by applicant .
Prosecution History for U.S. Appl. No. 11/701,125 including: Issue Notification dated Jan. 16, 2013, Notice of Allowance dated Oct. 2, 2012, Amendment dated Sep. 19, 2012, Final Office Action dated Aug. 29, 2012, Amendment dated May 21, 2012, Non-Final Office Action dated Mar. 13, 2012, Amendment with RCE dated Mar. 23, 2011, Final Office Action dated Nov. 15, 2010, Amendment dated Sep. 17, 2010, Non-Final Office Action dated Jun. 10, 2010, Part 1 of 2. cited by applicant .
Prosecution History for U.S. Appl. No. 11/701,125 including: Amendment with RCE dated Apr. 21, 2010, Final Office Action dated Feb. 16, 2010, Amendment dated Nov. 20, 2009, Non-Final Office Action dated Sep. 11, 2009 and Application and Drawings filed Feb. 1, 2007, Part 2 of 2, 256 pages. cited by applicant .
Prosecution History for U.S. Appl. No. 11/372,545 including: Issue Notification dated Mar. 23, 2011, Notice of Allowance dated Dec. 17, 2010, Amendment dated Sep. 21, 2010, Non-Final Office Action dated Jul. 8, 2010, Amendment with RCE dated May 14, 2010, Final Office Action dated Feb. 2, 2010, Amendment dated Nov. 17, 2009, Non-Final Office Aciton dated Aug. 3, 2009, Advisory Action dated Feb. 27, 2009, Amendment with RCE dated Feb. 17, 2009, Part 1 or 2. cited by applicant .
Prosecution History for U.S. Appl. No. 11/372,545 including: Final Office Action dated Nov. 28, 2008, Amendment dated Sep. 15, 2008, Non-Final Office Action dated Apr. 17, 2008 and Application and Drawings filed Mar. 10, 2006, Part 2 of 2, 178 pages. cited by applicant .
Prosecution History for U.S. Appl. No. 11/109,997 including: Issue Notification dated Mar. 31, 2010, Notice of Allowance dated Feb. 22, 2010, Amendment dated Dec. 14, 2009, Non-Final Office Action dated Sep. 24, 2009, Amendment with RCE dated Jun. 30, 2009, Final Office Action dated Mar. 20, 2009, Interview Summary dated Jan. 23, 2009, Amendment dated Jan. 15, 2009, Non-Final Office Action dated Oct. 15, 2008, Amerndment with RCE dated Jul. 29, 2008, Part 1 of 2. cited by applicant .
Prosecution History for U.S. Appl. No. 11/109,997 including: Advisory Action dated May 30, 2008, Amendment dated May 21, 2008, Interview Summary dated Apr. 17, 2008, Final Office Action dated Feb. 21, 2008, Amendment dated Nov. 21, 2007, Non-Final Office Action dated Sep. 7, 2007 and Application and Drawings filed Apr. 20, 2005, Part 2 of 2, 206 pages. cited by applicant .
Application and Drawings for U.S. Appl. No. 15/458,337, filed Mar. 14, 2017, 59 pages. cited by applicant .
"Notice of Allowance Issued in European Patent Application No. 05109794.7", dated Sep. 9, 2009, 9 Pages. cited by applicant .
"Office Action Issued in European Patent Application No. 05109794.7", dated Nov. 23, 2006, 4 Pages. cited by applicant .
"Notice of Allowance Issued in U.S. Appl. No. 10/990,345", dated Dec. 18, 2014, 2 Pages. cited by applicant .
"Notice of Allowance Issued in Korean Patent Application No. 10-2005-0089717", dated Feb. 27, 2012, 2 Pages. (W/o English Translation). cited by applicant .
"Decision on Rejection Issued in Chinese Patent Application No. 200510116339.9" dated Jul. 3, 2009, 5 Pages. cited by applicant .
"Notice of Allowance Issued in Chnese Patent Application No. 200510116339.9" dated Sep. 5, 2012,4 Pages. cited by applicant .
"Office Action Issued in Chinese Patent Application No. 200510116339.9", dated Jan. 9, 2012 7 Pages. cited by applicant .
"Notice of Allowane Issued in Japanese Patent Application No. 2005-299982", dated Sep. 27, 2011, 6 Pages. cited by applicant .
Notice of Allowance for U.S. Appl. No. 13/732,520 dated Jan. 25, 2017, 7 pages. cited by applicant .
Little & Miller, G&R.:, "Translating Keyword Commands into Executable Code", UIST'06, Oct. 15-18, 2006, Switzerland, 10 pages. cited by applicant .
Ng, et al., "Integrating Multiple Knowledge Sources to Disambiguate Word Sense: An Exemplar-Based Approach", In Proceedings of the 34th Annual Meeting on Association for Computational Linguistics, Jun. 24, 1996, pp. 40-47. cited by applicant .
"Non-Finai Office Action Issued in U.S Appl. No. 15/292,871", dated Dec. 15, 2017, 16 Pages. cited by applicant.
Primary Examiner: Ortiz Sanchez; Michael
The present application is a continuation of and claims priority of U.S. patent application Ser. No. 10/990,345, filed Nov. 16, 2004, the content of which is hereby incorporated by reference in its entirety.
1. A computing device implemented method comprising: receiving, at a module, first information corresponding to a first set of commands associated with a first application module; receiving, at the module, second information corresponding to a second set of commands associated with a second application module; obtaining, by the module, a voice command, recognized by processing a user input using a speech recognizer; based at least in part on the module determining, using the first information, that the voice command includes at least one word corresponding to a first object or a first action included in at least a first command included in the first set of commands, obtaining a first indication of the correspondence between the voice command and the first command; based at least in part on the module determining, using the second information, that the voice command includes at least one word corresponding to a second object or a second action included in at least a second command included in the second set of commands, obtaining a second indication of the correspondence between the voice command and the second command; based at least in part on the first indication and the second indication, preparing a list of third information corresponding, respectively, to the first command associated with the first application module and the second command associated with the second application module; visually rendering items in the list of third information on a display; receiving an indication of a user selection of one of the items in the list of third information, wherein the selected item corresponds to a selected one of the first or second command; and in response to the indication of the user selection, automatically executing the selected command associated with the first application module or the second application module.
2. The computing device implemented method of claim 1, and further comprising: receiving a voice input from the user; and processing, by the speech recognizer, the voice input to recognize the voice command.
3. The computing device implemented method of claim 2, wherein the module comprises an interface module that is communicatively coupled to the first application module, the second application module, and the speech recognizer.
4. The computing device implemented method of claim 1, wherein the module comprises a grammar generator module configured to generate a grammar based on the first and second information.
5. The computing device implemented method of claim 1, and further comprising determining that the voice command is ambiguous based on at least one of: a determination that the voice command can be executed to perform more than one action; or a determination that more one than one different application module can execute the voice command; and wherein the module obtains the first and second indications based on the determination that the voice command is ambiguous.
6. The computing device implemented method of claim 1, wherein each of the items in the list corresponds to an action performable by one of the first or second application modules.
7. The computing device implemented method of claim 6, wherein visually rendering the items in the list comprises: for each item in the list, visually identifying which of the first or second application modules corresponds to the item.
8. The computing device implemented method of claim 7, and further comprising: identifying an action, that corresponds to the item selected from the list; identifying one of the first or second application module that corresponds to the identified action; and providing an indication of the identified action to the identified application module.
9. A computing system comprising: a processor; and memory storing instructions executable by the processor, wherein the instructions, based at least on execution, configure the computing system to: receive first information corresponding to a first set of commands associated with a first application module; receive second information corresponding to a second set of commands associated with a second application module; store the first and second information in a data store; identify a voice command received from a user and recognized based at least in part on using a speech recognizer; based at least in part on a determination, using the stored first information, that the voice command includes at least one word corresponding to a first object or a first action included in at least a first command included in the first set of commands, obtain a first indication of the correspondence between the voice command and the first command; based at least in part on a determination, using the stored second information, that the voice command includes at least one word corresponding to a second object or a second action included in at least a second command included in the second set of commands, obtain a second indication of the correspondence between the voice command and the second command; based at least in part on the first indication and the second indication, prepare a list of third information corresponding, respectively, to the first command associated with the first application module and the second command associated with the second application module; visually render items in the list of third information on a display; for each item in the visually rendered list, visually identify on the display which of the first or second application modules corresponds to the item; receive an indication of a user selection of one of the items in the list of third information, wherein the selected item corresponds to a selected one of the first or second command; and based on the indication of the user selection, automatically execute the selected command.
10. The computing system of claim 9, wherein the instructions configure the computing system to: receive a voice input from a user; and process the voice input using a speech recognizer to recognize the voice command.
11. The computing system of claim 9, wherein the instructions configure the computing system to: generate a grammar based on the first and second information; and recognize the voice command based on the grammar.
12. The computing system of claim 9, wherein the instructions configure the computing system to: determine that the voice command is ambiguous, wherein the first and second indications are obtained based on the determination that the voice command is ambiguous.
13. The computing system of claim 12, wherein the voice command is determined to be ambiguous based on at least one of: a determination that the voice command can be executed to perform more than one action; or a determination that more than one different application module can execute the voice command.
14. The computing system of claim 9, wherein each of the items in the list corresponds to an action performable by one of the first or second application modules.
15. The computing system of claim 9, wherein the instructions configure the computing system to: identify an action that corresponds to the item selected from the list; identify one of the first or second application module that corresponds to the identified action; and provide an indication of the identified action to the identified application module.
16. A computer-implemented method comprising: receiving, at an interface module, first information corresponding to a first set of commands associated with a first application module; receiving, at the interface module, second information corresponding to a second set of commands associated with a second application module; obtaining a voice command recognized using a speech recognizer that processes a speech input from a user; obtaining, based on the first information, a first indication of a correspondence between the voice command and a first command associated with the first application module, wherein the first indication is based on a determination that the voice command includes at least one word corresponding to a first object or a first action included in the first command; obtaining, based on the second information, a second indication of a correspondence between the voice command and a second command associated with the second application module, wherein the second indication is based on a determination that the voice command includes at least one word corresponding to a second object or a second action included in the second command; based on the first and second indications, generating a list of items, each item in the list corresponding to at least one of the first or second commands; rendering the list of items to the user; receiving an indication of a user selection of one of the items in the list, wherein the selected item corresponds to a selected one of the first or second command; and in response to the indication of the user selection, automatically executing the selected command associated with the first application module or the second application module.
17. The computer-implemented method of claim 16, wherein each of the items in the list is associated with an action performable by one of the first or second application modules, and wherein rendering the list of items comprises: generating a user interface display that visually renders the list and, for each item in the list, visually identifies which of the first or second application modules corresponds to the action that is associated with the item.
18. The computer-implemented method of claim 17, and further comprising: identifying an action that corresponds to the item selected from the list; identifying one of the first or second application module that corresponds to the identified action; and sending an indication of the identified action to the identified application module.
19. The computer-implemented method of claim 17, wherein the user interface display includes a user interface element that identifies, for each item in the list, which of the first or second application modules corresponds to the item.
20. The computer-implemented method of claim 16, and further comprising: determining that the voice command is ambiguous based on at least one of: the voice command being executable to perform more than one action; or the voice command being executable by more than one different application module; and wherein the voice command is provided to the first and second application modules based on the determination.
A computing device can often interpret a voice command in multiple ways. In one aspect, a computing device may not understand which application the voice command is directed towards. For example, a voice command can include terms that are used across a plurality of application modules. In another aspect, an application may not understand what the user would like to execute because the voice command contains ambiguous information. For example, a voice command can include "play the Beatles" when there are multiple Beatles albums that the application could play. In still another example, the voice command can contain a misrecognition error. For example, the user may recite the command "insert ice cream" while the speech recognition system recognizes the command as "insert I scream".
Memory 204 includes an operating system 212, application programs 214 as well as an object store 216. During operation, operating system 212 is preferably executed by processor 202 from memory 204. Operating system 212, in one preferred embodiment, is a WINDOWS.RTM. CE brand operating system commercially available from Microsoft Corporation. Operating system 212 is preferably designed for mobile devices, and implements database features that can be utilized by applications 214 through a set of exposed application programming interfaces and methods. The objects in object store 216 are maintained by applications 214 and operating system 212, at least partially in response to calls to the exposed application programming interfaces and methods.
Grammar generator module 309 can also populate grammar 308 with alternative forms of various commands. These alternative forms of commands typically are received from application modules 310. For example, if a user desires to play the Beatles on a computing device, the user should utter "play the Beatles". It is possible, however, that the user may instruct the computing device to "start the Beatles" instead of properly instructing the computing device to "play the Beatles", which would accomplish the same task.
In accordance with one embodiment of the present invention, some or all of the application modules 310 determine that they can execute the recognized voice command. For example, a user utters the command "start settings". However, many of the application modules 310 that are installed in system 300 could have application aspects that include the term "settings". Therefore the recognized command 310 is ambiguous. It is unclear as to which application the user is referring.
In accordance with another embodiment of the present invention, only one of the plurality of application modules 310 determines that it can execute the recognized voice command. For example, a user utters the command "play the Indigo Girls". Only one application module 310, such as a media application, has a menu item that includes "play the Indigo Girls". However, upon the media application attempting to execute the command, it discovers that there are four possible Indigo Girls albums that can be played. Therefore, the recognized command is ambiguous. It is unclear to the media application what album the user meant to play.
In accordance with yet another embodiment of the present invention and as previously discussed, there can be ambiguity in what instance or application module of a particular application the user is referencing. For example, a user is using a spreadsheet application and utters the command "switch to the Internet browser". However, besides system 300 running one instance of the spreadsheet application, system 300 is also running two instances of the Internet browser application. Therefore, the recognized command is ambiguous. It is unclear which instance of the Internet browser the user is referring.
Each interpretation in the list of interpretations 402 and 502 are annotated with an identifier (i.e. "1", "2", "3", etc.). In the FIG. 4 and FIG. 5 embodiment, identifiers 404 and 504 are numeric symbols, although any form of symbol can be used. Upon viewing the list of interpretations 402 and 502, the user can determine which of the interpretations is the interpretation that corresponds to what the user meant. In one embodiment, the user can select the correct command by speaking the corresponding numeric symbol. In another embodiment, the user can select the correct command by using an input device 316 (illustrated in FIG. 3) to select desired interpretations. Input device 316 can be, but is not limited to, a selection apparatus, such as a mouse. Input device 316 can also be a keypad. Regardless of how the correct interpretation is selected, selection of an interpretation induces system 300 to execute the corresponding command by forwarding the command to the corresponding application.
In another embodiment, the user can indicate, in reference to FIG. 5, that playing all of the Indigo Girls albums is desired. Communication line 506 indicates that the user can either select an interpretation from the plurality of interpretations 502 or the user can utter "Play all" to play all of the albums. Regardless of what the user desired, selection of the correct interpretation induces system 300 to take action by providing the command information to the corresponding application.
Using a centralized interface module that visually renders a list of interpretations to a user when there is otherwise ambiguity in a voice command provides a way for system 300 to avoid arbitrarily guessing an appropriate command. A centralized interface module also provides a simple centralized panel for graphically representing possible user interactions, provides a highly efficient means of determining what the user meant and provides a consistent interface that is located outside of the plurality of application modules. In other words, each of the screenshots of a centralized panel illustrated in FIGS. 4, 5 and 8-10 (discussed below), look and operate similarly and can, therefore, be displayed in a consistent location. For example, FIG. 6 illustrates a screenshot 600 of a media player that is operating on an operating system. The user asks to "play Indigo Girls". As discussed above, there is more than one album of Indigo Girls that could be played. The centralized interface or panel 500 appears in the lower right hand corner of the screen and prompts the user to select what the user meant. In this manner, the user is less confused as to how to operate with the display even though the centralized panel can be used with one or more application modules 310.
If the user selects identifier "9", then the centralized interface module 312 visually renders a screenshot 900 in a centralized panel as illustrated in FIG. 9. The communication line 902 of screenshot 900 prompts the user to respeak the utterance. Upon respeaking the utterance, centralized interface module 312 will visually render a refreshed list of alternatives. Selecting identifier "9" is useful if system 300 has made a recognition error. If the user selects the identifier "10" in FIG. 8, then centralized interface module 312 visually renders screenshot 1000 in a centralized panel as illustrated in FIG. 10. Communication line 1002 of screenshot 1000 prompts the user to spell the utterance aloud. As the user spells the utterance, the letters will appear in block 1004. In addition, it is possible that a user could also type the correct spelling of the utterance in block 1004 using an input device.
In another embodiment of the present invention, FIG. 11 is a block diagram illustrating a method 1100 for manipulating characters displayed on a centralized panel, such as screenshot 1000 of a centralized panel illustrated in FIG. 10, after the user audibly spells a new alternative. For example, in FIG. 12, the user spells the following phrase, "intent". However, the speech recognizer 306 (FIG. 3) has "heard" the series of entered text 1204 as "i", "n", "v", "e", "n", "t" and thus, displays the word "invent" on panel 1200. The word "invent" needs to be corrected to be "intent."
Referring to FIG. 13, in order to correct this, the user invokes a spelling command to vocally communicate a command to modify "invent". Upon the system 300 receiving the spelling command, the selected word is identified and an appended word 1302 is created by appending at least one space 1304 to the selected word, wherein the appended word 1302 includes at least one character 1306 to be modified, as shown in operational block 1102 (FIG. 11). Each of the characters 1308 in the appended word are then assigned and correlated with a unique numerical value 1310, as shown in operational block 1104. The appended word 1302 is then displayed via screenshot 1300 of a centralized panel to visually communicate the correlation between each of the characters 1308 in the appended word 1302 and their assigned unique numerical value 1310. This correlation may be visually communicated by drawing a box 1312 around the appended word 1302 on screenshot 1300 and displaying each of the unique numerical values 1310 adjacent their assigned character 1308 in the appended word 1302. As such, each of the characters 1308 is "assigned" a unique numerical value 1310 to which each character 1308 is correlated. For example, a user who wants to change the word "invent" 1314 to "intent", would vocally enter a command to speech recognizer 306. This would cause a box 1312 to be displayed on screenshot 1300 around the word "invent" 1314. This also causes each letter in the word "invent" 1314 to be assigned a unique numerical value 1310 which is displayed adjacent its corresponding character 1308, both of which is shown in FIG. 13. This will allow the user to change and/or correct any letter in the word "invent."
At this point, a selection command and a modification command may be received by system 300, wherein the selection command is the unique numerical value 1310 corresponding to a selected character, as shown in operational block 1106. This indicates to system 300 which character in the appended word 1302 is to be changed. Once system 300 receives the modification command a modified word is generated by modifying the selected character responsive to the modification command, as shown in operational block 1108. It should be appreciated that the user may enter in a plurality of modification commands, such as "delete", "insert", or the letter/character to be added, wherein each of these situations are discussed below.
For example, referring to FIG. 14, consider the case above where the user would like to delete the letter "v" 1402 in the appended word "invent_" 1404 displayed on a screenshot 1400 of a centralized panel. As discussed herein, the user communicates the unique numerical value corresponding to the character to be change to system 300. It should be appreciated that although these unique numerical values 1408 start with the number 1 and increase by 1, any unique numerical values 1408 and increments may be assigned. As can been seen, the letter "v" 1402 in the appended word "invent_" 1404 is assigned a unique numerical value 1408 of "3". As such, the user would vocally communicate the number "3" to system 300. This selects the letter corresponding to and correlated with the number "3" as indicated by the caret 1410, which in this case is the letter "v" 1402 in the appended word "invent" 1404. The user may then enter the desired modification command, such as "delete" which will cause the letter "v" 1402 to be deleted from the appended word "invent" 1404, leaving the resultant "inent" 1412, as shown in screenshot 1500 of a centralized panel of FIG. 15. Thus, the modification command "delete" will remove the selected letter and its corresponding space from the appended word and the selection caret 1410 will select the next subsequent character, i.e. "e."
On the other hand, referring to FIG. 16, consider the case where the user wants to insert a character, such as a letter or space, between the letter "n" 1604 and the letter "v" 1606 in the word "invent" displayed on screenshot 1600 in a centralized panel. In essence, the user wants to insert a character into the spot corresponding to the unique numerical value "3". As discussed herein, the user may vocally communicate a command. This causes a space to be appended to the word "invent" to create an appended word "invent_" 1608 and a box 1610 to be displayed around the appended word "invent_" 1608 and unique numerical values 1612 to be assigned and displayed adjacent each of the characters in the appended word "invent_" 1608. As can be seen, the letter "v" 1606 in the appended word "invent_" 1608 is assigned a unique numerical value 1612 of "3". As such, the user can vocally communicate the number "3" to system 300 to "select" the letter corresponding to and correlated with the number "3" as indicated by the caret 1614, which in this case is the letter "v" 1606 in the appended word "invent_" 1608. The user may then enter in a modification command, causing system 300 to respond in an appropriate manner. For example if the user communicates the modification command "insert" and then communicates the word "space", then a space will be inserted between the letter "n" 1604 and the letter "v" 1606, effectively changing the appended word "invent_" 1608 to "in_vent_" 1616, as shown in screenshot 1700 in a centralized panel illustrated in FIG. 17. In this case the caret 1614 would remain in place to indicate that the space correlated with the unique numerical value "3" has been selected. However, if the user communicates the command "insert" and then communicates the letter "p", then the letter "p" will be inserted between the letter "n" 1604 and the letter "v" 1606, effectively changing the appended word "invent_" to "inpvent_" 1618, as shown in screenshot 1800 of a centralized panel illustrated in FIG. 18, and the selection caret 1614 will shift to the following character to indicate that the following character (i.e. the character corresponding to the unique numerical value "4") has been selected. Similarly, referring to FIG. 19, consider the case where the user simply wants to change a letter in the word "invent" displayed on screenshot 1900 of a centralized panel. As discussed herein, the user may vocally communicate a command. This causes a space to be appended to the word "invent" to create an appended word "invent_" 1902 and a box 1904 to be displayed around the appended word "invent_" 1902 and unique numerical values 1906 to be assigned and displayed adjacent each of the characters 1908 in the appended word "invent_" 1902. As can been seen, the letter "v" 1912 in the appended word "invent_" 1902 is assigned a unique numerical value 1906 of "3." As such, the user can vocally communicate the number "3" to system 300. This causes the letter corresponding to and correlated with the number "3" to be selected as indicated by the caret 1910, which in this case is the letter "v" 1912 in the appended word "invent_" 1902. The user may then enter in a modification command (in this case the command is simply a letter) causing system 300 to respond in an appropriate manner. For example if the user communicates the modification command "t" after the number "3", then the letter "v" 1912 will be replaced with the letter "t", effectively changing the appended word "invent_" 1902 to the word "intent" 1914, as shown in screenshot 2000 of a centralized panel illustrated in FIG. 20. At this point the selection caret 1910 will shift to the following character to indicate that the following character (i.e. the character corresponding to the unique numerical value "4") has been selected.
It should be appreciated that once a user enters the unique numerical value corresponding to the letter to be changed, a menu of suggested modification commands may be displayed, such as a dropdown menu, where each suggested action would be assigned is own unique numerical value. For example, referring to FIG. 21, consider the case where the user wants to change a letter in the word "invent" displayed on screenshot 2100 of a centralized panel. The user vocally communicates a command. This causes a space to be appended to the selected word "invent" to create an appended word "invent_" 2102 and a box 2104 to be displayed around the appended word "invent_" 2102 and unique numerical values 2106 to be displayed adjacent each of the letters in the appended word "invent_" 2102. As can been seen, the letter "v" 2108 in the appended word "invent_" 2102 is assigned a unique numerical value 2106 of "3." As such, the user would vocally communicate the number "3" to system 300 to "select" the character corresponding to and correlated with the unique numerical value "3" as indicated by the caret 2110, which in this case is the letter "v" 2108 in the appended word "invent_" 2102. Referring to FIG. 22, a menu 2112 may be displayed on centralized panel shot 2200 giving the user a number of modification command choices, each of which is assigned a second unique numerical value 2114. The user may then enter a modification command which is the second unique numerical value 2114 correlated with the desired modification command causing the speech recognition software application to respond in an appropriate manner. For example, if the user communicates the numerical value "4" after the number "3" then the letter "v" 2108 will be replaced with the letter "d" 2116, effectively changing the appended word "invent_" 2102 to the word "indent" 2118, as shown in screenshot 2300 of a centralized panel in FIG. 23. As above, the selection caret 2110 will shift to the following character to indicate that the following character (i.e. the character corresponding to the unique numerical value "4") has been selected.
It should be appreciated that the menu 2112 of suggested modification commands may include any modification commands suitable to the desired end purpose, such as a menu of characters/words that are acoustically similar to the selected letter or word, e.g. if "v" is selected, then the menu 2112 will contain "d", "t", "e", "g", "3". Additionally, the menu 1012 may also contain the capitalized form of the characters, e.g. "V", as well as an auto-complete list from a spell checker. As such, for this example the menu 1012 may include the words "indent", "intent", "amend." Furthermore, it should be appreciated that the present invention may include voice commands that performing multiple functions simultaneously, e.g. "Change 3 to "e" as in eagle," or "Change t to g" may change the letter "t" to the letter "g" if there were only one letter "t" in the selected word. If there were two letters "t" in the selected word, feedback may be provided to the user to be more precise. Moreover, other commands may also be provided such as "undo" which may revert previously changed characters to their previous state, e.g. if the user says "cap that" (to capitalize the selected letter), but the input is recognized as "caret", then the user may say "undo" to revert the letter to the previous state.
With reference to FIGS. 8-23, these figures are discussed in the context of a user dictating to a word processing application. However, those skilled in the art will recognize that the embodiments discussed in FIGS. 8-23 can also apply to a user uttering a voice command to system 300. If a voice command is ambiguous, centralized interface module 312 will visually render a list of interpretations and also visually render a list of further alternatives. The list of further alternatives include an alternative to respeak the command and an alternative to create a new command. For example, if the user would like to send email to David White, the user can audibly issue the command "insert the name David". However, if the user has previously sent email to David Black and David Green, but never has sent email to David White, there is an ambiguity as to what David to insert. Centralized interface module 312 visually renders a list of David interpretations. The list of interpretations includes David Black and David Green. In addition, centralized interface module 312 visually renders a list of further alternatives that includes an alternative to respeak the voice command and an alternative to create a new voice command. In this example, the user would choose to create a new command because the user desires to email David White, which is unfamiliar to system 300.
Previous Patent US 9,972,316 | Next Patent US 9,972,318