Source: https://patents.google.com/patent/US9600229B2/en
Timestamp: 2019-04-22 17:44:34+00:00

Document:
A method for receiving processed information at a remote device is described. The method includes transmitting from the remote device a verbal request to a first information provider and receiving a digital message from the first information provider in response to the transmitted verbal request. The digital message includes a symbolic representation indicator associated with a symbolic representation of the verbal request and data used to control an application. The method also includes transmitting, using the application, the symbolic representation indicator to a second information provider for generating results to be displayed on the remote device.
This application is a continuation application of and claims priority to U.S. application Ser. No. 14/150,134, filed on Jan. 8, 2014, which is a continuation of and claims priority to U.S. application Ser. No. 11/695,333, filed on Apr. 2, 2007, the entire contents of which are hereby incorporated by reference.
Implementations are described that relate to information retrieval, and more particularly to information retrieval based on verbal requests.
As computers and computer networks become more and more able to access a wide variety of information, people are demanding more ways to obtain that information. Specifically, people now expect to have access, on the road, in the home, or in the office, to information previously available only from a permanently-connected personal computer hooked to an appropriately provisioned network. They want graphical maps of locations from their cell phones, access to calendars from their personal digital assistants (PDAs), up-to-date contact information from their email devices, and timely, accurate search results from all their devices. They also want all of this information when traveling, whether locally, domestically, or internationally, in an easy-to-use, portable device.
Some systems provide information to a user when the user calls from a cell phone. For example, a user interested in the address for a particular business can call a directory service, where a customer service representative can provide the user with the business's address. This information, however, may not be provided in graphical form, for example, as a map. Additionally, other useful information, such as directions to the location and other similar businesses nearby may not be provided by the customer service representative. Also, the user may have to wait until a customer service representative is available to take the user's call.
In other systems, a user may enter a request for information, such as a map of a business location, by inputting the request using a keypad of a mobile device. Portability, however, generally requires a device small in size, which in turn limits the number of data entry keys and the amount of memory and available processing power. In addition, ultra portable devices often must be held in one hand or not held at all, so that data entry must be one-handed or no-handed. This can make entering information to obtain rich content, such as graphical maps, difficult.
Implementations are described that relate to information retrieval based on verbal requests.
In a first general aspect, a method for receiving processed information at a remote device is described. The method includes transmitting from the remote device a verbal request to a first information provider and receiving a digital message from the first information provider in response to the transmitted verbal request. The digital message includes a symbolic representation indicator associated with a symbolic representation of the verbal request and data used to control an application. The method also includes transmitting, using the application, the symbolic representation indicator to a second information provider for generating results to be displayed on the remote device.
In a second general aspect, a method for providing a user with location-based information is described. The method includes receiving a verbal request for locational information from a mobile device and converting the verbal request into a symbolic representation. The method also includes generating a message including the symbolic representation and an indicator used to control an application on the mobile device. Additionally, the method includes transmitting the message to the mobile device to cause the application to use the symbolic representation to display one or more locations associated with the symbolic representation.
In another general aspect, a method for receiving location-based information is described. The method includes transmitting from a mobile device a verbal request for locational information to a first information provider and receiving a Short Message Service message from the first information provider including a symbolic representation of the verbal request and an identifier used to control an application on the mobile device. The method also includes transmitting a data request including the symbolic representation of the verbal request for locational information to a second information provider using the application on the mobile device and displaying on the mobile device one or more locations associated with the symbolic representation using locational information transmitted by the second information provider in response to the data request.
In yet another general aspect, a method for providing processed data to a remote device is described. The method includes receiving a verbal request from the remote device and generating a data response for the verbal request. The data response includes data for controlling an application on the remote device and a conversion of the verbal request, wherein the converted verbal request is accessible by the application. The method also includes transmitting the data response to the remote device to cause the remote device to control the application, process at least a portion of the converted verbal request, and display results generated from the processing.
In another general aspect, a system for providing location-based information to a plurality of users is described. The system includes an interface for receiving verbal requests over a voice transmission line from a plurality of remote devices, a conversion module for converting the verbal requests into a symbolic representation of the verbal request, and means for outputting digital messages to the remote devices. Each message includes an identifier used to control an application on a remote device and a symbolic representation indicator used by the application to process the symbolic representation.
The systems and techniques described here may provide one or more of the following advantages. First, a system may increase the amount and type of information accessible to a remote device by passing, to an application on the remote device, seed information used to derive additional information. Second, a system decreases inconvenience of text entry on mobile devices. Third, a system provides increased efficiency and accuracy by offloading voice-recognition functions from a mobile device to a server. Fourth, a system increases efficiency because it enables transportation protocols of limited size be used to provide rich content to mobile devices.
The details of one or more implementations of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims.
FIG. 1 is a schematic diagram depicting an example of a system for processing a verbal request to present information on a remote device.
FIG. 2 is a schematic diagram of a system 200 depicting an automated implementation of the system 100 of FIG. 1.
FIG. 3 is a schematic diagram of a system 300 depicting a partially manual implementation of the system of FIG. 1.
FIG. 4 is a sequence diagram depicting an example of interactions between a directory service provider, a cellular phone, and a search engine.
FIG. 5 is a flow chart depicting an example of a method for displaying on a remote device information derived from a verbal request.
FIG. 6 is a schematic diagram depicting an example of a general computer system and a general mobile device that may be used in the systems and methods described in FIGS. 1-5.
System 100 can translate a verbal request 102 from a remote device, such as a voice telephone call from a cell phone, into a symbolic representation, such as text, which can be used by an application on the remote device to initiate a query for information, for example, a search query.
FIG. 1 is a schematic diagram depicting an example of the system 100 for processing the verbal request 102 to present information on the remote device 104. The system 100 includes a first information provider 106 and a second information provider 108. In certain implementations, the first information provider 106 is a directory assistance service, such as a “411” service, and the verbal request 102 is a call to the directory assistance service. In these and other implementations, the second information provider may be a search engine that processes search queries and returns search results based on the queries.
The remote device 104 (e.g., a cellular phone, a personal digital assistant, or an e-mail device) transmits the verbal request 102 to the first information provider 106. The verbal request 102 may include, for example, a request for a list of restaurants of a particular type in a particular location (e.g., “Pizza, Palo Alto”).
The first information provider 106 receives the verbal request 102 and converts the verbal request 102 into a symbolic representation. For example, the first information provider 106 may perform a speech-to-text operation on the verbal request 102, which converts the verbal request to ASCII (“American Standard Code for Information Interchange” text. In other examples, the first information provider 106 can convert the verbal request into other symbolic representations, such as binary or hexadecimal, which the remote device 104 can use in a query 114 described below.
In other implementations, the first information provider 106 can convert the verbal request to character sets other than ASCII. For example, in an international context, the verbal request can be converted into 150-8858-1 or UTF-8 (8-bit Unicode Transformation Format).
The first information provider 106 transmits a symbolic representation indicator 110 to the remote device 104 that indicates what the symbolic representation is or from where the symbolic representation may be retrieved. For example, the symbolic representation indicator can be substantially the same as the converted symbolic representation, or the symbolic representation indicator can be a pointer or address location (e.g., uniform resource locator) of the converted symbolic representation.
Additionally, the first information provider 106 transmits an application identifier 112 to the remote device 104. The remote device 104 uses the application identifier to launch an application, such as a mapping and directions application. The remote device 104 uses the application to transmit the symbolic representation indicator to the second information provider 108. For example, the remote device 104 may send the query 114 to the second information provider 108, including the symbolic representation indicator 110, requesting a map of restaurants in the Palo Alto area that serve pizza.
The second information provider 108 processes the query 114 and transmits results 116 of processing the query 114 to the remote device 104. For example, the second information provider 108 may transmit a map to the remote device 104 that highlights restaurants in Palo Alto that serve pizza. The remote device 104 receives the results 116 and presents displayed results 118, such as the map of restaurants, to a user.
Combining the examples above, a cell phone 104 can transmit a “411” request for “Pizza, Palo Alto,” as shown by arrow A. The request is received by a directory assistance provider 106, which converts the verbal request to text 110 and transmits the text 110 along with a binary message used to launch a mapping program on the remote device 104, as shown by arrow B. The remote device 104 receives the binary message, launches the mapping program by executing the binary message, and uses the mapping program to transmit to a mapping server 108 a query “Pizza, Palo Alto” 118, as shown by arrow C. The mapping server 108 processes the query and returns results 116 including a map highlighting pizza restaurants in Palo Alto, Calif., as shown by arrow D. Then remote device displays the map to the user.
FIG. 2 is a schematic diagram of a system 200 depicting an automated implementation of the system 100 of FIG. 1. The remote device 104, the first information provider 106, and the second information provider 108 are in communication via a network 202, such as a local area network, a wide area network, the Internet, a telephone network, or a combination of networks.
The first information provider 106 receives the verbal request 102 via an interface 204. In certain implementations, the remote device 104 transmits a recorded message as the verbal request 102. For example, a user may record the request before connection to the first information provider. In other implementations, the verbal request is made after an established connection with the first service provider 106.
The first information provider 106 includes a speech-to-text converter 206 that converts the verbal request 102 into text. An application selector 208 selects the application to be launched at the remote device 104 based on, for example, the content of the converted verbal request. For example, the application selector 208 may determine from the location name, “Palo Alto,” that the user is requesting a map of locations within Palo Alto. The application selector 208 generates the application identifier 112 based on the selected application. The application identifier 112, when received at the remote device 104, instructs the remote device 104 to launch the selected application.
In other implementations, the application selector is optional because the application identifier 112 is predetermined so that every verbal request is related to a single application. For example, every request made to the first service provider may be for mapping locations, so the application identifier 112 may be binary code that always launches a mapping application.
A message generator 210 uses the converted verbal request to generate a digital message 212 for transmission to the remote device. In certain implementations, the digital message 212 includes the application identifier 112 and the symbolic representation indicator. The first information provider 106 transmits the digital message 212 to the remote device 104. In certain implementations, the message generator 210 uses a Short Message Service (SMS) formatter 214 to format the digital message 212 as an SMS message. The application identifier 112 and/or the symbolic representation indicator 110 may include binary values or American Standard Code for Information Interchange (ASCII) values.
In certain implementations, the first information provider 106 includes a search engine 216 capable of performing a search using the symbolic representation of the verbal request 102. The search engine 216 outputs the search results to the message generator 210. The message generator 210 includes partial results 218 for the verbal request 102 in the digital message 212. For example, the partial results 218 may include in the SMS message an address for a pizza parlor, which was returned as the top result for the verbal request “Pizza, Palo Alto.” In another example, the partial results 281 include a list of the top 3, 5, or 10 results. A user may make a selection from the partial results 218 to be used in the query 114 to the second information provider 108, such as a particular pizza restaurant in Palo Alto to highlight on a map instead of all pizza restaurants in Palo Alto.
The remote device 104 uses the application identifier 112 to launch an application 220. In certain implementations, the application 220 is a calendar application and the verbal request 102 is a request to manage event information in a calendar. In other implementations, the application 220 is a mapping application and the verbal request 102 is a request to display one or more locations on a map. The locations may be identified, for example, by keywords (e.g., pizza) and/or addresses (e.g., Palo Alto). In still other implementations, the application 220 is a contact application and the verbal request 102 is a request to manage contact information. In other implementations, the application 220 is a search result user interface (e.g., web browser accessing an internet search page) and the verbal request 102 is a search query.
After launching, the application 220 formats the symbolic representation for inclusion in the query 114. The remote device 104 transmits the query 114, including the symbolic representation indicator 110, to the second information provider 108.
The second information provider 108 includes a symbolic representation processor 222 that processes the symbolic representation of the verbal request 102 received in the query 114. The symbolic representation processor 222 includes a search engine 224 that performs a search using the query 114, such as a search of pizza restaurants.
In certain implementations, the first information provider 106 places a symbolic representation 226 of the verbal request 102 in a storage location 228 and the symbolic representation indicator 110 indicates the location of the symbolic representation 226, such as with a Uniform Resource Locator (URI). For example, where the symbolic representation 226 of the verbal request is too large to be included in an SMS message, the symbolic representation 226 may be placed in the storage location 228. The second information provider 108 retrieves the symbolic representation 226 using the symbolic representation indicator 110 and processes the query 114 using the symbolic representation 226.
The remote device 104 receives the results 116 from the second information provider 108 and presents the displayed results 118 to a user. In certain implementations, results including map locations may also include directions to the locations. To calculate the directions, the starting location may be input by a user of the remote device 104, or the starting location may be provided by a global positioning component of the remote device 104. Additionally, a user may enter input which causes a displayed map to show more or less detail of the map, such as by zooming in or out on the map.
FIG. 3 is a schematic diagram of a system 300 depicting a partially manual implementation of the system of FIG. 1. In this example, a person 302 at the first information provider 104 receives the verbal request 102. Here, the interface 204 includes a telephone interface. The person 302 translates the verbal request 102 and inputs the symbolic representation indicator 110 into a computer device 304 using an input device 306, such as a keyboard, pointing device, or touch screen. In this example, the person 302 performs the operations of the speech-to-text converter 206. Additionally, the person 302 can input the application identifier 112 and/or the partial results 218 for the verbal request 102 performing the operations of the application selector 208 and the search engine 216, respectively.
In other implementations, the user may correct the translation of the verbal request by verbally repeating the request, by typing the text using a keypad, or by selecting from a list of possible translations of the verbal request.
The directory service provider 402 generates a binary SMS message 414 and transmits the message 414 to the cellular phone 404. The message 414 includes text, such as “Pizza, Palo Alto,” and an application identifier, such as binary code used to launch or execute a function of an internet browser or a custom application (e.g., Google Maps for Mobile developed by Google™ of Mountain View, Calif.) installed on the cellular phone 404.
The search engine 406 generates location search results 420 using the symbolic representation in the data request 418 (or from the storage location 228 as previously described). For example, the search engine 406 may determine that “Pizza” indicates a request for pizza restaurants and “Palo Alto” further indicates a request for pizza restaurant locations in the Palo Alto area. The search engine 406 transmits search results 422 to the cellular phone 404. The cellular phone 404 receives the results 422 and displays locations 424 using the map user interface.
FIG. 5 is a flow chart depicting an example of a method for displaying on a remote device information derived from a verbal request. The process 500 may be performed, for example, by a system such as the systems 100, 200, and 400. For clarity of presentation, the description that follows uses the systems 100, 200, and 400 as the basis of an example for describing the process 500. However, another system, or combination of systems, may be used to perform the process 500.
The process 500 begins with transmitting (502) a verbal request to a first information provider. For example, the remote device 104 transmits the verbal request 102, including “Pizza, Palo Alto,” to the first information provider 106 via the network 202.
Optionally, a confirmation of the verbal to symbolic conversion is transmitted (506). For example, a user replies “Yes, I did,” which is transmitted by the cellular phone 404 to the directory service provider 402.
An application identifier and a symbolic representation indicator is received (508). For example, the remoter device 104 receives the digital message 212 including the application identifier 112 and the symbolic representation indicator 110.
An application specified by the application identifier is launched (510). For example, the remote device 104 launches the application 220 specified by the application identifier 112.
If partial results are received in response to the verbal request 512, then the results an be displayed (514). If a user requests more results or the partial results were not received (516), then the process 500 transmits (518) the symbolic representation indicator to a second information provider.
For example, the remote device 104 may present the partial results 218 including addresses for locations associated with the verbal request and receive a user input requesting more results. If a user requests more results or partial results were not transmitted, the remote device 104 transmits the symbolic representation indicator 110, including “Pizza, Palo Alto,” to a search engine that returns mapped locations in response to location queries.
Processed results are received from the second information provider (520). For example, the remote device 104 receives the results 116 from the second information provider 108, including a map with locations of pizza restaurants in Palo Alto highlighted.
FIG. 6 is a block diagram of computing devices 600, 650 that may be used to implement the systems and methods described in this document, either as a client, as a server, or as a plurality of servers. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.
The high speed controller 608 manages bandwidth-intensive operations for the computing device 600, while the low speed controller 612 manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In one implementation, the high-speed controller 608 is coupled to memory 604, display 616 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 610, which may accept various expansion cards (not shown). In the implementation, low-speed controller 612 is coupled to storage device 606 and low-speed expansion port 614. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.
Processor 652 may communicate with a user through control interface 658 and display interface 656 coupled to a display 654. The display 654 may be, for example, a TFT LCD display or an OLED display, or other appropriate display technology. The display interface 656 may comprise appropriate circuitry for driving the display 654 to present graphical and other information to a user. The control interface 658 may receive commands from a user and convert them for submission to the processor 652. In addition, an external interface 662 may be provide in communication with processor 652, so as to enable near area communication of device 650 with other devices. External interface 662 may provide, for example, for wired communication (e.g., via a docking procedure) or for wireless communication (e.g., via Bluetooth or other such technologies).
The memory 664 stores information within the computing device 650. In one implementation, the memory 664 is a computer-readable medium. In one implementation, the memory 664 is a volatile memory unit or units. In another implementation, the memory 664 is a non-volatile memory unit or units. Expansion memory 674 may also be provided and connected to device 650 through expansion interface 672, which may include, for example, a SIMM card interface. Such expansion memory 674 may provide extra storage space for device 650, or may also store applications or other information for device 650. Specifically, expansion memory 674 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory 674 may be provide as a security module for device 650, and may be programmed with instructions that permit secure use of device 650. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.
The memory may include for example, flash memory and/or MRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 664, expansion memory 674, memory on processor 652, or a propagated signal.
Device 650 may also communication audibly using audio codec 660, which may receive spoken information from a user and convert it to usable digital information. Audio codex 660 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device 650. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device 650.
The computing device 650 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 680. It may also be implemented as part of a smart phone 682, personal digital assistant, or other similar mobile device.
A number of embodiments of the invention have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. For example, the first information provider 106 and the second information provider 108 may be the same system or may be included in a single system.
Additionally, the application identifier 112 does not have to launch the application. Instead, it can be used to initiate a function of an application that has been previously launched. For example, the application identifier can indicate that the a previously launched internet browser should navigate to a particular site identified by a URL included in the digital message 212.
In some implementations, the request 102 is a written request and the remote device 104 that transmits the request 102 is a personal computer, and the request 102 can be passed directly to the search engine 216 and the application selector without speech-to-text conversion.
Although SMS is used as an example in the above descriptions, in other implementations, the described systems and method may use a variety of mechanisms to transmit information to the remote device 104. For example, the first information provider 106 can use actual push mechanisms, such as session-initiated protocol (SIP) and SMS transmissions, or virtual push mechanisms, such as polling using hypertext transmission protocol (HTTP).
Additionally, in some implementations, the request 102 can split up into multiple requests. For example, the request 102 can be a two-phase interaction, where a user of the remote device requests a location first (e.g., Dallas, Tex.), which is confirmed by the first information provider. Next, the user transmits a second portion of the request, which specifies a particular category or item for which to search (e.g., barbeque).
In yet other implementations, various forms of the flows shown above may be used, with steps re-ordered, added, or removed. Also, although several applications of the systems and methods have been described, it should be recognized that numerous other applications are contemplated. Accordingly, other embodiments are within the scope of the following claims.
transmitting, using the application and to a second information provider different than the first information provider, a query based on the symbolic representation, such that the second information provider generates results, responsive to the query, for display by the computing device.
requesting confirmation from the user that the symbolic representation is accurate.
3. The method of claim 1, wherein the results include a map that indicates a location to which the verbal request was directed.
4. The method of claim 1, wherein the results include directions to a location to which the verbal request was directed.
5. The method of claim 4, wherein the directions indicate a starting location provided by the verbal request.
6. The method of claim 1, wherein the results indicate one or more street addresses for locations that correspond to the verbal request.
7. The method of claim 1, wherein the verbal request comprises a recorded message.
displaying, at the computing device, a map of a location associated with the user selection.
transmitting, using the application and to a second information provider different than the first information provider, a query based on the symbolic representation, such that the second information provider generates results, responsive to the query, for display by a computing device.
11. The system of claim 9, wherein the results include a map that indicates a location to which the verbal request was directed.
12. The system of claim 11, wherein the results include directions to a location to which the verbal request was directed.
13. The system of claim 12, wherein the directions indicate a starting location provided by the verbal request.
14. The system of claim 9, wherein the results indicate one or more street addresses for locations that correspond to the verbal request.
15. The system of claim 9, wherein the verbal request comprises a recorded message.
17. The tangible computer-readable media of claim 16, wherein the operations further comprise requesting confirmation from the user that the symbolic representation is accurate.
18. The tangible computer-readable media of claim 16, wherein the results include a map that indicates a location to which the verbal request was directed.
19. The tangible computer-readable media of claim 16, wherein the results include directions to a location to which the verbal request was directed.
20. The tangible computer-readable media of claim 19, wherein the directions indicate a starting location provided by the verbal request.
"Voice Activated Meeting Scheduling Mechanism," IBM Technical Disclosure Bulletin, International Business Machines Corp, 1994, 37(4A):9.
Australian Office Action in Australian Application No. 2008232435, dated Apr. 4, 2013, 3 pages.
Australian Office Action in Australian Application No. 2008232435, dated Oct. 3, 2013, 4 pages.
Australian Office Action in Australian Application No. 2014201912, dated Apr. 24, 2015, 3 pages.
European Search Report in European Application No. 08744910.4, dated Dec. 12, 2012, 9 pages.
Examination Report in Australian Application No. 2008232435, dated Mar. 30, 2012, 9 pages.
Glaser, M. and W.D. Tucker (2001). Web-based Telephony Bridges for the Deaf. Proceedings of the South African Telecommunications Networks & Applications Conference (SATNAC 2001), Wild Coast Sun, South Africa (5 pages.
Gruenstein et al., "Scalable and Protable Web-Based Multimodal Dialogue Interaction with Geographical Databases," in: Proc. Interspeech, 2006, pp. 453-456.
Hu et al., "Speech-Enabled Web Services for Mobile Devices," in: Proc. The International Conference on Semantic Web and Webservices, 2006, 7 pages.
Japanese Office Action in Japanese Application No. 2010-502253, dated Jan. 15, 2013, 12 pages.
Johnston et al, "Multimodal Language processing for mobile information Access," ICSLP 2002: 7th International Conference on Spoken Language Processing, 2002, 2237-2240.
Key Pousttchi, Martin Schurig, "Assessment of Today's Mobile Banking Applications from the View of Customer Requirements," hicss, p. 70184a, Proceedings of the 37th Annual Hawaii International Conference on System Sciences (HICSS'04)-Track 7, 2004 (10 pages).
Translation of Chinese Office Action in Chinese Application No. 200880017420.3, dated Nov. 30, 2011, 13 pages.

References: Application No. 2008232435
 Application No. 2008232435
 Application No. 2014201912
 Application No. 08744910
 Application No. 2008232435
 Application No. 2010
 Application No. 200880017420