Patent Description:
Implementations are described that relate to information retrieval, and more particularly to information retrieval based on verbal requests.

As computers and computer networks become more and more able to access a wide variety of information, people are demanding more ways to obtain that information. Specifically, people now expect to have access, on the road, in the home, or in the office, to information previously available only from a permanently-connected personal computer hooked to an appropriately provisioned network. They want graphical maps of locations from their cell phones, access to calendars from their personal digital assistants (PDAs), up-to-date contact information from their email devices, and timely, accurate search results from all their devices. They also want all of this information when traveling, whether locally, domestically, or internationally, in an easy-to-use, portable device.

Some systems provide information to a user when the user calls from a cell phone. For example, a user interested in the address for a particular business can call a directory service, where a customer service representative can provide the user with the business's address. This information, however, may not be provided in graphical form, for example, as a map. Additionally, other useful information, such as directions to the location and other similar businesses nearby may not be provided by the customer service representative. Also, the user may have to wait until a customer service representative is available to take the user's call.

In other systems, a user may enter a request for information, such as a map of a business location, by inputting the request using a keypad of a mobile device. Portability, however, generally requires a device small in size, which in turn limits the number of data entry keys and the amount of memory and available processing power. In addition, ultra portable devices often must be held in one hand or not held at all, so that data entry must be one-handed or no- handed. This can make entering information to obtain rich content, such as graphical maps, difficult. United States Patent Application Publication No. <CIT> presents systems and methods for receiving natural language queries and/or commands and executing the queries and/or commands. United States Patent Application Publication No. <CIT> presents a voice control application which transmits data to a voice server about a first execution state of an application program resident in the memory of an electronic device having a first graphical display.

Implementations are described that relate to information retrieval based on verbal requests.

The systems and techniques described here may provide one or more of the following advantages. First, a system may increase the amount and type of information accessible to a remote device by passing, to an application on the remote device, seed information used to derive additional information. Second, a system decreases inconvenience of text entry on mobile devices. Third, a system provides increased efficiency and accuracy by offloading voice-recognition functions from a mobile device to a server. Fourth, a system increases efficiency because it enables transportation protocols of limited size be used to provide rich content to mobile devices.

The details of one or more implementations of the invention are set forth in the accompanying drawings and the description below.

System <NUM> can translate a verbal request <NUM> from a remote device, such as a voice telephone call from a cell phone, into a symbolic representation, such as text, which can be used by an application on the remote device to initiate a query for information, for example, a search query.

<FIG> is a schematic diagram depicting an example of the system <NUM> for processing the verbal request <NUM> to present information on the remote device <NUM>. The system <NUM> includes a first information provider <NUM> and a second information provider <NUM>. In certain implementations, the first information provider <NUM> is a directory assistance service, such as a "<NUM>" service, and the verbal request <NUM> is a call to the directory assistance service. In these and other implementations, the second information provider may be a search engine that processes search queries and returns search results based on the queries.

The remote device <NUM> (e.g., a cellular phone, a personal digital assistant, or an e-mail device) transmits the verbal request <NUM> to the first information provider <NUM>. The verbal request <NUM> may include, for example, a request for a list of restaurants of a particular type in a particular location (e.g., "Pizza, Palo Alto").

The first information provider <NUM> receives the verbal request <NUM> and converts the verbal request <NUM> into a symbolic representation. For example, the first information provider <NUM> may perform a speech-to-text operation on the verbal request <NUM>, which converts the verbal request to ASCII ("American Standard Code for Information Interchange" text. In other examples, the first information provider <NUM> can convert the verbal request into other symbolic representations, such as binary or hexadecimal, which the remote device <NUM> can use in a query <NUM> described below.

In other implementations, the first information provider <NUM> can convert the verbal request to character sets other than ASCII. For example, in an international context, the verbal request can be converted into ISO-<NUM>-<NUM> or UTF-<NUM> (<NUM>-bit Unicode Transformation Format).

The first information provider <NUM> transmits a symbolic representation indicator <NUM> to the remote device <NUM> that indicates what the symbolic representation is or from where the symbolic representation may be retrieved. For example, the symbolic representation indicator can be substantially the same as the converted symbolic representation, or the symbolic representation indicator can be a pointer or address location (e.g., uniform resource locator) of the converted symbolic representation.

Additionally, the first information provider <NUM> transmits an application identifier <NUM> to the remote device <NUM>. The remote device <NUM> uses the application identifier to launch an application, such as a mapping and directions application. The remote device <NUM> uses the application to transmit the symbolic representation indicator to the second information provider <NUM>. For example, the remote device <NUM> may send the query <NUM> to the second information provider <NUM>, including the symbolic representation indicator <NUM>, requesting a map of restaurants in the Palo Alto area that serve pizza.

The second information provider <NUM> processes the query <NUM> and transmits results <NUM> of processing the query <NUM> to the remote device <NUM>. For example, the second information provider <NUM> may transmit a map to the remote device <NUM> that highlights restaurants in Palo Alto that serve pizza. The remote device <NUM> receives the results <NUM> and presents displayed results <NUM>, such as the map of restaurants, to a user.

Combining the examples above, a cell phone <NUM> can transmit a "<NUM>" request for "Pizza, Palo Alto," as shown by arrow A. The request is received by a directory assistance provider <NUM>, which converts the verbal request to text <NUM> and transmits the text <NUM> along with a binary message used to launch a mapping program on the remote device <NUM>, as shown by arrow B. The remote device <NUM> receives the binary message, launches the mapping program by executing the binary message, and uses the mapping program to transmit to a mapping server <NUM> a query "Pizza, Palo Alto" <NUM>, as shown by arrow C. The mapping server <NUM> processes the query and returns results <NUM> including a map highlighting pizza restaurants in Palo Alto, CA, as shown by arrow D. Then remote device displays the map to the user.

<FIG> is a schematic diagram of a system <NUM> depicting an automated implementation of the system <NUM> of <FIG>. The remote device <NUM>, the first information provider <NUM>, and the second information provider <NUM> are in communication via a network <NUM>, such as a local area network, a wide area network, the Internet, a telephone network, or a combination of networks.

The first information provider <NUM> receives the verbal request <NUM> via an interface <NUM>. In certain implementations, the remote device <NUM> transmits a recorded message as the verbal request <NUM>. For example, a user may record the request before connection to the first information provider. In other implementations, the verbal request is made after an established connection with the first service provider <NUM>.

The first information provider <NUM> includes a speech-to-text converter <NUM> that converts the verbal request <NUM> into text. An application selector <NUM> selects the application to be launched at the remote device <NUM> based on the content of the converted verbal request. For example, the application selector <NUM> may determine from the location name, "Palo Alto," that the user is requesting a map of locations within Palo Alto. The application selector <NUM> generates the application identifier <NUM> based on the selected application. The application identifier <NUM>, when received at the remote device <NUM>, instructs the remote device <NUM> to launch the selected application.

In other examples that do not form part of the invention, the application selector is optional because the application identifier <NUM> is predetermined so that every verbal request is related to a single application. For example, every request made to the first service provider may be for mapping locations, so the application identifier <NUM> may be binary code that always launches a mapping application.

A message generator <NUM> uses the converted verbal request to generate a digital message <NUM> for transmission to the remote device. In certain implementations, the digital message <NUM> includes the application identifier <NUM> and the symbolic representation indicator. The first information provider <NUM> transmits the digital message <NUM> to the remote device <NUM>. In certain implementations, the message generator <NUM> uses a Short Message Service (SMS) formatter <NUM> to format the digital message <NUM> as an SMS message. The application identifier <NUM> and/or the symbolic representation indicator <NUM> may include binary values or American Standard Code for Information Interchange (ASCII) values.

In certain implementations, the first information provider <NUM> includes a search engine <NUM> capable of performing a search using the symbolic representation of the verbal request <NUM>. The search engine <NUM> outputs the search results to the message generator <NUM>. The message generator <NUM> includes partial results <NUM> for the verbal request <NUM> in the digital message <NUM>. For example, the partial results <NUM> may include in the SMS message an address for a pizza parlor, which was returned as the top result for the verbal request "Pizza, Palo Alto. " In another example, the partial results <NUM> include a list of the top <NUM>, <NUM>, or <NUM> results. A user may make a selection from the partial results <NUM> to be used in the query <NUM> to the second information provider <NUM>, such as a particular pizza restaurant in Palo Alto to highlight on a map instead of all pizza restaurants in Palo Alto.

The remote device <NUM> uses the application identifier <NUM> to launch an application <NUM>. In certain implementations, the application <NUM> is a calendar application and the verbal request <NUM> is a request to manage event information in a calendar. In other implementations, the application <NUM> is a mapping application and the verbal request <NUM> is a request to display one or more locations on a map. The locations may be identified, for example, by keywords (e.g., pizza) and/or addresses (e.g., Palo Alto). In still other implementations, the application <NUM> is a contact application and the verbal request <NUM> is a request to manage contact information. In other implementations, the application <NUM> is a search result user interface (e.g., web browser accessing an internet search page) and the verbal request <NUM> is a search query.

After launching, the application <NUM> formats the symbolic representation for inclusion in the query <NUM>. The remote device <NUM> transmits the query <NUM>, including the symbolic representation indicator <NUM>, to the second information provider <NUM>.

The second information provider <NUM> includes a symbolic representation processor <NUM> that processes the symbolic representation of the verbal request <NUM> received in the query <NUM>. The symbolic representation processor <NUM> includes a search engine <NUM> that performs a search using the query <NUM>, such as a search of pizza restaurants.

In certain implementations, the first information provider <NUM> places a symbolic representation <NUM> of the verbal request <NUM> in a storage location <NUM> and the symbolic representation indicator <NUM> indicates the location of the symbolic representation <NUM>, such as with a Uniform Resource Locator (URI). For example, where the symbolic representation <NUM> of the verbal request is too large to be included in an SMS message, the symbolic representation <NUM> may be placed in the storage location <NUM>. The second information provider <NUM> retrieves the symbolic representation <NUM> using the symbolic representation indicator <NUM> and processes the query <NUM> using the symbolic representation <NUM>.

The remote device <NUM> receives the results <NUM> from the second information provider <NUM> and presents the displayed results <NUM> to a user. In certain implementations, results including map locations may also include directions to the locations. To calculate the directions, the starting location may be input by a user of the remote device <NUM>, or the starting location may be provided by a global positioning component of the remote device <NUM>. Additionally, a user may enter input which causes a displayed map to show more or less detail of the map, such as by zooming in or out on the map.

<FIG> is a schematic diagram of a system <NUM> depicting a partially manual implementation of the system of <FIG>. In this example, a person <NUM> at the first information provider <NUM> receives the verbal request <NUM>. Here, the interface <NUM> includes a telephone interface. The person <NUM> translates the verbal request <NUM> and inputs the symbolic representation indicator <NUM> into a computer device <NUM> using an input device <NUM>, such as a keyboard, pointing device, or touch screen. In this example, the person <NUM> performs the operations of the speech-to-text converter <NUM>. Additionally, the person <NUM> can input the application identifier <NUM> and/or the partial results <NUM> for the verbal request <NUM> performing the operations of the application selector <NUM> and the search engine <NUM>, respectively.

<FIG> is a sequence diagram depicting an example of interactions <NUM> between a directory service provider <NUM>, a cellular phone <NUM>, and a search engine <NUM>. The cellular phone <NUM> transmits a verbal request <NUM> to the directory service provider, such as "Pizza, Palo Alto.

The directory service provider <NUM> receives the verbal request <NUM> and, using a speech-to-text conversion, converts the verbal request <NUM> into a symbolic representation. In certain implementations, the directory service provider <NUM> transmits a confirmation request <NUM> to the cellular phone <NUM>. The confirmation request <NUM> can include computer-generated speech based on the symbolic representation, such as "Did you say, 'Pizza, Palo Alto?"' In other implementations, the directory service provider <NUM> may transmit a confirmation request <NUM> including a text query containing the symbolic representation. For example, the confirmation request <NUM> may be an SMS message with the text "Did you say, 'Pizza, Palo Alto?" The cellular phone <NUM> transmits a confirmation response <NUM> to the directory service provider <NUM> from the user, such as "Yes, I did. " The confirmation response <NUM> may be verbal or symbolic, such pressing the key '<NUM>' on a keypad to indicate "Yes" and pressing the key '<NUM>' to indicate "No.".

In other implementations, the user may correct the translation of the verbal request by verbally repeating the request, by typing the text using a keypad, or by selecting from a list of possible translations of the verbal request.

The directory service provider <NUM> generates a binary SMS message <NUM> and transmits the message <NUM> to the cellular phone <NUM>. The message <NUM> includes text, such as "Pizza, Palo Alto," and an application identifier, such as binary code used to launch or execute a function of an internet browser or a custom application (e.g., Google Maps for Mobile developed by Google™ of Mountain View, CA) installed on the cellular phone <NUM>.

The cellular phone <NUM> receives the message <NUM> and launches the browser <NUM> (or the custom application) with a map user interface (UI) based on the included application identifier. The browser <NUM> with map user interface transmits a data request <NUM> to the search engine <NUM> including the symbolic representation, such as "Pizza, Palo Alto.

The search engine <NUM> generates location search results <NUM> using the symbolic representation in the data request <NUM> (or from the storage location <NUM> as previously described). For example, the search engine <NUM> may determine that "Pizza" indicates a request for pizza restaurants and "Palo Alto" further indicates a request for pizza restaurant locations in the Palo Alto area. The search engine <NUM> transmits search results <NUM> to the cellular phone <NUM>. The cellular phone <NUM> receives the results <NUM> and displays locations <NUM> using the map user interface.

<FIG> is a flow chart depicting an example of a method for displaying on a remote device information derived from a verbal request. The process <NUM> may be performed, for example, by a system such as the systems <NUM>, <NUM>, and <NUM>. For clarity of presentation, the description that follows uses the systems <NUM>, <NUM>, and <NUM> as the basis of an example for describing the process <NUM>. However, another system, or combination of systems, may be used to perform the process <NUM>.

The process <NUM> begins with transmitting (<NUM>) a verbal request toa first information provider. For example, the remote device <NUM> transmits the verbal request <NUM>, including "Pizza, Palo Alto," to the first information provider <NUM> via the network <NUM>.

Optionally, a confirmation request is received (<NUM>) from the first information provider. The confirmation request seeks to confirm that a conversion of the verbal request to a symbolic representation, such as speech-to-text, is accurate. For example, the cellular phone <NUM> receives, from the directory service provider <NUM>, the confirmation request <NUM> "Did you say, 'Pizza, Palo Alto?'".

Optionally, a confirmation of the verbal to symbolic conversion is transmitted (<NUM>). For example, a user replies "Yes, I did," which is transmitted by the cellular phone <NUM> to the directory service provider <NUM>.

An application identifier and a symbolic representation indicator is received (<NUM>). For example, the remoter device <NUM> receives the digital message <NUM> including the application identifier <NUM> and the symbolic representation indicator <NUM>. An application specified by the application identifier is launched (<NUM>). For example, the remote device <NUM> launches the application <NUM> specified by the application identifier <NUM>.

If partial results are received in response to the verbal request <NUM>, then the results an be displayed (<NUM>). If a user requests more results or the partial results were not received (<NUM>), then the process <NUM> transmits (<NUM>) the symbolic representation indicator to a second information provider.

For example, the remote device <NUM> may present the partial results <NUM> including addresses for locations associated with the verbal request and receive a user input requesting more results. If a user requests more results or partial results were not transmitted, the remote device <NUM> transmits the symbolic representation indicator <NUM>, including "Pizza, Palo Alto," to a search engine that returns mapped locations in response to location queries.

Processed results are received from the second information provider (<NUM>). For example, the remote device <NUM> receives the results <NUM> from the second information provider <NUM>, including a map with locations of pizza restaurants in Palo Alto highlighted.

<FIG> is a block diagram of computing devices <NUM>, <NUM> that may be used to implement the systems and methods described in this document, either as a client, as a server, or as a plurality of servers. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

The storage device <NUM> is capable of providing mass storage forthe computing device <NUM>. In one implementation, the storage device <NUM> isa computer-readable medium. In various different implementations, the storage device <NUM> may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. The information carrier is a computer- or machine-readable medium, such as the memory <NUM>, the storage device <NUM>, memory on processor <NUM>, or a propagated signal.

The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM>. The processor may also include separate analog and digital processors.

The display <NUM> may be, for example, a TFT LCD display or an OLEO display, or other appropriate display technology. External interface <NUM> may provide, for example, for wired communication (e.g., via a docking procedure) or for wireless communication (e.g., via Bluetooth or other such technologies).

Expansion memory <NUM> may also be provided and connected to device <NUM> through expansion interface <NUM>, which may include, for example, a SIMM card interface.

The memory may include for example, flash memory and/or MRAM memory, as discussed below. The information carrier is a computer- or machine-readable medium, such as the memory <NUM>, expansion memory <NUM>, memory on processor <NUM>, or a propagated signal.

Communication interface <NUM> may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TOMA, PDC, WCDMA, CDMA2000, or GPRS, among others. In addition, GPS receiver module <NUM> may provide additional wireless data to device <NUM>, which may be used as appropriate by applications running on device <NUM>.

Device <NUM> may also communication audibly using audio codec <NUM>, which may receive spoken information from a user and convert it to usable digital information. Audio codex <NUM> may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device <NUM>.

A number of embodiments of the invention have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the invention. For example, the first information provider <NUM> and the second information provider <NUM> may be the same system or may be included in a single system.

Additionally, the application identifier <NUM> does not have to launch the application. Instead, it can be used to initiate a function of an application that has been previously launched. For example, the application identifier can indicate that the a previously launched internet browser should navigate to a particular site identified by a URL included in the digital message <NUM>.

In some implementations, the request <NUM> is a written request and the remote device <NUM> that transmits the request <NUM> is a personal computer, and the request <NUM> can be passed directly to the search engine <NUM> and the application selector without speech-to-text conversion.

Although SMS is used as an example in the above descriptions, in other implementations, the described systems and method may use a variety of mechanisms to transmit information to the remote device <NUM>. For example, the first information provider <NUM> can use actual push mechanisms, such as session-initiated protocol (SIP) and SMS transmissions, or virtual push mechanisms, such as polling using hypertext transmission protocol (HTTP).

Additionally, in some implementations, the request <NUM> can split up into multiple requests. For example, the request <NUM> can be a two-phase interaction, where a user of the remote device requests a location first (e.g., Dallas, Tx), which is confirmed by the first information provider. Next, the user transmits a second portion of the request, which specifies a particular category or item for which to search (e.g., barbeque).

Claim 1:
A method comprising:
receiving, at a computing device, a message from a server of a first remote service provider, the message comprising a symbolic representation of content of a verbal request spoken by a user of the computing device and an application identifier selected based on the content of the verbal request;
in response to receiving the message:
establishing, by the computing device, a connection with a server of a second remote service provider, wherein the server of the second remote service provider corresponds to an application on the computing device, and wherein the application on the computing device corresponds to the application identifier in the received message; and
transmitting, by the computing device and using the application corresponding to the application identifier, a query to the server of the second remote service provider, the query based on the symbolic representation of content of the verbal request;
receiving, by the computing device, a response to the query from the server of the second remote service provider; and
presenting, by the computing device, the response to the query through an interface of the computing device.