VIRTUAL KEYBOARD WITH INTENT-BASED, DYNAMICALLY GENERATED TASK ICONS

Systems, methods, and computer media for intent-based, dynamic generation and display of task icons within virtual keyboards are provided herein. A system can include a processor, an intent classifier, and a user interface generator. The intent classifier can be configured to determine user intent candidates based on contextual information. A user interface generator can be configured to generate the virtual keyboard for display and, upon receiving an indication of a user intent determined based on the user intent candidates, generate a task icon within the virtual keyboard. The task icon represents functionality associated with the determined user intent. Interaction with the task icon in the virtual keyboard can launch functionality associated with the determined intent.

BACKGROUND

With the advent of touchscreens and mobile devices, virtual keyboards have become commonplace. Virtual keyboards are typically displayed, for example, when a user taps the screen to enter text while using an application on a touchscreen device. Despite their advantages, virtual keyboards are often difficult and cumbersome to use for certain activities.

SUMMARY

Examples described herein relate to intent-based, dynamic generation and display of task icons within virtual keyboards. An example system can include a processor, an intent classifier, a ranker, and a user interface generator. The intent classifier can be configured to determine, by the processor, one or more user intent candidates based on contextual information. Contextual information can be, for example, text entered via a virtual keyboard, information relating to an application that is active while the virtual keyboard is displayed, text received in a conversation in the active application, etc.

User intent candidates can be selected in different ways. For example, a ranker can be configured to, by the processor, rank the one or more user intent candidates, and based on the ranking, select a user intent candidate as a determined user intent. A user interface generator can be configured to, by the processor, generate the virtual keyboard for display. Upon receiving an indication of the determined user intent, the user interface generator can also be configured to generate a task icon within the virtual keyboard based on the determined user intent. The task icon can be displayed, for example, in the input method editor (IME) of the virtual keyboard. Selection of or other interaction with the task icon in the virtual keyboard can launch functionality associated with the determined intent. User intent can be updated based on additional contextual information, and the task icon can be removed from the virtual keyboard if the task icon no longer reflects the updated user intent.

The foregoing and other objects, features, and advantages of the claimed subject matter will become more apparent from the following detailed description, which proceeds with reference to the accompanying figures.

DETAILED DESCRIPTION

Using the systems, methods, and computer-readable media described herein, an intent of a user interacting with a user interface of a computing device can be dynamically determined based on contextual information (e.g., text entered by the user or received from another user), and a task icon reflecting the user intent can be generated and displayed within a virtual keyboard in the user interface. Interaction with the task icon can cause a task icon user interface to be displayed in place of a portion of the overall user interface (e.g., displayed above the virtual keyboard or in place of a portion of the virtual keyboard). The task icon user interface provides convenient access (e.g., via an application user interface, links, deep links, etc.) to functionality corresponding to the user intent.

Unlike conventional approaches, the dynamic, intent-based approaches described herein allow users to access desired functionality directly from the virtual keyboard and/or task icon user interface without forcing the user to exit an application, open another application to perform an action, and then switch back to the original application.

As an example, the described approaches can be used to determine user intent while a user is having a conversation in a messaging application. In an example conversation, the user receives “feel like grabbing a bite to eat?” The user then enters, via a virtual keyboard, “sure, where?” One or both of these questions can be analyzed to determine that the user intent is to find a restaurant at which to meet for dinner. After this user intent is determined, a mapping task icon is generated and presented in the virtual keyboard. A selection, swipe, or other interaction with the mapping task icon causes a mapping task icon user interface to be displayed. The mapping task icon user interface can show locations of nearby restaurants, provide links to webpages of the restaurants, provide shareable restaurant information (e.g., a name and address that can be inserted into the messaging conversation), etc.

Thus, the mapping task icon user interface provides the user with access to mapping functionality (searching for restaurants near the user's current location) without the user having to exit the messaging application, launch a mapping application, locate restaurants, copy information, switch back to the messaging application, etc. The user can simply select a restaurant from the mapping task icon user interface, causing the restaurant's information to be added to the conversation, and continue with the uninterrupted flow of conversation.

In the described examples, the computational complexity of performing a desired action is reduced through the dynamic, intent-based approaches, which eliminates the computational cost of exiting/launching/relaunching applications and navigating through a user interface to locate desired applications. Examples are described below with reference toFIGS. 1-21.

FIG. 1illustrates a system100implemented on one or more computing device(s)102. Computing device102includes at least one processor104. Computing device102can be, for example, a mobile device, such as a smartphone or tablet, a personal computer, such as a desktop, laptop, or notebook computer, or other device.

A user interface generator106is configured to, by the at least one processor104, generate a virtual keyboard108for display in a user interface. The user interface is presented on a display of computing device(s)102. As used herein a “virtual keyboard” refers to a user interface having numbers, letters, etc. corresponding to those of a physical keyboard (e.g., a laptop or PC keyboard). Typically, the characters of a virtual keyboard are arranged similarly to those of a physical keyboard. The virtual keyboard is typically displayed on a touchscreen, and individual characters are selected through touch selection, hover selection, or other interaction with the touchscreen. Projection keyboards, AirType keyboards, and other non-physical keyboards are also considered to be virtual keyboards.

User interface generator106is also configured to, by the at least one processor104and upon receiving an indication of a determined user intent110, generate a task icon112within virtual keyboard108based on determined user intent110. As used herein, a “task icon” refers to a graphic, image, text, symbol, or other user interface element that represents functionality. The appearance of task icon112can reflect determined user intent110. For example, if determined user intent110is to see a movie, task icon112can be a graphic or image of a movie studio, theater, ticket, or projector, the text “movies,” etc. In some examples, multiple user intents are determined (e.g., multiple user intents that exceed a confidence or probability threshold), and multiple task icons are presented. Examples of task icons are shown inFIGS. 4A-15.

In some examples, task icon112is presented in the input method editor (IME) portion114of virtual keyboard108. IME portion114is shown inFIG. 1as being at the top of virtual keyboard108. IME portion114can be the portion of virtual keyboard108where autocorrect or word suggestions are displayed. IME portion114can contain various positions of importance. Using the example of autocorrect suggestions appearing in the IME, a most likely suggestion can be presented in IME portion114on the left in a first position, a second most likely suggestion can be presented to the right of the first position in a second position, etc. Task icon112can be presented in any position within IME portion114. Task icon112can also be presented in a different portion of virtual keyboard108, such as the left or right side, bottom left or right, etc. Task icon112can also be partially occluded as a “peek” of additional information which can be obtained with a swipe. In some examples, user interface generator106is further configured to, by the at least one processor104, remove task icon112upon receiving an indication that determined user intent110has been updated (e.g., based on additional contextual information or after a time threshold has elapsed).

Interaction with task icon112in virtual keyboard108launches functionality associated with determined user intent110. Interaction with task icon112can be, for example, a touch or hover selection, swipe to the right (or left, up, or down), pinch, select and hold for a threshold amount of time, or other interaction.

User interface generator106can be configured to launch the functionality associated with determined user intent110in a task icon user interface (illustrated, e.g., inFIGS. 4B, 6B, 7B, 8D, 9B, 12G, 12H, 13Band other figures). For example, user interface generator106can be configured to, upon receiving an indication of an interaction with task icon112, replace a portion of the user interface with a task icon user interface. The functionality associated with determined user intent110is accessible via the task icon user interface. In some examples, the task icon user interface is displayed in place of a portion of virtual keyboard108. As a specific example, the task icon user interface can be displayed in place of the portion of virtual keyboard108below IME portion114(illustrated, e.g., inFIGS. 4Band6B). In some examples, a portion of the user interface other than virtual keyboard108is replaced with the task icon user interface. This is illustrated, for example, inFIGS. 12B and 13B. In various examples, virtual keyboard108can continue to be displayed while the task icon user interface is displayed.

The task icon user interface can include an application user interface for an application launched by interaction with task icon112or a link or deep link to functionality of an application or functionality of a web service. The application launched by interaction with task icon112can be, for example, a mapping application, an organization application, a funds transfer application, a media application (audio, video, and/or image), or a user review application. In some examples, the application user interface for the application launched by interaction with task icon112comprises shareable content generated by the application and related to determined user intent110. Shareable content can include, for example, an estimated arrival or departure time, an event start time or end time, a restaurant suggestion, a movie suggestion, a calendar item, a suggested meeting time or meeting location, transit information, traffic information, weather or temperature information, or an instant answer result. Task icon user interfaces are discussed in detail with respect toFIGS. 4B-15.

User interface generator106can be implemented, for example, as computer-executable instructions (e.g., software) stored in memory (not shown) and executable by the at least one processor104. User interface generator106can also be implemented at least in part using programmable logic devices (PLDs) such as field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other computing hardware.

The determined user intent110that is provided to user interface generator106is determined by an intent classifier116and, in some examples, a ranker118. Intent classifier116is configured to determine, by the at least one processor104, one or more user intent candidates based on contextual information120. In some examples, system100includes multiple intent classifiers. Contextual information120can include, for example, text entered via virtual keyboard108, text received from a remote computing device, commands received through voice recognition, information relating to an application that is active while virtual keyboard108is displayed, task icons previously interacted with, a location of a user interacting with the user interface, a current time, day, or date, or history or preference information.

As an example, if a messaging application is active while virtual keyboard108is displayed, contextual information120can include: messages entered via virtual keyboard108; messages received from another user; the fact that the messaging application is active; preferences or history associated with the other user with whom the user is communicating; or preferences or history associated with conversations between the two users (e.g., a history of the two users meeting for coffee). Contextual information120can also include actions recently performed by the user (e.g., a recent search for “coffee” in a mapping application), reminders/alarms, calendar or organizer items (e.g., “have coffee with Kyle” stored as an upcoming appointment in a calendar application), etc. As used herein, a messaging application includes dedicated “chat” applications, chat or messaging functionality in other applications, email applications, or other applications in which messages are sent between users.

Intent classifier116can determine user intent candidates through a number of approaches, including artificial intelligence and machine learning approaches such as natural language understanding (NLU). NLU involves parsing, organizing, and classifying human language (whether received through voice recognition, touch/type input, or received in an electronic message or other electronic communication). NLU can be performed, for example, using a template matching approach in which text is analyzed to identify particular co-located strings that correspond to a known intent. For example, a template of “(airport_code_1) to (airport_code_2)” can correspond to an intent to purchase airline tickets. In template matching approaches, received text can be compared to a number of different templates.

Intent classification can also be performed through the use of statistical models such as logistic regression, boosted decision trees, neural networks, conditional Markov language models or conditional random fields. In such approaches, a training set of text portions that are tagged with a known intent are used to build statistical models that are then used to predict the intent of other text encountered at run-time. Collecting a variety and large amount of training data can improve the performance of such approaches.

In some examples, system100includes different intent classifiers for different types of functionality that can be associated with and accessed via task icon112. For example, system100can include an intent classifier for restaurant recommendations, an intent classifier for directions, an intent classifier for media items, etc. In examples where template matching techniques are used, different intent classifiers can have different associated templates. Intent classifier116can be trained using training data based on previously entered text and subsequent actions taken (for the user and/or for other users). For example, if a user receives the text “Hungry?” and replies “starving,” and then opens a restaurant review or mapping application, this data can be used as training data to match future received incidences of “hungry” and “starving” with the intent to find a restaurant at which to eat. Training can also be done based on user interactions with, or lack of user interactions with, previously presented task icons. For example, if a task icon is presented based on a determined intent, and the user selects, swipes, or otherwise interacts with the task icon, an interpretation is that the determined user intent was accurate for the contextual information. Conversely, if a task icon is presented but not interacted with by the user, an interpretation is that the determined user intent was not accurate. In some examples, training data can be aggregated across users and stored, for example, in a cloud environment where the training data can be accessed by different users. In other examples, training data is user specific.

A search tool can also be used to train intent classifier116. User searches and corresponding subsequent actions taken or results selected are used to inform future intent classification. The search tool (illustrated, e.g., inFIGS. 13A-13D, 14A-14D, 15A-15D, and 16) can be included in virtual keyboard108. The search tool can be presented, for example, in IME portion114, and can be accessed by interacting with a search icon (e.g., a magnifying glass, question mark, etc.) or by performing a particular gesture or combination of gestures. For example, a swipe of IME portion114(e.g., swipe left or right) can cause the search tool having a text entry area to be displayed in IME portion114. A swipe of IME114in an opposite direction or selection of an exit button can cause the search tool to disappear. The search tool can also be a speech recognition search tool that begins “listening” when a search icon is interacted with. In some examples, the search tool is displayed when virtual keyboard108is displayed. As a specific example, when virtual keyboard108is launched, the search tool appears in or under IME portion114. In some examples where the search tool is presented in IME portion114, when task icon112is generated and presented (or when autocorrect suggestions are generated) due to use of the application with which virtual keyboard108is being used, the search tool disappears. System100can include a search engine (not shown) configured to perform searches received via the search tool, and user interface generator106can be configured to present the search tool.

Intent classifier116can be implemented, for example, as computer-executable instructions (e.g., software) stored in memory (not shown) and executable by the at least one processor104. Intent classifier116can also be implemented at least in part using programmable logic devices (PLDs) such as field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other computing hardware. Intent classifier116can also be implemented using neural networks (e.g., deep neural networks, convolutional neural networks, etc.). In some examples, intent classifier116can be implemented in a cloud computing environment (e.g., such as cloud2110ofFIG. 21), and system100is in communication with intent classifier116through a network such as the Internet. In some examples, some functionality of intent classifier116can be implemented in system100while other functionality of intent classifier116is implemented in the cloud.

In some examples intent classifier116performs ranking of or otherwise selects (e.g., selects a candidate with a highest probability, etc.) one or more of the user intent candidates as determined user intent110. In other examples, ranker118is configured to, by the at least one processor104, rank the one or more user intent candidates determined by intent classifier116and, based on the ranking, select at least one user intent candidate as determined user intent110. Ranker118can be configured to apply a variety of ranking approaches, such as to select a user intent candidate with a highest confidence level (e.g., probability of being correct). In some examples, the functionality of ranker118is combined with intent classifier116. Ranking can be done, for example, by calibrating intent classifier116using isotonic or logistic regression and then sorting by classifier outputs, using boosted decision trees or neural networks trained under ranking loss functions, or other approaches. Selection of determined user intent110can be done, for example, by thresholding the scores used for ranking.

Ranker118can be implemented, for example, as computer-executable instructions (e.g., software) stored in memory (not shown) and executable by the at least one processor104. Ranker118can also be implemented at least in part using programmable logic devices (PLDs) such as field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other computing hardware.

FIG. 2illustrates a system200implemented on one or more computing device(s)202. Computing device202includes at least one processor204. Computing device202can be similar to computing device102ofFIG. 1. User interface generator206and ranker208can also be similar to the corresponding components inFIG. 1. System200includes multiple intent classifiers210. A federator212is configured to, by the at least one processor204, distribute contextual information to intent classifiers210. Federator212is also configured to determine an aggregated group of user intent candidates based on the user intent candidates determined by intent classifiers210. Ranker208is further configured to, by the at least one processor204, rank the user intent candidates in the aggregated group.

User interface generator206, ranker208, and federator212can be implemented, for example, as computer-executable instructions (e.g., software) stored in memory (not shown) and executable by the at least one processor204. User interface generator206, ranker208, and federator212can also be implemented at least in part using programmable logic devices (PLDs) such as field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other computing hardware.

Decoder214is configured to, by the at least one processor204, interpret and recognize touch and hover input to the virtual keyboard (not shown). Decoder214can be configured to recognize touches/taps as well as swipes. Decoder214can be configured to interpret input according to one or more touch models216and language models218(e.g., such as a user's language model). Touch models216are configured to evaluate how well various hypotheses about which word a user intends, as the user is entering text, match the touch and hover input. Language models218are configured to evaluate how well these hypotheses fit words already entered. Autocorrector220is configured to, by the at least one processor204, provide autocorrect suggestions, spelling suggestions, etc., that can be presented, for example, in the IME of the virtual keyboard.

Similar to user interface generator106ofFIG. 1, user interface generator206is configured to generate a task icon, within the virtual keyboard, reflecting a determined user intent, and upon receiving an indication of an interaction with the task icon, generate a task icon user interface providing access to functionality corresponding to the determined intent. The task icon user interface can include links or deep links to one or more local services or applications222or web services or applications224. The task icon user interface can also include an application interface for the one or more local services or applications222or web services or applications224.

FIG. 3illustrates an example method300of reconfiguring a user interface. In process block302, a virtual keyboard is generated. In process block304, contextual information is received. Contextual information can include, for example, text entered via the virtual keyboard, text received from a remote computing device, commands received through voice recognition, or information relating to an application that is active while the virtual keyboard is displayed. A user intent is determined in process block306. User intent can be determined, for example, using an intent classifier and ranker. In process block308, a task icon is generated within the virtual keyboard. The task icon can be presented in the IME of the keyboard, for example. Method300can be performed, for example, using system100ofFIG. 1or system200ofFIG. 2.

FIGS. 4A-4Dillustrate determination of user intent and presentation of a calendar task icon. In user interface400ofFIG. 4A, a messaging application is active and a conversation402is displayed. User interface400also includes a virtual keyboard404having an IME portion406. Based on contextual information obtained from conversation402(e.g., the question “Want to go for dinner this week?” and the response “I'm free at”), a user intent to identify an available time to meet for dinner is determined. Contextual information can also include, for example, whether or not “Ellen” is a frequent contact (and is therefore someone with whom it is more likely the user would meet up for dinner), the user's statement “Happy Birthday” and accompanying birthday cake emoji, whether an email indicated Ellen would be in the same location as the user, etc. After this intent is determined, a task icon408is generated and presented in IME portion406of virtual keyboard404. A text entry box409is also shown in user interface400. Text entry box409is part of the messaging application and is not a part of virtual keyboard404.

As shown inFIG. 4B, interaction with task icon408(such as a selection, swipe to the left or right, etc.) causes a task icon user interface410to be presented in place of a portion of user interface400. InFIG. 4B, task icon user interface410is presented in place of a portion of virtual keyboard404. Specifically, the portion of virtual keyboard404below IME portion406has been replaced by task icon user interface410, but IME portion406remains displayed. The portion of user interface400in which conversation402is displayed remains unchanged. The functionality associated with the determined intent is accessible via task icon user interface410. Task icon user interface410includes an application user interface of a calendar application in which blocks of time for different days are shown. The appearance of task icon408(a month view of a calendar) reflects the determined user intent and the functionality that can be launched in task icon user interface410by interacting with task icon408. Task icon408can be accentuated (e.g., underlined and bolded as shown inFIGS. 4A and 4B, distorted, or otherwise emphasized) to indicate that interaction with task icon408launches functionality. A keyboard icon412is presented in IME portion406to allow the user to replace task icon user interface410with the character keys of virtual keyboard404. In task icon user interface410, blocks of time are selectable and can be added to conversation402as shown inFIGS. 4C and 4D.

InFIG. 4C, two possible times, time block414and time block416, have been selected. Text entry box409indicates “Selected Events:2.” Time block414and416are shareable content that can be selected and added to conversation402. By taking another action (such as pressing “Send,” dragging and dropping, etc.) an “Available times:” conversation entry418including the times of time block414and time block416is added to conversation402as shown inFIG. 4D. After conversation entry418has been added, task icon408is no longer displayed in IME portion406. AsFIGS. 4A-4Dillustrate, dynamic determination of user intent and generation and presentation of a task icon corresponding to the user intent allow the user to perform actions and access other applications without interrupting the flow of a conversation.

FIG. 5illustrates a user interface500that includes a conversation502being conducted in a messaging application. A virtual keyboard504includes an IME portion506. A user intent to open a file from a web service is determined based on the statement “Hey I just updated salespitch.pptx” in conversation502. A web service task icon508is presented in IME portion506, along with autosuggestions510(“In a meeting”) and512(“I'm busy”). InFIG. 5, autosuggestion510is in IME position one, autosuggestion512is in IME position two, and task icon508is in IME position three. Interaction with task icon508launches functionality associated with the web service (not shown), such as a link or deep link to the “salespitch.pptx” file, to a shared work area or file folder, a web service user interface, etc.

FIG. 6Aillustrates a user interface600that includes a conversation602being conducted in a messaging application. A virtual keyboard604includes an IME portion606. A user intent to share a current location (and/or to provide an estimated time of arrival, etc.) is determined based on the statements “Hey where are you, just got a table . . . ” and/or “I'm at” in conversation602. A mapping task icon608is then presented in IME portion606to reflect the determined intent. Interaction with mapping task icon608causes mapping task icon user interface610to be presented in place of a portion of virtual keyboard604, as shown inFIG. 6B.

Mapping task icon user interface610includes a map of the user's current location and destination as well as shareable content items612,614, and616that indicate the user's estimated time of arrival by car, bus, or walking, respectively. InFIG. 6C, shareable content item614has been selected (as indicated by the bolding of shareable content item614), and the bus route taken between the current location of the user and the destination is shown in mapping task icon user interface610. InFIG. 6D, conversation entry618has been added to conversation602. Conversation entry618reflects shareable content item614.

FIG. 7Aillustrates a user interface700that includes a conversation702being conducted in a messaging application. A virtual keyboard704includes an IME portion706. A user intent to meet at a location (e.g., for dinner at a restaurant) is determined based on the statements “Want to get together tonight?” and/or “Sure, how about” in conversation702. A mapping task icon708is presented in IME portion706. Interaction with mapping task icon708causes mapping task icon user interface710to be presented in place of a portion of virtual keyboard704, as shown inFIG. 7B. Mapping task icon user interface710displays the user's current location and lists nearby restaurants.

As shown inFIGS. 7A and 7B, IME portion706also includes additional task icons generated based on contextual information, such as movie task icon712, which is partially obscured inFIG. 7Abut is visible inFIG. 7B. In some cases, multiple user intents are possible. Based on the statements in conversation702, a user might be interested in meeting for dinner, meeting for coffee, meeting for a movie, meeting to return an item, etc. As a result, multiple task icons can be generated and presented within virtual keyboard704. The multiple task icons can be presented in an order of likelihood determined, for example, based on confidence level, user history, etc. As shown inFIG. 7B, mapping task icon708, movie task icon712, restaurant task icon714and other task icons are presented in IME portion706.

In some examples, a first task icon can be associated with other task icons representing a subset of functionality of the first task icon. For example, mapping task icon708can launch a variety of different mapping functionality (e.g., estimated time of arrival, location of restaurants, location of stores, etc.). Accordingly, as illustrated inFIG. 7B, mapping task icon708has been selected, and mapping task icon user interface710is associated with the restaurant aspect of mapping represented by restaurant task icon714. In user interface700, a user can swipe or select other task icons to change the task icon user interface that is displayed below IME portion706.

InFIG. 7C, a shareable content item716(the restaurant “Mamnoon”) has been selected (as indicated by the bolding of shareable content item716). InFIG. 7D, a conversation entry718has been added to conversation702. Conversation entry702reflects shareable content item718and lists the name and address of the restaurant. The task icons, including task icons708and712, have been removed from IME portion706after conversation entry702was added.

FIGS. 8A and 8Brelate to intent-based instant answers. In user interface800ofFIG. 8A, a new message is being composed (e.g., in an email or messaging application) that includes the text “I'm inviting 25 kids x $9 meal x $4 drinks.” Some or all of this text can be used to determine a user intent of calculating a total cost. An instant answer result802indicating the total cost is displayed within IME portion804of virtual keyboard806. Similarly, in user interface808ofFIG. 8B, a new message is being composed that includes the text “For a souvenir for ε45.” Some or all of this text can be used to determine a user intent of calculating a U.S. dollar equivalent price. An instant answer result810indicating the U.S. dollar price is displayed within IME portion804of virtual keyboard806.

FIGS. 9A and 9Billustrate a user interface900in which a conversation902is displayed. Based on contextual information (e.g., the conversation entry “Did you see that Seahawks game???”) a user intent to view or share a video of a Seattle Seahawks football game is determined. An instant answer result904of a recent game score is provided in IME portion906of virtual keyboard908. One or more task icons can also be generated and displayed in virtual keyboard908, as is illustrated inFIG. 9B. Additional task icons can be revealed, for example, by swiping instant answer result904.FIG. 9Bshows multiple task icons including football media task icon910, which when interacted with causes a replacement of a portion of virtual keyboard908with football media task user interface912, which displays shareable and/or viewable football game video clips and/or images. A user, for example, can select or drag and drop a thumbnail image of a video clip into conversation902.

FIGS. 10A and 10Billustrate a user interface1000in which a conversation1002is displayed. Based on contextual information (e.g., the conversation entries “We could also see a movie after dinner” and/or “Yes! Let's get tix for Star Wars”) a user intent to go see the movie “Star Wars” is determined. A movie task icon1004is presented in IME portion1006of virtual keyboard1008. Interaction with movie task icon1004causes movie task icon user interface1010to be presented in place of a portion of virtual keyboard1008, as shown inFIG. 10B. Movie task icon user interface1010displays show times for “Star Wars” at theaters near the user's current locations. Movie task icon user interface1010can be a movie ticket purchase/reservation service application user interface and/or can contain links to a movie service or deep links to purchase tickets for a particular show. InFIG. 10B, the theater “iPic Redmond (2D)” is selected and appears as a text entry in text entry box1012. This text entry can then be added to conversation1002.

FIG. 11illustrates a user interface1100in which a conversation1102is displayed. Based on contextual information (e.g., the conversation entries “Hey got tix for us $25/each” and/or “Thanx! Let me send you the money”) a user intent to transfer funds is determined. A funds transfer is considered a transaction service; a transactional service application can be a funds transfer application or other transaction-based application. A funds transfer task icon1104is presented in IME portion1106of a virtual keyboard (only IME portion1106is shown inFIG. 11).FIG. 11also shows a funds transfer task icon user interface1108that replaced a portion of the virtual keyboard after interaction with funds transfer task icon1104. Funds transfer task icon user interface contains deep links1110and1112that can be selected to transfer funds using different funds transfer services. An indication that funds were sent can then be added to conversation1102.

FIGS. 12A-12Hillustrate examples in which a task icon user interface replaces a portion of the overall user interface (above the virtual keyboard) rather than replacing a portion of the virtual keyboard.FIG. 12Aillustrates a user interface1200in which “I'm meeting Kyle” has been entered into a text entry box1202of a messaging application. Based on contextual information (e.g., the text entry “I'm meeting Kyle”) a user intent to access contact information for Kyle is determined. A contacts task icon1204is presented in IME portion1206of virtual keyboard1208. Interaction with task icon1204causes a portion of user interface1200(the portion immediately above IME portion1206) to be replaced with a contacts task icon user interface1210, as shown inFIG. 12B. Contacts task icon user interface1210comprises functionality of a contacts or organizer application and displays contact information for Kyle.

FIG. 12Cillustrates an expanded contacts task icon user interface1212that is presented upon interaction with contacts task icon user interface1210inFIG. 12C. Expanded contacts task icon user interface1212is presented in place of a portion of virtual keyboard1208(i.e., in place of the character keys). IME portion1206is moved to the bottom of user interface1200. InFIG. 12C, the portion of user interface1200available for displaying messages remains the same as inFIG. 12Bbefore presentation of expanded contacts task icon user interface1212.

InFIG. 12D, the user has exited expanded contacts task icon user interface1212and continued typing in text entry box1204, which now reads “I'm meeting Kyle for lunch at Din Tai Fung.” An updated user intent of determining/sharing the location of a restaurant is determined based on updated contextual information (the additional text “for lunch at Din Tai Fung”). A restaurant task icon1214is displayed in IME portion1206to reflect the updated user intent, and contacts task icon1204is removed from IME portion1206. In some examples, task icons that were presented but subsequently removed because of an updated user intent can be represented by an indicator (e.g., a numeral or other indicator in the IME or elsewhere in the virtual keyboard), and these task icons can be redisplayed upon user interaction with the indicator.

InFIG. 12E, a user interaction with restaurant task icon1214causes a portion of user interface1200to be replaced with a restaurant task icon user interface1216that provides contact information for the restaurant “Din Tai Fung.” InFIG. 12F, user interface1200reflects that a user has selected exit button1218illustrated inFIG. 12E, and restaurant task icon user interface1216and restaurant task icon1214have disappeared.

In some examples, if a task icon is determined to still reflect a determined user intent after additional contextual information is received but an additional intent is also determined based on the additional contextual information, then an additional task icon can be presented with the original task icon. InFIGS. 12G and 12H, both contacts task icon1204and restaurant task icon1214are presented in IME portion1206based on updated contextual information (the additional text “for lunch at Din Tai Fung, wanna join?”). A user interaction with contacts task icon1204causes a contacts task icon user interface1220to replace a portion of user interface1200. InFIG. 12G, multiple task icon user interfaces are present to correspond to the multiple task icons. A portion of a restaurant task icon user interface1222is visible next to contacts task icon user interface1220. The different task icon user interfaces can be scrollable. For example, a user can swipe or scroll contacts task icon user interface1220to the left or right or swipe/select restaurant task icon1214to display restaurant task icon user interface1222.FIG. 12Hillustrates an example in which restaurant task icon user interface1222is selected and then, upon user interaction with restaurant task icon user interface1222, a portion of virtual keyboard1208is replaced by extended restaurant task icon user interface1224.

FIG. 13Aillustrates a user interface1300in which “I'm meeting Kyle for lunch at Din Tai Fung” has been entered into a text entry box1302of a messaging application. Based on contextual information (e.g., the text entry “I'm meeting Kyle for lunch at Din Tai Fung”) a user intent to access/share restaurant information is determined. A restaurant task icon1304is presented in IME portion1306of virtual keyboard1308. Interaction with task icon1304causes a portion of user interface1300(the portion above IME portion1206) to be replaced with a restaurant task icon user interface1310, as shown inFIG. 13B. Virtual keyboard1308continues to be displayed when restaurant task icon user interface1310is presented.

FIGS. 14A-16illustrate various example user interfaces in which a search tool is presented within a virtual keyboard. InFIG. 14A, user interface1400contains a virtual keyboard1402having an IME portion1404. A search tool, represented by a magnifying glass icon1406is presented in IME portion1404. In some examples, the search tool is accessed (and magnifying glass icon1406is displayed) by swiping the IME or selecting another icon or character displayed within virtual keyboard1402. InFIG. 14A, a user has entered “Pizza.” A search result user interface1408is then displayed that shows results for various categories such as emoji, restaurants, etc.

InFIG. 14B, different task icons are displayed in IME portion1404corresponding to the various categories. InFIG. 14B, an emoji icon1410is selected, and search results user interface1408displays only emoji results for “pizza.” Emoji icon1410is not considered to be a task icon because the emoji is not associated with functionality. InFIG. 14C, a restaurant task icon1412is selected, and search results user interface1408displays only restaurant results. Similarly, inFIG. 14D, an image task icon1414is selected, and search results user interface1408displays only image results. As with emoji icon1410, in some examples, image task icon1414is not considered to be a task icon unless functionality is associated with image task icon1414. Search results user interface1408can provide access to functionality (e.g., via links, deep links, or application user interfaces) similar to a task icon user interface.

FIG. 15Aillustrates an example user interface1500including a virtual keyboard1502having an IME portion1504.FIG. 15Aillustrates an example of the initial display of a search tool (as represented by magnifying glass icon1506). The search tool can be presented, for example, after a user swipes or otherwise interacts with IME portion1504. The search tool can be hidden by selecting exit button1508. In some examples, the search tool provides an option to select search results that correspond to various categories.FIG. 15Billustrates a variety of task icons corresponding to the various categories presented within IME portion1504. The category-based search can be an alternative to the arrangement displayed inFIG. 15A, or the arrangements inFIGS. 15A and 15Bcan be toggled or selected between.

FIG. 15Cillustrates a user selection of a particular search category. InFIG. 15C, an image search is selected, as indicated by image task icon1510shown in IME portion1504. InFIG. 15D, “Sounders” is searched for, and a search results user interface1512is presented that includes image results.

FIG. 16illustrates a user interface1600in which a search results user interface1602is presented above a virtual keyboard1604in place of a portion of user interface1600.

InFIGS. 4A through 16, task icon user interfaces, extended task icon user interfaces, and search result user interfaces are presented in different locations and replace different portions of the overall user interface. It is contemplated that any of the positions described herein of the task icon user interfaces can be used with any of the examples. Specific configurations and examples were chosen for explanatory purposes and are not meant to be limiting.

In some examples, task icons are generated and presented based on previous manual searches. For example, task icons corresponding to previous searches entered through a search tool (e.g., in the IME portion of a virtual keyboard) can be presented in the virtual keyboard. In some examples, task icons corresponding to previous searches can be presented before a current user intent is determined.

FIG. 17illustrates a method1700for reconfiguring a user interface on a computing device. In process block1702, a virtual keyboard is presented in the graphical user interface. In process block1704, one or more text entries are received. In process block1706, based at least in part on the one or more text entries, a user intent is determined using one or more intent classifiers. Upon determining the user intent, a task icon representing functionality corresponding to the user intent is presented within the virtual keyboard in process block1708. In process block1710, after a user selection of the task icon, a task icon user interface that provides access to the functionality corresponding to the user intent is presented in place of a portion of the graphical user interface.

FIG. 18illustrates a method1800for reconfiguring a graphical user interface. In process block1802, while a first application is active, a virtual keyboard is presented in the graphical user interface. The virtual keyboard has an input method editor (IME) portion. In process block1804, based at least in part on contextual information for the first application, a user intent is determined. The contextual information includes at least one of text entered via the virtual keyboard, text received via the first application, or information relating to the first application. A task icon is presented within the IME portion of the virtual keyboard in process block1806. The task icon is linked to functionality reflecting the user intent. In process block1808, upon receiving an indication of a selection of the task icon, a task icon user interface is presented in place of a portion of the virtual keyboard. The task icon user interface comprises at least one of: an application user interface for a second application, shareable content generated by the second application, or a deep link to functionality of the second application or functionality of a web service.

Example Computing Systems

FIG. 19depicts a generalized example of a suitable computing system1900in which the described innovations may be implemented. The computing system1900is not intended to suggest any limitation as to scope of use or functionality, as the innovations may be implemented in diverse general-purpose or special-purpose computing systems.

With reference toFIG. 19, the computing system1900includes one or more processing units1910,1915and memory1920,1925. InFIG. 19, this basic configuration1930is included within a dashed line. The processing units1910,1915execute computer-executable instructions. A processing unit can be a general-purpose central processing unit (CPU), processor in an application-specific integrated circuit (ASIC), or any other type of processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. For example,FIG. 19shows a central processing unit1910as well as a graphics processing unit or co-processing unit1915. The tangible memory1920,1925may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two, accessible by the processing unit(s). The memory1920,1925stores software1980implementing one or more innovations described herein, in the form of computer-executable instructions suitable for execution by the processing unit(s). For example, memory1920,1925can store intent classifier116, ranker118, and/or user interface generator106ofFIG. 1and/or user interface generator206, ranker208, federator212, decoder214, autocorrector220, and/or intent classifiers210ofFIG. 2.

A computing system may have additional features. For example, the computing system1900includes storage1940, one or more input devices1950, one or more output devices1960, and one or more communication connections1970. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system1900. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing system1900, and coordinates activities of the components of the computing system1900.

The tangible storage1940may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium which can be used to store information and which can be accessed within the computing system1900. The storage1940stores instructions for the software1980implementing one or more innovations described herein. For example, storage1940can store intent classifier116, ranker118, and/or user interface generator106ofFIG. 1and/or user interface generator206, ranker208, federator212, decoder214, autocorrector220, and/or intent classifiers210ofFIG. 2.

The input device(s)1950may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing system1900. For video encoding, the input device(s)1950may be a camera, video card, TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM or CD-RW that reads video samples into the computing system1900. The output device(s)1960may be a display, printer, speaker, CD-writer, or another device that provides output from the computing system1900.

Example Mobile Devices

FIG. 20is a system diagram depicting an example mobile device2000including a variety of optional hardware and software components, shown generally at2002. Any components2002in the mobile device can communicate with any other component, although not all connections are shown, for ease of illustration. The mobile device can be any of a variety of computing devices (e.g., cell phone, smartphone, handheld computer, Personal Digital Assistant (PDA), etc.) and can allow wireless two-way communications with one or more mobile communications networks2004, such as a cellular, satellite, or other network.

The illustrated mobile device2000can include a controller or processor2010(e.g., signal processor, microprocessor, ASIC, or other control and processing logic circuitry) for performing such tasks as signal coding, data processing, input/output processing, power control, and/or other functions. An operating system2012can control the allocation and usage of the components2002and support for one or more application programs2014. The application programs can include common mobile computing applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications), or any other computing application. The application programs2014can also include virtual keyboard, task icon, and user interface reconfiguration technology. Functionality2013for accessing an application store can also be used for acquiring and updating application programs2014.

The illustrated mobile device2000can include memory2020. Memory2020can include non-removable memory2022and/or removable memory2024. The non-removable memory2022can include RAM, ROM, flash memory, a hard disk, or other well-known memory storage technologies. The removable memory2024can include flash memory or a Subscriber Identity Module (SIM) card, which is well known in GSM communication systems, or other well-known memory storage technologies, such as “smart cards.” The memory2020can be used for storing data and/or code for running the operating system2012and the applications2014. Example data can include web pages, text, images, sound files, video data, or other data sets to be sent to and/or received from one or more network servers or other devices via one or more wired or wireless networks. The memory2020can be used to store a subscriber identifier, such as an International Mobile Subscriber Identity (IMSI), and an equipment identifier, such as an International Mobile Equipment Identifier (IMEI). Such identifiers can be transmitted to a network server to identify users and equipment.

The mobile device2000can support one or more input devices2030, such as a touchscreen2032, microphone2034, camera2036, physical keyboard2038and/or trackball2040and one or more output devices2050, such as a speaker2052and a display2054. Other possible output devices (not shown) can include piezoelectric or other haptic output devices. Some devices can serve more than one input/output function. For example, touchscreen2032and display2054can be combined in a single input/output device.

The input devices2030can include a Natural User Interface (NUI). An NUI is any interface technology that enables a user to interact with a device in a “natural” manner, free from artificial constraints imposed by input devices such as mice, keyboards, remote controls, and the like. Examples of NUI methods include those relying on speech recognition, touch and stylus recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, and machine intelligence. Other examples of a NUI include motion gesture detection using accelerometers/gyroscopes, facial recognition, 3D displays, head, eye, and gaze tracking, immersive augmented reality and virtual reality systems, all of which provide a more natural interface, as well as technologies for sensing brain activity using electric field sensing electrodes (EEG and related methods). Thus, in one specific example, the operating system2012or applications2014can comprise speech-recognition software as part of a voice user interface that allows a user to operate the device2000via voice commands. Further, the device2000can comprise input devices and software that allows for user interaction via a user's spatial gestures, such as detecting and interpreting gestures to provide input to a gaming application.

A wireless modem2060can be coupled to an antenna (not shown) and can support two-way communications between the processor2010and external devices, as is well understood in the art. The modem2060is shown generically and can include a cellular modem for communicating with the mobile communication network2004and/or other radio-based modems (e.g., Bluetooth2064or Wi-Fi2062). The wireless modem2060is typically configured for communication with one or more cellular networks, such as a GSM network for data and voice communications within a single cellular network, between cellular networks, or between the mobile device and a public switched telephone network (PSTN).

The mobile device can further include at least one input/output port2080, a power supply2082, a satellite navigation system receiver2084, such as a Global Positioning System (GPS) receiver, an accelerometer2086, and/or a physical connector2090, which can be a USB port, IEEE 1394 (FireWire) port, and/or RS-232 port. The illustrated components2002are not required or all-inclusive, as any components can be deleted and other components can be added.

Example Cloud-Supported Environments

FIG. 21illustrates a generalized example of a suitable cloud-supported environment2100in which described embodiments, techniques, and technologies may be implemented. In the example environment2100, various types of services (e.g., computing services) are provided by a cloud2110. For example, the cloud2110can comprise a collection of computing devices, which may be located centrally or distributed, that provide cloud-based services to various types of users and devices connected via a network such as the Internet. The implementation environment2100can be used in different ways to accomplish computing tasks. For example, some tasks (e.g., processing user input and presenting a user interface) can be performed on local computing devices (e.g., connected devices2130,2140,2150) while other tasks (e.g., storage of data to be used in subsequent processing) can be performed in the cloud2110.

In example environment2100, the cloud2110provides services for connected devices2130,2140,2150with a variety of screen capabilities. Connected device2130represents a device with a computer screen2135(e.g., a mid-size screen). For example, connected device2130can be a personal computer such as desktop computer, laptop, notebook, netbook, or the like. Connected device2140represents a device with a mobile device screen2145(e.g., a small size screen). For example, connected device2140can be a mobile phone, smart phone, personal digital assistant, tablet computer, and the like. Connected device2150represents a device with a large screen2155. For example, connected device2150can be a television screen (e.g., a smart television) or another device connected to a television (e.g., a set-top box or gaming console) or the like. One or more of the connected devices2130,2140,2150can include touchscreen capabilities. Touchscreens can accept input in different ways. For example, capacitive touchscreens detect touch input when an object (e.g., a fingertip or stylus) distorts or interrupts an electrical current running across the surface. As another example, touchscreens can use optical sensors to detect touch input when beams from the optical sensors are interrupted. Physical contact with the surface of the screen is not necessary for input to be detected by some touchscreens. Devices without screen capabilities also can be used in example environment2100. For example, the cloud2110can provide services for one or more computers (e.g., server computers) without displays.

Services can be provided by the cloud2110through service providers2120, or through other providers of online services (not depicted). For example, cloud services can be customized to the screen size, display capability, and/or touchscreen capability of a particular connected device (e.g., connected devices2130,2140,2150).

In example environment2100, the cloud2110provides the technologies and solutions described herein to the various connected devices2130,2140,2150using, at least in part, the service providers2120. For example, the service providers2120can provide a centralized solution for various cloud-based services. The service providers2120can manage service subscriptions for users and/or devices (e.g., for the connected devices2130,2140,2150and/or their respective users). The cloud2110can store training data2160used in user intent determination as described herein. An intent classifier2162, which can be, for example, similar to intent classifier116ofFIG. 1, can also be implemented in cloud2110.

Example Implementations