Patent Description:
Modern computing devices support a variety of user input modalities such as voice control and spatial gesturing, in addition to traditional touch and keyboard-based input modes. While a vast improvement over the options of the past, the same general framework persists where one modality is utilized at a time to control events.

As an example, many devices in the marketplace today support the voice invocation of digital assistants. Users can interact with digital assistants in a conversational manner by speaking a digital assistant's name followed by a command or request. In another example, spatial gestures made with a device (e.g. making a particular motion with a phone) can be used to unlock the device, control a video game, or the like.

The content capture experience has likewise been improved by the advent of new input modalities. Users can reliably generate content using the speech-to-text transcription capabilities of many applications and devices, as well as their handwriting recognition capabilities coupled with digital inking implements.

Nevertheless, the single-modality framework relegates the user to supplying input by one modality or another: a user is either speaking to a device or using touch or spatial gestures; typing on a keyboard or inking with a stylus. Such a framework limits the creative capacity of developers to continue to improve the user experience.

Some frameworks do utilize multiple modes of user input at the same time, but not to control events. For example, some messaging applications include a feature whereby users can hold down a button while speaking, in order to make a recording of their voice which can be sent to a recipient in a text message. The first modality in this example is touch, while the second modality is voice, although it is the touch that controls events - the recording is just content. <CIT> describes a system, method and computer-readable storage devices for multi-modal interactions with a system via a long-touch gesture on a touch-sensitive display. A system can receive a multi-modal input comprising speech and a touch on a display, wherein the speech comprises a pronoun. When the touch on the display has a duration longer than a threshold duration, the system can identify an object within a threshold distance of the touch, associate the object with the pronoun in the speech, to yield an association, and perform an action based on the speech and the association.

Technology disclosed herein relates to a new multi-mode user input framework that enhances the user experience with respect to the content capture capabilities of computing devices and applications. In an implementation, a combined user input comprises a voice signal and a touch gesture sustained at least partially coincident with the voice signal. An occurrence of the combined user input triggers the identification of an associated content object which may then be associated with a captured version of the voice signal. Such an advance provides users with a new framework for interacting with their devices, applications, and surroundings.

This Overview is provided to introduce a selection of concepts in a simplified form that are further described below in the Technical Disclosure. It may be understood that this Overview is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Many aspects of the disclosure may be better understood with reference to the following drawings. Moreover, like reference numerals in the drawings designate corresponding parts throughout the several views. While several embodiments are described in connection with these drawings, the disclosure is not limited to the embodiments disclosed herein.

A new multi-input mode framework is disclosed herein that allows users to enter and exit a multi-input mode paradigm that, when engaged, allows the users to control the content capture experience by a combination of user inputs. The multi-input mode may be selectively engaged by the user, although it may be provided in some cases as the default mode of operation. In other cases, multi-input mode may be one of two or more supported frameworks on a device or alternatively it may be the only framework supported on a device.

The multi-input mode framework centers on the concept of combined user inputs. A combined user input comprises a voice signal and a touch gesture sustained at least partially coincident with the voice signal. An occurrence of a combined user input triggers the identification of an associated content object. The content object may then be associated with a captured version of the voice signal such as an audio and/or video recording, a transcription, or the like, such that the content object and a representation of the voice signal can be presented and accessed together.

In some implementations, a combined object may be produced that associates the captured version of the voice signal with the content object voice object and that can be displayed or otherwise presented to the user. The captured version of the voice signal can be stored in association with the content object targeted by a touch gesture. In some cases, a single file structure holds both the captured version of the voice and the content object, whose data types may differ. In other cases, the file structure includes meta data that points to or otherwise references one or both of the captured voice and the content object.

Examples of content objects include but are not limited to individual emails, documents, calendar events, screen captures, images, and videos. Examples of captured versions of the voice signals include but are not limited to transcriptions of spoken utterances encoded in the voice signals, audio or video recordings of voice signals, and the like.

As mentioned, the captured versions of the voice signals may be transcriptions of utterances encoded in the voice signals. In some scenarios, the transcriptions of the utterances in no association with any other objects. That is, just the transcription may be stored such that a user may later consult what was said. The transcription may be stored in the context of an application that allows the user to browse both stand-alone transcriptions (such as the one just described), as well as those stored in association with other objects (as described above).

Combined user inputs may be detected by monitoring for them when in a multi-input mode, as opposed to when not in a multi-input mode. If a given environment is always in a multi-input mode, then the monitoring may be continuous - or nearly so. The multi-input mode may be entered into via a selection or command in a user interface to an environment. For example, the selection of the multi-input mode may be a touch-and-drag gesture made with respect to an icon associated with the multi-input mode. The gesture "drags" the icon from an initial position to a position on a content canvas in the user interface. Press-and-hold gestures made with respect to the same icon when in the initial position, generate captured versions of voice signals that are received coincident with the press-and-hold gestures.

In some scenarios, a slide gesture made with the icon transitions the user experience from one application to another while in the multi-input mode. As an example, a user browsing email may select an email for voice annotation, which causes a "listening" window to surface. In this example, the listening graph includes a button for launching the camera application. The user can slide the multi-input icon onto the camera button which causes the camera application to load in the user interface - in the multi-input mode. The user may then proceed to annotate images and/or video captured by the camera.

Referring now to the drawings, <FIG> illustrates an operational scenario <NUM> in an implementation of combined user inputs. Operational scenario <NUM> relates to computing device <NUM>, examples of which include - but are not limited to - mobile phones, tablet computers, laptop computers, desktop computers, wearable devices (e.g. watches), and any other computing devices, combination of devices, or variation thereof. Computing device <NUM> may be implemented in accordance with any suitable computing architecture of which the architecture of computing device <NUM> in <FIG> is representative.

Computing device <NUM> includes one or more user interface devices, systems, or sub-systems through which a user may interact with software and services running on the device. Examples include display <NUM> and microphone <NUM>.

Operational scenario <NUM> begins with an inbox <NUM> and feature menu <NUM> displayed on display <NUM>. Inbox <NUM> is representative of a screen presented by a personal information management application for users to view the contents of the email, e.g. email <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. A personal information management application (e.g. an email application) is illustrated for exemplary purposes, although it may be appreciated that the concepts disclosed herein with respect to combined user input apply to any other applications.

Feature menu <NUM> includes various options for navigating to different parts or features of the application such as a compose mail option, a search option, and a calendar option. Feature menu <NUM> also includes an option for transitioning to multi-input mode, represented by capture button <NUM>. Capture button <NUM> comprises an icon and associated controls that respond to various user inputs to drive the behavior of the application.

For instance, touch and dragging capture button <NUM> via user input <NUM> onto the content canvas of inbox <NUM> (or the content canvas of any other application) triggers the application to transition to multi-input mode. Feature menu <NUM> disappears from display <NUM> upon the transition to multi-input mode, thereby giving the user a visual clue to the modal change. Capture button <NUM> also lands elsewhere on the content canvas to provide the user with further contextual awareness of the mode.

Once in the multi-input mode, computing device <NUM> employs combined input process 200A, illustrated in <FIG>, to enhance the user experience. Combined input process 200A may be implemented in program instructions in the context of any of the modules, components, or other such elements of the operating system and/or application(s) supported by computing device <NUM>. The program instructions direct computing device <NUM> to operate as described for combined input process 200A, referring parenthetically to the steps in <FIG>.

To begin, computing device <NUM> identifies an occurrence of a combined user input (step <NUM>). Examples of combined user inputs include spoken utterances (speech) that are carried by voice signals that occur at least partly coincident with touch gestures, spatial gestures, or the like. While two user inputs are disclosed herein, a combined user input may include more than two user inputs in some implementations (e.g. voice, touch, and spatial inputs).

Next, computing device <NUM> identifies a content object associated with the combined user input (step <NUM>). This may include, for instance: identifying an email, a document, an object in a photo or video; an image on a page, a paragraph of text, or any other type of content object that is the target of a sustained touch gesture (press and hold).

Having identified the content object, computing device <NUM> associates the decoded utterance with the content object (step <NUM>). For example, computing device <NUM> may annotate the content object with the words of the spoken utterance or store an audio recording of the utterance in association with the content object. In another example, a new combined-input object may be created that includes both the content object and the captured version of the utterance. The combined-input object may include meta data that points to both the content object and the captured version of the utterance, although it may also include the actual data for the content object and the utterance. In any case, the content object and captured version of the utterance are associated such that a combination of the two items may be retrieved and presented together in a later context such as an application, a module of an application, or the like.

<FIG> illustrates a functional architecture 200B for implementing combined input process 200A. Functional architecture 200B includes speech-to-text engine <NUM>, gesture engine <NUM>, process module <NUM>, which may be implemented in hardware, software, firmware, or any combination thereof. Functional architecture 200B may be implemented on a single device (e.g. computing device <NUM>) or distributed over multiple devices, co-located or remote.

Speech-to-text engine <NUM> is operatively coupled with process module <NUM>, as is gesture engine <NUM>. In operation, speech-to-text engine <NUM> receives audio data captured by an audio sub-system of computing device <NUM>. The audio data comprises a symbolic representation of an analog voice signal having a spoken utterance encoded therein. The audio data may be formatted in accordance with any suitable protocol and in an uncompressed or compressed state. Non-limiting examples of such protocols include WAV, PCM, MP3, and MP4. Speech-to-text engine <NUM> analyzes the audio data to identify and extract utterances within the recordings, which may then be provided to process module <NUM>.

Process module <NUM> also receives gesture information from gesture engine <NUM>. Gesture engine <NUM> receives touch data from one or more sub-systems of computing device <NUM> and may itself be comprised of one or more hardware, software, or firmware components. The touch data may indicate a location on a display screen where a user initiated a touch gesture, as well as timing data that indicates the duration of the gesture. Gesture engine <NUM> uses the touch data to determine whether a content object was the target of a touch gesture and if so - for how long. Gesture engine <NUM> passes the gesture information to process module <NUM> for consideration in the context of combined input process 200A.

Process module <NUM> takes-in the utterances and gesture information from speech-to-text engine <NUM> and gesture engine <NUM> respectively and analyzes them - per combined input process 200A - to determine whether a combined user input has occurred that would result in one of several associations of content objects with utterances. For example, process module <NUM> may determine to associate a photo with a transcription, a video with a transcription, a document with a transcription, or an email with a transcription. In some cases, a combined user input may occur that results in transcription that lacks any association with a content object. Process module <NUM> produces a note, meta data indicative of a note, or both. The resulting note(s) may be presented in a variety of contexts such as an application for managing and interfacing with a collection of notes.

<FIG> briefly illustrates an object architecture 200C in an exemplary implementation. Object architecture 200C includes content object <NUM>, utterance object <NUM>, combined object <NUM>, and one or more applications, represented by application <NUM>.

Content object <NUM> represents one or more software objects that underpin a given content item such as an email, an image, or a document. Content object <NUM> may be the subject or target of a touch gesture in the context of multi-input mode operation.

Utterance object <NUM> is representative of one or more software objects that form a data representation of a spoken utterance. Utterance object <NUM> may be, for example, a transcription, a recording, or both.

Combined object <NUM> represents one or more data structures that combines one or more elements of each of content object <NUM> and utterance object <NUM>. That is, combined object <NUM> draws from both of the other objects such that a rendering of combined object <NUM> by an application would include visual references to both of the other objects.

Application <NUM> represents any application capable of utilizing combined object <NUM> and - optionally - content object <NUM> and utterance object <NUM>. In an example, a combined object may include (or has meta data that points to) the text in an utterance object and a file icon in a content object, in addition to other graphical elements of the combined object itself.

Application <NUM> would be capable of rendering a visualization of the combined object that includes the text, the file icon, and its other graphical elements (a border, background, etc., which may also be drawn from a graphics library). Application <NUM> may draw the text and file icon directly from the combined object. Alternatively, the combined object may include meta data that points to the text and/or the file icon, in which case application <NUM> would draw the text and file icon from the utterance object and the content object themselves.

<FIG> illustrates an operational scenario <NUM> in an implementation of combined input process <NUM> by computing device <NUM>. In operation, computing device <NUM> has entered into multi-input mode by virtue of a user dragging capture button <NUM> from feature menu <NUM> onto the canvas of inbox <NUM>, which includes emails <NUM>-<NUM>.

Having entered the device into multi-input mode, the user proceeds to supply user input <NUM>, which is representative of a touch-and-hold gesture. While user input <NUM> persists, computing device <NUM> is triggered by the gesture to surface an expanded view <NUM> of email <NUM> (i.e. content object). Additionally, computing device <NUM> surfaces a supplemental window <NUM> that indicates that the device is listening for voice input coincident with user input <NUM>.

The user proceeds to speak, thereby producing voice signal <NUM>. Microphone <NUM> receives the voice signal <NUM>, which an audio sub-system on computing device <NUM> captures and converts to a digital representation. A speech-to-text engine onboard computing device <NUM> (or possible remote from the device, however unlikely) analyzes the digital representation of voice signal <NUM> to extract and transcribe one or more utterances carried by the voice signal.

The transcription is surfaced on supplemental window <NUM> for the user to view. When the user ceases user input <NUM>, a new combined object is produced and supplied to one or more applications for display in a list or in some other context. Capture button <NUM> is updated on display <NUM> to indicate that a new combined object has been created and is available. For example, the new combined object may be represented using the metaphor of a note within the context of a note application, a task within a task application, or the like. Updating capture button <NUM> provides the user with a visual hint that the note has been persisted to the application.

<FIG> illustrates another operational scenario <NUM> in an additional implementation of combined input process <NUM>. In operation, computing device <NUM> has again entered into multi-input mode, but this time within the context of a messaging, conversation, or other such collaboration and communication application or tool.

Capture button <NUM> in this scenario resides above a conversation view <NUM> of the various messages or conversations carried on via the application or tool. For example, conversation view <NUM> includes conversation <NUM>, conversation <NUM>, and conversation <NUM>. Conversation <NUM> includes a preview <NUM> of a document (e.g. a presentation deck) that was attached to or otherwise referenced in the conversation. The user may expand the preview <NUM> of the document by supplying user input <NUM> such as a touch gesture, a spoken command, a mouse click, or the like.

While the preview <NUM> is displayed in its expanded state, the user proceeds to supply user input <NUM>, which is a touch-and-hold gesture made with respect to capture button <NUM>. User input <NUM> causes a window <NUM> to surface in association with the preview <NUM>, to give the user the indication that the device is monitoring for the user to speak.

While the user continues to touch-and-hold capture button <NUM>, the user begins to speak, producing voice signal <NUM>. Computing device <NUM> captures voice signal <NUM> and transcribes an utterance encoded in the signal. The utterance then surfaces in window <NUM>. The user may then release capture button <NUM> which stops the transcriptions process and returns preview <NUM> to its original state. In addition, a combined object is created from all or portions of the transcribed utterance and the document and capture button <NUM> is updated to reflect that a new object has been created and persisted.

In <FIG>, two variations on an implementation of combined user input are illustrated. In operational scenario 500A, user input <NUM> comprises a touch-and-hold gesture made with respect to capture button <NUM>. That is, the user presses and holds capture button <NUM>, as opposed to touching one of the emails in inbox <NUM>. User input <NUM> causes computing device <NUM> to surface window <NUM>. Window <NUM> includes the same capture button as well as a camera icon <NUM>.

In operational scenario 500A, the user proceeds to maintain user input while also speaking into microphone <NUM>. Microphone <NUM> captures voice signal <NUM> which is decoded and analyzed for a spoken utterance. A transcription of the utterance surfaces in window <NUM>. Once the user ceases user input <NUM>, the transcription of the utterance is made available to one or more applications and capture button <NUM> is updated to reflect the recent addition of the transcription to a collection of notes, tasks, or other such items accessible through an application that may be reached by selecting capture button <NUM>.

Operational scenario 500B, illustrated in <FIG>, gives an alternative example whereby the user navigates to the camera function or application on computing device <NUM>. The again provides a user input <NUM> comprises a touch-and-hold gesture made with respect to capture button <NUM>. User input <NUM> brings up window <NUM>, which includes the same capture button as well as a camera icon <NUM>. However, rather than maintaining the touch-and-hold gesture, the user performs a new gesture <NUM> by sliding the touch point onto camera icon <NUM>.

Sliding the touch point to camera icon <NUM> causes display <NUM> to transition of a view <NUM> provided by a camera application, function, feature, or the like. View <NUM> includes an image of whatever scene is presently in-view of the camera lens of computing device <NUM>. The scene in this example is of a white board with various notes scrawled on its canvas. View <NUM> also includes capture button <NUM> hovering over the canvas of the view. The user may proceed to take a photo, change to a video or slow-motion mode, or - as described with respect to <FIG> utilize combined user input to enhance the content capture experience.

<FIG> illustrates operational scenario 600A in which display <NUM> presents a view <NUM> supplied by a camera application. View <NUM> includes an image of scene <NUM> taken in by the camera lens and sub-system of computing device <NUM>. Scene <NUM> includes a whiteboard with various notes, symbols, and graphs written on it.

The user proceeds to pan the camera to the left such that the left-most part of the whiteboard in view <NUM> comes into full view while the right-most portion slides partially out of view. The user then supplies user input <NUM> (a touch-and-hold gesture) on an area of scene <NUM> that includes an exemplary formula. The gesture causes window <NUM> to surface on display <NUM>.

While maintaining user input <NUM>, the user vocalizes an utterance which is carried by voice signal <NUM> and received by microphone <NUM>. A speech-to-text engine on computing device <NUM> extracts the utterance from a digital representation of the signal and transcribes the utterance into text. Computing device <NUM> then displays the text within window <NUM>.

Once the user releases the touch-and-hold gesture, a sub-view <NUM> of the region of interest in scene <NUM> that was the target of user input <NUM> is displayed over the scene. The image (or sub-image) that is used to create the sub-view <NUM> is a content object which may then be merged with the transcription to form a new combined object. The new combined object may be stored in association with the other such notes and tasks. Window <NUM> and sub-view <NUM> disappear from view <NUM> upon subsequent user input, a timer expiring, or the like.

<FIG> illustrates operational scenario 600B in which display <NUM> presents a view <NUM> of video supplied by a camera application. View <NUM> includes an image of scene <NUM> captured by the camera lens and sub-system of computing device <NUM>. As mentioned, scene <NUM> includes a whiteboard with various notes, symbols, and graphs written on it. View <NUM> includes a button <NUM> to change view <NUM> from an outward perspective to a self-facing perspective. Button <NUM> allows the user to start and stop recording. View <NUM> also includes capture button <NUM> that would navigate the user to a sift application where combined objects may be consumed.

In operation, the user proceeds to pan the camera to the left such that the left-most part of the whiteboard in view <NUM> comes into full view while the right-most portion slides partially out of view. The user then supplies user input <NUM> (a touch-and-hold gesture) on an area of scene <NUM> that includes the exemplary formula. The gesture again causes window <NUM> to surface on display <NUM>.

Once the user releases the touch-and-hold gesture, the sub-view <NUM> of the region of interest in scene <NUM> that was the target of user input <NUM> is displayed over the scene. The image (or sub-image) that is used to create the sub-view <NUM> is a content object which may then be merged with the transcription to form a new combined object. The new combined object may be stored in association with the other such notes and tasks. Window <NUM> and sub-view <NUM> disappear from view <NUM> upon subsequent user input, a timer expiring, or the like.

In addition, capture button <NUM> is updated to indicate that a new combined object (e.g. a note or task) is available via the sift application. The user can touch, click-on, or otherwise select capture button <NUM> to navigate to the application and/or list of objects. The combined object in this example would be comprised of the transcription of the utterance in voice signal <NUM> and the image(s) used to populate sub-view <NUM>.

Operational scenario 600C in <FIG> illustrates a user experience in which display <NUM> again presents view <NUM> of video supplied by a camera application. View <NUM> includes an image of scene <NUM> captured by the camera lens and sub-system of computing device <NUM>. As mentioned, scene <NUM> includes a whiteboard with various notes, symbols, and graphs.

In operation, he user then supplies user input <NUM> (a touch-and-hold gesture) on an area of scene <NUM> that includes an exemplary graph or chart. The gesture again causes window <NUM> to surface on display <NUM>.

While maintaining user input <NUM>, the user vocalizes an utterance which is carried by voice signal <NUM> and received by microphone <NUM>. A speech-to-text engine on computing device <NUM> extracts the utterance from a digital representation of the signal and transcribes the utterance into text. Computing device <NUM> then displays the text within window <NUM>. Once the user releases the touch-and-hold gesture, a sub-view <NUM> of the region of interest in scene <NUM> that was the target of user input <NUM> is displayed over the scene.

Window <NUM> in this scenario includes an inking button <NUM> that, when selected, allows the user to annotate images with digital ink. User input <NUM> (e.g. a touch gesture) made with respect to inking button <NUM> causes window <NUM> to recede or disappear, to be replaced by an expanded version of sub-view <NUM> on which the user may ink with a digital inking implement <NUM>. The user provides digital ink <NUM> on the graph in sub-view <NUM>. The digital ink <NUM> is persisted to a combined object along with the image (or sub-image) that is used to create the sub-view <NUM> and the transcription of the utterance in voice signal <NUM>.

The new combined object may be stored in association with other such notes and tasks. Sub-view <NUM> disappears from view <NUM> upon subsequent user input, a timer expiring, or the like. In addition, capture button <NUM> is updated to indicate that a new combined object is available via the sift application. The user can touch, click-on, or otherwise select capture button <NUM> to navigate to the application and/or list of objects.

<FIG> illustrates operational scenario <NUM> in an implementation of a capture application. In operational scenario <NUM>, a view of inbox is displayed on display <NUM>. The user supplies user input <NUM> with respect to capture button <NUM> to navigate to captures view <NUM>. While illustrated with respect to inbox <NUM>, in may be appreciated that the user could navigate to captures view <NUM> from anywhere capture button <NUM> may reside or via other actions such as selecting a captures icon to launch the captures application.

Captures view <NUM> includes all of the combined objects captured in the preceding operational scenarios. For example, captures view <NUM> includes combined object <NUM> created by the combination of a transcribed voice annotation (transcription) and email content in operational scenario <NUM>. Combined object <NUM> includes the combination of the transcription and document in operational scenario <NUM>. Combined object <NUM> includes the transcription captured in operational scenario 500A. Lastly, combined object <NUM> includes the transcription captured in operational scenario 600B, as well as a preview <NUM> of the image.

Note that combined object <NUM> includes an email icon <NUM> which provides a link to the email object associated with the combined object. The user can select email icon <NUM> to open the associated email.

Combined object <NUM> includes a document icon <NUM> that links to the document associated with combined object <NUM>, and combined object <NUM> includes the preview <NUM> of its associated image. Document icon <NUM> can be selected to open the associated document, as can preview <NUM> to open the associated image.

As an example, user input <NUM> comprises a selection of combined object <NUM>, causing an expanded view <NUM> of the combined object to be opened. The expanded view <NUM> also includes email icon <NUM>. Selecting email icon <NUM> via user input <NUM> causes computing device <NUM> to open email view <NUM>. Email view <NUM> includes the contents of the associated email such as a subject area <NUM>, a content canvas <NUM>, and a reply button <NUM>.

Note also that combined object <NUM> includes a calendar icon <NUM>. The calendar icon <NUM> is present when computing device <NUM> is able to determine that a given transcription pertained to a schedulable event referenced in the transcribed utterance. For example, computing device <NUM> in this example recognized the key word "appointment" and can automatically provide a link via calendar icon <NUM> to create a calendar event.

Additional combined objects may be present but out of view below the bottom of display <NUM>. For instance, the user could navigate to the combined object created in the context of operational scenario 600C. However, users may also access combined objects by other means such as that illustrated in <FIG>.

Operational scenario <NUM> in <FIG> pertains to photo gallery <NUM>. Photo gallery <NUM> includes thumbnails <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> of photos and/or videos taken with a camera application on computing device <NUM>. For those of the images that were the subject of a combined user input - or otherwise contributed to a combined object - icons embedded or associated with the thumbnails provide a mechanism for navigating to the associated combined objects. For example, thumbnail <NUM> includes icon <NUM>, and thumbnail <NUM> includes icon <NUM>. A user may select a given icon to navigate to a corresponding combined object.

For instance, a selection <NUM> of icon <NUM> launches the user to a view <NUM> of the combined object created in the context of operational scenario 600C. View <NUM> includes the image and inking captured during the scenario, as well as a play button <NUM> to play out the associated video. View <NUM> also includes a capture button <NUM> to navigate to captures view <NUM>. A selection <NUM> of capture button <NUM> causes computing device <NUM> to navigate the user to captures view <NUM>, whereby the user can proceed to view and consume the list of combined objects as discussed with respect to <FIG>.

<FIG> illustrates computing device <NUM> that is representative of any device or collection of devices in which the various processes, programs, services, and scenarios disclosed herein may be implemented. Examples of computing device <NUM> include, but are not limited to, mobile phones, tablets, laptops, desktops, watches and other wearable devices, and Internet of Things (IoT) devices. Some or all aspects of the various processes, programs, services, and scenarios disclosed herein may also be implemented on server computers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machine, container, and any variation or combination thereof.

Computing device <NUM> may be implemented as a single apparatus, system, or device or may be implemented in a distributed manner as multiple apparatuses, systems, or devices. Computing device <NUM> includes, but is not limited to, processing system <NUM>, storage system <NUM>, software <NUM>, communication interface system <NUM>, and user interface system <NUM>. Processing system <NUM> is operatively coupled with storage system <NUM>, communication interface system <NUM>, and user interface system <NUM>.

Processing system <NUM> loads and executes software <NUM> from storage system <NUM>. Software <NUM> includes and implements combined input process <NUM>, which is representative of the combined input processes discussed with respect to the preceding Figures. When executed by processing system <NUM> to enhance the user experience with respect to content capture experiences, software <NUM> directs processing system <NUM> to operate as described herein for at least the various processes, operational scenarios, and sequences discussed in the foregoing implementations. Computing device <NUM> may optionally include additional devices, features, or functionality not discussed for purposes of brevity.

Referring still to <FIG>, processing system <NUM> may comprise a microprocessor and other circuitry that retrieves and executes software <NUM> from storage system <NUM>. Processing system <NUM> may be implemented within a single processing device but may also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing system <NUM> include general purpose central processing units, graphical processing units, application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof.

Storage system <NUM> may comprise any computer readable storage media readable by processing system <NUM> and capable of storing software <NUM>. Storage system <NUM> may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read only memory, magnetic disks, optical disks, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage media. In no case is the computer readable storage media a propagated signal.

Software <NUM> (including combined input process <NUM>) may be implemented in program instructions and among other functions may, when executed by processing system <NUM>, direct processing system <NUM> to operate as described with respect to the various operational scenarios, sequences, and processes illustrated herein. For example, software <NUM> may include program instructions for implementing a combined input process as described herein.

In particular, the program instructions may include various components or modules that cooperate or otherwise interact to carry out the various processes and operational scenarios described herein. The various components or modules may be embodied in compiled or interpreted instructions, or in some other variation or combination of instructions. The various components or modules may be executed in a synchronous or asynchronous manner, serially or in parallel, in a single threaded environment or multi-threaded, or in accordance with any other suitable execution paradigm, variation, or combination thereof. Software <NUM> may include additional processes, programs, or components, such as operating system software, virtualization software, or other application software. Software <NUM> may also comprise firmware or some other form of machine-readable processing instructions executable by processing system <NUM>.

In general, software <NUM> may, when loaded into processing system <NUM> and executed, transform a suitable apparatus, system, or device (of which computing device <NUM> is representative) overall from a general-purpose computing system into a special-purpose computing system customized to provide enhanced content capture capabilities. Indeed, encoding software <NUM> on storage system <NUM> may transform the physical structure of storage system <NUM>. The specific transformation of the physical structure may depend on various factors in different implementations of this description. Examples of such factors may include, but are not limited to, the technology used to implement the storage media of storage system <NUM> and whether the computer-storage media are characterized as primary or secondary storage, as well as other factors.

For example, if the computer readable storage media are implemented as semiconductor-based memory, software <NUM> may transform the physical state of the semiconductor memory when the program instructions are encoded therein, such as by transforming the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. A similar transformation may occur with respect to magnetic or optical media. Other transformations of physical media are possible without departing from the scope of the present description, with the foregoing examples provided only to facilitate the present discussion.

Communication interface system <NUM> may include communication connections and devices that allow for communication with other computing systems (not shown) over communication networks (not shown). Examples of connections and devices that together allow for inter-system communication may include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry. The connections and devices may communicate over communication media to exchange communications with other computing systems or networks of systems, such as metal, glass, air, or any other suitable communication media. The aforementioned media, connections, and devices are well known and need not be discussed at length here.

Communication between computing device <NUM> and other computing systems (not shown), may occur over a communication network or networks and in accordance with various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, internets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software defined networks, data center buses and backplanes, or any other type of network, combination of network, or variation thereof. The aforementioned communication networks and protocols are well known and need not be discussed at length here.

User interface system <NUM> may include a keyboard, a mouse, a voice input device, a touch input device for receiving a touch gesture from a user, a motion input device for detecting non-touch gestures and other motions by a user, and other comparable input devices and associated processing elements capable of receiving user input from a user. Output devices such as a display, speakers, haptic devices, and other types of output devices may also be included in user interface system <NUM>. In some cases, the input and output devices may be combined in a single device, such as a display capable of displaying images and receiving touch gestures. The aforementioned user input and output devices are well known in the art and need not be discussed at length here.

User interface system <NUM> may also include associated user interface software executable by processing system <NUM> in support of the various user input and output devices discussed above. Separately or in conjunction with each other and other hardware and software elements, the user interface software and user interface devices may support a graphical user interface, a natural user interface, a conversational user interface, or any other type of user interface.

Claim 1:
A computing apparatus comprising:
one or more computer readable storage media;
one or more processors operatively coupled with the one or more computer readable storage media; and
program instructions stored on the one or more computer readable storage media that, when executed by the one or more processors, direct the computing apparatus, when in a multi-input mode, to at least:
display, on a display screen of the computing apparatus, a content object in a user interface of an application;
monitor for occurrences of combined modality inputs;
detect an occurrence of a combined modality input (<NUM>, <NUM>), wherein the combined modality input comprises an utterance (<NUM>) captured via a microphone of the computing apparatus and a touch gesture (<NUM>) sustained on the display screen of the computing apparatus, wherein the touch gesture occurs at least partially coincident with the utterance;
in response to detecting the occurrence of the combined modality input, identify the content object (<NUM>) associated with the touch gesture; and
generate a combined object comprising a transcription of a captured version of the utterance and an icon for opening the content object with the content object in the application.