Multi-threaded conditional processing of user interactions for gesture processing using rendering thread or gesture processing thread based on threshold latency

Embodiments isolate gesture processing for improved user interactivity with content. A first thread detects user interaction with a document via a touch sensor associated with a computing device. A latency associated with a second thread is calculated. The calculated latency is compared to a threshold latency. Based on the comparison, the first thread determines whether to transfer processing of the detected user interaction from the first thread to the second thread. The detected user interaction is processed by the first thread or by the second thread based on the determination. In some embodiments, users are provided with smooth interactivity such as panning and scrolling when browsing web content with a browser executing on a mobile computing device.

BACKGROUND

Mobile devices provide increasing functionality and high-quality graphics. The functionality is accessible by users via interfaces such as touch screens that accept gestures input by the users. Responsiveness from the touch screens suffers due to other processes executing on the mobile devices. For example, many applications executing on the mobile devices are single-threaded. For such applications, if a processor-intensive operation is performed (e.g., execution of a script), responsiveness of user interactivity with the mobile device suffers. As such, while some existing systems can process user interactions in approximately 800 milliseconds, such a delay is noticeable to users who can detect delays of 150 milliseconds.

SUMMARY

Embodiments of the disclosure separate gesture processing from rendering processing. A first thread detects user interaction with a document via a touch sensor associated with a computing device. The first thread handles gesture processing. A latency associated with a second thread is calculated. The second thread handles rendering of the document. The calculated latency is compared to a threshold latency. Based on the comparison, the first thread determines whether to transfer processing of the detected user interaction from the first thread to the second thread. The detected user interaction is processed by the first thread or by the second thread based on the determination.

DETAILED DESCRIPTION

Referring to the figures, embodiments of the disclosure dynamically isolate gesture processing from rendering processing and other processing by using a multi-threaded architecture conditioned on latency. In some embodiments, touch interactivity such as scrolling performance for web content is enhanced by faster display of content. Additional touch interactivity benefitting from the disclosure includes gestures such as pan gestures, pinch/stretch gestures, flick gestures, and double-tap zoom gestures. However, aspects of the disclosure are operable with any user interactions. That is, aspects of the disclosure are not limited to particular user interactions, or to a pre-determined or pre-designated subset of user interactions or processes. In operation, aspects of the disclosure improve user interaction with documents (e.g., documents120), applications (e.g., applications118), and other content by, for example, enabling a user104to interact with a web page in a browser instantaneously with smooth panning, scrolling, and the like with reduced checkerboard.

Referring next toFIG. 1, an exemplary block diagram illustrates a computing device102having computer-executable components for separating gesture processing from rendering processing. In the example ofFIG. 1, the computing device102is associated with the user104. The computing device102represents any device executing instructions (e.g., as application programs, operating system functionality, or both) to implement the operations and functionality associated with the computing device102. The computing device102may include a mobile computing device or any other portable device. In some embodiments, the mobile computing device includes a mobile telephone, laptop, tablet, computing pad, netbook, gaming device, and/or portable media player. The computing device102may also include less portable devices such as desktop personal computers, kiosks, and tabletop devices. Additionally, the computing device102may represent a group of processing units or other computing devices.

The computing device102has at least one processor110, a plurality of threads, a memory area116, and at least one user interface113. The processor110includes any quantity of processing units, and is programmed to execute computer-executable instructions for implementing aspects of the disclosure. The instructions may be performed by the processor110or by multiple processors executing within the computing device102, or performed by a processor external to the computing device102. In some embodiments, the processor110is programmed to execute instructions such as those illustrated in the figures (e.g.,FIG. 2andFIG. 3).

The computer device further has a plurality of threads or processes such as a first thread112and a second thread114. Each of the threads represents a unit of processing that is capable of being scheduled for performance or execution by the computing device102.

The computing device102further has one or more computer readable media such as the memory area116. The memory area116includes any quantity of media associated with or accessible by the computing device102. The memory area116may be internal to the computing device102(as shown inFIG. 1), external to the computing device102(not shown), or both (not shown).

The memory area116stores, among other data, one or more applications118. The applications118, when executed by the processor110, operate to perform functionality on the computing device102. Exemplary applications118include mail application programs, web browsers, calendar application programs, address book application programs, messaging programs, media applications, location-based services, search programs, and the like. The applications118may communicate with counterpart applications or services such as web services accessible via a network108. For example, the applications118may represent downloaded client-side applications that correspond to server-side services executing in a cloud.

The applications118may also interact with the user104by receiving or displaying data via, for example, a touch screen or other user interface113associated with the computing device102. The touch screen represents one or more touch sensors in any arrangement, collection, configuration, or organization for communicating with the user104. For example, the touch screen may be a touch pad associated with a laptop, a full screen of a mobile computing device or tablet device, and/or a fingerprint biometric pad (e.g., for logging in).

The memory area116further stores or caches one or more documents120or portions thereof. Exemplary documents120include web pages, hypertext markup language documents, word processing documents, or the like. Each of the documents120may have one or more document elements122associated therewith. For example, the documents120may include frames, panels, subpanel, popups, and the like. The document elements122are adjustable or configurable by the user104via, in some embodiments, the user interface113(e.g., touch screen). For example, the document elements122may be capable of being scrolled, zoomed, panned, re-sized, etc. In some embodiments, the documents120are received from a content provider106(e.g., server) via network108, such as when the user104browses web sites.

The memory area116further stores a threshold latency124. The threshold latency124represents a value or criteria used to determine whether or not to assign a requested user interaction to a particular thread, and/or whether or not to transfer processing of the requested user interaction from one thread to another. The threshold latency124is based upon, for example, the ability of the user104to perceive or otherwise notice delays associated with interaction with the computing device102. An exemplary latency is 50 milliseconds. Use of the threshold latency124is described below with reference toFIG. 2andFIG. 3.

The memory area116further stores one or more computer-executable components that, when executed by the processor110, cause the processor110to separate gesture processing from rendering processing. Exemplary components include a user interface component126, a touch point component128, a measurement component130, and an overlay component132. Execution of the computer-executable components is described below with reference toFIG. 2.

Referring next toFIG. 2, an exemplary flow chart illustrates operation of the computing device102to conditionally select a thread to process a user interaction with the document120. The document120may include, for example, a web page and/or data displayed by an executing application program (e.g., application118). In the example ofFIG. 2, the first thread112handles gesture processing and the second thread114handles rendering processing. In some embodiments, the first thread112corresponds to a fast response thread while the second thread114corresponds to a slow response thread. If user interaction with the document120via a touch sensor is detected at202by the first thread112, a latency associated with the second thread114is calculated at204. For example, detecting the user interaction includes detecting user interaction with a web page.

In some embodiments, calculating the latency includes requesting, from the second thread114, touch point data associated with the detected user interaction, or otherwise performing a “hit test” of the second thread114. An elapsed time since the request is calculated. The latency is then determined based at least on the calculated elapsed time. In embodiments in which portions of the touch point data are provided by the second thread114asynchronously, the latency may be determined when a particular element of the touch point data is received (e.g., specific to a particular gesture), when a particular quantity of elements are received, or the like.

The calculated latency is compared to the threshold latency124at206. Based on the comparison, the computing device102determines whether to transfer processing of the user interaction to the second thread114at208. If the computing device102decides to transfer processing to the second thread114, the second thread114processes the user interaction at210. For example, if the second thread114provides the requested touch point data or otherwise responds to the first thread112within the latency threshold, the first thread112may proceed to transfer processing of the user interaction to the second thread114. In some embodiments, the second thread114processes the detected user interaction if the calculated latency is less than the threshold latency124.

If the computing device102decides not to transfer processing to the second thread114, the first thread112processes the user interaction at212. For example, if the second thread114fails to provide the requested touch point data or otherwise respond to the first thread112, the first thread112proceeds to process the user interaction. In some embodiments, the first thread112processes the detected user interaction if the calculated latency is greater than the threshold latency124.

Depending on how the threshold latency124is defined, the decision at208may be performed using any relative comparison of the threshold latency124and the calculated latency. Exemplary comparison criteria include one or more of the following: greater than, greater than or equal to, less than, and less than or equal to.

In some embodiments, the first thread112determines an operation associated with the detected user interaction by analyzing the touch point data received from the second thread114. For example, the first thread112may correlate the touch point data with a gesture. Based on the gesture, the first thread112determines what operation to perform on the document120. In instances in which the first thread112processes the detected user interaction, the first thread112may apply the operation to a version of the document120cached by the computing device102. In instances in which the second thread114processes the detected user interaction, the first thread112identifies the determined operation to the second thread114during transfer of the user interaction processing to the second thread114. The second thread114then applies the operation to the document120. For example, the second thread114applies the operation to a “live” or fresh version of the document120stored by another computing device (e.g., content provider106).

The computer-executable components illustrated inFIG. 1operate to implement the functionality illustrated inFIG. 2. For example, the user interface component126, when executed by the processor110, causes the processor110to detect, by the fast response thread, user interaction with the document120via the touch sensor associated with the computing device102. In some embodiments, the user interface component126includes a graphics card for displaying data to the user104and receiving data from the user104. The user interface component126may also include computer-executable instructions (e.g., a driver) for operating the graphics card. Further, the user interface component126may include a display (e.g., a touch screen display) and/or computer-executable instructions (e.g., a driver) for operating the display. The user interface component126may also include one or more of the following to provide data to the user104or receive data from the user104: speakers, a sound card, a camera, a microphone, a vibration motor, one or more accelerometers, a BLUETOOTH brand communication module, global positioning system (GPS) hardware, and a photoreceptive light sensor. For example, the user104may input commands or manipulate data by moving the computing device102in a particular way.

The touch point component128, when executed by the processor110, causes the processor110to request, from the slow response thread by the fast response thread, touch point data corresponding to the user interaction detected by the user interface component126. The requested touch point data is associated with an operation to be applied to at least a portion of the document120. For example, the user interaction corresponds to a gesture and the operation includes a scrolling operation.

The measurement component130, when executed by the processor110, causes the processor110to compare, by the fast response thread, a latency associated with the slow response thread to the threshold latency124. The overlay component132, when executed by the processor110, causes the processor110to apply, based on the comparison performed by the measurement component130, the operation by the fast response thread to the entire document120or by the slow response thread to only the portion of the document120associated with the received touch point data. For example, the overlay component132applies the operation by the fast response thread to the entire document120if the latency violates the threshold latency124. In another example, the overlay component132applies the operation by the slow response thread to only the portion of the document120if the latency satisfies the threshold latency124.

Referring next toFIG. 3, an exemplary flow chart illustrates operation of the computing device102to select a thread to scroll a web page. The web page may be displayed from, for example, a web site on the Internet. In the example ofFIG. 3, a gesture processing thread handles gesture processing and a rendering processing thread handles rendering processing. In some embodiments, the gesture processing thread corresponds to a fast response thread while the rendering processing thread corresponds to a slow response thread. If a request from the user104to scroll the web page via a touch sensor is detected by the gesture processing thread at302, touch point data is requested from the rendering processing thread at304. The touch point data describes one or more of the following: whether the touch point is scrollable, whether the touch point is editable, and the shape of the touch point. Further, the touch point data identifies, or may be correlated to, one or more of the scrollable elements within the web page.

A latency associated with the rendering processing thread is next calculated based on the time elapsed since the request for the touch point data was sent. The calculated latency is compared to the threshold latency124. The gesture processing thread waits until the elapsed request time exceeds the threshold latency124or the requested touch point data is received. If the elapsed request time exceeds the threshold latency124at306, the gesture processing thread scrolls the entire web page at308. If the requested touch point data is received at310, the rendering processing thread scrolls only the scrollable elements identified by, or corresponding to, to the touch point data at312. For example, the gesture processing thread “hands off” or otherwise transfers processing of the request from the user104to scroll the web page to the rendering processing thread.

Referring next toFIG. 4, an exemplary sequence diagram illustrates thread selection for processing gesture inputs. In the example ofFIG. 4, the gesture input is processed by the gesture processing thread402or the rendering thread404, depending on the latency of the rendering thread404as next described. After the gesture processing thread402detects the gesture input at406(e.g., a scroll request or other user input request), the gesture processing thread402requests touch point data at408corresponding to the detected gesture input from the rendering thread404. If the rendering thread404provides the requested touch point data at410to the gesture processing thread402within the threshold latency124, the gesture processing thread402transfers processing of the gesture input to the rendering thread404at412. The rendering thread404then applies or performs the gesture to the particular subpage element identified in the touch point data at414(e.g., a frame or other scrollable element), and not to the entire web page.

Alternatively, if the rendering thread404fails to provide the requested touch point data to the gesture processing thread402within the threshold latency124(e.g., 50 milliseconds), the gesture processing thread402processes the gesture input by applying or performing the gesture to the entire web page at416, rather than to any subpage element.

Referring next toFIG. 5, an exemplary block diagram illustrates the computing device102with a multi-threaded architecture for separating gesture processing from rendering processing. In the example configuration ofFIG. 5, the fast response thread generally handles gesture processing, while the slow response thread handles rendering processing. While the threads are described as “fast” and “slow” to distinguish between the threads, aspects of the disclosure are operable with any threads that can be distinguished from each other in any way. For example, both threads may respond very quickly, but one thread responds slightly faster than the other does. The slightly faster thread may be designated the “fast response thread” while the slightly slower thread may be designated the “slow response thread.”

The fast response thread is responsible for an overlay window502, ICS504, animation engine506, scroll indicator508, and gesture controller510. The animation engine506directs animation results (e.g., frame updates via a frame renderer) directly to ICS504and scroll position to the scroll indicator508. The gesture controller510uses a proxy512to cross a thread boundary to access document support logic520on the slow response thread. The slow response thread is further responsible for a mobile optimization service514and an ICS manager516.

In the example ofFIG. 5, the fast response thread operates on a cached version of a document517, while the slow response thread operates on a “live” version of a document518obtained from, for example, the content provider106.

Additional Examples

In some embodiments, the disclosure is operable with content other than document120. For example, aspects of the disclosure are operable with applications (e.g., applications118), media files, games, and/or any other content. In such embodiments, the operation corresponding to the input user interaction is applied to the content. For example, the user104may scroll a portion of the user interface113displayed by one of the applications118.

While embodiments have been described herein with reference to selecting a thread to process particular user interactions, aspects of the disclosure are operable to select a thread to process any user interaction.

In an example scenario, aspects of the disclosure execute to process gestures within 50 milliseconds of input by the user104. For example, the operations illustrated inFIG. 3execute within the 50 milliseconds.

At least a portion of the functionality of the various elements inFIG. 1may be performed by other elements inFIG. 1, or an entity (e.g., processor, web service, server, application program, computing device, etc.) not shown inFIG. 1.

In some embodiments, the operations illustrated inFIG. 2andFIG. 3may be implemented as software instructions encoded on a computer readable medium, in hardware programmed or designed to perform the operations, or both. For example, aspects of the disclosure may be implemented as a system on a chip.

While no personally identifiable information is tracked by aspects of the disclosure, embodiments have been described with reference to data monitored and/or collected from users104. In such embodiments, notice is provided to the users104of the collection of the data (e.g., via a dialog box or preference setting) and users104are given the opportunity to give or deny consent for the monitoring and/or collection. The consent may take the form of opt-in consent or opt-out consent.

Exemplary Operating Environment

Exemplary computer readable media include flash memory drives, digital versatile discs (DVDs), compact discs (CDs), floppy disks, and tape cassettes. By way of example and not limitation, computer readable media comprise computer readable storage media and communication media. Computer readable storage media store information such as computer readable instructions, data structures, program modules or other data. Computer readable storage media exclude propagated data signals. Communication media typically embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media.

Although described in connection with an exemplary computing system environment, embodiments of the invention are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with aspects of the invention include, but are not limited to, mobile computing devices, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

Embodiments of the invention may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. The computer-executable instructions may be organized into one or more computer-executable components or modules. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types. Aspects of the invention may be implemented with any number and organization of such components or modules. For example, aspects of the invention are not limited to the specific computer-executable instructions or the specific components or modules illustrated in the figures and described herein. Other embodiments of the invention may include different computer-executable instructions or components having more or less functionality than illustrated and described herein.

Aspects of the invention transform a general-purpose computer into a special-purpose computing device when configured to execute the instructions described herein.

The embodiments illustrated and described herein as well as embodiments not specifically described herein but within the scope of aspects of the invention constitute exemplary means for separating gesture processing from rendering processing, and exemplary means for latency-based conditional transfer of gesture processing from one thread to another.