Detecting gesture struggle

A computer implemented method is disclosed. The method includes obtaining, by at least one processor, data of user interaction with an application in a user session in a given computing environment. The method further includes identifying a user gesture interacting with a user interface of the application on a screen of a computing device based on the data. The method further includes calculating a struggle factor score for the user gesture based on the data. The method further includes comparing the struggle factor score with a baseline value, wherein the baseline value is derived from historical information of past user sessions in a computer environment like the given computing environment. The method further includes determining that the user gesture indicates a user struggle in response to a determination that the struggle factor score is above the baseline value.

BACKGROUND

The present disclosure relates generally to the field of user interaction analysis, and more particularly to methods, systems, and computer program products for analyzing screen gestures representing user struggles in interacting with the user interfaces.

User interfaces (UIs) are provided on screens of computing devices, such as smart phones, to provide a way for users of the devices to interact with software and/or applications (apps) that are being run on the devices. A smart phone may be equipped with apps provided by different industries, such as retail, hotel, and airlines, and each of these apps may utilize a UI. A user may interact with the UIs on a touch screen of the smart phone to conduct transactions, for example, to finish a payment via an app running on the smart phone.

Technologies have been developed to capture real-time user sessions to understand user interaction with apps. Such technologies answer questions about what a user is doing, or seeing, on each page during a user session. This information about user interaction can be used to detect user gesture struggles.

SUMMARY

Embodiments of the present disclosure include a method, computer program product, and system for analyzing screen gestures representing user struggles in interacting with the user interfaces.

According to some embodiments of the present disclosure, a computer implemented method includes obtaining, by at least one processor, data of user interaction with an application in a user session in a given computing environment. The computer implemented method further includes identifying a user gesture interacting with a user interface of the application on a screen of a computing device based on the data. The computer implemented method further includes calculating a struggle factor score for the user gesture based on the data. The computer implemented method further includes comparing the struggle factor score with a baseline value, wherein the baseline value is derived from historical information of past user sessions in a computer environment like the given computing environment. The computer implemented method further includes determining that the user gesture indicates a user struggle in response to a determination that the struggle factor score is above the baseline value.

According to another embodiment of the present disclosure, a system includes a computer-readable storage medium and at least one processor coupled to the computer-readable storage medium. The computer-readable storage medium includes instructions, and the at least one processor is configured such that, in response to executing the instructions, the at least one processor performs the method set forth above.

According to another embodiment of the present disclosure, a computer program product includes a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a processor to cause the processor to perform the method set forth above.

DETAILED DESCRIPTION

Aspects of the present disclosure relate generally to the field of user interaction analysis, and in particular to methods, systems, and computer program products for analyzing screen gestures representing user struggles in interacting with the user interfaces. While the present disclosure is not necessarily limited to such applications, various aspects of the disclosure may be appreciated through a discussion of various examples using this context.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Referring now toFIG. 1, a schematic of an example of a cloud computing node10is shown. Cloud computing node10is only one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the present disclosure. Regardless, cloud computing node10may be used in implementing one or more of the methods, tools, and modules, and any related functions, described herein, in accordance with embodiments of the present disclosure.

With reference now to accompanying drawings, exemplary embodiments of the present disclosure will be described. The exemplary embodiments are directed to a method, system and computer program product for detecting gesture struggle.

It should be noted that the processing of the method for detecting gesture struggle of the present disclosure may be implemented by computer system/server12ofFIG. 1.

The present disclosure proposes a method for detecting a gesture struggle by analyzing user gestures on UIs among most sessions, calculating the normal behavior criteria and detecting outliers. In view of existing technologies, there is room for improved automatic struggle analysis on users' gestures on user interfaces of computing devices to enhance the user experience. It is desirable for UI designers to have improved technologies that accurately detect user struggle in interacting with UIs associated with particular computing environments.

It is to be understood that the aforementioned advantages are example advantages and should not be construed as limiting. Embodiments of the present disclosure can contain all, some, or none of the aforementioned advantages while remaining within the spirit and scope of the present disclosure.

FIG. 4depicts a flowchart of a computer-implemented method400for detecting a user gesture struggle according to some embodiments of the present disclosure. The method generally comprises steps410to450, each of which may be performed by at least one processor, as follows. Step410of the method comprises obtaining data pertaining to a user's interaction with an application in a given user session in a given computing environment. Step420of the method comprises identifying a “user gesture” based on the data pertaining to the user's interaction. A “user gesture” is a gesture that the user performs to interact with a UI on a screen of a computing device. Step430of the method comprises calculating a “struggle factor score” for the user gesture based on the data pertaining to the user's interaction. A “struggle factor score” is a numerical value which represents the extent of a possible struggle that the user experiences while interacting with the UI. Step440of the method comprises comparing the struggle factor score with a baseline value, wherein the baseline value is derived from historical information of past user sessions in a computing environment that is similar to the given computing environment. Finally, step450of the method comprises determining whether the user gesture represents a user struggle based on the result of the comparison of the struggle factor score with the baseline value. In particular, if the struggle factor score is greater than the baseline value, then step450of the method determines that the user gesture does represent a user struggle. Conversely, if the struggle factor score is less than the baseline value, then step450of the method determines that the user gesture does not represent a user struggle. Some embodiments of each step of method400will be described in the following paragraphs.

As noted above, method400starts with step410, which comprises obtaining data pertaining to a user's interaction with an app in a given user session in a given computing environment. It is to be assumed that in a given user session, a user is interacting with a UI provided by an app running on a computing device. In the context of the present disclosure, it may be said that there is a user interaction with the app via the UI in a user session. The user interaction may be detected, and the data pertaining to the user interaction may be captured by the computing device. Such data in different sessions may be stored in a storage, such as a database, either locally or remotely. For brevity, the term “data pertaining to user interaction” is also referred to as “data of user interaction” or “user interaction data” in the following description.

Specifically, by way of example, assume that a user is in an online-shopping session, and the user interacts with a “checkout” page, which is a form of UI, of a web-based app on a touch screen of a smart phone functioning as a computing device. The checkout page is generated or provided by the web-based app of the “retail” industry sector. The user may use his or her hand(s) to serve as an input device in order to interact with elements in the UI, such as the checkout page, during the online-shopping session. The actions of the user's hand(s) may be captured by sensors in the smart phone. Signals sensed by the sensors may be processed, resulting in a part of the user interaction data.

In step410of the method400, the user interaction data may be obtained from the storage (e.g., a computer-readable storage medium in a database) as described above. Alternatively, the user interaction data may be obtained directly from the smart phone, which may capture the user interaction data in real time during the user session.

Following step410, step420of the method400comprises identifying a user gesture based on the user interaction data. In other words, step420of the method includes identifying a gesture that the user has used to interact with a UI of the app on a screen of a computing device based on the user interaction data. As mentioned, known techniques exist for analyzing user interaction data to understand the user behavior. Such techniques may be employed in step420to identify any user gesture from the user interaction data obtained in step410.

There are different types of user gestures that may be used by a user to interact with a UI. Hand operations, in particular, are user gestures that are typically used to interact with a UI of an app on a screen of a computing device. Typical user gestures that are hand operations include “dwelling,” “repeated gesture,” “single-handed opposite gesture,” and “slider,” among others.

The term “dwelling” refers to instances wherein one of the user's fingers remains on a given UI element. The element is often an input field such as, for example, text, checkbox, calendar, radio box, email, and the like. The length of time that the user's finger remains on the given UI element may serve as an indicator of abnormal user operation behavior. For example, if the user is visiting a web page and takes much more time on a given text field than other users take on the same text field, there is a high probability that the user is experiencing some struggle in utilizing the UI. The extent of the struggle indicated by the user's dwelling may be measured by a struggle factor score called “dwell time,” which refers to the length of time that the dwelling lasts. It shall be appreciated there may be instances wherein dwelling on some UI elements, such as two-dimensional code, is a requirement designed for completing a function, and thus does not necessarily indicate a user struggle.

The term “repeated gesture” refers to instances wherein the user repeats an action (or a “gesture”) several times. More specifically, “repeated gesture” refers to subset of instances wherein the user unnecessarily performs a gesture repeatedly with respect to either a page or a UI element, in which case the repeated gesture may be indicative of a struggle in utilizing the UI that the user is experiencing. The extent of the struggle indicated by the user's repeated gesture may be measured by a struggle factor score called “repeated gesture,” which refers to a count of the number of times the gesture is repeated.

The term “single-handed opposite gesture” refers to instances wherein the user holds one side of the computing device with one hand and attempts to perform a gesture with the same hand in a location on the screen that is near the opposite side. Assuming that the location on the screen is out of a comfortable zone that is easily accessible for the single hand, this type of gesture struggle exists due to the distance from the side on which the computing device is held to the position where the operation is desired. The extent of a struggle indicated by a user's single-handed opposite gesture may be measured by a struggle factor score called “single-handed opposite gesture,” which refers to a count of the number of times that the user attempts to perform opposite side operations.

The term “slider” refers to instances wherein one of the user's fingers taps and holds at a start position on the screen of the computing device, and while maintaining contact with the screen, moves the finger from the start position to an end position on the screen. The pattern of this movement is tracked and can be referred to as a movement pattern. During an interaction with a UI, the movement pattern of a user's finger typically does not include a winding path, such as a back-and-forth path. However, if the movement pattern does contain a winding path, the user's gesture may indicate that the user is experiencing a struggle in the interaction with the UI. As another example, if the speed of the movement of the user's finger slows significantly near the end position, the user's gesture may also indicate that the user is experiencing a struggle in the interaction with the UI. For brevity, user struggles indicated by a winding path and/or a slowing in the user's movement pattern will be referred to as “slider struggle.” The extent of a slider struggle may be measured by a struggle factor score called “slider ratio,” which represents an extent of the winding path and/or slow-down in the user's movement pattern.

For the following example, assume that in step420of the method the user gesture is identified as a slider. In the following paragraphs, the slider gesture is used as an example for the purpose of illustrating subsequent steps430to450of the method400. Following identification of the user gesture in step420, step430is performed. Thus, a struggle factor score is calculated for the identified user gesture based on the data obtained in step410.

For exemplary instances wherein the user gesture is a slider, the struggle factor score for the slider gesture, also referred to as slider ratio, may be calculated according to the formula (1) below:

As mentioned above, the slider ratio may serve as an indicator of the extent of a user's struggle with a UI while applying the slider gesture. More specifically, the slider ratio may be directly related to the extent of the struggle. In such instances, a larger slider ratio indicates a more difficult struggle experienced by the user and a smaller slider ratio indicates a less difficult struggle experienced by the user.

As previously discussed, struggle factor scores may be defined for various user gestures. Struggle factor scores for respective user gestures identified may be measured and/or calculated based on user interaction data with predefined schemes according to existing techniques or future techniques.

It is also to be noted that although step430and step420are described as being performed sequentially, they may alternatively be performed virtually in parallel, or simultaneously.

Following steps420and430, step440of the method comprises comparing the calculated struggle factor score with a baseline value, wherein the baseline value is derived from historical information of past user sessions in a computing environment that is similar to the given computing environment. In the instant example, wherein the struggle factor score is slider ratio, the baseline value is a baseline value of the struggle factor score slider ratio derived from historical information of such past user sessions. In the context of the present disclosure, historical information of past user sessions in a computing environment that is similar to the given computing environment refers to information or data aggregated for user sessions that occurred in the past in a computer environment that is similar to the computing environment in which the session of interest in step410is occurring. In some embodiments, the historical information is information pertaining only to the user, whereas in other embodiments, the historical information contains information pertaining to one or more other users, in addition to, or instead of, information about the user.

According to some embodiments of the present disclosure, the computing environment may be defined by one or more of the following parameters: the type of UI, the platform of the app, the device model of the computing device, and the industry sector of the app. The type of UI may be, for example, a checkout page, a search page, and the like. The device model may be, for example, iPhone 7®, Huawei Mate 10®, and the like. The platform of the app may be, for example, web, iOS®, Android®, and the like. The industry sector of the app may be, for example, retail, airlines, hotels, and the like.

According to some embodiments of the present disclosure, the one or more parameters defining the computing environment may have been collected along with the user interaction data and stored in association with the user interaction data. The parameters may be used for defining a computing environment in aggregating historical information of past user sessions, as will be further discussed later in the disclosure.

In the instant case, parameters describing the computing environment of the current user session may be collected before step440is performed in method400. Parameters of the given computing environment for the current session are illustrated, as an example, in a table500inFIG. 5. As shown, the first row of the table contains one of the following parameter names in each of the four columns: “UI-Type” (501), “App Platform” (502), “Device Model” (503), and “Industry Sector” (504). The second row of the table contains the following parameters in the corresponding columns: <checkout page, Firefox web, iPhone 7, retail. Accordingly, the table500indicates that the user is interacting with a “checkout page” type of UI on an “iPhone 7” model computing device, that the app is provided by a “retail” industry sector, and that the app is running on a “Firefox web” platform.

As mentioned above, the historical information of past user sessions in a like computing environment refers to information or data aggregated for past user sessions that occurred in a computing environment that is similar to that of the current session. In this instance, the historical information of past user sessions in a like computing environment refers to the information aggregated for past user sessions in a computing environment complying with the parameters: <checkout page, Firefox web, iPhone 7, retail> for the current session, as shown in table500.

The following paragraphs describe some embodiments of aggregating the information of past user sessions in a like computing environment and how to derive a baseline value for a gesture. Before that, assume that the baseline value derived from the information of past user sessions is 215%.

Following step440, step450comprises determining whether the user gesture represents a user struggle based on the result of the comparison of the struggle factor score with the baseline value. In the instant example, at step440the struggle factor score for the slider gesture (slider ratio of 250%) is compared to the baseline value for the slider gesture (215%). Because the slider ratio is greater than the baseline value for the slider gesture, at step450of the method400, it is determined that the slider gesture represents a user struggle. In response to determining that a user is struggling, or that a particular aspect of the GUI frequently encounters user struggles, the GUI may be modified. For example, if a GUI element is frequently mis-clicked (e.g., users try to click that element multiple times before successfully doing so), the GUI element may be enlarged or moved.

Embodiments of the general process of method400inFIG. 4have been described. Further implementation details and examples will be described below. As stated in the description of step440, the baseline value may be derived from the historical information of past user sessions in the like computing environment. The following paragraphs provide further details regarding how historical information may be aggregated and how baseline values may be derived for different gestures from the historical information.

FIG. 6, described in further detail below, depicts a flowchart of the general process of a method600for aggregating historical information of past user sessions according to some embodiments of the present disclosure.FIG. 8, also described in further detail below, depicts another table800illustrating intermediate results of the performance of the method600ofFIG. 6according to some embodiments of the present disclosure.

Method600may be performed with respect to the historical information of past user sessions in a given computing environment. For example, given a computing environment as defined by parameters shown in table500inFIG. 5, the historical information of past user sessions may be aggregated in advance according to the given computing environment with a suitable aggregating algorithm. The aggregation may be made based on a given number of most recent days' incoming sessions, for example, and the aggregated historical information may be updated with the inclusion of any incoming sessions. The following steps may be carried out with respect to the aggregated historical information.

Step610comprises calculating and/or obtaining struggle factor scores for various types of gestures for each of the past user sessions associated with the given computing environment. The struggle factor scores for respective gestures may be calculated in the same or a similar manner as in step430of method400. Alternatively, the struggle factor scores may be obtained directly if they have been calculated and stored for this or other purposes. Intermediate results of performance of the method600, following performance of step610, are illustratively shown in a table700inFIG. 7.

Table700shows calculated and/or obtained struggle factor scores for only two sessions, session1and session2, as indicated in column701. The columns702,703,704and705contain struggle factor scores: slider ratio, dwell time, repeated times and single-handed opposite gestures, respectively, for each of the sessions.

As shown, for session1: the slider ratio is 210%; the dwell time is 15 minutes, indicating that the user's finger dwelt on an element (e.g., the whole form) in the UI (e.g., checkout page) for 15 minutes; the repeated gesture is 2 times, indicating that a gesture was repeated twice on some element in the checkout page; and the single-handed opposite gesture is 10 times, indicating that ten single-hand opposite operations were performed.

Analogously, for session2: the slider ratio is 220%; the dwell time is 5 minutes; the repeated gesture is 2 times; and the single-handed opposite gesture is 5 times. Accordingly, such struggle factor scores may be calculated and stored as shown inFIG. 7for all past user sessions available for analysis.

Following the calculation and/or obtention of struggle factor scores in step610of the method600, the distribution of each struggle factor score is calculated in step620, based on the result of step610. More specifically, an overall percentile for each struggle factor score is calculated. The results of performance of step620are displayed in table800ofFIG. 8. As shown, columns811,812,813,814and815represent five percentiles (namely 25th, 50th, 75th, 95th, and maximum (e.g., 100th)), respectively, for each of the struggle factor scores. Rows802,803,804and805correspond with columns702,703,704and705of table700. For example, column812of table800displays the 50thpercentiles of the struggle factor scores, and indicates that the slider ratio (shown in row802) of 215 (percent) ranks in the 50thpercentile of the overall slider ratio scores. Similarly, the dwell time (shown in row803) of 5 (minutes) ranks in the 50thpercentile of the overall dwell time scores. Analogously, column813of the table800displays the 75thpercentiles of the struggle factor scores, and indicates that the repeated gesture (shown in row804) of 6 (times) ranks in the 75thpercentile of the overall repeated gesture scores, and that the single-handed opposite gesture (shown in row805) of 4 (times) ranks in the 75thpercentile of the overall single-handed opposite gesture scores.

Following the distribution calculation of each struggle factor score in step620of the method600, a baseline value for a struggle factor score is selected in step630, based on the distribution calculated in step620. For example, if the 50thpercentile is selected to define the baseline value, then corresponding struggle factor scores (from column812of the table800) are: slider ratio of 215%; dwell time of 5 minutes; repeated gesture of 3 times; and single hand opposite gesture of 2 times. Therefore, in this example, the baseline value for the struggle factor score slider ratio is defined to be 215%.

The above examples illustrate that the baseline value may be defined or selected according to a distribution of struggle factor scores for the same or similar user gestures calculated from the historical information of past user sessions aggregated according to the computing environment. Further, the baseline value for a struggle factor score may be defined by a selected percentile in the distribution.

It is to be noted that the selection of the baseline value described above is only an illustrative, non-limiting example. Alternatively, the baseline value may be selected or defined according to a combination of several of the percentiles, for example, an average of the 50thand 75thpercentiles.

Referring back to the method400shown inFIG. 4, as previously stated, if the struggle factor score is larger than the corresponding baseline value, then the user gesture is determined to represent a user struggle. In the instant example, wherein the user gesture is identified to be a slider gesture (in step420), and the struggle factor score slider ratio is calculated to be 250% (in step430), which is compared to the baseline value of 215% (in step440), in step450, the slider gesture is determined to represent a user struggle.

The determination resulting from step450of the method400may be used by UI designers, for example, to understand that there is a high probability that a user has experienced some kind of struggle due to the user's performance of an abnormal number or type of actions compared to historical user behaviors in like computing environments.

It is to be noted that in the above description, slider is taken as an example of user gesture. However, the principle of embodiments of the present disclosure may also be applicable to other types of user gesture.

Embodiments of the present disclosure provide a solution for detecting user struggles. The solution involves an unsupervised learning process for aggregating historical user interaction data, which may evolve struggle-judging criteria and identify struggling sessions automatically. Implementing embodiments of the present disclosure may enable or facilitate UI analysts providing analysis services to enterprises by comparing their apps with peer apps. Such comparisons may indicate to UI designers where to improve the apps for the enterprises in order to alleviate user struggles.

Embodiments of the present disclosure have been described above, particularly with respect to the detection of gesture struggles as an example. However, the principle of the present disclosure is by no means limited to the example. As noted above, those skilled in the art shall appreciate that, as design choices, a variety of equivalent or alternative methods and techniques in accordance with the principle of the present disclosure may be implemented for the operation of computing systems with configuration and protocol similar or equivalent to the exemplary mobile application. Therefore, those design choices shall be construed as falling into the scope and spirit of the present disclosure.

In addition to embodiments described above, other embodiments having fewer operational steps, more operational steps, or different operational steps are contemplated. Also, some embodiments may perform some or all of the above operational steps in a different order. Furthermore, multiple operations may occur at the same time or as an internal part of a larger process. The modules are listed and described illustratively according to some embodiments and are not meant to indicate necessity of a particular module or exclusivity of other potential modules (or functions/purposes as applied to a specific module).

When different reference numbers comprise a common number followed by differing letters (e.g.,100a,100b,100c) or punctuation followed by differing numbers (e.g.,100-1,100-2, or100.1,100.2), use of the reference character only without the letter or following numbers (e.g.,100) may refer to the group of elements as a whole, any subset of the group, or an example specimen of the group.