Ergonomic and sensor analysis based user experience design

Ergonomic and sensor analysis based user experience design improvement is provided in various embodiments by: detecting interactions between a user and an element of a user interface of a software application executing on a computing device; gathering biometric data for the user from interface devices of the computing device; identifying a task performed in the software application that corresponds to the interactions and associates the biometric data with the interactions; adjusting a user interface setting for the element to reduce a strain on the user, wherein the user interface setting comprises at least one of: a location of the element relative to a second element of the user interface involved in the task; a relative size of the element in the user interface; a color balance of the element; and a size and location of a hitbox of the element for registering user selection of the element.

BACKGROUND

The present invention relates to user experience improvement in software systems, and more specifically, to gathering ergonomic and sensor data to affect the improvement. Users of software programs interact with those programs via physical devices, such as monitors and displays that output visual information to the user, speakers that output audio information, haptic devices that output tactile information, controllers that accept tactile input (e.g., keyboards, touch screens, mice/trackballs/trackpads), microphones that accept audio input, and cameras that accept visual input. The arrangement of these physical components in the physical environment may impart strains on the user when using the software program, which may be exacerbated by the layout or accessibility of elements of the software program.

SUMMARY

According to one embodiment of the present invention, a method for ergonomic and sensor analysis based user experience design improvement is provided, the method comprising: detecting interactions between a user and a first element of a user interface of a software application executing on a computing device; gathering biometric data for the user from interface devices of the computing device; identifying a task performed in the software application that corresponds to the interactions and associates the biometric data with the interactions; in response to determining that a strain on the user when performing the task exceeds an ergonomic threshold, adjusting a user interface setting for the first element to reduce the strain, wherein the user interface setting comprises at least one of: a location of the first element relative to a second element of the user interface involved in the task; a relative size of the first element in the user interface; a color balance of the first element; and a size and location of a hitbox of the first element for registering user selection of the first element.

According to one embodiment of the present invention, a computer readable medium including instructions that when executed by a processor enable the processor to perform an operation for ergonomic and sensor analysis based user experience design improvement is provided, the operation comprising: detecting interactions between a user and a first element of a user interface of a software application running on a computing device; gathering biometric data for the user from interface devices of the computing device; identifying a task performed in the software application that corresponds to the interactions and associates the biometric data with the interactions; in response to determining that a strain on the user in performing the task exceeds an ergonomic threshold, adjusting a user interface setting for the first element to reduce the strain, wherein the user interface setting comprises at least one of: a location of the first element relative to a second element of the user interface involved in the task; a relative size of the first element in the user interface; a color balance of the first element; and a size and location of a hitbox of the first element for registering user selection of the first element.

According to one embodiment of the present invention, a system for ergonomic and sensor analysis based user experience design improvement is provided, the system comprising: a processor; and a memory storage device, including instructions that when executed by the processor enable the system to: detect interactions between a user and a first element of a user interface of a software application running on a computing device; gather biometric data for the user from interface devices of the computing device; identify a task performed in the software application that corresponds to the interactions and associates the biometric data with the interactions; in response to determining that a strain on the user in performing the task exceeds an ergonomic threshold, adjust a user interface setting for the first element to reduce the strain, wherein the user interface setting comprises at least one of: a location of the first element relative to a second element of the user interface involved in the task; a relative size of the first element in the user interface; a color balance of the first element; and a size and location of a hitbox of the first element for registering user selection of the first element.

DETAILED DESCRIPTION

The present disclosure provides for User Experience (UX) design improvements via ergonomic and sensor analysis. Although design teams may seek to provide a UX that is optimized for most users performing most tasks in a software application, some users may find the originally provided UX to be non-optimal (e.g., due to hardware or physical constraints, due to performing atypical tasks, or due to not being familiar with the interface). Embodiments of the present disclosure gather data related to an application's use of text, images, and audio to evaluate the current position and actions of the user to determine when repetitive motions, eye focus, body position, and other ergonomic measures for the user may be improved by adjusting the UX of the application. These data may be forwarded to the design team for later use in redesign of the application, or may affect local options for the application to improve the UX for the particular user.

FIGS. 1A and 1Billustrate two viewpoints for a first use case101for a user interacting with a computer110running a software application.FIG. 1Cillustrates a viewpoint for a second use case102for a user interacting with a computer110running a software application. The user interacts with the software application by various physical input and output devices, such as, for example, a display device120, a camera130, a microphone140, a keyboard150, a pointing device160(e.g., a mouse, a trackball, a touch screen), etc. The input/output devices may be separate from the computer110(as perFIGS. 1A and 1B) or may be integrated in the computer110(as perFIG. 1C). Other input/output devices, more input/output devices, or fewer input/output devices may be provided in other embodiments.

The input devices include various sensors to receive inputs from the user and to measure various conditions of use that are collectively referred to as biometric data. For example, a keyboard150may include various switches to record when a particular key has been pressed by the user, and to measure the force at which the user presses the particular key. In another example, a camera130includes software for measuring a distance170between the user's eyes and the display device120, angles180and185at which the user's head is rotated relative to the user's body in various planes, and software for identifying a facial expression190. In a further example, a pointing device160includes pressure sensors, to measure how hard the user is pressing various buttons on the pointing device, and an accelerometer, to measure how quickly the user moves the pointing device160(e.g., via a trackball, scroll wheel, or via whole-mouse movement).

As will be appreciated, the position of portions of the user relative to the display device120and input devices may affect the comfort of the user in using the computer110, especially during prolonged use sessions. For example, the angle at which the user's arms and wrists are held may ease or exacerbate joint or tendon discomfort. Similarly, the distance170and angles180,185of the user relative to the display device120may influence whether the user experiences neck or eye strain. However, the positions, arrangements, and color balances (including contrasts, brightnesses, and hue schemas) of various elements presented on the display device120may encourage or require the user to perform more frequent actions, longer actions, and actions in non-optimal postures than is comfortable in the course of completing a task, thereby defeating an otherwise ergonomically optimized physical layout for using the computer110. For example, a poorly chosen font size or color versus a background may encourage a user to lean in closer to the display device120than a well-chosen font size or color. In another example, elements of a user interface separated by a larger distance may require larger real-world motions to select than elements separated by a smaller distance.

The actions performed by the user may include both gross motor movements (e.g., moving an arm to affect a movement of a pointing device160or rotating or extending a neck to track the head) and fine motor movements (e.g., moving a finger to affect a click, key press, or button hold; squinting the eyes; or tracking the eyes) to affect an interaction in the software or to observe actions taken in the software. Actions performed by the user may also include actions taken in response to or independent from the actions intended to interact with software (e.g., a facial expression coinciding with a software effect; cracking joints to release pressure after performing interactive actions; stretching; leaving workspace including the computer110). Sensors included in various input devices track these user actions in relation to software tasks to improve the design of the user experience.

FIG. 2illustrates a computing system200, such as computer100, which may be a personal computer, a laptop, a tablet, a smartphone, etc. As shown, the computing system200includes, without limitation, a central processing unit (CPU)205, a network interface215, an interconnect220, a memory260, and storage230. The computing system200may also include an I/O device interface210connecting I/O devices212(e.g., keyboard, display and mouse devices) to the computing system200.

The CPU205retrieves and executes programming instructions stored in the memory260. Similarly, the CPU205stores and retrieves application data residing in the memory260. The interconnect220facilitates transmission, such as of programming instructions and application data, between the CPU205, I/O device interface210, storage230, network interface215, and memory260. CPU205is included to be representative of a single CPU, multiple CPUs, a single CPU having multiple processing cores, and the like. And the memory260is generally included to be representative of a random access memory. The storage230may be a disk drive storage device. Although shown as a single unit, the storage220may be a combination of fixed and/or removable storage devices, such as magnetic disk drives, flash drives, removable memory cards or optical storage, network attached storage (NAS), or a storage area-network (SAN). Further, computing system200is included to be representative of a physical computing system as well as virtual machine instances hosted on a set of underlying physical computing systems. Further still, although shown as a single computing system, one of ordinary skill in the art will recognized that the components of the computing system200shown inFIG. 2may be distributed across multiple computing systems connected by a data communications network.

As shown, the memory260includes an operating system261(e.g., Microsoft's Windows® Operating System), at least one a software application280, and an ergonomic tracker290linked with the software application280to provide improvements of the UX for the software application280based on ergonomic and sensor analysis.

As the user provides inputs, the software application280translates those inputs into interactions within the framework of the software application280. For example, a word processing application translates inputs from a keyboard150into characters in a typing interface and translates inputs from a pointing device160into selections of various user interface (UI) elements, such as, for example, positions in typed text, typeface options, special character insertions, file save commands, etc. In another example, an Operating System (OS) application translates inputs from a pointing device160to move files within a storage schema, launch selected applications, arranging UI elements within a graphical user interface (GUI), etc.

The ergonomic tracker290gathers the biometric data from the input devices and detects interactions in a UI of one or more software applications280running on the computer110that the ergonomic tracker290is linked to. The ergonomic tracker290associates the biometric data with the interaction data in a time series and identifies various tasks completed by a user in the software application280.

FIG. 3A-Dillustrate a series of interactions with a user interface to perform a task in a software application. The illustrated task is a drag-and-drop task that includes interactions in which the user attempts to select a first element320aof the UI310, drag the first element320ato a second element320b, and release the first element320aover the second element320b. However, due to non-optimized UI design, the interactions with the UI310may require physical actions that are more frequent (e.g., multiple attempts), longer, or more forceful that are ergonomically advisable. The present disclosure is applicable to improvements in UI design beyond drag-and-drop tasks, which is provided as a non-limiting example of how the present disclosure improves computing devices.

InFIGS. 3A-D, a UI310is presented with a first element320a, a second element320b, and a third element320c(generally, elements320), respective hitboxes330a-c(generally, hitbox330) for the elements320a-c, and a cursor340. More or fewer elements320of different sizes, shapes, arrangements, labeling, coloring, are possible in other embodiments.

The elements320may include various widgets, menus, folders, icons, and software defined buttons in the UI310for performing various tasks or representing various files or objects. In the illustrated examples, the first element320ais presented as a five-pointed star, the second element320bis presented as a white folder with a label350bof a white five-pointed start, and the third element320cis presented as a white folder with a label350cof a white six-pointed star.

A hitbox330represents the area in the UI310that is selectable by the user to interact with an associated element320. InFIGS. 3A-D, the hitboxes330are shown via dotted lines defining an area in the UI310, but may be invisible to the user. Hitboxes330may be rectangular, circular, shaped according to the associated element320, or irregularly shaped, and may be sized to match an area of the associated element320, extend past the associated element320, to be smaller than the associated element320, and may be aligned to be centered or offset from an anchor point of the associated element320.

The cursor340is shown as a graphical pointer in the UI310that may be associated with a pointer device160. In other embodiments, such as a computer110using a touch screen, the cursor340represents a point of contact from the user touching the touch screen, which may be shown or invisible to the user.

FIG. 3Aillustrates the user unsuccessfully attempting to select the first element320a, with a cursor340positioned on the displayed icon for the first element320a, but not on the hitbox330aassociated with the first element320a.FIG. 3Billustrates the user successfully attempting to select the first element320a, with a cursor340positioned on the displayed icon and hitbox330afor the first element320a. The ergonomic tracker290may track the number of miss-clicks that the user needs to select the element320as attempted interactions that are part of a task, and track the force made during those clicks, relative locations of those clicks to the associated hitbox330and other hitboxes320, the time between successive clicks, whether the click is a single click/double click/click-and-hold, a body position during the click, etc. Attempted interactions include those interactions made by the user that do not result in an interaction with the intended element320, including actions that do not result in an interaction (e.g., a missed click, typing without focus in a document) or that select a different element320than is used in the following task. For example, the ergonomic tracker290may determines that a user who selects a first element320a, then selects a second element320b, and then performs a task using the second element320b(but not the first element320a) made an attempted interaction with the second element320bas part of the task when the first element320awas selected.

FIG. 3Cillustrates the user dragging the first element320afrom an initial position to a final position over the second element320band the third element320c. In the illustrated example, the second element320band the third element320care folders, that may accept and store the file represented by the first element320when the first element320ais dragged over the respective hitbox330and de-selected (e.g., “dropped”) by the user. FIG.4D illustrates the UI310once the user has dropped the first element320ainto one of the folders represented by the second element320band the third element320c. Due to the proximity of the folders, the user may not know which folder the first element320ahas been dropped into, and may have to access the folders to verify where the file represented by the first element320ahas been stored. The ergonomic tracker290may determine how far the user moves the first element320afrom the initial position to the final position within the UI310, a speed and motion used to perform the actions in the real world required to affect the interactions in the software application280, whether the user performs follow-up interactions (e.g., to determine which folder the file was stored in), a length of time to perform the interaction, etc.

FIGS. 4A and 4Bpresent the UI310as illustrated inFIGS. 3A-Dthat are adjusted to improve the UX thereof in performing a drag-and-drop operation, according to embodiments of the present disclosure. The UI310is presented with a first element320a, a second element320b, and a third element320c, respective hitboxes330a-cfor the elements320a-c, and a cursor340. More or fewer elements320of different sizes, shapes, arrangements, labeling, coloring, are possible in other embodiments. Additionally, a shortcut element410and an indicator element420are presented as part of the improved UX.

InFIG. 4A, the first element320ais presented as a five-pointed star, and the hitbox330athereof has been improved to match the shape of the first element320a. In some embodiments, the ergonomic tracker290automatically adjusts the hitbox330ain response to identifying user strain in attempting to interact with the hitbox330(e.g., more than n miss-clicks, click force over a threshold pressure). In other embodiments, the ergonomic tracker290notes the user strain in attempting to interact with the hitbox330and includes the strain determination in a strain report transmitted to the development team for the software application280.

InFIGS. 4A and 4B, the second element320bis presented as a white folder with a label350bof a darker five-pointed start, and the third element320cis presented as a white folder with a label350cof a darker six-pointed star; increasing the contrast between the elements and associated labels350. Additionally, the relative sizes of the labels350are increased relative to the associated elements320in increase readability. In various embodiments, the ergonomic tracker290automatically adjusts the color balance and/or sizing of the second and third elements320relative to the associated labels350in response to identifying user strain in attempting to interact with the second and third elements320(e.g., the user squinting, the user moving closer to the display device120, the user re-doing an action). In other embodiments, the ergonomic tracker290notes the user strain in attempting to interact with the second and third elements320and includes the strain determination in a strain report transmitted to the development team for the software application280.

The position of the second and third elements320are also adjusted fromFIGS. 3A-DtoFIGS. 4A and 4B, so that the distance between locations that the user moves the first element320aacross may be reduced. Additionally, the distance between the second and third elements320to reduce the likelihood of dropping the first element320ainto an undesired folder. The ergonomic tracker290may make the positional adjustments to the elements320based on identified user strain when attempting to move (e.g., drag and drop) the first element320ato one of the second and third elements320. Alternatively or additionally, the ergonomic tracker290may include the positional data for the drag and drop task in a strain report transmitted to the development team for the software application280.

The shortcut element410presented inFIG. 4Ais one example of the ergonomic tracker290providing the user with an additional control in the UI310to complete a task with fewer actions to thereby reduce strain on the user. For example, after identifying that the user moves the first object320ato the second object320b, when the user re-selects the first object320a, the ergonomic tracker290may cause the software application280to display the shortcut element410. The shortcut element410, when selected by the user, may cause the software application to perform a specified task, with the user performing fewer or less strenuous actions to affect the task in the software application280. For example, instead of dragging the first element320aover to the second element320b(including selecting, moving across the UI310, and deselecting) the user may instead select the first element310and select the shortcut element410to store the file represented by the first element320ain the folder represented by the second element320b.

The indicator element420presented inFIG. 4Bis one example of the ergonomic tracker290providing the user with a visual indication in the UI310to complete a task with fewer action to thereby reduce strain on the user. For example, after identifying that the user moves the first element320ato the second element320band double checks the folder represented by the second element320bactually contains the file represented by first element320a, the ergonomic tracker290may cause the software application280to display the indicator element420in response to the user dropping an element310into another element320. The indicator element420provided feedback to the user that a task has been performed, and may eliminate interactions from future performances of the task that the ergonomic tracker290(or a development team alerted by the ergonomic tracker290) deemed as superfluous.

FIG. 5is a flowchart of a method500for implementing ergonomic and sensor analysis based UX design improvements. Method500begins with block510, where the ergonomic tracker290detects interactions made with a software application280using various input devices of a computer110.

At block520, the ergonomic tracker290gathers biometric data from various sensors associated with the input devices of the computer110being used by the user. The biometric data may include measurements of how many time the user clicks a button, how long the user pauses after a performance of a button click, the pressure the user applies to an input device to perform an action, how many times the user needs to drag and drop items on the UI310, how many times/how far the user has to stretch to look the at display device120(e.g., in order to see the information clearly), how many times the user frowned/squinted/grimaced in front of a UI310, how many times the user moved back and force or up and down to each direction of the screen to search for information in the UI310, etc. The biometric data may be continuously gathered from sensors embedded in input devices and/or from cameras focused on the user.

At block530, the ergonomic tracker290identifies a task performed in the software280and the associated biometric data and interactions performed in the software application280within a time window during which the task was performed. The ergonomic tracker290may identify distinct tasks within a series of tasks based on definitions of certain interactions in the software application290corresponding to a start or an end interaction for a task. For example, a selection of a particular element320in the UI310may signify a start of a task, while the selection of a different element320in the UI310may signify the end of a task. In various embodiments, the ergonomic tracker290may include interactions and biometric data gathered from after the end-interaction of a first task as part of the next task to capture data related to unsuccessful or strenuous attempts to start the next task. In some embodiments, the ergonomic tracker290may include interactions and biometric data gathered from after the end-interaction of a given task as part of the given task to capture data related to verifying the successful completion of the task.

At block540, the ergonomic tracker290determines whether a strain on the user in performing the task exceeds an ergonomic threshold. In response to determining that the strain on the user is below or has not yet exceeded the ergonomic threshold, method500returns to block510for the ergonomic tracker290to continue monitoring the user's actions in relation to the software application280. In response to determining that the strain on the user has exceeded the ergonomic threshold, method500proceeds to at least one of block550and block560. As used herein, when selecting at least one element of the group of A and B, it will be understood that a selection may be only element A, only element B, or element A and element B. Stated in relation to method500, the ergonomic tracker290may perform one or both of block550and block560in various embodiments in response to the strain exceeding the ergonomic threshold.

The ergonomic threshold represents one or more measures of the actions performed by the user in the real-world as translated to interactions in the software application280that are correlated to ranges of motion, frequencies of performance, and/or forces/speeds of performance. Each user may specify different ergonomic thresholds based on personal preference, or the ergonomic tracker290may develop a personalized baseline for the user based on historic user actions and an initial behavioral model. In various embodiments, the ergonomic tracker290may user a population-based baseline for the ergonomic thresholds until a personalized baseline is specified or developed for the user.

In one example, an ergonomic threshold may relate to a field of vision, such that an element320that travels more than n degrees of the user's field of vision over the UI310(vertically, horizontally, or both), where each user may specify a personalized value for n. In some embodiments, a field of vision ergonomic threshold may include a measure of the distance170between the user and a display device120displaying the UI310during performance of the task, an angle180,185between the user and the display device210, and a travel between an initial position and a final position of at least one element320during the task.

In one example, an ergonomic threshold may relate to a number of interactions with a pointer device160during performance of the task or within a predefined period of time. In a further example, a length of a drag operation of an element320within the UI310during performance of the task. In one example, a force of an action (e.g., as measured by a pressure sensor, force meter, or the like integrated with an input device) when performing of the task is used as an ergonomic threshold. In one example, an ergonomic threshold is based on a facial expression of the user, such that a pained expression, squinting, or confused expression satisfies the ergonomic threshold, while neutral, happy, or concentrating expressions do not satisfy the ergonomic threshold.

At block550, the ergonomic tracker290adjusts a UI setting for the element320used in the task to reduce a level of strain in future performances of that task. Based on the type of ergonomic threshold exceeded (and the associated strain measured), the ergonomic tracker290may adjust different UI settings for the element320. For example, when an ergonomic threshold related to eye strain, squinting, or neck strain is exceeded, the ergonomic tracker290may adjust the size or color balance of an element320. In another example, when an ergonomic threshold related to arm or finger strain is exceeded, the ergonomic tracker290may adjust a location of the element320in the UI310. In a further example, when an ergonomic threshold related to a frequency of interactions is exceeded, the ergonomic tracker290may adjust a size and/or a location of a hitbox330for an element320.

In some embodiments, the ergonomic tracker290identifies tasks that are repeated by the user or that include several interactions in a multi-step task, and generates a shortcut element410for display in the UI310. In one example, the shortcut element410is a contextual menu that allows the user to eliminate some interactions (reducing the number of clicks, drags, scrolls, etc.) to perform the task. In a further example, the shortcut element410is a hyperlink to text or other elements320elsewhere in the UI310so that the user is presented with relevant information in closer proximity to a current point of visual focus than the information is otherwise located in the UI310.

In some embodiments, the ergonomic tracker290identifies that additional interactions are taken as part of the task, and generates an indicator element420to visually or audibly highlight information in the UI310to reduce the actions taken by the user in a future performance of the task. For example, an indicator element420of a chime/tone/beep, an animation, or the like may be presented to the user to highlight a UI element320with which the task has been performed to eliminate interactions associated with confirming the first task was completed as desired. In another example, the indicator element420identifies a more efficient element310for the user to select in the UI310for performing the task, so as to eliminate clicks/drags/typing or other actions by the user in performing the task.

At block560, the ergonomic tracker290generates and transmits a strain report to a development team associated with the software application280. The strain report may contain information related with which ergonomic threshold have been exceeded, a frequency at which the thresholds are exceeded, a severity by which the thresholds were exceeded, tasks performed at the time a threshold was exceeded, etc.

Method500may return to block510from block550or block560for the ergonomic tracker290to continue monitoring the user's actions in relation to the software application280. The ergonomic tracker290may make incremental adjustments to one or more elements320of a UI310through several loops of method500, and may batch data for inclusion in a strain report that are collected over several loops of method500.