Detecting accessibility patterns to modify the user interface of an application

Disclosed herein are system, method, and computer program product embodiments for modifying a graphical user interface (GUI) in response to detecting an accessibility pattern. In an embodiment, a user interacting with an application of a mobile phone may experience difficulty viewing the GUI elements or supplying commands to the application. Data from sensors on the mobile phone may be recorded and analyzed to detect an accessibility pattern indicating that the user is experiencing difficulty interacting with the application. For example, a proximity sensor may indicate that the user has placed the mobile device close to the user's face. In response to detecting an accessibility pattern, the application may modify the GUI of the application according to the particular accessibility pattern detected.

BACKGROUND

Users may experience accessibility issues when interacting with applications on a mobile device. For example, font sizes may be too small and difficult to read. Further, some users may have difficulty manipulating or selecting buttons or icons on a graphical user interface of an application as displayed on the mobile device. Some users may attempt to alleviate these issues by setting default accessibility options on their mobile device. For example, an operating system of the device may include options such as a screen reader, visibility enhancements for text or color, hearing enhancements, or manipulation enhancements aiding with interactions.

While these techniques have been proposed to aid with accessibility, these techniques do not provide particular assistance to different applications on the mobile device. For example, different applications may require different techniques to aid in accessibility that are not addressed by the default settings provided by the mobile device. Further, some users may still require assistance despite not enabling an accessibility setting on the mobile device.

BRIEF SUMMARY

Disclosed herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for modifying a user interface of an application in response to detecting an accessibility pattern.

In an embodiment, a mobile device may detect a mobile device accessibility setting and/or sensor data to determine an accessibility pattern. The accessibility pattern may be one or more instances of data gathered from mobile device sensors indicating that the user is experiencing difficulty interacting with an application. Upon detecting the accessibility pattern, the application may modify a user interface such as a graphical user interface (GUI) in accordance with the detected accessibility pattern. This modification may be tailored to the particular application and/or the detected accessibility pattern to aid a user in interacting with the application.

In some embodiments, a computer-implemented method for modifying a graphical user interface (GUI) of an application may include recording data from a sensor of the mobile device to detect an accessibility pattern indicating that a user is experiencing difficulty interacting with the application due to an impairment of the user. In response to determining the accessibility pattern, a graphical user interface (GUI) of the application as displayed on the mobile device may be modified according to the detected accessibility pattern.

In some embodiments, a system for modifying a graphical user interface (GUI) of an application may comprise a memory device (including software and/or hardware) and at least one processor coupled to the memory device. The processor may be configured to The processor may record data from a sensor of the mobile device to detect an accessibility pattern indicating that a user is experiencing difficulty interacting with the application on the mobile device due to an impairment of the user. In response to detecting the accessibility pattern, the processor may modify a GUI of the application as displayed on the mobile device according to the detected accessibility pattern, wherein no accessibility setting of the mobile device has been set.

In some embodiments, a non-transitory computer-readable device is disclosed, the non-transitory computer-readable device may have instructions stored thereon that, when executed by at least one computing device, may cause the at least one computing device to perform operations including recording data from a sensor of the mobile device to detect an accessibility pattern indicating that a user is experiencing difficulty interacting with the application on the mobile device due to an impairment of the user. In response to detecting the accessibility pattern, a GUI of the application as displayed on the mobile device may be modified according to the detected accessibility pattern by enlarging a GUI data object displaying a real-time camera view.

DETAILED DESCRIPTION OF THE INVENTION

Provided herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for modifying a user interface of an application in response to detecting an accessibility pattern.

In an embodiment, an accessibility pattern may be pattern identified data collected from a device executing the application, such as, for example, a mobile device. The accessibility pattern may indicate that a user is experiencing difficult interacting with the application. The accessibility pattern may be due to an impairment of the user. For example, the text or images displayed on a graphical user interface (GUI) of the application may be difficult to read due to the user's impairment. Based on the user's manipulation of the mobile device, sensors of the mobile device may gather data and a processor of the mobile device may identify patterns of the data as representing an accessibility pattern.

Impairments related to vision, manual dexterity, cognition, age, and/or other physical disabilities may result in a user experiencing difficulty interacting with an application executing on a mobile device. For example, if a user is near-sighted, the user may experience difficulty reading text of the application. The user may bring the mobile device close to the user's eyes so that the user can better read the text. As the application executes on the processor of the mobile device, the application may detect data from sensors of the mobile device to identify this motion and categorize the motion as an identified accessibility pattern.

As will be further explained below, examples of sensors that can be used to identify an accessibility pattern may include a proximity sensor, one or more cameras, and/or an accelerometer. The proximity sensor may be a light or infrared sensor on the mobile device. In some embodiments, the proximity sensor may be the same sensor used to lock a display screen when a user attempts to make a phone call. The application may identify that the user is holding the mobile device close to the user's face based on a distance determined from the proximity sensor. Based on the data collected from the proximity sensor, the application may identify an accessibility pattern.

In some embodiments, the application may utilize one or more cameras of the mobile device. For example, the application may compare image data to identify a change in proximity of the mobile device to a user's face. The application may identify a facial feature in a front-facing camera and detect that the size of the face increases. This data may indicate that the user has moved the mobile device closer to the user's face and may represent an accessibility pattern. Similarly, a rear-facing camera may be used. The rear-facing camera may detect that object captured in an image becomes smaller. This change in image data may also indicate that the user is moving the camera toward the user's face and away from an external object.

Another sensor that may be used by the mobile device may be an accelerometer. The accelerometer may measure forces applied to the mobile device. Similar to the proximity sensor and/or the cameras, the accelerometer may measure a movement of the mobile device that may indicate an accessibility pattern, such as, for example determining that the mobile device is shaking. In some embodiments, the accelerometer may be a standalone sensor and/or may be combined with the proximity sensor and/or cameras to identify an accessibility pattern. The accelerometer may identify a movement of the mobile device such as bringing the mobile device closer to a user's face.

While the movement of the mobile device closer to a user's face has been discussed as an accessibility pattern, other types of accessibility patterns may be detected. For example, the data from the proximity sensor, cameras, and/or accelerometer may indicate that the user is unable to hold the mobile device steadily. For example, the mobile device may be shaking and/or may cause fluctuations in proximity, images, and/or accelerometer readings. This pattern may indicate that the user may be elderly and may also experience difficulties interacting with the application.

While sensors of the mobile device may detect different accessibility patterns based on the programming of the application and/or processor, the application and/or processor may also consider an accessibility setting of the device. For example, a user may have selected an option to display larger text for the application of the mobile device. The application and/or processor may identify this setting and/or identify the sensor data to define an accessibility pattern. For example, if the user has selected a setting, the application may not utilize the sensor data. In some embodiments, the application may use the sensor data to further enhance the setting selected by the user. In some embodiments, the application may identify the selected setting but may operate independently from the setting and modify a user interface according to the configuration of the application.

After identifying an accessibility pattern, the application and/or the processor executing the application may modify a user interface according to the accessibility pattern detected. For example, the application may modify a graphical user interface (GUI) of the application. In response to detecting an accessibility pattern indicating a visual impairment, the application may increase the size of the text of the GUI. In some embodiments, the application content displayed on the GUI may be altered to aid with accessibility. For example, a GUI displaying a list of items or input boxes may be altered depending on the detected accessibility pattern. For example, items on the list may be arranged or ordered differently depending on the accessibility pattern. Similarly, items may be added or removed from the list.

Content of a GUI may also be modified by splitting the content into multiple screens or separate GUIs. For example, upon detecting an accessibility pattern, the application may split a list of items into multiple GUI screens or slides. A user may interact with individual screens which may include larger font. This interaction may reduce confusion and be more helpful for a user as each screen may display less information. A user may navigate through each screen to interact with the application.

In some embodiments, the user interface may be modified by providing auditory feedback and/or accepting voice controls. For example, the application may trigger a screen reading process to generate audio data reading the contents of a screen. In some embodiments, the detection of an accessibility pattern may initiate a playing of an audio file that may be independent from the content displayed on a GUI. The audio file may be targeted to aid the user in interacting with the application. For example, the audio file may describe content displayed on a GUI beyond reading the text displayed on the GUI.

In an example embodiment, the GUI modification process may aid users attempting to capture an image using a mobile device. For example, the application may require verification or image capture and may request that a user supply information and/or capture an image of a government issued identification. The identification may be a document or a card. In attempting to supply user data and/or an image, the user may be presented with interactive icons, buttons, and/or a list of input boxes in a GUI requesting identification information. Based on the quantity of text and/or information requested, user that are visually impaired may face difficulty reading the text and/or interacting with the elements of the GUI. For example, the user may move the GUI closer the user's eyes to better read and/or view the information. Using the sensors of the mobile device and/or the accessibility settings, the application may detect an accessibility pattern and that the user is experiencing difficulties supplying the requested information.

In response to detecting the accessibility pattern, the application may modify the GUI to provide assistance. This modification may result from a configuration of the application, content of the currently displayed GUI, and/or the parameters detected to identify the accessibility pattern. For example, if a user is viewing text information, the text may be enlarged. If the user is attempting to capture an image, instead of having a box on the GUI represent the camera's view, the camera's view may be expanded to occupy a larger portion of the GUI. In some embodiments, the camera's view may be expanded to occupy the entire display screen. Similarly, rather than a manual capture, the application may switch to an automatic capture mode.

If a user is viewing a list of items providing options for verification, the application may determine that some options may be difficult to complete for users with impairments. The application may then modify the GUI to remove these options or reorder the list to highlight simpler options. In some embodiments, the application may modify a list of input forms and split the list into multiple GUI screens or slides. For example, rather than requesting a user name and a password in a single screen, the application may generate a first GUI requesting the user name. After the user has supplied the user name, the application may generate a second GUI requesting the password. In this manner, a user may more easily view and interact with the application.

In view of the described embodiments and as will be further described below, the disclosed modification of GUIs may aid users in viewing and/or interacting with a GUI. This modification may aid in more efficient data entry as well as image capture. In particular, fewer processing steps are wasted attempting to capture images of poor quality or analyzing incorrect data. For example, less computer processing may be needed to analyze this information, thus computing resources such as processing and memory may be used more effectively. The modification of the GUI may be directly correlated with the detected accessibility pattern through a machine learning or decision tree configuration. These configurations may allow for a fast modification of the GUI as displayed on a mobile device. In this manner, the described embodiments result in a faster data capture process.

Various embodiments of these features will now be discussed with respect to the corresponding figures.

FIG. 1Adepicts a block diagram of an action100A indicating a desire for accessibility assistance, according to some embodiments. User110may be interacting with a mobile device that may be executing an application. User110may be interacting with the mobile device at a first position120A. As user110interacts with the mobile device and/or applications executing on the mobile device and displayed in a graphical user interface (GUI), user110may experience difficulties. For example, user110may experience a visual impairment and/or the text, buttons, icons, Tillable forms, and/or other data objects may become difficult for the user to read and/or manipulate. For example, user110may have limited sight and/or may be elderly.

When user110experiences the difficulty, user110may perform action100A by moving the mobile device from the first position120A to a second position120B. First position120A may represent a position of the mobile device at a particular distance from user110. Second position120B may represent a position of the mobile device that is closer to the face or eyes of user110relative to first position120A. For example, the user may be near-sighted and moving the mobile device via action100A may be an attempt to better read the mobile device.

Action100A may represent an accessibility pattern based on the sensor data captured by the mobile device. As will be further described with reference toFIG. 1B, data from one or more sensors of the mobile device may be gathered and analyzed to identify action100A. This analysis may be performed via machine learning, artificial intelligence, and/or decision trees. In some embodiments, movement from first position120A to second position120B may indicate the accessibility pattern. In some embodiments, user110may move the mobile device back and forth between first position120A and second position120B, in an attempt to find an ocular focus to view the GUI. This type of motion may also represent an accessibility pattern. Similarly, the user110may squint and/or adjust their eyelids in an attempt to read the GUI. In this case, the application may also identify an accessibility pattern.

In some embodiments, user110may be far-sighted. User110may attempt to move the mobile device further away from the user's110face or eyes to better view the GUI. In these cases, the application may also identify an accessibility pattern and modify the GUI accordingly. In view of the nearsighted and farsighted examples, the application may modify the GUI in the same way and/or different ways. For example, the application may identify different accessibility patterns while enlarging text in response to the patterns.

FIG. 1Bdepicts a block diagram of components of a mobile device120, according to some embodiments. Mobile device120may include one or more cameras130, a proximity sensor140, an accelerometer150, a processor160, and/or a graphical user interface (GUI) display170. These components may aid in determining an accessibility pattern and/or modifying a GUI as displayed by an application executed by processor160.

Camera130A and camera130B may be cameras located in different positions on mobile device120. For example, camera130A may be a front-facing camera pointing toward a user when a user is interacting with mobile device120and/or GUI display170. Camera130B may be a rear-facing camera, such as a camera used to capture images of objects at a distance. One or both of camera130A and camera130B may be used to determine an accessibility pattern. For example, camera130A may be used to capture images of the user's face or eyes. If the user has brought mobile device120closer to the user's face or eye, images captured by camera130A may indicate a change in face size. For example, multiple images may be captured and compared. If a pattern is detected that indicates that the user's face or eyes become larger, the user may be bringing mobile device120closer to their face. Processor160may identify this pattern using image recognition, facial recognition, and/or classification techniques.

Similarly, camera130A may also monitor the eyes of a user. In some embodiments, a user may attempt to squint or blink if the user is experiencing difficulty viewing a GUI. In some embodiments, the dilation of the user's eyes may also be tracked. Camera130A may capture images while processor160may be configured to identify these images and/or classify the eye patterns of the user. In response to detecting an accessibility pattern, processor160may modify the GUI accordingly.

Camera130B may be used in conjunction with and/or independently from camera130A to detect an accessibility pattern. Camera130B may capture images of objects while processor160may process the images in a similar manner to processing images from camera130A. For example, camera130B may be pointed at an object such as a government identification card. The user may be attempting to capture an image of the card. Due to difficulties viewing the card through GUI display170on mobile device120, the user may attempt to move mobile device120closer to the user's eyes. This action may cause camera130B to move further away from the card and/or may result in an image where the card is smaller. Processor160may identify this action as an accessibility pattern.

Similarly, camera130B may be pointed at an object at a further distance such as a tree or car while the user is attempting to interact with mobile device120. As the user moves mobile device120, the images of the tree and/or car may become larger or smaller. Processor160may identify these images and/or categorize them as an accessibility pattern. For example, when the user is interacting with an application, the user may be farsighted and move mobile device120further away from the user's face. In these cases, objects in the images may appear larger.

Proximity sensor140may also be used to identify an accessibility pattern. Proximity sensor140may be a light or infrared (IR) sensor on mobile device120. In some embodiments, proximity sensor140may be the same sensor used to lock a display screen when a user attempts to make a phone call by placing mobile device120next to the user's ear. The application may identify that the user is holding mobile device120close to the user's face based on a distance determined from proximity sensor140.

Similar to the operation of camera130A, proximity sensor140may be used to determine a distance of mobile device120from the user's face or eyes. For example, proximity sensor140may transmit a light and/or IR signal and measure a reflected signal from the user's face. An application executing on processor160may perform this transmission and measurement using proximity sensor140. Upon receiving a reflected signal, processor160may determine a distance between mobile device120and a user's face. In some embodiments, proximity sensor140may not detect a reflected signal indicating that mobile device120is located at a far distance from the user's face.

Based on the measured distance from proximity sensor140, processor160may determine an accessibility pattern. For example, as user moves mobile device120closer to the user's face in an attempt to better view GUI display170, proximity sensor140may detect a shorter distance. Processor160may detect this short distance as an accessibility pattern. For example, if the distance measure falls below a threshold, an accessibility pattern may be detected. In some embodiments, processor160may measure the change in distance to detect an accessibility pattern. For example, if a user is moving the mobile device120, the distance measured by proximity sensor140may decrease over time. Different measurements may be sampled to detect this decrease in distance. In response to the sampling, processor160may detect an accessibility pattern.

Similarly, another accessibility pattern may relate to the user moving mobile device120closer and then farther away from the user's face. The user may be attempting to find a desired distance for reading by varying the distance between mobile device120and the user's face. Processor160may sample these distance measurements, measure the variation of distance, and detect a corresponding accessibility pattern indicating that the user is attempting to find an optimal distance.

Accelerometer150may also be used to detect an accessibility pattern. For example, accelerometer150may detect that the user is unable to hold mobile device120steadily. The user may be elderly and may experience difficulty viewing the content of the application and/or interacting with the application. In response to the data gathered from accelerometer150, processor160may detect an accessibility pattern. Similarly, processor160may detect an accessibility pattern based on interactions with GUI display170. GUI display170may be a display screen of mobile device120. GUI display170may be a touch-screen display. Similar to accelerometer150, processor160may detect difficulties interacting with GUI display170as an accessibility pattern. For example, a user may experience difficulty pressing buttons or icons. Processor160may monitor these interactions with GUI display170and detect an accessibility pattern to provide assistance to the user.

While the examples described may trigger an accessibility pattern independent from other sensors, in some embodiments, processor160may identify an accessibility pattern based on a combination of data from sensors. For example, proximity sensor140may indicate that the user has moved mobile device120closer to the user's face. Camera130A may also indicate the same motion via an analysis of the images captured by the front-facing camera130A. In response to both of these conditions being met, processor160may identify an accessibility pattern. While multiple sensors need not be used to identify an accessibility pattern, using multiple sensors may provide processor160with redundancy so that false-positives are not triggered.

While processor160may interact with the sensors to detect an accessibility pattern, processor160may also execute an application and modify a GUI of the application in response to the detected accessibility pattern. In some embodiments, the application may supply commands to processor160to modify the GUI displayed on GUI display170. In some embodiments, the type of modification may depend on the accessibility pattern detected. For example, if a user is near-sighted and/or moving mobile device120closer to the user's face, a particular modification to the GUI may be applied for near-sightedness, such as, for example, enlarging the text. Similarly, if a user is far-sighted, the text may also be enlarged or a different modification may be applied. In some embodiments, the corresponding modification may be pre-programmed by the application.

In some embodiments, machine learning and/or artificial intelligence may be used to determine a preferred modification. Processors160and/or the application executing on processor160may communicate with an external server to gather data and/or provide updates to determining a GUI modification. For example, test data presenting GUI modifications to users may be collected to determine effective GUI modifications. In some embodiments, this data may correlate a particular modification to a particular accessibility pattern or detected sensor pattern. Effectiveness of a particular GUI modification may be a measurement of time or attempts by a user to perform a task prescribed by the application.

For example, the task may be to capture an image of a government identification card or to supply information after reading a paragraph of text. If a user is experiencing difficulty, a GUI modification may be applied and the time to success and/or attempts to success may be measured. Initially, the type of GUI modification as well as the degree of a modification may be randomized and applied to the user applications. For example, icons and/or buttons may be rearranged and/or a paragraph of text may be divided into different portions to aid in gathering data related to effectiveness. This data may be gathered among multiple users to determine the effectiveness of a GUI modification. These GUI modifications may then be supplied to users in response to detecting particular accessibility patterns based on measured sensor patterns. Example embodiments of GUI modifications are further described with reference toFIGS. 3A-3C.

FIG. 1Cdepicts a block diagram of a graphical user interface (GUI) display170on a mobile device120, according to some embodiments. Mobile device120may be similar to mobile device120as described with reference toFIG. 1B.FIG. 1Bdepicts an embodiment of mobile device120. Mobile device120may include a front-facing camera130A and/or a proximity sensor140. In some embodiments, front-facing camera130A and/or proximity sensor140may reside on the same face of mobile device120as GUI display170.

GUI display170may display text, images, videos, icons, buttons, Tillable forms, lists, and/or other visual data objects generated by an application. Upon detecting an accessibility pattern via the sensors of mobile device120, the application as executed on a processor of mobile device120may modify the data options displayed on GUI display170.

In some embodiments, GUI display170may include a camera data object180. Camera data object180may display images captured by a rear-facing camera on mobile device120. GUI display170may display these images to aid a user in capturing an image with mobile device120. In some embodiments, an application may request and/or prompt a user to capture an image via camera data object180. The user may experience difficulty capturing the image by being unable to accurately view the image in camera data object180and/or align the object to be captured. The object to be captured may be, for example, a government identification.

As previously described, sensors or a processor of mobile device120may detect an accessibility pattern. For example, the accessibility pattern may be detected in response to a user accessing the application and/or accessing a portion of the application prompting the user to capture an image. The application may display camera data object180on GUI display170during a first state of the application. Upon detection of the accessibility pattern, the application may modify GUI display170to enlarge camera data object180to aid the user in capturing the image. This process may enlarge camera data object180to display a larger real-time camera view. For example, a larger portion of GUI display170may display camera data object180. This enlarging of camera data object180may be a modification of the GUI.

As previously explained, this GUI modification may be specific to and/or controlled by the application. The application may control this modification rather than a default setting of mobile device120. Further mobile device120may be incapable of detecting this type of modification. In this manner, the embodiments described are able to tailor specific GUI modifications to specific applications and/or application interactions.

FIG. 2depicts a flow diagram illustrating a method200for modifying a graphical user interface (GUI) of an application according a detected accessibility pattern, according to some embodiments. Method200shall be described with reference toFIG. 1B; however, method200is not limited to that example embodiment.

In some embodiments, processor160of mobile device120may utilize method200to modify a GUI displayed on GUI display170in response to detecting an accessibility pattern. While method200is described with reference to mobile device120, method200may be executed on any computing device, such as, for example, the computer system described with reference toFIG. 4and/or processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof.

It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown inFIG. 2, as will be understood by a person of ordinary skill in the art.

At202, processor160of mobile device120may receive a command to access an application. A user may interact with mobile device120via GUI display170and select a particular application. Upon receiving the selection, mobile device120may retrieve software elements from memory and/or download application data from an external computer or server. Processor160may then execute the application and generate GUI data object visualized on GUI display170.

At204, processor160may detect a mobile device accessibility setting. At206, processor160may record data from a sensor of mobile device120to detect an accessibility pattern. Using the mobile device accessibility setting and/or the recorded data, processor160may determine whether an accessibility pattern is detected at208. In some embodiments, user profile information may also be retrieved to aid in determining an accessibility pattern.

The mobile device's accessibility setting may be a setting of the operating system. Using a mobile device's accessibility setting to detect an accessibility pattern may be optional. In some embodiments, however, the accessibility setting may be used in conjunction with the recorded sensor data to determine an accessibility pattern. For example, an accessibility setting may indicate that applications of the mobile device120are to display text with a larger font size. The application, as executed by processor160, may use this information and adjust the font size accordingly. In some embodiments, this accessibility setting may result in a first modification to a GUI while recorded data may cause further or additional modifications based on the detected accessibility pattern. In some embodiments, processor160may determine that an accessibility setting of mobile device120has not been set. Processor160may still apply GUI modifications in response to detecting an accessibility pattern even if no default accessibility setting has been set.

At206, data may be recorded from sensors of the mobile device120such as camera130A, camera130B, proximity sensor140, accelerometer150, and/or GUI display170. Upon recording data, processor160may analyze the data gathered from the one or more sensors at208to determine whether an accessibility pattern is detected. The types of sensor data gathered as well as the different accessibility patterns were previously discussed with reference toFIG. 1BProcessor160may be pre-configured to detect an accessibility pattern from the pattern identified from the sensors. When the application is updated, processor160may also update the patterns and/or conditions used to detect an accessibility pattern.

At210, processor160may determine whether to proceed due to detecting an accessibility pattern or return to206to continue monitoring recorded data from the one or more sensors. If an accessibility pattern is not detected, processor160may continue to record data until an accessibility pattern is detected. In some embodiments, a user may complete a task or finish interacting with the application without detection of an accessibility pattern.

If an accessibility pattern is detected, however, processor160may modify a GUI of the application as displayed on mobile device120according to the detected accessibility condition at212. In some embodiments, the modification of the GUI may be performed according to the accessibility pattern and/or underlying data recorded. For example, if processor160detects an accessibility pattern based on data recorded from proximity sensor140and/or camera130A, processor160may perform a particular modification corresponding to the accessibility pattern, such as, for example, enlarging text.

In some embodiments, processor160may detect an accessibility pattern from other sensors, such as accelerometer150or GUI display170. Accelerometer150may detect that a user is not holding mobile device120steadily and/or that mobile device120is shaking. Processor160may also detect that the user is experiencing difficulty interacting with GUI display170. In response to one or more of these accessibility patterns, a different GUI modification may be applied, such as, for example, rearranging data objects depicted in GUI display170and/or splitting the content into multiple GUI screens. This accessibility may modify the content of the application to aid the user in interacting with the application beyond enlarging text. In this manner, the GUI may be modified in response to the accessibility pattern detected. Example GUI modifications are further described with reference toFIGS. 3A-3C.

In some embodiments, the mapping of a particular GUI modification to a particular accessibility pattern may be pre-configured by the application as executed on processor160. In some embodiments and as previously described, the mapping of a particular GUI modification to a particular accessibility pattern may be determined using machine learning and/or artificial intelligence. For example, the test data may be collected from multiple instances of the application executing on multiple mobile devices120. Different GUI modifications may be recommended based on different accessibility patterns detected. The application may determine the efficacy of the GUI modification based on how quickly the user completes a task or operation on the application and/or the number of interactions used to complete the operation. Based on this gathered information, GUI modifications may be mapped using the machine learning to minimize the amount of time or interactions that elapse prior to a user completing an interaction. Upon detecting a subsequent accessibility pattern, the particularly mapped GUI modification may be executed.

FIG. 3Adepicts a block diagram of a GUI modification process300A displaying enlarged text within an application, according to some embodiments. A mobile device may include a GUI display310with a text data object320A. For example, text data object320A may be application content, a message prompt, instructions, icons, buttons, and/or other text generated by the application. In an example embodiment, the text data object320A may be an instruction for capturing an image of a government identification that states “Please Go Inside To Capture Your License.”

In response to detecting an accessibility pattern, a processor of the mobile device may execute GUI modification process300A to enlarge text data object320A to display a larger version of the text as text data object320B. While the content of the text may be the same, the size may be increased to aid a user experiencing difficulties with interacting with the application. In some embodiments, GUI modification process300A may also be applied to images. After detecting an accessibility pattern, the processor may enlarge different types of data objects, such as images, windows, graphics, and/or other visual data to assist the user.

FIG. 3Bdepicts a block diagram of a GUI modification process300B displaying list330modifications, according to some embodiments. As the application executes, the application may display a list330A. List330A may include data objects340A,345A. Data objects340A,345A may be interactive icons and/or buttons used to interact with the application. WhileFIG. 3Bdisplays the arrangement of data objects340A,345A in a list format, the data objects may be arranged in other formats such as a grid or free-form floating icons viewable with the GUI. In an example embodiment, data objects340A,345A may be selectable options to proceed with a “two-step verification.” For example, when selected, data object340A may allow a user to capture an image of an identification. When data object345A is selected, the application may be command to send a text message requesting that a temporary be transmitted to the mobile device. List330A may include other data objects and may represent different commands available for interacting with the application.

Upon detecting an accessibility pattern, a processor of the mobile device may modify the GUI and alter list330A. This modification may cause the GUI to display list330B. List330B may alter the location of data objects as seen from data objects340B,345B. For example, the option to capture an image may be listed second as data object345B while the option to request a text message may be listed first as data object340B. This GUI modification may be performed based on an accessibility pattern detected by the processor of the mobile device. For example, the processor may determine that the user is unable to steadily hold the mobile device from accelerometer data and/or camera data. In this case, the user may experience difficulty capturing an image. The processor may then modify the GUI to highlight an option that may be easier for the user to perform the verification.

In some embodiments, a GUI displaying list330A or list330B may be altered to display list330C. List330C may remove data objects, rearrange, and/or reorder the location of data objects in response to a detected accessibility pattern. For example, if a user is detected as having difficulty navigating the application, some data objects may be removed leaving data objects340C,345C as the options available for selection. For example, to complete a verification process, data object340C may highlight touching a contact card that may operate using near-field communications to perform a verification. In some embodiments, other hardware authentication devices may be recommended, such as hardware tokens and/or devices implementing a FIDO protocol. Some options, such as capturing an image may be removed as potentially being too difficult for a user to perform. For example, a user may register failed attempts at interacting with the application and/or may be unable to steadily hold the mobile device. In this case, the detected accessibility pattern may indicate that the user may not be capable of capturing an image. The GUI modification may then remove this option.

As previously explained, the GUI modification may rearrange and/or reorder data objects that may appear on a list330and/or that may be free-form. Similarly, some data objects may be removed or hidden. In some embodiments, other visual distinctions may be performed to identify a data object such as colors, highlighting, or enlarging text.

FIG. 3Cdepicts a block diagram of a GUI modification process300C displaying a slide-based modification, according to some embodiments. GUI modification process300C may rearrange content of an application to aid a user in completing a process. In some embodiments, portions of a GUI may be displayed so that a user is not overwhelmed or unable to view and/or interact with data objects on the GUI. This modification may modify data objects into different “slides” representing portions of the GUI.

For example, a GUI may be divided into a first portion and a second portion. A second GUI may be generated display the first portion such that the second GUI occupies the display area. A third GUI may then be generated displaying the second portion of the GUI. This third GUI may replace the second GUI such that the third GUI occupies the display area. In an embodiment, this process may be referred to as “slides” and will be further discussed with reference to list350and data objects360A,370A as depicted inFIG. 3C.

In some embodiments, an application executing on the mobile device may display a list350of data objects360A,370A. The data objects360A,370A may be forms used by the user to supply information to the application. For example, the application may request identification information such as a username, password, zip code, and/or the last four digits of a social security number. The data objects360A,370A may allow users to enter this information while interacting with list350.

Upon detecting an accessibility pattern, however, the application as executed on a processor of the mobile device may identify that the user is experiencing difficulty interacting with the data objects360A,370A and/or supplying information. In this case, a slide-based modification may be applied to list350. Instead of display list350, the GUI may display portions of list350so that less text occupies the GUI and/or so that fewer data objects are visible to avoid overwhelming a user.

For example, GUI modification may generate slide380. Slide380may be a GUI display representing data object360A as a slide data object360B. Slide data object360B may be an enlarged version of data object360A and/or may occupy more space on a display screen of the mobile device. Similarly, the text of slide data object360B may be enlarged to aid a user in viewing the text. Other interactive icons and/or buttons may also be provided to aid with navigation to view other data objects. In some embodiments, slide data object360B may request that a user supply a username. Upon receipt of the username, slide380may advance to slide390.

Slide390may be a GUI display representing data object370A as a slide data object370B. Similar to slide data object360B, slide data object370B may represent an enlarged version of data object370A. In some embodiments, the user may be requested to supply a password using slide data object370B. Upon completion, the user may continue to interact with subsequent slides so that the application may receive the requested information. These GUI modifications in response to detecting an accessibility pattern may aid a user in interacting with the application. Further, this modification may reduce the number of failed attempts to supply information leading to an increase in data processing speeds.

In view of the modifications described with reference toFIGS. 3A-3C, the application may modify GUI screens to adjust an application experience for a user. Certain data objects may be adjusted to avoid distractions and to aid with completion of a task. These GUI modifications may be applied to many varieties of different applications performing different tasks.

WhileFIGS. 3A-3Chave described user interface modifications to a GUI, other user interface modifications may also be performed such as auditory modifications. In some embodiments, a screen reader may be employed and/or another audio file may be played to aid the user. For example, the screen reader may read the content of data objects. The audio aids may be played independently from the GUI modification and/or in conjunction with the GUI modifications. In some embodiments, upon detecting an accessibility pattern, the application may play an audio file that provides further instructions beyond reading the text on the screen. For example, the audio file may explain that the process will be divided into slides. In some embodiments, if a user is experiencing difficulty interacting with the application, a microphone of the mobile device may be activated allowing a user to navigate the application with voice commands.

FIG. 4depicts an example computer system useful for implementing various embodiments.

Various embodiments may be implemented, for example, using one or more well-known computer systems, such as computer system400shown inFIG. 4. One or more computer systems400may be used, for example, to implement any of the embodiments discussed herein, as well as combinations and sub-combinations thereof.

Computer system400may include one or more processors (also called central processing units, or CPUs), such as a processor404. Processor404may be connected to a communication infrastructure or bus406.

Computer system400may also include user input/output device(s)403, such as monitors, keyboards, pointing devices, etc., which may communicate with communication infrastructure406through user input/output interface(s)402.

Computer system400may also include a main or primary memory408, such as random access memory (RAM). Main memory408may include one or more levels of cache. Main memory408may have stored therein control logic (i.e., computer software) and/or data.

Computer system400may also include one or more secondary storage devices or memory410. Secondary memory410may include, for example, a hard disk drive412and/or a removable storage device or drive414. Removable storage drive414may be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, tape backup device, and/or any other storage device/drive.

Removable storage drive414may interact with a removable storage unit418. Removable storage unit418may include a computer usable or readable storage device having stored thereon computer software (control logic) and/or data. Removable storage unit418may be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and/any other computer data storage device. Removable storage drive414may read from and/or write to removable storage unit418.

Secondary memory410may include other means, devices, components, instrumentalities or other approaches for allowing computer programs and/or other instructions and/or data to be accessed by computer system400. Such means, devices, components, instrumentalities or other approaches may include, for example, a removable storage unit422and an interface420. Examples of the removable storage unit422and the interface420may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and/or any other removable storage unit and associated interface.

Computer system400may further include a communication or network interface424. Communication interface424may enable computer system400to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively referenced by reference number428). For example, communication interface424may allow computer system400to communicate with external or remote devices428over communications path426, which may be wired and/or wireless (or a combination thereof), and which may include any combination of LANs, WANs, the Internet, etc. Control logic and/or data may be transmitted to and from computer system400via communication path426.