System and method for providing a customized graphical user interface based on user inputs

A system and method that provides customized graphical user interfaces on mobile devices based on user inputs. An exemplary method includes detecting a computing device remotely connected to a remote server over a network and having an active session of a software application running on the remote server. Moreover, the method further includes identifying and selecting one or more hotkey buttons based on the detected software application, transmitting the one or more hotkey buttons to the computing device to be displayed in a customized interface while the software application is active, detecting an activation of the one or more hotkey buttons displayed on the computing device, and executing, by the remote server, an operation for the active software application in response to the activation of the one or more hotkey button by the user.

TECHNICAL FIELD

The present disclosure generally relates to graphical user interfaces on computer displays, and, more particularly, to a system and method for providing a customized graphical user interface based on user inputs.

BACKGROUND

As mobile devices and other electronic appliances continue to develop with increasingly powerful processing capabilities and dynamic touchscreen displays, the software applications available on these devices also continues to get more complex. However, with these increasingly complex software applications comes a significant complexity in the number of keystrokes and other various inputs for users to interact with these applications, especially in the context of electronic gaming, for example. Moreover, while the processing abilities continue to increase for such mobile devices, the complex sequences of user interaction within such software application environments still consume significant processing resources.

Therefore, a customized and easy interface is still needed enabling users to easily and quickly interact with such sophisticated software applications. Moreover, it is desired to provide a system and method that minimizes resource consumption by the end client device.

SUMMARY

Accordingly, a system and method is provided a customized graphical user interface based on user inputs. An exemplary method includes detecting, by a remote server, a computing device remotely connected to the remote server over a network and having an active session of a software application running on the remote server; identifying and selecting, in a hotkey database of the remote server, at least one hotkey button based on the detected active session of the software application on the computing device; transmitting, by the remote server, the identified and selected at least one hotkey button to the computing device to be displayed in a customized interface on a display screen thereon in connection with the active session of the software application; detecting, by the remote server, an activation of the identified and displayed at least one hotkey button by a user of the computing device; and executing, by the remote server, an operation for the software application running on the remote server in response to the activation of the identified and displayed at least one hotkey button by the user.

According to another exemplary aspect, the method includes analyzing, by the remote server, combinations of input sequences received from a plurality of computing devices running active sessions of the software application; generating, by the remote server, the at least one hotkey button based on respective frequencies of the combinations of input sequences; and storing, by the remote server, the generated the at least one hotkey button in the hotkey database.

According to another exemplary aspect, the method includes detecting, by the remote server, a new computing device having another active session of the software application; determining device parameters of the new computing device that include a device type and touchscreen input dimensions of the new computing device; selecting, by the remote server, the at least one hotkey button stored in the hotkey database based on the device type and touchscreen input dimensions of the new computing device; and transmitting, by the remote server, the selected at least one hotkey button to the new computing device to be presented as a customized interface on a display screen in connection with the active session of the software application.

According to another exemplary aspect, the identifying and selecting, by the remote server, of the at least one hotkey button based on the detected active session of the software application comprises identifying the hotkey button based on device parameters of the computing device including a device type and touchscreen input dimensions of the computing device.

According to another exemplary aspect, the method includes logging, by the computing device, combinations of input sequences entered by a user of the computing device operating the active session of the software application based on at least one of keystroke modifiers and functional keys selected by the user.

According to another exemplary aspect, the method includes transmitting, by the computing device, the logged combinations of input sequences to the remote server; and generating, by the remote server, the at least one hotkey button based on the logged combinations of input sequences received from the computing device.

According to another exemplary aspect, the method includes determining, by the remote server, device parameters of the computing device; and generating, by the remote server, the at least one hotkey button based on the logged combinations of input sequences received from the computing device and the determined device parameters.

According to another exemplary aspect, the method includes dynamically updating, by the remote server, the at least one hotkey button selected for the active session of the software application based on combinations of input sequences received from other computing devices running other active sessions of the same software application as the computing device; and transmitting, by the remote server, the dynamically updated at least one hotkey button to the computing device to be presented in the customized interface on the display screen in connection with the active session of the software application.

According to another exemplary aspect, the method includes logging, by the computing device, combinations of input sequences entered by the user of the computing device operating the active session of the software application based on at least one of keystroke modifiers and functional keys selected by the user; determining, by the computing device, whether one of the logged combination of input sequences exceeds a predetermined threshold; and identifying and displaying, by the computing device, in the customized interface on the display screen at least one hotkey button that corresponds to the logged combination of input sequences when the predetermined threshold is exceeded.

According to another exemplary aspect of the method, the identified and selected at least one hotkey button displayed in the customized interface on the display screen comprises an expandable menu of a plurality of selectable hotkeys that each correspond to a different selectable combination of input sequences for the software application that is active on the computing device.

According to another exemplary aspect of the method, the expandable menu comprises a first ring having a plurality of user actions associated with the software application and a second ring having the plurality of selectable hotkeys that is displayed on the customized interface upon the user selecting at least one of the plurality of the user actions in the first ring.

In yet another aspect, a system provides a customized graphical user interface based on user inputs. In this aspect, the system includes a hotkey database configured to store at least one hotkey button for a software application; and a remote server with a processor configured to detect a computing device that is remotely connected to the remote server over a network and that has an active session of the software application running on the remote server, identify and select, in the hotkey database, at least one hotkey button based on the detected active session of the software application on the computing device, transmit the identified and selected at least one hotkey button to the computing device to be displayed in a customized interface on a display screen thereon in connection with the active session of the software application, detect an activation of the identified and displayed at least one hotkey button by a user of the computing device, and execute an operation for the software application running on the remote server in response to the activation of the identified and displayed at least one hotkey button by the user.

DETAILED DESCRIPTION

Various aspects are now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to promote a thorough understanding of one or more aspects. It may be evident in some or all instances, however, that any aspect described below can be practiced without adopting the specific design details described below. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate description of one or more aspects. The following presents a simplified summary of one or more aspects in order to provide a basic understanding of the aspects. This summary is not an extensive overview of all contemplated aspects, and is not intended to identify key or critical elements of all aspects nor delineate the scope of any or all aspects.

FIG. 1illustrates a block diagram of a system100for providing a customized graphical user interface based on user inputs according to an exemplary aspect. As shown, the system100is generally comprised of one or more client devices110A-110D, a key combination generation and execution server120(hereinafter referred to as “server” or “remote server”120) and a network130that communicatively couples the client devices110A-110D to server120. According to the exemplary aspect, the client devices110A-110D are each a mobile computing device, such as a smartphone or tablet, for example. It is well known to those skilled in the art that such mobile devices typically have smaller display screens (relative to conventional desktop personal computers), and, therefore, it can be frustrating or difficult for many users to quickly enter user input operations (e.g., keystroke combinations) using these smaller screens. Thus, as will be readily apparent from the disclosure herein, the exemplary system and method enable the user(s) of each mobile device110A-100D to quickly and easily input user operations using a customized graphical user interface provided on the respective display screens of each mobile device.

It should also be appreciated that while mobile devices are described as the client devices110A to110D according to the exemplary aspect, the exemplary system and method described herein can be provided for any type of computing device, such as a desktop, laptop or the like, as would be appreciated to one skilled in the art. Moreover, while only four client devices110A-110D are shown according to the exemplary system100, it is noted that the exemplary system and method can be provided for any number of client devices. Moreover, in one aspect, a large number of client devices (e.g., 100 or even 1000 or more mobile devices) provide more analysis points for the server120to evaluate user inputs and generate “hotkeys” as will be described in more detail below.

As noted above, each client device110A-110D of the system100is configured to communicate data across one or more networks130to server120according to the exemplary aspect. The applicable network130can be any network for communicating data and data operations and can include one or more specific communication systems (not shown) that connect the various components of the system100by wire, cable, fiber optic, and/or wireless links facilitated by various types of well-known network elements, such as hubs, switches, routers, and the like. It should be appreciated that the network130may employ various well-known protocols to communicate information amongst the network resources. In one aspect, the network130can be part of the Internet or intranet using various communications infrastructure such as Ethernet, WiFi, mobile telecommunication networks, and the like.

According to the exemplary aspect, each client device110A to110D is configured to install a client/server application (e.g., Parallels® Remote Application Server Client) on the device that is configured to communicate via network130with the server120. In turn, server120can be an application server (e.g., Parallels® Remote Application Server) that, among other things, provides the functions of key combination (i.e., “hotkey”) generation and execution and can operate in a coordinate manner with the client/server application installed on each client device110A-110D, as will be described in more detail below.

In the exemplary aspect, remote server120is configured to generate and provide customized button(s) (referred to as “hotkeys”) that can be customized and configured to be displayed on each specific client device enabling the respective user of the device to perform frequently used key combinations using the hotkey presented on the customized interface. For example, in an exemplary aspect, these button(s) may represent shortcuts of controls selected for different applications, threads or combinations of keystrokes, or the like.

In one aspect, the client/server application that is running on a client device (e.g., one or more of client devices110A-110D) is configured to maintain a hotkey database of shortcuts received from remote server120. Thus, during operation, the client/server application can monitor and analyze the combination of such keystrokes and also measure usage frequency for the active user application also running on the client device. When the usage frequency of a particular keystroke combination reaches a certain threshold (which can be configured by the device user or a system administration, for example), the client/server application can query the local hotkey database to determine whether the specific shortcut is already known in the database and can be displayed as a single on-screen element (i.e., actionable hotkey button) with a human readable label and/or icon, which can in turn execute the shortcut as a single user touch action, for example. This label or icon can be displayed automatically in one aspect. In another aspect, the user can be presented with an option as to whether he or she would like to add this shortcut as an on-screen button to facilitate further usage (i.e., for providing the single touch action). Yet further, if the shortcut is not already known in the database, the user can be presented with an option of whether he or she would like to create a new on-screen element, provide it with a customizable name, and the like, which can be added to the local database and saved for further usage.

In yet a further aspect, these identified frequently used combinations can be transmitted/posted from the client/server application on the mobile device to the remote server120by network130. The server120is configured to receive these combinations of keystrokes and determine desired/useful key combinations for each client device based on the statistics of which shortcuts are used more frequently in the user applications, for example. When the remote server120determines that the shortcut usage frequency for each combination reaches a defined threshold (also configurable by the system administrator, for example), the remote server120can generate a suggestion for configuring a new-well known on-screen button (assuming this button has not been previously generated. The newly generated key combination (i.e., “hotkey”) button can then be transmitted back to client device and added to the hotkey database, for example. Thus, during subsequent operation, this new hotkey can be presented on the user display as a form of a customized graphical user interface (“GUI”) using the process described above.

Furthermore, in the exemplary aspect, the server120may be remotely running the application for one or more of the client devices110A-110D, such that each client device can run an active session of the remote software application that is primarily executed by the remote server120. Upon receiving a user input of a key combination button on the client device, the button input/activation can be transmitted back to the server120for execution of the operations associated with the hotkey button. By doing so, the exemplary system100can provide higher security and hardware independence for the client devices110A-110D according to an exemplary aspect. The details and advantages of the system100will now be described as follows.

FIG. 2illustrates a block diagram of a client device110for providing a customized graphical user interface based on user inputs according to an exemplary aspect. Client device110generally represents any one of client devices110A to110D described above. In general, client device110can be any type of mobile device, for example, and will include an operating system and a central processing unit (“CPU”)210provided to, among other things, execute client/server application220, which as noted above can be Parallels® Remote Application Server Client, for example. In this aspect, the client/server application220can be downloaded from server120and installed and includes software code (e.g., processor executable instructions) in memory, which may be configured to execute/facilitate the key combinations on the customized GUI of the client device110according to the exemplary aspects described herein.

According to an exemplary aspect, the client/server application220downloaded on client device110is configured to execute a plurality of module, including input collection module230and input analysis module240. These modules may be part of client/server application220in one exemplary aspect or separate software component(s) according to another exemplary aspect. As used herein, the term “module” refers to a software service or application executed on one or more computers, including client devices110A-110D, components, or arrangement of components implemented using hardware, such as by an application specific integrated circuit (ASIC) or field-programmable gate array (FPGA), for example, or as a combination of hardware and software, such as by a microprocessor system and a set of instructions to implement the module's functionality, which (while being executed) transform the microprocessor system into a special-purpose device. A module can also be implemented as a combination of the two, with certain functions facilitated by hardware alone, and other functions facilitated by a combination of hardware and software. In certain implementations, at least a portion, and in some cases, all, of a module can be executed on the processor of a general purpose computer. Accordingly, each module can be realized in a variety of suitable configurations, and should not be limited to any example implementation exemplified herein. For purposes of this disclosure below with respect to the exemplary algorithms, the disclosure generally refers to the client device110and/or CPU210and/or one of the specific modules as shown to perform the various steps of the algorithm.

As is the case for all mobile devices, client device110is generally configured to execute and/or provide a plurality of software applications for the user of the client device110. For example, as described above, the client/server application220can be provided by server120, as a remote application server that remotely provides software applications (e.g., a remote desktop and remote gaming environment, etc.) on client device110. In this configuration, the server120remotely executes the desired application for the client device110, which is then transmitted and displayed on the client device110over network130, as should be appreciated to one skilled in the art. For example, in one particular aspect, the server120can provide a “game mode” that allows a user of client device110to conveniently play desktop games by the customized GUI presented on the display260(e.g., LCD screen) on the client device110, where the game is running on a remote session by server120.

In the exemplary aspect, the input collection module230is activated by client/server application220, for example, and is configured to receive keyboard and/or mouse inputs from the user of the client device110running the remote application. Assuming the client device110is a mobile device, such as a iPhone®, for example, the input collection module230is configured to monitor and track/record inputs of the electronic keyboard (or other movement/touch strokes) input by the user via the user interface of the application that is presented on display260of the device110. In this aspect, the input collection module230is configured to log the usage of key combinations, and the like, and associate them with the particular type of client device and also for the current active application, in which the user of the client device110is working, playing, etc. In one aspect, these “hotkeys” can be tracked when the user selects one or more modifier keys with a regular (i.e., non-modifier) key. Modifier keys can be, for example, one or more of: “ctrl”, “alt”, “shift”, “altGr”, “win”, “command”, “Fn”, “media keys”, or the like. Moreover, the client device110can determine the current active application from the software agent installed on the remote server120, for example.

According to the exemplary aspect, the selection of one of the modifier keys by the user activates the input collection module230to begin collecting the keystroke combinations, for example. In other aspects, the input collection module230can continuously monitor any non-letter keystroke inputs, for example, and log combinations. In one aspect, if a user is playing a game, the input collection module230can monitor particular keystroke sequences of arrow inputs relevant to the game. In another aspect, if the user is playing a game, such as a combat game or first person shooter game, the collection module230can monitor particular combinations of action strokes required to perform a particular action in the game. For example, a specific combination of directional swipes on the touchscreen of the display260may be required to perform a particular fighting combination move in a game. The collection module230is configured to track the repeated combination of such direction strokes in this aspect.

Moreover, the information collected and logged by the input collection module230can be provided to analysis module240in the exemplary aspect. In turn, analysis module240is configured to analyse and compare the collected data, including client device type (e.g., model and operating software version), specific application(s) running, keystroke combination, and frequency of use. As the analysis module240detects shortcut combinations, the analysis module240can store them as local records, where each record contains the shortcut key sequence, a timestamp when the shortcut has been detected, and an active remote application identifier (i.e., which application was running when the user entered the keystroke combination). Having those records, the analysis module240can also calculate how many times a specific keystroke combination (i.e., a shortcut) has been activated in a given active application during a predefined time frame, which can be defined the device user and/or by an administrator of the client/server application220, for example. This calculation can be used to determine a shortcut usage frequency for a given keystroke combination.

As further described above, a hotkey database270can be downloaded from remote server120and stored on the client device110. In one aspect, the downloaded hotkey database270can be synchronized with a current hotkey database maintained by the remote server120to receive the best or optimal hotkeys for a given user application installed on the client device110. Moreover, when a measured shortcut usage frequency for a given keystroke combination exceeds a given threshold (e.g.,5instances of the keystroke combination in one minute), the analysis module240can query the hotkey database270to determine whether the specific shortcut is already known in this local database270. If so, the CPU210can cause a single on-screen element with a human readable label and/or icon that corresponds to the identified hotkey to be displayed on display260of the client device110while the specific user application is “active” (i.e., currently being operated by the user). This single on-screen button can be configured to execute the identified keystroke combination upon as a single user touch action during use of the active application. In a refinement of this aspect, the user of mobile device110can be presented with an option as to whether he or she would like to add this shortcut as an on-screen button to facilitate further usage (i.e., the single touch action). Yet further, if the shortcut is not already known in the database, the user can be presented with an option of whether he or she would like to create a new on-screen element, provide it with a customizable name, and the like, which can be added to the database270and saved for further usage.

Furthermore, the analysis module240is configured to cause the collected data (both known and/or new keystroke combinations, for example) to be transmitted to remote server120for hotkey identification, statistical analysis and/or hotkey generation according to an exemplary aspect. For example, the collected data can be transmitted at a particular frequency n, for example, periodically such as once a day, or every time before closing connection by the client/server application220to the server120. Moreover, in one refinement of the exemplary aspect, the analysis module240is configured to filter the data collected by input collection module230and only transmitted relevant (i.e., possible hot keys and keystroke combinations) while not transmitting user inputs that are determined to not be possible keystroke combinations. For example, if a user is typing an email, these types of user inputs will either not be collected by input collection module230and/or will not be transmitted by analysis module240to server120. As further shown, client device110is configured to transmit the collected and analyzed data across the network130using any convention I/O device260, such as a network interface controller or the like.

FIG. 3illustrates a block diagram of a remote server120for providing a customized graphical user interface based on user inputs according to an exemplary aspect. In general, server120includes a database310of hotkey buttons, a plurality of modules configured to perform the exemplary algorithms described herein, and I/O interface360configured to communicate with the plurality of client devices110A-110D over network130, as described above. Although not shown, it should be appreciated that the server120is configured to execute applications and the software modules described herein and includes the necessary hardware and software components that would be known to one skilled in the art, such as one or more computer processing units, for example.

In one exemplary aspect, the database310is configured to store a plurality of custom on-screen controls, such customized buttons for keyboard stroke combinations, mouse stroke combinations, touchscreen combinations, joysticks movement combinations, and the like. The remote server120further includes a hotkey selection module320that is configured to select particular hotkey buttons, from hotkey database310, for example, to be provided to a client device running a particular client/server application and being run on a particular device. These customized hotkey buttons can be selected and transmitted to the client device110when the remote server120detects the mobile device's launching/activation of the specific client/server application. Moreover, as described above, the client device110can maintain a hotkey database270for storing hotkey buttons for the particular applications installed thereon. In this regard, the remote server120can synchronize with the client device110to identify the active (or frequently used) applications, for example, and download the relevant hotkeys to be stored in the hotkey database270.

Referring back toFIG. 2, the client/server application220further includes a hotkey display module250configured to receive and customized hotkey buttons received from server120for the particular application. The hotkey display module250can cause these buttons to be displayed on display260, for example, when the user is playing a game application, which enables the user to conveniently play the desktop game that running in a remote session from the remote server120, using the client device110, which can be an iPhone or iPad, for example. Thus, when the user interacts with these customized buttons (i.e., activates or clicks the hotkey button displayed on display260), those interactions are transformed into the desired keys/shortcuts presses, mouse movements/clicks, for the particular application.

It should be appreciated that different game applications require different set of controls to play conveniently, but games of similar genres (e.g., first person shooters, RPGs, strategies, simulators, etc.), often use the same or similar set of keyboard and mouse actions to play. In other words, if one shooter has a certain set of controls, then another shooter most probably has a very similar set of controls, for example. In the exemplary aspect, when client device110downloads and installs the client/server application220, it can be provided with a limited set of predefined on-screen controls (i.e., “pre-sets” or “pre-set hotkeys”) in the hotkey database270, so that the user does not have to setup game controls from the scratch for each and every game. In this aspect, the pre-set hotkeys can either be presented automatically based on the particular game and/or particular device and operating software or the user can select these pre-set hotkeys manually.

It should also be appreciated that as the amount of games is huge and continues to grow, and the various games can differ in terms of “best” on-screen controls configuration, the remote server120can figure be configured to add and maintain the pre-set hotkeys centrally at the hotkey database310, for example. Therefore, in this exemplary aspect, the remote server120(e.g., Parallels® cloud or within a serve farm) can store predefined the pre-set hotkeys in hotkey database310for on-screen controls for various games/genres. When a user launches a game via the client/server application220, the identification of the game can be transmitted to remote server120. If the game is already registered with remote server120, the server120can return the best pre-set hotkeys for the game for hotkey database270, which can be determined from previous game users using the method described herein, for example. In contrast, if this game is not registered yet, the remote server120can further be configured to identify information about the game on the Internet, for example, determine the game's genre, and the like, and return the optimal pre-set hotkeys for the game based on similar games registered with the remote server120, for example.

Thus, according to an exemplary aspect, the pre-set hotkeys can be updated in the hotkey database310when a popular game is released or is about to be released. In addition, the hotkey database310can also be updated based on users' pre-sets analysis. This analysis is aimed to find common configuration patterns in pre-sets (e.g., which keys are assigned to on-screen controls, what are the mouse sensitivity settings, etc.) for the same games, and deduce the optimal default pre-set hotkey buttons for each game (or other application). As a result, when a user of a client device110launches a particular game for the first time, the remote server120will be informed of the launching/activation/installation of the game and will identified the “best known” optimal pre-set hotkeys automatically as described above. Moreover, while the user is playing the particular game (or other application), the input collection module230will collect user input controls, for example, if the user customizes the pre-sets, or creates a new pre-set, and have this information sent to remote server120. This information will enable remote server120to dynamically build and update the “best known” pre-set hotkey buttons in the hotkey database310for subsequent users of the same game (or other application).

As further shown inFIG. 3, remote server120also includes key combination analysis module330that is configured to receive the input information collected by the input collection module230of the client device110and determine the most frequent combinations of keystrokes to determine the best key combinations for hotkeys. For example, this determination can be based on frequency of use of the combinations (compared with other combinations), i.e., the shortcut usage frequency, for the specific application and also the specific type of mobile device (which may include the type of input, size of touchscreen, etc.). The analysed information that can then be provided to hotkey generation module340that is configured to generate one or a plurality of hotkey buttons for the particular application (if the hotkey has not already been created). These hotkey buttons generated by hotkey generation module340can be stored as software objects in hotkey database310. Moreover, in one aspect, the remote server120is configured to determine and provide hotkey buttons back to client device110in real-time (while the application on the device is currently running). As shown, the hotkey selection module230is also coupled to receive the output of the hotkey generation module340. Thus, as hotkey buttons are generated by the hotkey generation module340, these buttons can be accessed by hotkey selection module320and transmitted back to client device110to be displayed by hotkey display module250in real-time for the running application according to one exemplary aspect.

Otherwise, the generated hotkey buttons can also be stored as software objects in hotkey database310and will include metadata associated each generated button with the particular application (or genre of applications, for example), and the particular device. In this aspect, when a different client device subsequently launches the same (or a similar application), the hotkey selection module320can identify and transmit the optimal hotkey buttons for this new/activated application as further described above.

As further described above, the remote server120may be remotely running the application for one or more of the client devices110A-110D. These remotes applications are generally shown as remote applications370, which can be remote desktop applications, remote gaming applications, or the like. Moreover, during operation, the one or more client devices can configured to run an active session of the remote application370. As a result, the processing of the software application will be primarily executed by the remote server120, such that when the user enters inputs (e.g., keystrokes, mouse clicks, press inputs, hotkeys, etc.), these inputs are transmitted to the remote server120where they are executed for the remote application370.

With regard to the exemplary aspect, the remote server120further includes a hotkey execution module350that is configured to receive the user's input activation of a hotkey button presented on the customized GUI on display260of the client device110during an active session of the software application. Upon receiving this activation instruction, the remote server120is configured to perform the requested key stroke combination (i.e., the input operation) of the particular remote application370(as described above), which is also received by the client device110in real-time during the execution of the software application by the remote server120. By doing so, the remote server120can provide higher security and hardware independence for the client devices110, effectively minimizing CPU processing requirements and resource consumption.

As further described above, the remote server120is configured to continuously and dynamically build the database310of hotkey buttons. In an exemplary aspect, the information (i.e., the keystroke combinations and frequency) can be collected from a plurality of users. For example, this information can be received and analyzed by key combination analysis module330from a plurality of associated client device, such as, devices in different companies, different countries, or within RAS Infrastructure of organization for single company, for example. Moreover, the received information can be analyzed key combination analysis module330for the frequency of use considering types of applications and type of client devices (other parameters can be taken into account) to determine the optimal hotkey buttons for client devices in a particular group, for example. Thus, as described above, each new user will be provided with the customized hotkey buttons that are optimal for the specific application and for the specific device. The information about the current relevant buttons (and/or the buttons themselves) can be sent to client device110with a certain frequency and suggested use, for example.

FIGS. 4A and 4Billustrates a flowchart for a method400for providing a customized graphical user interface based on user inputs according to an exemplary aspect. As shown, initially at step405, keystroke combinations and other user inputs (e.g., touch screen swipes, etc.) can be logged by input collection modules230for one or a plurality of mobile devices (e.g., client devices110A-110D). The related data for these applications can then be identified, which includes identifying the specific application and device parameters (e.g., device model, operating system, touchscreen dimensions, input functions, etc.) at step410. The collected information can then be analyzed at step415to identify specific input combinations that are and/or may be preferred keystroke combinations to be used for customized hotkeys. In the exemplary aspect, this initial analysis can be performed by input analysis module240of client device110. In another aspect, the analysis is performed by key combination analysis module330of remote server120. In yet another aspect, the input analysis module240of client device110performs a first pass to filter out meaningless inputs and then key combination analysis module330of remote server120performs the more sophisticated analysis of input combinations to determine frequency of most used/important keystroke and input combinations.

In either case, once the input analysis module240of client device110performs its analysis of the collected data, it is transmitted to server120on a periodic basis, for example, at step420. The remote server120is then configured to generate one or more hotkey buttons at step425using the algorithms described above. In one exemplary aspect, the key combination analysis module330performs this analysis taking into account collected data from a plurality of devices using this information that can be associated with each other by some specific criteria, such as country, company, device type, or the like. Finally, as further shown inFIG. 4A, the generate hotkey buttons can be stored as pre-set hotkey combinations in database310, for example, at step430.

FIG. 4Billustrates the providing and execution environment of the hotkey buttons according to an exemplary aspect. As shown, at step435, the remote server120will detect an application (e.g., a remote gaming application) running on a client device110, for example. In an alternative aspect, the remote server120may detect an active session on the client device110of the software application (e.g., the remote application) on the remote server120.

In either case, the remote server120will then confirm at step440whether the application is known to the server, i.e., whether it has pre-set hotkey combination buttons. If not, the method proceeds to step445causing the client/server application220of the client device to begin logging input combinations and keystrokes as described above. Otherwise, if the application is known to the server, the remote server120will further obtain the device parameters at step450. These parameters can include the type of device, operating system, screen dimensions, and any other applicable information that can be used to determine the optimal hotkey combination buttons to customize the display of the client device.

As further shown, at step455, the remote server120will identify and transmit the hotkey buttons to the client device. For example, based on the application and device type, the remote server120can access hotkey database310to identify the one or more optimal buttons to be sent to the client device. These buttons are then displayed on the client device110by hotkey display module250during the running of the application. During operation, the client/server application220will monitor and check for input/activation of the customized hotkey buttons on the display260of the client device110and transmit indication of the activation to remote serve120. Finally, at step465, the remote server120will perform the operations based on the activated button for the remote application running on the client device110according to the exemplary aspect. As noted above, the exemplary method minimizes resource and computing functions by the client device. Moreover, the customized GUI provided on each client device110enables each respective user to have a dynamic and easy experience for the particular application (e.g., remote gaming application) with customized hotkey buttons provide to minimize keystrokes and easily activate commonly used keystroke combinations required for the application.

As described above, the exemplary system and method for providing customized GUI with hotkey buttons can be implemented for any type of software applications, such as games, word processing applications, browsers, and the like. Thus, keystroke combinations can be monitored while these applications are active and being used by the user to identify, display and/or generate new hotkey buttons.

In general, it should be appreciated that the vast majority of applications already provide some type of hotkey combinations that include at least one modifier key (e.g., Shift, Alt/Option, Ctrl, Win/Cmd and the like) and/or functional keys (F1, F2, and the like). As described above, the client/server application220(and the related modules) is configured to receive the user input events from a native, third-party, or its own custom keyboard, and to transform these user inputs to key press/release events that can be sent to a remote session. Thus, in the exemplary aspect, when the particular client application is active, it has full visibility on keystrokes received from the user.

Moreover, it is also known that the modifier and functional keys either have specific, well-known, documented key codes (e.g., if the user utilizes a hardware keyboard attached to the device) or those keys are implemented as a custom keys by the client application itself, since native on-screen iOS/Android keyboards, for example, do not have those keys (except for “Shift”) so their press/release events can be detected by the client software application easily and efficiently. Therefore, in the exemplary aspect, as keystrokes are processed, the input collection module230, for example, detects when those “special” keys (e.g., modifier or functional keys) are pressed and which other keys are pressed/released until the “special” keys are released. When all “special” keys are released, the input analysis module240can deduce the shortcut from the sequence of key press/release events recorded (i.e., the keystroke combination) as further described above. For example, if the user first presses “Ctrl” and then pressed and released “C”, and then releases “Ctrl”, the shortcut “Ctrl+C” will be detected by the input analysis module240. In more complex cases, shortcuts can be comprised from multiple “special” keys, such as “Option+Shift+Command+V” in macOS, for example, which helps to paste a text matching the style of the current document. As further described above, these keystroke combinations can then be compared with hotkey buttons in the hotkey database270and/or further transmitted to the remote server for further statistical analysis and hotkey generation as described in detail above.

Moreover, the exemplary aspects described above consider a single on-screen button that can be displayed on the client device110for the “active” application, which enables the user to simply touch (or input by a mouse, for example) the single button to execute the otherwise complicated keystroke combination. Thus, as an example, if the user is working on a client device with macOS and Microsoft Word is the active application, the user may often use the text selection (e.g., “Shift+Command+Up Arrow”), while in a Browser application, the user may frequently use the hotkey to switch windows in the current program (“Shift+Command+(˜)”). Thus, according to this example, when the user opens the Word document, the button “Text selection” (as a hotkey) can be accessed from the hotkey database and provided to him as part of the customizable display. However, when the user switches to Browser, the button “Text selection” will disappear and the new button “Switch windows” (also accessed from the hotkey database) will be presented to the user in a dynamic manner, for example. Thus, it should be appreciated that when the input analysis module240is analyzing the keystroke combinations obtained by the input collection module230and referencing them with the hotkey database270, the input analysis module240will only be accessing the specific hotkeys in the hotkey database270for the “active” application in an exemplary aspect.

Moreover, although the exemplary aspect described above contemplates a single on-screen button (or a plurality of individual buttons displayed next to each other on the display260) that is configured to execute a single keystroke combination as a hotkey, the hotkey presented on the user interface of the client device110can also be an expandable menu to present a plurality of related hotkey functions in an alternative aspect.

In particular,FIG. 5illustrates an configurable hotkey button displayed on a user interface of a client device according to an additional aspect. As described above, the system and method described herein is provided to identify the “active” application running on the client device110. According to this example, it is contemplated that the active or working application is a word processing application, such as Microsoft Word, for example. However, it should be appreciated that such configurable user interface can be provided for any type of user application. The exemplary aspect provides a method for presenting alternative configurable options for the software application (e.g., menu, accelerators, context menu, and the like).

Upon detecting the “active” application, the input analysis module can access the hotkey database270, for example, and display a single (but expandable) on-screen button, which ultimately provides an adjustable menu of hotkeys. For example, as shown inFIG. 5, the first view505of the user interface (e.g., display260) displays a first button512that can be displayed to simply provide the user with the option to access “short cut actions”. According to an exemplary aspect, the menu505can be contextual (i.e., to be a called button) or to appear depending on the context of the active application. In addition, the dashed box518A is provided to illustrate a user's first selection to view the first tier (or ring) of customized actions for the active application. That is, upon selection of the button on the customized interface for “short cut actions”, the button will be automatically expanded in the display260to provide a plurality of additional actions associated with the active application.

In that event, as further shown, a second view510is displayed that includes a plurality of different types of actions/elements selected for the active application. In other words, the customized user interface provides a circular representation of the set of actions that provides the user with fast access to a large number of elements of action/sub-actions with a minimum amount of manipulation by the user. According to the example, actions514A to514D are shown and correspond to “copy”, “paste”, “cut” and “view”, which are each possible first actions that can be used in the word processing application as would be readily appreciated to one skilled in the art. In addition, the customizable menu/view510is shown according to the example as a half circle menu. In this regard, additional view controls516A and516B are provided that enable the user to scroll through other possible actions/elements (in addition to actions514A to514D) in either a clockwise or counterclockwise manner. Of course in a refinement of this aspect, the menu with the first of actions/elements can be provided as a full circle with a complete listing of available actions. Such decisions regarding the specific display configurations and dimensions can be tailored based on the screen size and device type of the client device110and/or user preferences, for example.

Furthermore, it should be appreciated that the hotkeys presented in this first tier can be selected and presented using the techniques described above. In other words, the remote server120, for example, can build the customizable interface to include actions514A to514D based on the statistics accumulated from a plurality of mobile devices using the application as further described above. Thus, the four actions514A to514D can be selected and displayed as the four most frequently used actions for the active application.

As further shown, the view or menu510presented on the user interface that is composed of the elements (i.e., the actions514A to514D) can be expanded into additional sub-items according to another aspect. Thus, the first menu/view510provides a single ring-level of selected actions. When the user selects one of these items, a secondary ring can be opened/expanded and displayed as an additional view, such as menu520. Thus, in this example, the user has selected the action for “View”514D (the user's selection is denoted by the dashed box518B), which expands the menu to view520. Again, the view520is shown as a half circle in order to minimize the size of the customized menu, assuming the application is working on a Tablet or Smartphone, for example. Thus, this menu can also include additional view controls524A and524B enabling the user to scroll the available options for actions in either a clockwise or counterclockwise direction.

As further shown in the view520, a plurality of sub-actions relating to the selected “View”514D can be displayed. For example, these views can include “Split”522A, “Expand”522B, “Collapse”522C, as well as additional actions “Act4”522D, “Act5”522E, and “Act6”522F. Similarly as described above, the presented actions/elements522A to522F can be selected based on the accumulated statistics to provide the most frequently used hotkeys corresponding to the most frequently used keystroke combinations for the active software application. Thus, the selection of these actions/elements can be determined using the principles described above, for example. Based on these actions, the user can select one of the particular actions (e.g., “Split”522A) which executes the keystroke combination to split the view of the document, which would otherwise require the user to enter a complicated keystroke combination (especially in the context of the application being presented on a device with only a touchscreen such as a Tablet or Smartphone). Thus, the customizable and expandable menu of hotkeys shown inFIG. 5provides an alternative and additional technique that enables the user to easily and efficiently work with a software application on a device having limited screen size using the principles described herein.

FIG. 6illustrates a block diagram of an example of a general-purpose computer system on which the disclosed system and method can be implemented according to an example aspect. As shown, a general purpose computing device is provided in the form of a computer system20or the like including a processing unit21, a system memory22, and a system bus23that couples various system components including the system memory to the processing unit21. It should be appreciated that computer system20can correspond to client device110and/or remote server120described above, processing unit21can correspond to the CPU210, system memory22can correspond to hotkey database310according to various exemplary aspect.

Moreover, the system bus23may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes read-only memory (ROM)24and random access memory (RAM)25. A basic input/output system26(BIOS), containing the basic routines that help transfer information between elements within the computer104, such as during start-up, is stored in ROM24.

The computer20may further include the hard disk drive27for reading from and writing to a hard disk, a magnetic disk drive28for reading from or writing to a removable magnetic disk29, and an optical disk drive30for reading from or writing to a removable optical disk31such as a CD-ROM, DVD-ROM or other optical media. The hard disk drive27, magnetic disk drive28, and optical disk drive30are connected to the system bus23by a hard disk drive interface32, a magnetic disk drive interface33, and an optical drive interface34, respectively. The drives and their associated computer-readable media provide non-volatile storage of computer readable instructions, data structures, program modules and other data for the computer20.

A number of program modules may be stored on the hard disk, magnetic disk29, optical disk31, ROM24or RAM25, including an operating system35. The computer20includes a file system36associated with or included within the operating system35, one or more application programs37, other program modules38and program data39. A user may enter commands and information into the computer20through input devices such as a keyboard40(which can correspond to display260) and pointing device42. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner or the like.

These and other input devices are often connected to the processing unit21through a serial port interface46that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, game port or universal serial bus (USB). A monitor47or other type of display device is also connected to the system bus23via an interface, such as a video adapter48. In addition to the monitor47, personal computers typically include other peripheral output devices (not shown), such as speakers and printers.

The computer20may operate in a networked environment using logical connections to one or more remote computers49. The remote computer (or computers)49may be another computer, a server (e.g., servers24A to24N), a router, a network PC, a peer device, physical equipment30and/or other common network node, and typically includes many or all of the elements described above relative to the computer20. The logical connections include a network interface51and connected to a local area network (i.e., LAN)51, for example, and/or a wide area network (not shown). Such networking environments are commonplace in offices, enterprise-wide computer networks, Intranets and the Internet. It should be appreciated in one aspect that when computer20corresponds to client device110, remote computer49can correspond to remote server120and the network can correspond to network130.

In the interest of clarity, not all of the routine features of the aspects are disclosed herein. It will be appreciated that in the development of any actual implementation of the present disclosure, numerous implementation-specific decisions must be made in order to achieve the developer's specific goals, and that these specific goals will vary for different implementations and different developers. It will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skill in the art having the benefit of this disclosure.