Reward-based recommendations of actions using machine-learning on telemetry data

System and methods for automatically providing action recommendations are described. A method may include collecting a set of telemetry data from a client application. The set of telemetry data contains a plurality of pages generated based on a plurality of user actions performed on the client application. The method may include generating a plurality of prior probabilities corresponding to the plurality of pages and the plurality of user actions. In response to the client application displaying a first page, the method may generating a plurality of posterior probabilities for a subset of user actions that can be invoked in the client application from the first page, and selecting a plurality of recommended actions from the subset of user actions for having the highest corresponding posterior probabilities among the plurality of posterior probabilities.

RELATED APPLICATIONS

Benefit is claimed under 35 U.S.C. 119(a)-(d) to Foreign Application Serial No. 201741032628 filed in India entitled “REWARD-BASED RECOMMENDATIONS OF ACTIONS USING MACHINE-LEARNING ON TELEMETRY DATA”, on Sep. 14, 2017, by VMware, Inc., which is herein incorporated in its entirety by reference for all purposes

BACKGROUND

In a cloud computing environment, applications and systems are becoming more and more complex, as many of the functions in these applications and systems are delivered by cloud-based services such as database-as-a-service, search-as-a-service, etc. Users of a seemingly simple web-based application may easily be overwhelmed by the application's large amount of functions and services. Thus, users are more inclined towards an application which can provide recommendations in selecting any of its functions and services. However, many applications are designed with a preconfigured set of recommendations, without considering the users' preferences or historical usage patterns. Further, even when some applications may be designed to record usage data generated by users of the application, these usage data may become too large for effective processing.

DETAILED DESCRIPTION

FIG.1illustrates a block diagram of a system configured to provide reward-based recommendations of actions using machine-learning on telemetry data in a cloud environment, according to one or more embodiments of the present disclosure. InFIG.1, a cloud environment may include one or more clouds140. The cloud environment may be managed by an action recommendation manager160. Users111and113may utilize the client applications130to access the various functions and services provided by the cloud140. The action recommendation manager160may be configured to collect telemetry data135from the users111and113's during their previous accessing of the client applications130, and to predict and provide to the users111and113subsequent recommended actions137that can be performed on the client applications130based on the collected telemetry data and the starting page135of the client applications130the users111and113are accessing.

In some embodiments, a “cloud”140in the cloud environment may be a network-based computing architecture that provides shared pools of cloud resources on demand. A “virtual machine cloud” (or VM cloud) in the cloud environment may be a cloud implemented using virtualized computing resources. The cloud140may contain, among other components, one or more virtual machines (VMs)141and/or physical machines145. Further, each cloud may include a cloud manager143configured for implementing the various cloud functionalities such as resource pooling, resource allocating, high-availability, and automation etc. In some embodiments, the cloud140may be constructed using products such as VMWARE® vCloud, and the cloud manager143may be implemented using a VMWARE vRealize Suite. For example, the cloud140may be configured to implement VMWARE VRealize Automation (“VRA”), configured to perform VMWARE VRealize Operations (“VROPS”), or configured with a VMWARE VSPHERE server. Alternatively, the cloud140may be implemented using any commercial cloud products, such as OpenStack® Cloud, and/or AMAZON® S3 Cloud.

In some embodiments, the cloud manager143may include a VM manager (not shown inFIG.1) to create one or more VMs141based on one or more physical machines145. The physical machine145may be a physical computer system having a “physical hardware platform” (e.g., an x86 architecture platform). The physical machine145may include a “hypervisor”, which is configured to construct a “virtual hardware platform” for the VM141based on the physical machine145's physical hardware platform. In other words, a “virtual machine” (VM)141may be an abstraction of an actual physical machine145. The VM manager may coordinate the multiple hypervisors associated with the VMs141together to form a distributed (virtual) system (e.g., the cloud140). Thus, the cloud140may be constructed using multiple physical machines145, and using multiple VMs141that are created based on some of the physical machines145.

In some embodiments, the physical hardware platform of the physical machines145may include various “physical hardware components” such as, without limitation, one or more physical Central Processing Units (CPUs), physical memory, physical storage (e.g., hard drive), physical Network Interface Card (NIC), and/or additional electronic circuit components (all of which are not shown inFIG.1). The VM manager may configure the virtual hardware platform of the VM141with one or more “virtual hardware components” such as, without limitation, one or more virtual CPUs, virtual memory, virtual storage, virtual NIC, and/or additional virtual components. With help from the VM manager, the virtual hardware components may emulate the behaviors and the computing capabilities of the corresponding physical hardware components, thereby allowing the VM141to function as if it were a physical machine145.

In some embodiments, the cloud140(as well as the VMs141and the physical machines145contained therein) may be configured to provide various computational services to the client applications130. The client applications130, which may include various software-based clients131and web-based clients133, may be any graphical or non-graphical user-interfacing applications (or modules/components) designed to interact with the users111and113. The software-based clients131may be any conventional software interfaces designed to display information and accept user inputs from the users111and113. Examples of software-based clients131may include the client partitions of software applications such as, without limitations, word processor, spreadsheet, accounting application, database, media player, graphic editor, game, or photo editor. The software-based clients131may also include the client partitions of mission-critical enterprise management and production engineering applications, such as enterprise resource planning (ERP), document management, business workflow management, hardware engineering and software engineering applications. The web-based clients133may be provided via web browsers such as, without limitations, Internet Explorer®, Firefox®, and Chrome®.

In some embodiments, the client applications130may present to the users111and113with page-by-page of display and information. Each “page” may refer to a specific arrangement of information and graphic elements displayed on a specific client application130. Further, the users111and113may interact with the client applications130via various user actions121and123. A “user action” may refer to a user-interface operation available on a particular page of a specific client application130, and may be activated by the users111and113. Examples of user actions121and123may include, without limitations, keyboard inputs, mouse inputs (e.g., single mouse click, double click), and touch-screen inputs.

In some embodiments, a single page of a client application130may include one or more user-action options, each of which can be activated by users111and113. For example, when a user111accessing a specific word-processor client application130(e.g., MICROSOFT® WORDS), the word-processor may display to the user111a first page which may be a document-editing page. The document-editing page may provide user-action options such as “Save”, “Edit”, “Insert”, etc. The user111may select and issue a specific user action111(e.g., Save) based on these user-action options. Alternatively, a client application130may be a web browser including user-action options such as a text-input field for inputting web addresses and multiple menu items. A user113may input a user action123which is a keyboard input of a web address into the web browser, or input a user action123which is a mouse clicking on a menu item or a button in the web browser.

In some embodiments, different pages of a single client application130may be associated with each other by various user actions121and123. In other words, after a user111activates a specific user action121on a first page of the client application130, the client application130may process the user action121and display a second page to the user111in response to such user action121. The “first” page may be referred to as “original” or “starting” page, and the “second” page may be referred to as a “resulting” or “next” page. Thus, the second page may be related to the first page via the specific user action121. Further, a single client application130may generate different resulting pages from a single starting page based on different user actions121and123, and may generate a same resulting page from different starting pages based on different user actions121and123. For example, an email client application130displaying a starting page showing an email's detail may generate an email-editing resulting page when a user113issues a “reply” user action123, or generate an email-removed resulting page when the user113issues a “delete” user action123. Similarly, this email client application130may generate the same inbox resulting page when the user113issues a corresponding “return-to-inbox” user action123from the email-editing starting page or the email-deleting starting page. Also, a starting page and a resulting page from a user action123may be the same page.

In some embodiments, the “telemetry data” may refer to past or historical usage patterns of users111and113utilizing the client applications130. Specifically, telemetry data may include a set of user actions121perform by a user111on a particular client application130, and include a sequence of pages (each page may be a starting page of a subsequent page in the sequence, and/or a resulting page of a previous page in the sequence) generated by the particular client application130in response to the set of user actions121by the user111. Such telemetry data may be used for predicting subsequent/future user actions121(from any particular starting page of the client application130) that may be selected by the user111. The telemetry data may also include historical user actions121and123performed by multiple users111and113on multiple client applications130, as well as the starting and resulting pages associated with these user actions121and123.

In some embodiments, an action recommendation manager160may be configured to collect telemetry data135generated from users111and113accessing the client applications130, analysis the telemetry data135, and predict/generate recommended actions137to the client applications130. The client applications130may present the recommended actions137to the users111and113by displaying e.g., a “recommendation pop-up window” in the client application130, in order to help the users111and113in selecting their likely follow-up user actions121and123. The users111and113may select one of the recommended actions137to issue the next user action121, without having to go through the normal path of locating and invoking the user action121from multiple user-action options.

In some embodiments, the action recommendation manager160may be configured with, among other components, a telemetry-data-collector161, a recommendation-module163, and a machine-learning module165. The telemetry-data-collector161may be configured to collect past usage patterns of all the users111and113utilizing the client applications130, and store the collected usage patterns as telemetry data135in a database147in the cloud140. The recommendation-module163may study/analyze the telemetry data in the database147by using a classifier (e.g., a Naive Bayes Classifier). Depending on a specific starting page136of a particular client application130currently being displayed to a user111, the recommendation-module163may generate a set of recommended actions137as the predicted most feasible set of user actions the specific user111may subsequently select from and perform. Further, based on the feedback received from the client application130with respect to the actual user action121performed by the specific user111(regardless of whether the user111selected from the recommended actions137or not), the machine-learning module165may perform reinforcement learning to reward or penalize the predicted result as per user's feedback.

For example, a performance-intensive client application130may have a lot of features that are used by different users111and113depending on their roles and permissions. The users111and113may not be able to keep track of all the actions they performed on the client applications130. Therefore, the telemetry-data-collector161may be configured to record the telemetry data including all the user actions121and123performed by the users111and113on the client application130. The recommendation-module163may then use these telemetry data to predict the next courses of action as recommended actions137to the users111and113, and may present the recommended actions137to the users111and113to simplify user selection. In other words, the future course of action can be inferred from the past usage history of the users, or it can also be deduced from other users having similar pattern of usages.

In some embodiments, the action recommendation manager160may be implemented as a service in the cloud140. For example, the action recommendation manager160may be a VMWARE vRealize Operations Manager (“vROps”) with predictive analysis and smart alerts capabilities. The action recommendation manager160may not only ensure optimal performance and availability of the applications, services, and infrastructure in the cloud140, but also provide monitoring capability across different client applications130and record telemetry data135generated by the users111and113. Further, the action recommendation manager160may utilize parallelism and in-memory processing to handle exponential growth of telemetry data135.

FIG.2illustrates a process for collecting telemetry data and generating recommended actions, in accordance with one or more embodiments of the present disclosure. InFIG.2, an action recommendation manager (similar to the action recommendation manager160ofFIG.1) may be configured to perform a process including the following operations: pre-processing210, the probability computing & recommendation220, and the machine-learning230. Specifically, a telemetry-data-collector of the action recommendation manager may perform the pre-processing210to collect telemetry data201(similar to the telemetry data135ofFIG.1) from client applications in a cloud environment, and store the telemetry data201in a database205(similar to the database147ofFIG.1). A recommendation-module of the action recommendation manager may perform the probability-computing & recommendation220to generate a set of recommended actions241for a user based on a starting page202of a specific client application the user is accessing. Afterward, the action recommendation manager may transmit the set of recommended actions241to the client application in response to the starting page202. Based on a subsequent user action209performed on the client application in view of the set of recommended actions241, a machine-learning module of the action recommendation manager may perform the machine-learning230to generate a machine-learning matrix235, and store the machine-learning matrix235in the database205. The machine-learning module may further refine and optimize the machine-learning matrix235based on the performed user action209as feedback.

In some embodiments, the telemetry-data-collector may perform the following pre-processing operations: including without limitation, gathering-telemetry-data211, grouping-telemetry-data213, and computing-page-probabilities215. Specifically, in the gathering-telemetry-data operation211, the telemetry-data-collector may collect telemetry data generated by one or more users accessing one or more client applications. For example, the telemetry-data-collector may use GOOGLE Chrome®′ HTTP Trace plugin to gather user actions performed in the cloud environment. Optionally, the telemetry-data-collector may filter the user actions, and retain those telemetry data that are related to page accessing and page updating. The collected telemetry data may be stored in the database205in a form as shown in Table 1 below.

In Table 1, column “User” may refer to the IDs of users accessing the client applications; column “Timestamp” may refer to the time stamp of a specific user action; and column “Telemetry ID” may refer to the identification of the pages displayed to the users or accessed by the users. In other words, a specific telemetry ID may be an identifier of a starting page. A specific telemetry ID may also be associated with a resulting page generated based on a user action performed on a specific starting page. For example, “Action-Environment-Overview” may be an identifier of a starting page showing an environment overview window; and “Object-Summary” may be a resulting page after a user issued a user action “Get Summary” on a starting page.

In some embodiments, in the grouping-telemetry-data operation213, the telemetry-data-collector may sort the telemetry data based on individual users, and generate a set of tables each of which is associated with one of the users. In other words, the telemetry-data-collector may divide the telemetry data for all users into non-overlapping groups based on users. The Table 2 below may show the divided telemetry data associated with user “User-1”:

In some embodiments, in the computing-page-frequencies operation215, the telemetry-data-collector may count the occurrences of each unique telemetry ID for all the users in the telemetry data, and store such occurrences information as page frequencies203. Afterward, the telemetry-data-collector may store the page frequencies203in the database205. Examples of the page frequencies203may be shown in Table 3 below. In Table 3, the page with telemetry ID “Admin-Solutions” may have a frequency count of 11, meaning this page may be accessed 11 times by a specific user in the telemetry data.

In some embodiments, based on the various information gathered and summarized by the telemetry-data-collector, the recommendation-module may perform the following “probability-computing and recommendation”220operations: including without limitation, computing-prior-probabilities operation221, computing-posterior-probabilities operation223, and generating-recommendations operation225. When a user interacts with a client application, he may move from a starting page to a resulting page by performing a specific user action. From any particular starting page, there are a limited number of user actions the user can invoke. For example, if the client application is on an “Administration” starting page, the user may perform a user action to go to “Policies” page or “Licensing or Alerts” page, but he may not have access to the “Alert Definitions” page. Thus, the recommendation-module may define all the possible user actions which can be performed from a starting page, and count the corresponding frequencies of these user actions accordingly.

In some embodiments, the recommendation-module may perform the computing-prior-probabilities operation221by generating a transaction-frequency table for each user. Specifically, the recommendation-module may take into account all the user actions a specific user may have permissions to perform in a client application, and generate a frequency count for each user action transition, as shown in Table 4 below.

TABLE 4Starting PageResulting PageFrequencyTelemetry IDTelemetry IDCountAdmin-SolutionsAdmin-Environment-4OverviewDashboardAdmin-Solutions3Admin-Environment-Object-Summary3OverviewContent-DashboardsAdmin-Solutions2DashboardAlerts-All2
In Table 4's example, based on the recommendation-module's analysis of the telemetry data, the specific user performed 4 times of a user action to reach “Admin-Environment-Overview” resulting page from the starting page “Admin-Solutions”, and performed 3 times of another user action to reach “Admin-Solutions” resulting page from the starting page of “Dashboard”. Further, the specific user may perform different user actions to reach two different resulting pages (“Admin-Solutions” and “Alert-All”) from the same starting page of “Dashboard”, or reach the same resulting page (“Admin-Solutions”) from different starting pages (“Dashboard” and “Content-Dashboards”).

In some embodiments, once the recommendation-module obtained all the frequency tables as shown above, the recommendation-module may compute the probabilities of the possible user actions from each starting page using a Naive Bayes Classifier. “Naive Bayes Classifier” assumes that the presence of a particular feature in a class is unrelated to the presence of any other feature. For example, a fruit may be considered to be an apple if it is red, round, and about 3 inches in diameter. Even if these features depend on each other or upon the existence of the other features, all of these properties independently contribute to the probability that this fruit is an apple, and that is the reason such a probability classification may be referred to as ‘Naive’.

In some embodiments, the recommendation-module may calculate a prior probability table for all the pages and user actions associated with a particular client application, as shown in Table 5 below. In Table 5's example, based on a number of times (11 times) the user accessed the page “Admin-Solutions” and a total number of pages accessing by the user (e.g., 52 times) in the telemetry data, the recommendation-module may calculate that this page may have a prior-probability value of 11/52=20.8%.

In some embodiments, the recommendation-module may perform the computing-posterior-probabilities operation223to calculate posterior probabilities for all the user actions that are allowed to be performed from a starting page. Specifically, the recommendation-module may utilize a Bayes theorem formula to calculate the posterior probability P, as shown below:

P⁡(c|X)=P⁡(X|c)⁢P⁡(c)P⁡(X)
where P(c) is the prior probability of class c, P(X) is the prior probability of predictor X, P(X|c) is the prior probability of predictor Xgiven class c, and P(c|X) is the posterior probability of class (c) given predictor (X). In other words, P(X) and P(c) may be deemed the prior probabilities of a specific starting page and a specific resulting page, P(X|c) may be deemed the prior probability of a corresponding user action associated with the specific starting page and the specific resulting page, and P(c|X) may be deemed the “posterior probability” of the corresponding user action associated with the specific starting page and the specific resulting page.

For example, assuming a user may have the options to perform a first user action U from a starting page A to a resulting page B, and a second user action V from the starting page A to a resulting page C. The recommendation-module may first calculate the posterior probability of the user action U by first retrieving from Table 5 above the prior probabilities of the starting page A and the resulting page B (i.e., P(X) and P(c)). Afterward, the recommendation-module may calculate a prior probability (i.e., P(X|c)) for the user action U by determining the probability of the resulting page B being generated from the starting page A using the Table 4 and Table 5 above. Based on these three prior probabilities, the recommendation-module may calculate the posterior probability of the user action U based on the above Bayes theorem formula. Likewise, the recommendation-module may use the Bayes theorem formula above to calculate the posterior probability of the user action V, based on the prior probabilities of the pages A and C, as well as the prior probability of the user action V.

In some embodiments, the recommendation-module may perform the generating-recommendations225based on the above formula. Specifically, the recommendation-module may receive from the client application a starting page202of a client application currently being accessed by a user. The recommendation-module may calculate the posterior probabilities of all the possible user actions that can be invoked from the starting page202. Assuming the user is accessing a starting page “Admin-Solutions”, the recommendation-module may calculate the posterior probability of the user actions that can lead to resulting pages “Admin-Environment Overview”, “Dashboard”, and “Admin-policies”. Afterward, the recommendation-module may select from all the possible user actions one or more user actions with the highest posterior probabilities as the recommended actions241. Alternatively, the recommendation-module may select the user actions having the top-n (e.g., top-5) highest posterior probabilities as the most likely user actions (i.e., the recommended actions241) the user may undertake from the starting page202.

In some embodiments, after the recommended actions241were provided to the client application, the user of the client application may perform a subsequent user action which may or may not be one of the recommended actions241. Regardless, the client application may transmit the user's performed user action209as a feedback to the action recommendation manager. The machine-learning module of the action recommendation manager may perform subsequent machine-learning operation230which includes, without limitations, a constructing-learning-matrix operation231and a rewarding-or-penalizing operation233.

In some embodiments, the machine-learning module may first perform the constructing-learning-matrix operation231to generate a machine-learning matrix235based on telemetry data retrieved from the database205. Specifically, the “machine-learning matrix”235may contain a set of starting pages and a set of resulting pages generated by a particular client application. The machine-learning matrix235may also include user actions each of which associates one of the starting pages with one of the resulting pages contained therein. Further, the machine-learning matrix235may assign a corresponding preference value to each of the user actions (or each of the associations between the starting pages and the user actions). Each “preference value” may be a numeric value for determining a user's preference to the corresponding user action. The machine-learning module may utilize the Table 2 and Table 4 above to populate the starting/resulting pages and user actions in the machine-learning matrix235, assign an initial value (e.g., zero) to all the preference values in the machine-learning matrix235, and store the generated machine-learning matrix235in the database205.

In some embodiments, the machine-learning module may utilize the posterior probabilities calculated above as the preference values for the user actions in the machine-learning matrix235. Specifically, for each starting page in the machine-learning matrix235, the machine-learning module may identify all the user actions and resulting pages associated with this starting page. The machine-learning module may then utilize the recommendation-module to calculate the prior probabilities of the starting page and the resulting pages, as well as the prior probabilities of these user actions. Afterward, the machine-learning module may generate a corresponding posterior probability for each of the user actions using the Bayes theorem formula shown above, and assign the corresponding posterior probability as the preference value for each of the user actions (or associations between the starting pages and the user actions) in the machine-learning matrix235.

In some embodiments, based on the machine-learning matrix235stored in the database205, the recommendation-module may provide a set of recommended actions241without having to dynamically calculate the posterior possibilities of the user actions. In other words, the recommendation-module may retrieve the starting page202from the machine-learning matrix235, and directly retrieve a set of user actions that are associated with this starting page202. Afterward, the recommendation-module may evaluate the preference values associated with the set of user actions, and select a number of the user actions having the top-n highest preference values as the recommended actions241. Such an approach allows quick generating of the recommendation actions241for the client application.

In some embodiments, the machine-learning module may use a reinforcement learning approach to determine whether to reward or penalize the recommended actions241provided by the recommendation-module. In other words, if the performed user action209indicates that the user selected one of the recommended actions241, the machine-learning module may reward the above algorithm and the machine-learning matrix235from which the recommended actions241are generated. If the performed user action209shows that the user didn't choose from the recommended actions241and performed a different user action instead, the machine-learning module may penalize the algorithm and adjust the machine-learning matrix235, in order to subsequently generate better recommended actions241.

In some embodiments, the reinforcement learning approach adapted by the machine-learning module may maximize the notion of cumulative reward by using a Q-learning algorithm. Q-Learning algorithm may utilize an action-value function that gives the expected utility of taking a given action in a given state. In other words, the Q-Learning algorithm can determine how good an action is given a certain state based on a maximize function Q(s, a) as defined:
Q(s, a)=immediate reward for making an action+best utility (Q) for the resulting state.
Formally, the function Q(s, a) may be defined as:
Q(s,a)=r(s,a)+γ maxa′(Q(s′, a′)
where r(s, a) denotes immediate reward; γ denotes to a relative value (between 0 to 1) of delayed vs. immediate rewards; a denotes an action; a′ denotes a new action; s denotes to a state; and s′ denotes to a new state after action a.

In some embodiments, the machine-learning module may assign γ with a value 0.5. And Q(s, a) may be deemed the machine-learning matrix235. When a user performs a user action209that is chosen from the recommended actions241, the preference value for this user action209in the machine-learning matrix235may be rewarded a fixed value (e.g., 10 points), while the other choices in the recommended actions241are penalized with a fixed value (e.g., 0 point or negative points) in the machine-learning matrix235. The machine-learning matrix235Q(s, a) may be adjusted and updated accordingly during the subsequent iterations of action recommendations, till the machine-learning matrix235Q(s, a) converges towards optimal solution, meaning additional rounds of reinforcement learning would not significant change the machine-learning matrix235.

In some embodiments, once the machine-learning matrix235converges, the recommendation-module may generate the recommended actions241directly from the machine-learning matrix235Q(s, a). Alternatively, after a certain period (e.g., one month), the machine-learning module may discard the machine-learning matrix235in the database205, and generate a new machine-learning matrix235based on telemetry data201, as described above. In some embodiments, the telemetry data and the machine-learning matrix235may contain different users in the cloud environment having similar roles. In this way, the action recommendation manager may study the usage patterns of these different users, and can deduce the user actions based on collaboration from these users.

FIG.3shows a flow diagram illustrating a process for automatically providing action recommendations in a cloud environment, according to one or more embodiments of the present disclosure. The processes301may set forth various functional blocks or actions that may be described as processing steps, functional operations, events, and/or acts, which may be performed by hardware, software, and/or firmware. Those skilled in the art in light of the present disclosure will recognize that numerous alternatives to the functional blocks shown inFIG.3may be practiced in various implementations.

At block310, an action recommendation manager operating in the cloud environment may be configured to collect a set of telemetry data from a client application operating in the cloud environment. The set of telemetry data may contain a plurality of pages generated based on a plurality of user actions performed on the client application. In some embodiments, for a subset of pages selected from the plurality of pages and a subset of user actions selected from the plurality of user actions, the client application's performing of each of the subset of user actions from the first page of the client application may generate a corresponding one of the subset of pages. In other words, the first page may be referred to as a starting page, and each of the subset of pages may be referred to as a resulting page generated after the client application performed a specific user action from the first page.

At block320, the action recommendation manager may generate a plurality of prior probabilities corresponding to the plurality of pages and the plurality of user actions. Specifically, the action recommendation manager may generate prior probabilities for the first page and its resulting pages (e.g., the subset of pages) based on the telemetry data, and generate prior probabilities for the subset of user actions selected from the plurality of user actions based on the prior probabilities of the first page and its associated resulting pages.

At block330, in response to the client application displaying the first page which is selected from the plurality of pages, the action recommendation manager may generate a plurality of posterior probabilities for the subset of user actions that can be invoked in the client application from the first page. Specifically, the action recommendation manager may generate a corresponding posterior probability for each of the subset of user actions based on the corresponding prior probabilities of the first page, the subset of pages, and the subset of user actions.

At block340, the action recommendation manager may select a plurality of recommended actions from the subset of user actions for having the highest corresponding posterior probabilities among the plurality of posterior probabilities. In some embodiments, the plurality of recommended actions may have the top-n highest corresponding posterior probabilities among the subset of user actions. Specifically, the action recommendation manager may sort from the highest(largest) to the lowest(smallest) the subset of user actions based on their corresponding posterior probabilities, and select the first n (e.g., 5) number of user actions from the sorted subset of user actions as the plurality of recommended actions for having the top-n highest corresponding posterior probabilities.

At block350, the action recommendation manager may transmit the plurality of recommended actions to the client application, and receive a performed user action from the client application in response to the transmitting of the plurality of recommended actions.

At block360, the action recommendation manager may generate a machine-learning matrix based on the performed user action and the plurality of recommended actions. Specifically, the machine-learning matrix contains associations among the plurality of pages and the plurality of user actions, and contains preference values corresponding to the associations.

FIG.4shows a flow diagram illustrating a process for automatically providing action recommendations based on a machine-learning matrix, according to one or more embodiments of the present disclosure. The processes401may set forth various functional blocks or actions that may be described as processing steps, functional operations, events, and/or acts, which may be performed by hardware, software, and/or firmware. Those skilled in the art in light of the present disclosure will recognize that numerous alternatives to the functional blocks shown inFIG.4may be practiced in various implementations.

At block410, an action recommendation manager operating in the cloud environment may collect a set of telemetry data from a client application operating in the cloud environment. The set of telemetry data contains a plurality of pages generated based on a plurality of user actions performed on the client application.

At block420, the action recommendation manager may generate a machine-learning matrix based on the set of telemetry data. Specifically, the machine-learning matrix contains associations among the plurality of pages and the plurality of user actions, and contains preference values corresponding to the associations. Further, the action recommendation manager may generate a plurality of prior probabilities corresponding to the plurality of pages and the plurality of user actions, generate a plurality of posterior probabilities for the plurality of user actions based on the plurality of prior probabilities, and assign the preference values for the associations among the plurality of pages and the plurality of user actions using the plurality of posterior probabilities.

At block430, in response to the client application displaying a starting page selected from the plurality of pages, the action recommendation manager may generate a plurality of recommended actions based on the machine-learning matrix. The plurality of recommended actions may have the highest preference values (or the top-n highest preference values) based on the starting page.

At block440, the action recommendation manager may receive a performed user action from the client application in response to a transmitting of the plurality of recommended actions to the client application.

At block450, the action recommendation manager may adjust the machine-learning matrix by evaluating the performed user action against the plurality of recommended actions.

At block460, when the performed user action is one of the plurality of recommended actions, the action recommendation manager may increase (reward) a corresponding preference value of an association between the first page and the performed user action in the machine-learning matrix. In some embodiments, if the machine-learning matrix converges, the action recommendation manager may stop adjusting the machine-learning matrix.

When the performed user action is not one of the plurality of recommended actions, at block470, the action recommendation manager may decrease (penalize) a corresponding preference value of an association between the first page and the performed user action in the machine-learning matrix. After awarding or penalizing, the action recommendation manager may repeat the operation in the above block430, in order to generate additional recommended actions based on the adjusted machine-learning matrix.

Thus, systems and methods for automatically providing action recommendations in a cloud environment have been disclosed. The various embodiments described herein may employ various computer-implemented operations involving data stored in computer systems. For example, these operations may require physical manipulation of physical quantities usually, though not necessarily, these quantities may take the form of electrical or magnetic signals where they, or representations of them, are capable of being stored, transferred, combined, compared, or otherwise manipulated. Further, such manipulations are often referred to in terms, such as producing, identifying, determining, or comparing. Any operations described herein that form part of one or more embodiments of the disclosure may be useful machine operations.

In addition, one or more embodiments of the disclosure also relate to a device or an apparatus for performing these operations. The apparatus may be specially constructed for specific required purposes, or it may be a general purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations. The various embodiments described herein may be practiced with other computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like.

In addition, while described virtualization methods have generally assumed that virtual machines present interfaces consistent with a particular hardware system, persons of ordinary skill in the art will recognize that the methods described may be used in conjunction with virtualizations that do not correspond directly to any particular hardware system. Virtualization systems in accordance with the various embodiments, implemented as hosted embodiments, non-hosted embodiments, or as embodiments that tend to blur distinctions between the two, are all envisioned. Furthermore, various virtualization operations may be wholly or partially implemented in hardware. For example, a hardware implementation may employ a look-up table for modification of storage access requests to secure non-disk data.