DETERMINING TRUE UTILIZATION

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining true utilization based on productive behaviors of operators in a digital environment. One of the methods includes automatically determining productivity for each of multiple users of an organization. Timing data for a user is received that indicates available work time for the user. Interaction data is received for the user for interactions by the user during the available work time for the user. Productivity rules for an organization of the user are received that define productivity of interactions by users of the organization. Productivity of interactions of the user is determined based on the interaction data and the productivity rules. True utilization for the user is determined based on the productivity of the interactions and the timing data. Action data is generated based on the determined true utilization action is taken based on the action data.

BACKGROUND

Technical Field

This specification relates to determining true utilization based on productive behaviors of operators in a digital environment.

Background

Computer users often multitask, doing multiple things at once. A computer user may be a customer support agent who may split their time across multiple cases, such as to provide support to customers phoning in or using a chat box to obtain support. The customer support representative may spend different portions of time helping each of the customers. The customer support representative may also perform other actions on the computer that are not related to any of the customers' cases.

SUMMARY

This specification describes technologies for determining true utilization based on productive behaviors of operators in a digital environment. True utilization can be defined as productive time divided by available operator time. These technologies include determining true utilization by using a configurable method of defining which types of interactions or sequences of interactions are productive and identifying operator interactions that are either productive or nonproductive based on configured or learned rules. All actions performed by operators can be evaluated to determine productive yield across operator time and activity, on a per-service basis.

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of: automatically determining productivity for each of multiple users of an organization, wherein determining productivity for a respective user comprises: receiving timing data for the user indicating available work time for the user; receiving interaction data for the user for interactions occurring in multiple software services used by the user during the available work time for the user; receiving productivity rules for an organization of the user that include conditions defining productivity of interactions by users of the organization; determining productivity of interactions of the user based on the interaction data and the productivity rules; determining true utilization for the user based on the productivity of the interactions and the timing data; and generating action data based on the determined true utilization; and taking action based on the action data.

Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. True utilization can be determined and tracked using a low-overhead, always-on solution that captures a complete picture of operator work and objectively and consistently categorizes that work into multiple buckets (e.g., productive, nonproductive, or other categories). True utilization tracking can capture all information about productive actions and otherwise untracked time and activity to generate a complete picture of employee productivity and utilization. Productivity can be determined across systems, applications, and services. Productivity can be determined based on sequences of interactions that are determined to be productive. Different customers can configure different rules for determining productivity. Each respective customer can configure productivity rules based on what is considered productive work for the customer. Productivity and true utilization can be determined at least in part by a machine learning system. Employers can save resources by not needing to manually shadow or audit employee work days or timesheets. The system can be used to calculate or ensure accurate work hours. Employers can use calculated true utilization rather than aggregating work-related data from multiple systems and data sources to generate an estimate of employee working time. The solution can be applied, for example, to as low as 10′s of operators handling hundreds of events over a specified period (e.g., minutes or hours) and more typically at least hundreds of operators handling at least thousands of events over a specified period of time (e.g., minutes or hours). The solution can provide objective assessment of work according to a set of rules, in contrast to manual efforts that involve human users who may provide subjective assessments, especially at scale or when suffering from distraction, exhaustion, etc. The solution, by constant, allows for fair and consistent assessment of work. Productivity rules can be managed and modified, and changed productivity rules can be automatically applied to rapidly re-assess productivity retroactively or for future interactions for thousands of representatives.

DETAILED DESCRIPTION

Previous systems for tracking operator time have historically not had a method for accurately and automatically converting employee actions into a measure of productivity without manual oversight. Previous solutions include manual shadowing and tabulation of employee activities (e.g., time-in-motion studies). Time-in-motion studies are manual processes that can only be applied while a human observer is available to observe, so samples may be limited and additional resources (e.g., the human observer) are required.

Other previous solutions involve manual time tracking (e.g., employees self-reporting on timesheets), or naive, imprecise, and only partial time tracking performed using software with tracking restricted to the user's time within only that software's own system. Some prior solutions only observe time spent within the software's direct system or predefined partner systems, so organizations lack visibility into a full operator work day and corresponding actions. Some previous solutions only track time, not actions, and not more refined tracking such as productive actions. Additionally, previous software systems do not categorize work into productive or other buckets, rather naively assuming all time spent within their system is to be counted, or relying on the operator to manually indicate their status (which is highly inaccurate, inconsistent, and subjective).

To solve the above and other problems, a true utilization engine can be used that can analyze each operator or representative action and compare the action against historical data, learned models, and any prescribed action categorization by the operator's employer, to determine whether the action is productive. For example, sending replies to customer tickets within a case management system may get categorized as productive but typing personal emails may get categorized as unproductive. As another example, entering bug reports within an internal system may get categorized as productive. Reading blogs on unsanctioned websites may get categorized as unproductive while reading articles within a knowledge base can be categorized as productive.

True utilization metrics can be calculated to determine what percentage of a day a representative is performing productive work, such as actively working on a case and engaged productively with approved or recommended tools in order to solve the case. True utilization can be used in any digital work environment where human operators perform tasks. True utilization can be calculated across each system, application or service an operator uses, rather than within a single system or selected applications. True utilization can be used to aid understanding of whether operator actions are productive relative to the operator's broader tasks at hand such as assigned cases.

Further details and advantages of the true utilization approach are described below. For example,FIGS. 1-15provide details regarding a workforce analytics manager for recording and managing interactions.FIGS. 16-19provide additional details regarding determining true utilization based on productive behaviors of operators in a digital environment.

The following detailed description describes techniques for discretizing time spent by users (e.g., customer service agents) doing specific tasks on computers. These technologies generally involve associating identifiers (IDs) from different systems while users spend time handling a case spanning multiple pages and applications of the different systems. Various modifications, alterations, and permutations of the disclosed implementations can be made and will be readily apparent to those of ordinary skill in the art, and the general principles defined may be applied to other implementations and applications, without departing from the scope of the disclosure. In some instances, details unnecessary to obtain an understanding of the described subject matter may be omitted so as to not obscure one or more described implementations with unnecessary detail and inasmuch as such details are within the skill of one of ordinary skill in the art. The present disclosure is not intended to be limited to the described or illustrated implementations, but to be accorded the widest scope consistent with the described principles and features.

The techniques of the present disclosure can be used to assign each user action to a single “case” that a customer service agent is working on when the customer service agent is working simultaneously on more than one case. For example, the customer service agent can be a customer representative agent that handles Customer Relationship Management (CRM) cases that arrive at a CRM system by phone call, chat session, or online portal.

In some implementations, discretizing time can include setting identifier threshold rules, so as to define finer-grain criteria used to identify events that count as being associated with a case. Rules can also be used to define and access a set of identifiers used in set of systems that are to be tracked. Techniques of the present disclosure can be used to disregard time spent associated with identifiers that are not included in a tracked subset of systems to be tracked. Moreover, techniques of the present disclosure can be used to disregard identifiers corresponding to events that last less than a threshold event duration. Doing so can provide the benefit of avoiding an interruption of a current count of work being discretized.

Identifiers from multiple systems can be linked by observing an expected behavioral pattern of a user, such as a customer support agent. As an example, the system can determine that customer support agents generally follow a certain workflow on a given case. The identifiers used in the different systems that are accessed during the workflow can be linked together even if their linkage was previously unknown. For example, a chat application (or app) for chatting with customers may have a chat ID which is used as the case ID of the case. In a new chat, the customer support agent may use their own internal CRM system where they look up the customer. The internal CRM system may have a completely different set of identifiers, different from the chat app. If it is known that the customer support agent is always going to look up the customer in a certain amount of time after getting a new customer chat request, then the identifiers can be automatically associated or linked.

In some implementations, input context intervals (ICIs) can be used to improve the tracking of events in a more efficient way. An ICI is defined as a time interval having beginning and ending timestamps corresponding to a user action having a context (e.g., associated with a specific case). For example, events can be tracked by recording keystrokes. If the customer support agent is working on multiple cases at the same time, techniques of the present disclosure can be used to determine which case gets precedence. If a customer support agent is switching between systems, as noted above techniques of the present disclosure can link two systems that have their own case IDs but that are linked by the workflow. In order to be more efficient in linking cases and tracking time spent by customer support agents on each case, techniques of the present disclosure can be used to only allow one case to interrupt a second case if the duration of the interrupting event is above a threshold time. The threshold time can be a variable by specific situation and the system(s) that are involved. In computer systems that implement the techniques of the present disclosure, computer-implemented methods can be implemented for determining the primary task on which an agent is working when it appears that the agent is working on multiple simultaneous tasks. The computer-implemented methods can use configurable rules.

A browser in which the chat app executes can use an application programming interface (API) to send a data stream to a back end system for interpretation. APIs can be programmed to notice events that occur inside a browser or outside a browser. For example, a browser (e.g., Chrome) plugin can be implemented such that whenever an agent switches windows within a browser and visits a new page, the system records the event (e.g., the event data is sent to the backend system). A similar API can exist in Windows, for example, when an agent switches to a different window, sending event data to a server/backend. For example, the event data can indicate that the agent spent a specified amount of time on website V, or the agent spent a specified amount of time in application window X with page title Y.

In some implementations, ICIs can be implemented through the use of recording timestamps instead of just recording a time duration. In this way, the timestamps can additionally be used to correct durations corresponding to the start and end times spent on a webpage by a customer support agent. As an example, the timestamps can be fitted to key strokes that occur when a customer support agent is on a particular web page.

FIG. 1Ashows an example of a workforce analytics system100that can be used to determine discrete time spent by customer service agents on different tasks across different systems and produce reports based on discretized time, according to some implementations of the present disclosure. The workforce analytics system100includes a workforce analytics manager102that interfaces with one or more customer relationship systems104. Each customer relationship system104includes one or more customer relationship applications106, such as CRM systems. Users (such as CRM agents) can use the customer relationship system104, for example, by accessing webpages108and using desktop applications110.

While an agent is using the customer relationship systems104, a data stream112is sent to the workforce analytics manager102for interpretation. The data stream112can include discretized time data captured by browsers using APIs to send the data stream to a back end for analysis. The workforce analytics manager102can store the received data stream112as analytics data116. The workforce analytics manager102can use the analytics data116to generate reports. The report can include, for example, reports containing information described with reference toFIGS. 3-11. Techniques by which the data stream112captures data include parameters and set-up operations described with reference toFIGS. 12-14C. Components of the workforce analytics system100are connected using a network114that includes, for example, combinations of the Internet, one or more wide area networks (WANs), and one or more local area networks (LANs).

Examples of reports that can be produced using discretized time data can include focus events. Focus events can be used, for example, to assign each action performed by an agent to a single “case.” An action that is assigned to a case can be disambiguated from actions performed on other cases. Discretizing the time and assigning events to specific cases can be based on cross-platform tagging for each active session. Automatic matching can occur, for example, when an agent opens a specific document within a specific period of time after opening a case. The automatic matching can use agent behavior pattern recognition that incorporates logic for timeouts, accesses to specific pages and documents, and automatic linking of identifiers from disparate systems.

The workforce analytics system100can perform tracking in the context of multiple workflows and multiple customers. For example, a customer service agent may have a workflow to provide a customer refund that requires the customer service agent to access a number of different systems. Based on a list or pattern of the different systems necessary for a particular type of task, workforce analytics system100can insure that the customer service agent follows a proper procedure while collecting metadata from each system that the customer service agent accesses and linking the metadata together.

A customer service agent may be handling multiple, simultaneously customer service cases (for example, chats) at once. Even though the time is overlapping for each of the associated customers, the workforce analytics system100can determine how much of their time is actually spent on each customer. The time that is tracked includes not only how much time the customer service agent is chatting with that customer, but how much time the customer service agent is spending working on that customer versus working on actions associated with another customer. The workforce analytics system100can use clustering algorithms and other techniques to identify that an agent is working on the same case across different systems. The clustering can occur, for example, using text copied from one box into another and based on patterns of access of different systems when handling a case.

FIG. 1Bshows an example of the workforce analytics manager (WAM)102ofFIG. 1A, according to some implementations of the present disclosure. The WAM102includes a WAM front end152that provides a user interface for a user to request reports154, for example, using analytics data156. The analytics data156can include data described with reference toFIGS. 2-9. Report requests154can be made by a user through a web user interface (UI). Example reports can include viewing true utilization and viewing true handle time. Using the UI, the user can apply filters, including user filters and date filters (e.g., date range = last week). The analytics data156, including user actions and event data, can serve as data input to a query engine158accessible through the UI for accessing relevant data for requested insights. Calculated insights160can be used to display report insights162. For example, for a report providing true utilization (including user efficiency and time spent on cases), the insights can be used to create a ratio of hours active on cases and hours ready for work. Displayed reports can be displayed, for example, as table results, bar graphs, pie charts, and flow charts. Example reports are described with reference toFIGS. 2-10B.

FIG. 2is a screenshot200of a customer screen202for handling cases in the workforce analytics system100, according to some implementations of the present disclosure. The customer screen202can be an interface used by a user (for example, a customer service agent). The customer screen202can be one of many screens available and used in the user's browser or on the user's desktop to handle cases, including another page204that may present a user interface for specific products or services. An originating call, such as a chat, may originate on the customer screen202used by an agent. The agent may immediately or subsequently navigate to other resources, such as other pages204, to look up the customer or perform some other action related to the case.

Working areas206in customer screens202and other pages204can include several pages208a-208d(or specific screens), accessible through browsers, for example, each with corresponding identifiers210a-210d.Other resources accessed by the customer service agent can include documents such as word documents and spreadsheets for presenting and recording information associated with a case. The identifiers210a-210dmay be completely different across the systems associated with the pages208a-208d.However, the workforce analytics system100can use the analytics data116to associate an identifier with work done on various uncoordinated systems, which in turn can link together time spent on those different systems for the same case. The various uncoordinated systems can provide multiple software services such as web pages, documents, spreadsheets, workflows, desktop applications, and conversations on communication devices. The multiple software services include at least a software service of a first type and a software service of a second type, where the software service of the first type and the software service of the second type are uncoordinated software services lacking inter-service communication and a common identification labelling system.

In some implementations, the following steps can be used for assigning an event to a case. First, the system determines a location of a case ID or other identifier. For example, the identifier may only be seen on webpages matching specific Uniform Resource Locator (URL) patterns or using specific desktop apps. Such identifiers can be extracted from the URL, from a page/app title, or from a specific region in the HTML hierarchy of the webpage.

Each website or desktop app where an ID can be extracted is known as a service. By associating observed identifiers together with multiple services, events from multiple services can be associated together under a single case ID. The case ID can originate from whichever service the system determines to be the primary service.

To associate a first identifier with a second identifier, a sequence of events can be defined that represents the observation of identifiers in a particular order, within a bounded time-frame. The system can use this defined sequence of events to link events and their respective identifiers. Such a defined sequence can be a sequence of pages, for example, that are always, or nearly always, visited, in order and in a time pattern, when a new case is originated and handled by a customer service agent. Whenever a linked identifier is determined, that event and any subsequent events are associated with the case as identified by the identifier from the primary service.

In a working example, consider a customer service agent that engages in multiple simultaneous chats and uses a separate CRM service to look up customers and make changes to their accounts. Since the customer service agent switches between the chat windows and the CRM service, there is a need to know, specifically, how much time is spent on each customer and case. The following sequence of events can be defined.

First, the customer service agent receives a new chat box, for example, entitled “Chat123” on a website that is considered as the primary service. The new Chat ID123is created, and the Case ID is marked with the Chat ID. Second, within a threshold time period (e.g., 60 seconds), the customer service agent searches the CRM system for the customer.

Third, within another 60 seconds, the customer service agent lands on the customer's page within the CRM that matches the URL pattern (for example, crm.site.com/customers/234). The CRM ID234is recognized, and the ID234is linked with Case ID123.

Fourth, the customer service agent responds to another customer and enters a chat box, for example, with Chat ID567. This action and subsequent actions in this chat box are not associated events with Chat123, but instead are associated with Chat567.

Fifth, the customer service agent goes back to the CRM system on page crm.site.com/customers/234. This surfaces CRM234which is linked with Chat123, associating that event and subsequent events with case123until the next time case123is interrupted.

Note that, if the customer service agent performs other events at the same time as the sequence of events described above, such additional events do not affect the system's ability to recognize system operation. This is because certain implementations do not require that the set of events is exclusively limited to the chat and CRM events noted above.

In some implementations, the functionality of the techniques of the present disclosure can be represented in pseudocode. Assume that event stream is a variable that represents a time-ordered list of the following types of events: 1) webpage visits with URLs and page titles, 2) desktop application window events with page titles, and 3) clicks, events, and interactions within a web page on a particular webpage element or region that has its own descriptors. A case ID can be defined as any identifier associated with a service that is the primary tool used for customer communications. In such a case, pseudocode describing operation of the workforce analytics manager ofFIG. 1Acan include:identifier_mappings =<Mapping of Case ID to list of linked identifiers>all_possible_sequences =<List of event sequences already in process for each rules pertaining to possible sequences of events>current_case_id = None

At a high level, the pseudocode links events (e.g., customer service agent actions) to corresponding cases and captures event information (e.g., clicks, customer service agent inputs) for the events, e.g., by stepping through a sequence of events that have occurred. Once the system has analyzed agent events and assigned those events to various cases, the system can provide a variety of useful functions. For example,FIG. 3is a screen shot of an example of a search analytics page300for looking up customer service agent events, according to some implementations of the present disclosure. The search analytics page300includes search controls that facilitate searching for particular types of customer service agent data, for example, for actions and events by one or more specific customer service agents. The filters can be used to select customer service agent events by combinations of customer service agent name, case type, and case ID. Sorting controls can allow a user of the search analytics page300to sort the output of filtered information in different ways.

The search analytics page300displays data stream information that can be collected to identify how customer service agents are spending their time on particular cases. The information that is displayed can include case type (for example, printer fires) or specific application (for example, ZENDESK).

FIG. 4is a screen shot of an example of a dashboard400for customer service agent time spent on cases, according to some implementations of the present disclosure. A cases information area402can list different cases, each case's case type (e.g., “My printer is on fire”), and other information for each case.

A video playback area404can allow the user of the dashboard400to open a video corresponding to focus events for a particular case. The case session video playback area404can include a video status bar, a case sessions bar, and a page visits bar. Each bar is displayed relative to time, for example, from opening a case until handling of the case is complete.

A video status bar in the dashboard400can allow the user to display a video of what has occurred on overlapping cases. For example, playing the video in high speed can show the overlapping case sessions on which a customer service agent has worked. The video can show, for example, that the customer service agent was working on case X, then looking at a different case, then working on case X again.

FIG. 5is a screen shot of an example of a focus events structure500, according to some implementations of the present disclosure. The focus events structure500can be used to capture and store information about page events. This can include information such as every single time the customer service agent switches pages or looks at a new resource, what case is associated, and which case session? The information can include multiple case sessions, the working customer service agent, how much time was spent, page refreshes, key presses, paste actions, and mouse scrolls.

FIGS. 6A-6Bare screen shots collectively showing an example of a true utilization page600, according to some implementations of the present disclosure. The information includes heartbeats indicating, for every 60 seconds, how the CPU is performing, whether the customer service agent was active, page load events, page load times, open tabs, and slow times.

FIGS. 7A-7Bare screen shots collectively showing an example of a true handle time page700, according to some implementations of the present disclosure.

The system uses a Document Object Model (DOM) to monitor clicks, scrolls, and actual IDs of objects accessed, down to the class names. The DOM is a cross-platform and language-independent interface that treats an XML or HTML document as a tree structure, where each node is an object representing a part of the document. The DOM represents a document with a logical tree. Each branch of the tree ends in a node, and each node contains objects. DOM methods allow programmatic access to the tree. Nodes can have event handlers attached to them. Once an event is triggered, the event handlers are executed. The DOM information provides tracking of clicks, and the workflow analytics system can attach the tracked clicks and active page events to a corresponding case. This connection of clicks and active page events to a specified case can be used to understand, for each customer service agent, how active they are, and what opportunities exist for improving true handle times for a particular customer service agent.

FIGS. 8A-8Bare screen shots collectively showing an example of an outliers page800, according to some implementations of the present disclosure. The outliers can identify the cases that are taking the longest.

FIG. 9is a screen shot of an example of a customer service agent summary page900, according to some implementations of the present disclosure. The customer service agent summary page900includes a customer service agent summary area902that provides customer service agent profile information and productivity statistics for a customer service agent. A daily usage area904includes graphs showing customer service agent activity statistics (e.g., in hours) over time, identifying customer service agent shift hours, observed hours, and active hours.

FIG. 10Ais a screen shot of an example of a process flow diagram1000, according to some implementations of the present disclosure. The diagram shows the most likely path to close a ticket based on active page events. Blocks in the diagram are arranged over time and indicate a progression of systems used by the customer service agent to work on the ticket. The blocks can be annotated with one or more of descriptive labels, shading, and color, for example, to indicate that the ticket started in CRM, moved to an administration (admin) tool, and then back to CRM.

FIG. 10Bis a screen shot of an example of a process timeline1050, according to some implementations of the present disclosure. The timeline can show the resources that were used and which ones were used a higher percentage of the time. This can identify where customer service agents are getting stuck and what is likely their next step.

FIG. 11is a screen shot of an example of a resource analysis page1100, according to some implementations of the present disclosure.

FIG. 12is a screen shot of an example of a trigger event definition page1200, according to some implementations of the present disclosure. Trigger events effectively provide the ability to parse pages visited by a customer service agent while working on a given case. A trigger event can be used to parse the URL, for example, when a page matches XYZ, to extract the case ID out of it. This information can be used to identify how a new case is started. For example, a trigger condition can be set when the URL of a page (e.g., viewed by the customer service agent) matches a particular URL pattern, or when the title of the page matches a particular title pattern. Conditions defined on the trigger event definition page1200can be defined using Boolean operators for matches of URLS, titles, and HTML elements found on the page.

As an example, a trigger event can be defined for triggering a new case (or being associated with a current case) when a customer service agent navigates to a web page such as page208a,having a specific URL. The page208acan correspond to the first block inFIG. 10A, for example. Using defined trigger events a sequence of events can be tracked that occur in multiple software services being accessed by a customer service agent. The sequence of events can include one or more events from each case of a group of cases handled by the customer service agent. Using information extracted from one or more interactions of the customer service agent with at least one service, focus events can be determined that identify which case in the group of cases is being worked on by the customer service agent at various points in time, with each focus event having a focus event duration. Each focus event can be assigned to a particular case using the extracted information. A total period of time spent by the customer service agent on the particular case can be determined based on a sum of focus events durations for the focus events assigned to the particular case.

In an example, when a trigger (e.g., a page view) occurs, additional controls that are available from the trigger event definition page1200can be used to define certain responses that are to happen (or be triggered, in addition to logging the event). The responses can include, for example, creating an activity (e.g., marking this moment, or timestamp, in time), sending an email, sending a workbook, providing a Chrome notification, or redacting video. Marking the moment can cause the moment to be labeled on the timeline of the video playback area404, for example.

FIG. 13is a screen shot of an example of a clickstream page1300, according to some implementations of the present disclosure. The clickstream page1300can be used to identify specific data that is to be monitored and collected. Monitoring performed by the clickstream page1300can be filtered or limited based on opt-in and opt-out preferences set for specific customer service agents. Settings in the clickstream page1300can be used to define specific pages (e.g., by URL) that are to be used in monitoring and to specify a data retention time (e.g., a number of days) for monitored information. Clickstream deletion time fields can be used to control when existing clickstream data is to be deleted.

FIG. 14Ais a screen shot of an example of a case defining services per organization page1400, according to some implementations of the present disclosure. The page1400allows a user (such as an Administrator of agent monitoring) to identify which services apply to an organization's cases. The definitions identify the names of services and time thresholds (e.g., start, timeout, and break times) that are used to link customer service agent actions to a case. For example, for the case currently defined on the page1400, fields of the page define a Gmail app1402and a Front app1404, with time thresholds1406, as being the apps used for cases in Organization ABC1408. The start time indicates a minimum time that a case is in view before the case is considered being worked on by the customer support agent. Doing this avoids assigning time to a case when the customer support agent tabs through information for the case for only a moment.

FIG. 14Bis a screen shot showing an example of a DOM tools page1440, according to some implementations of the present disclosure. For a currently-selected DOM monitor option1442, a URLs list1444identifies the current list of URLs that DOM settings apply to for the organization1408. A rules section1446facilitates the definition of rules associated with DOM monitoring, including write a rule, for example, that defines a path of an element that starts or ends a specific case.

FIG. 14Cis a screen shot showing an example of an add DOM fingerprint page1460, according to some implementations of the present disclosure. The DOM fingerprint page provides the ability to define a fingerprint that effectively captures the path of an element effectively starting and/or ending a specific case. The fingerprint can apply, for example, to a key press, entry in a field, starting a recording, or some other action.

FIG. 15is a flowchart of an example of a method1500l for determining time spent by the customer service agent on the particular case, according to some implementations of the present disclosure. For example, the system200can be used to perform the method1500. For clarity of presentation, the description that follows generally describes method1500in the context of the other figures in this description. However, it will be understood that method1500can be performed, for example, by any suitable system, environment, software, and hardware, or a combination of systems, environments, software, and hardware, as appropriate. In some implementations, various steps of method1500can be run in parallel, in combination, in loops, or in any order.

At1502, a sequence of events occurring in multiple software services being accessed by a user (e.g., a customer service agent) is tracked. The multiple software services can include web pages, documents, spreadsheets, workflows, desktop applications, and conversations on communication devices. As an example, the multiple software services can include web pages used by the user within a CRM system, and the user can be a customer service representative. The sequence of events includes one or more events from each case of a group of cases handled by the user. For example, tracking the sequence of events can include the following. In some implementations, the multiple software services can include at least a software service of a first type and a software service of a second type, where the first type is CRM software and the second type is a search engine.

Focus events are recorded that identify page switches by the customer service agent, views of a new resource by the customer service agent, where each focus event identifies the customer service agent, an associated case, an associated session, a time spent on a particular page, whether the particular page was refreshed, keys that were pressed, copy-paste actions that were taken, and mouse scrolls that occurred. Heartbeats are recorded at a threshold heartbeat interval (for example, once every 60 seconds). The heartbeats can indicate CPU performance and whether the customer service agent has been active (and to what degree). Page load events are recorded including identifying a time to process a page load request, a time to finish loading the page, a number of tabs that are open, and whether a page load was slow. DOM events are recorded, including clicks by the customer service agent, scrolling by the customer service agent, an identifier of a software service, a class name and a subclass name of the software service, and content of text typed into the software service.

In some implementations, tracking the sequence of events can include setting identifier threshold rules defining a set of identifiers used in a set of systems that are to be tracked, disregarding identifiers not included in a tracked subset of the multiple software services, recording timestamps for start and end times on a particular software service, and disregarding, using the start and end times, identifiers corresponding to events that last less than a threshold event duration.

In some implementations, tracking the sequence of events can include collecting active page events, page level events, machine heartbeats, DOM events, video, audio, times when the customer service agent is speaking versus not speaking, times when the customer service agent is using video, entries written to documents, desktop application events, and entries extracted from the documents. From1502, method1500proceeds to1504.

At1504, focus events identifying which case in the group of cases is being worked on by the customer service agent at various points in time are determined using information extracted from one or more interactions of the customer service agent with at least one service, where each focus event includes a focus event duration. From1504,method1500proceeds to1506.

At1506, each focus event of the focus events is assigned to a particular case using the extracted information. For example, assigning each focus event of the focus events to a particular case can include linking previously unlinked identifiers from the software services by observing an expected behavioral pattern for using the multiple software services in a particular order pattern to respond to and close the particular case. In some implementations, the expected behavioral pattern can be company-dependent. In some implementations, the expected behavioral pattern can include ICIs including a timeframe defining an amount of time between a start time of the particular case and a next step performed by the customer service agent on the particular case. From1506, method1500proceeds to1508.

At1508, a total period of time spent by the customer service agent on the particular case is determined based on a sum of focus event durations for the focus events assigned to the particular case. As an example, assigning a focus event to the particular case can include using clustering algorithms to identify and cluster a same customer corresponding to the particular case across the multiple software services. After1508, method1500can stop.

FIG. 16is a block diagram illustrating an environment1600that includes a true utilization engine1601. The true utilization engine1601can be a part of the workforce analytics manager101,for example. As described above, support representatives, such as a support representative1602, can provide support for users, such as a user1604. The support representative1602can perform support interactions using a computing device1606and/or other devices(s). The user1604can contact a support system using a device1608, which may be a smartphone as illustrated (e.g., using a voice call and/or browser or application connection) or some other type of device.

The true utilization engine1601(or another system) can retrieve interaction information1610that describes interactions that occur during various support cases for the support representative1602for a particular time period (e.g., day, week, month). The true utilization engine1601can also obtain timing data for the support representative1602that indicates available work time for the support representative1602during the time period.

A productivity rules evaluator1612can evaluate productivity rules1614that have been defined or configured for an organization of the support representative1602(or in some cases learned by a machine learning engine, as described in more detail below). The productivity rules1614can include conditions defining productivity of interactions by users of the organization. The productivity rule evaluator1612can evaluate the productivity rules1614with respect to the interaction information1610for the support representative1602to determining productivity of the interactions taken by the support representative1602during the time period.

A true utilization determination engine1616can determine true utilization for the support representative1602based on the productivity of the interactions and the available work time for the support representative1602during the time period. For example, the true utilization determination engine1616can determine true utilization by determining a sum of productive time and dividing the sum of productive time by the available work time. A true utilization presentation component1618can present true utilization information, such as to a supervisor1620on a supervisor device1622. As described in more detail below, other outputs can be produced and other actions can be taken, by the true utilization engine1601or by other system(s), based on determined true utilization information.

FIG. 17is a diagram of the components of an exemplary system1700for determining true utilization based on productive behaviors of operators in a digital environment. The system1700includes a true utilization engine1702. As described in more detail below, productivity rules used by the true utilization engine1702can be specified or defined using a productivity rule configuration portion1704of an administrative/configuration application1706. For example, an administrator of the true utilization engine1702and/or an administrator for a particular customer may define productivity rules1708that are provided as input(s) to the true utilization engine1702. In some implementations, the administrator of the true utilization engine1702can define default productivity rules1710that can be used (after possibly being customized) by different customers. As another example, a customer administrator and/or the administrator of the true utilization engine1702can define customer-specific productivity rules1712that are specific to and used by specific customers. For example, particular customer-specific productivity rules1712amay be only used by one particular customer.

In general, the productivity rules1708specify condition(s) for which representative interactions or sequences of interactions are productive or non-productive. The productivity rules1702can describe patterns of productive (or non-productive) behavior, for example. Although some rules may define which behaviors are productive or non-productive as a binary classification, other rules can describe a degree of productivity. For example, the customer-specific productivity rules1712ainclude weights1714that can be applied to classify certain behaviors as partially productive. Weights are described in more detail below with respect toFIG. 18.

For each customer, a time/interaction tracker1716can track representative interactions and activity to produce timing data1718and interaction data1720. The time/interaction tracker1716can be or include software on representative computing devices that logs operator time available with the digital workspace of the representative. The timing data1718can include representative time available for work and durations of tracked interactions. The time/interaction tracker1716can track and log representative/operator behavior such as typing, mouse movements, item selections, etc., including selections of objects, to generate the interaction data1720. The interaction data1720can also include information describing sequences of interactions1722.

The timing data1718, the interaction data1720, and the productivity rules1708can be provided to a productivity determination engine1724included in the true utilization engine1702. The productivity determination engine1724can determine (e.g., based on the timing data1718, the interaction data1720, and the productivity rules1708), productive interactions1726, unproductive interactions1728, productive time1730(e.g., out of available time), and unproductive time1732. The productivity determination engine1724can apply the productivity rules1708to the interaction data1720, for example, to match interactions in the interaction data1720to productivity rules1708that define productive and/or nonproductive behavior.

The productivity determination engine1724can determine the productive time1730by determining a net yield of productive time spent by the representative. Although productivity rules are defined, in some cases, instead of or in addition to rules for productivity or non-productivity, other rules can be defined that specify conditions for other classification buckets to which interactions, sequences, or work time can be attributed. In addition or alternatively to determining the productive interactions1726and the unproductive interactions1728using prescribed indicators reflected by the productivity rules1708, the productivity determination engine1724can be included in, used, or otherwise be associated with a learning engine1734that can determine productivity using learned model(s).

The productivity determination engine1724can be configured to perform multiple, parallel assessments of productivity, to increase an efficiency of productivity determination. For example, the productivity determination engine1724can evaluate different productivity rules1708simultaneously, in different processes or threads. Output from multiple parallel processes can be evaluated as a final step, to aggregate outputs of evaluating different productivity rules1708.

The learning engine1734can be trained using ground truth in a variety of ways. For example, an administrator can use a learning engine configuration application1736to provide ground truth training data1738to the learning engine1734. The ground truth training data1738can include known productive sequences of interactions1740(or known productive interactions) and known unproductive sequences of interactions1742(or known unproductive interactions).

One example machine learning approach can include using a random forest approach to assign a probability of productivity/utilization for a given event or sequence. Input features can include whether the event or sequence occurred during scheduled work hours, a time offset from a last seen unit of work (e.g., +2300 milliseconds), a classification of an interacting resource (e.g., “core,” “assistive,” “communication,” “distraction,” etc.) A classification of an interacting resource may be made by other classification model(s) that evaluate and classify all resources the operator interacts with, or may be predetermined by the customer. Other features may include a nature of Interaction (e.g., duration, mouse click, keystroke, mouse move, sequence progress, etc.) or a value of value of interaction (e.g., 10 ms, 3 clicks, 57 keypresses, 100 pixels, 60% of sequence completed, etc.). For training input data, an input parameter can be an indication whether the training data corresponds to productive or unproductive activity. Hyperparameters of the machine learning approach can include a regression type (e.g., MSE (Mean Squared Error)), a sample with replacement setting, an initial depth (e.g., ten), a number of features (e.g., an initial feature count may be four), a number of estimators (e.g., an initial estimator count may be five hundred), a number of samples to leave (e.g., starting with one), and a number of samples to split (e.g., starting with two).

The learning engine configuration application1736can include a user interface that can enable an administrator to inspect specific focus events or event sequences and to define (e.g., as productive sequences1740or unproductive sequences1742), a designation of productivity or utilization for a given event or sequence, for training or re-training the learning engine1734. The learning engine configuration application1736can also be used to designate attributes of focus events and/or event sequences as features for the machine learning model used by the learning engine1734. Features, as mentioned, can include duration, count of keypresses of focus events, etc., and can be used as machine learning features in combination with the event or sequence productivity designation to affect future machine learning.

In a testing phase, the learning engine1734, once trained, can generate candidate categorizations1744of productive/nonproductive interactions and/or sequences which can be provided to a user (e.g., test, administrator) in the learning engine configuration application1736. The user can provide feedback1746on the candidate categorizations1744, with the feedback1746being used to further train the learning engine1734.

Regarding the training data1738, in some cases, some sequences may be known to be productive and other sequences that don't match the known productive sequences may be initially assumed to be non-productive. Similarly, some sequences may be known to be unproductive and other sequences that don't match the known unproductive sequences may be initially considered as productive.

In some cases, the learning engine1734, either during testing or post-testing use, outputs uncategorized interactions1748(e.g., interactions or sequences for which the learning engine1734isn't able to generate a productivity categorization or score with at least a certain level of confidence). The uncategorized interactions1748can be provided to a user in the learning engine configuration application1736, with the user providing feedback1746regarding the uncategorized interactions1748and the learning engine1734being further updated with improved learning model(s) based on the feedback1746.

Once trained and tested, the learning engine1734can identify representative actions that are productive based on learning models. In some cases, the productivity determination engine1724generates a first set of productivity scores or weights for interactions, based on prescribed productivity rules1708and the learning engine1734generates separate productivity scores or weights, using learned models. An overall combined productivity score or weight for an interaction or sequence can be determined based on respective scores from the productivity determination engine1724and the learning engine1734. For example, an average score can be determined, or weighted scores can be factored into the overall score based on a confidence score (e.g., produced by the learning engine1734) or a match score (e.g., produced by the productivity determination engine1724).

Information regarding productive time1730and unproductive time1732can be provided to a true utilization determination engine1750. The true utilization determination engine1750can determine true utilization metrics1752based on the productive time1730and unproductive time1732information. The true utilization metrics1752can represent, for one or more representatives, a net yield of productive time spent by the representative, for example as a fraction of the representative's time available for work. The true utilization determination engine1750can, for example, calculate a true utilization metric1752for a representative by dividing a sum of representative time spent on productive work by a sum of representative time available within a digital workspace. True utilization metrics1752can be calculated for all representatives observed within a particular time range (e.g., day, week, month, year).

The true utilization metrics1752can be provided to an action engine1754and the action engine1754can generate action data and/or perform one or more actions based on the true utilization metrics1752. For example, the true utilization metrics1752, as well as information about productive interactions1726, nonproductive interactions1728, productive time1730, and unproductive time1732, can be provided to a reporting engine1756for inclusion in one or more reports1758generated by the reporting engine1756. The reports1758can be provided to one or more entities, such as for presentation on a supervisor device1760.

In some implementations, action rules1762are configured, for example using an action rule configuration application1764(e.g., by a customer or true utilization engine1702administrator). Action rules1762can specify which action(s) the action engine1754is to perform in response to generation of true utilization metrics1752. Some action rules1762may specify that certain actions are to be performed in response to generation of certain types of true utilization metrics1752. For example, certain reports1758may be generated and sent to certain supervisor devices1760based on configured supervisor-employee relationships. As another example, action rules1762can specify that certain action(s) are to be performed when a true utilization metric1752is above or below a threshold. For example, an action rule1762can specify that if a true utilization metric1752is below a specified threshold that a notification generator1766is to generate one or more notifications and/or warnings1768which can be provided to specified supervisor devices1760and/or to specified representative devices1770.

Other than reporting and notifications/warnings1768, in some implementations, the action engine1754can perform one or more other actions1771based on the true utilization metrics1752(and/or based on other productivity-related information). For example, if utilization drops beneath a particular threshold, an operator could be blocked from performing additional work on a current task, and the task can be automatically routed to a next available operator. Such automatic rerouting can be used to help promote more rapid resolution of work, or more focused (and likely higher quality) resolution of work.

In some cases, the action engine1754generates action data1772which can be provided to one or more other action performers1774. The action data1772can be raw true utilization metrics1752, related productivity information, or can be filtered or aggregated true utilization or productivity information. For example, action data1772can include true utilization and representative information for representatives who have true utilization metrics1752above or below certain thresholds. For example, action data1772can include information for representatives who have true utilization less than 50%. The action data1772can be provided to supervising and/or training entities who can perform one or more training actions1776, such as adjusting training materials or recommending or scheduling a representative for certain training. Low levels of true utilization across a large pool of tasks or operators can be used to self-identify new work streams which may not have been previously known to an organization. For example, when self-similar patterns of work across a large number of interactions are identified as unproductive, the organization can be alerted that there may an opportunity to discover new processes to categorize, evaluate, and formalize within the workflows of the organization. In some cases action data1772can include information about use of certain tools that are known or have been determined to be unproductive. Such action data1772can be provided to one or more supervisor entities who may perform one or more tool change actions1778, such as removing certain tool access or limiting tool use except for authorized uses.

When the action data1772includes true utilization and representative information for representatives who have true utilization metrics1752above or below certain thresholds, various personnel actions1780can be performed, such as promotions, demotions, compensation adjustments, awards, etc. For example, productivity triggers can be used to automate payroll or billable hour accounting or auditing for hourly pay models. For instance, when productivity is detected, billing for operators can start (and potentially only at a net utilization rate), and when productivity stops or goes below a threshold utility rate, corresponding time can be automatically excluded from accounting until an acceptable level of productivity resumes. Accordingly, true utilization tracking can enable precise billing models with clear audit trails.

Other actions and use of true utilization can occur. For example, true utilization information can be used to understand downstream implications of an operator's actions. For instance, if customers of an organization generate lifetime value of $5000 when the operators working on their support requests are at least 80% utilized, but only provide $3000 lifetime value when operators exhibit utilization less than 80%, true utilization information can be used as a leading indicator for customer success, satisfaction, or dissatisfaction, and can trigger preemptive measures to help retain or increase downstream engagement of the customers of the organization.

FIG. 18is a diagram1800that illustrates exemplary productivity rules1802for different customers. A server can maintain a productivity rules repository1802. The server can store, in the productivity rules repository1802, different productivity rules for different customers. For example, the productivity rules repository1802can store productivity rules1804for a customer ABC1806, productivity rules1808for a customer DEF1810, and productivity rules1812for a customer HIJ1814. Although shown as being stored in a same repository, in some implementations, different customer productivity rules are stored in different repositories.

Each productivity rule in the productivity rules1802can specify a condition for how productive time is measured for customer service agents of a given customer. Various types of productivity rules can be defined, with different conditions, to enable each customer to define specific productivity rules that are tailored to desires or needs of the particular customer. The server can retrieve and evaluate customer-specific productivity rules, at runtime, to evaluate, for a given customer, whether customer representative time is considered productive.

For a given customer, different productivity rules can be defined for different services. For example: the productivity rules1804for the customer ABC1806relate to a Service1, a Service2, a Service3, and a Service4; the productivity rules1808for the customer DEF1812relate to Service1, Service2, Service3, Service4, and a Service6; and the productivity rules1812for the customer HIJ1814relate to Service1, Service3, Service4, and a Service5.

The productivity rules repository1802can store different productivity rules for different customers for a same service. For example, the productivity rules1804,1808, and1812for the customer ABC1806, the customer DEF1810, and the customer HIJ1814, respectively, each include productivity rules for Service1but with different conditions. Service1may be a service used by multiple customers, such as email service or a knowledge base service that is available to different customers, but each customer can define what type of use of Service1is considered productive.

As another example, the productivity rules repository1802can store productivity rules for services that are unique to a given customer. For instance, Service6, which is only referenced in the productivity rules1808for the customer DEF1810, may be a service that is only used by the customer DEF1810and not other customers. Only the customer DEF1810may have Service6installed and configured for use, for instance. As another example, Service5is only referenced in the productivity rules1812for the customer HIJ1814. Service5may be a service developed by the customer HIJ1814, for instance (and only used by the customer HIJ1814and not other customers).

In some cases, productivity rules can specify whether general use of certain services are productive or non-productive. For example, productivity rules1816and1818for the customer DEF1810specify that use of Servicel is productive and use of Service4is not productive, for customer service agents of the customer DEF1810, respectively. For more refined productivity rules, for some services and for some (or all) customers, productivity rules can be based on sequences of actions.

For example, productivity rules1820,1822, and1824for the customer ABC1806specify that a SequenceA in Service1is productive, a SequenceB in Service1is not productive, and a SequenceC in Service1is productive, for customer service agents of the customer ABC1806, respectively. As illustrated by the productivity rules1820,1822, and1824, different productivity rules for different sequences can be defined, for a same service, with some productivity rules defining which sequence(s) of interactions with the service are productive and other productivity rules defining which sequence(s) of interactions with the service are not productive. As another example, a single rule can specify which sequences for a service are productive (and/or non-productive).

In some implementations, different customers may define a same productivity rule or similar productivity rules. As another example, a default productivity rule, currently used by multiple customers, may include a same or similar set of condition(s). In some implementations, a same (or substantially similar) sequence can be used in different productivity rules, for different customers, for a same or different service. For instance, both the productivity rule1822for the customer ABC1806and a productivity rule1826for the customer HIJ1814specify that SequenceB in Service1is not productive. As another example, a productivity rule1828for the customer DEF1810specifies that SequenceB in Service2is productive.

In general, different customers may define different or same conditions, for a same sequence and/or a same service. For example, a productivity rule1830for the customer HIJ1814specifies that SequenceA in servicel is not productive when performed by customer service agents of the customer HIJ1814. In contrast, for customer ABC1806, the SequenceA in Service1is construed as productive.

When defining productivity rules, a same (or similar) sequence may be defined as productive in one service but unproductive in another service. For instance, a customer may instruct and prefer that customer service agents use a particular (e.g., preferred or standard) service rather than an alternative service. For example, a productivity rule1832for the customer HIJ1814specifies that SequenceA is productive in Service3, in contrast to SequenceA being identified, in the productivity rule1830, as unproductive, for customer service agents of the customer HIJ1814. As another example, a productivity rule could be added to the productivity rules1812that specifies that all use of the Servicel is unproductive for customer service agents of the customer HIJ1814.

Different types of rule conditions, different complexities of rule conditions, and various types of syntax or approaches for defining rule conditions can be used. For example, a productivity rule1836for the customer ABC1806specifies that interaction sequences for Service3that include “InteractionX” are productive. A productivity rule1838specifies that interaction sequences for Service4that include “InteractionY” are not productive. A productivity rule1840specifies that interaction sequences for Service2that do not include “InteractionZ” are productive.

As another example, a productivity rule1842for the customer DEF1810specifies that sequences of interactions in Service2other than SequenceB are productive. As yet another example, a productivity rule1844for the customer HIJ1814specifies that only SequenceD, SequenceE, or SequenceF in Service3are productive (with other interactions in Service3being non-productive). A productivity rule1846specifies that only SequenceG and SequenceH are non-productive in Service4(with other sequences in Service4being productive).

Instead of or in addition to defining productivity rules based on whether a service or a sequence is productive, a productivity rule can specify a productivity weight that indicates a level of productivity (e.g., as a percentage) of a sequence or use of a service. For example, a productivity rule1850for the customer ABC1806specifies that Sequences B and C in Service3are 100% productive and other interactions with Service3are 50% productive. As another example, a productivity rule1852for the customer DEF1810specifies that, for Service3,a sequence SeqA is 100% productive, a sequence SeqB is 80% productive, a sequence SeqC is 20% productive, and other uses of Service3are 10% productive. A productivity rule1854for the customer DEF1810specifies that, for Service6, sequences that include “InteractionX” (e.g., a particular interaction) are 100% productive while other use of Service6is 50% productive.

In some implementations, whether (or how much) service representative interactions are considered productive is based on a degree of match of service representative interactions to a defined sequence. For example, a productivity rule1856for the customer HIJ1814specifies that service representative interactions with Service4that match a particular sequence with greater than an 80% match are considered productive while other interactions with Service4are considered nonproductive. A match or degree of match can be determined in different ways. For example, for some sequences, a match may be determined to occur if service representative interactions match a same number of steps in a same order as that defined by a productivity rule. In other cases, a certain threshold number of steps are to match (e.g., four out of five) for an 80% match to occur, for example, regardless of which four of the five steps match. In other cases, an amount of time spent performing a certain step is to occur for the interaction to be considered as a match to the productivity rule.

In some cases, a productivity level can be reduced by accounting for “counterproductive” actions. For instance, a subset of representative interactions may match those included in a defined sequence at a 100% match, but other representative interactions may be observed that are considered counterproductive (or a degree or weight of counterproductive). Accordingly, a 100% match or 100% productivity level may be reduced to account for the counterproductive actions. Counterproductive actions may be other sequences or events, or other aggregated data. For example, the system could detect that a representative deleted seventy characters out of one hundred initially-typed characters.

As another example, a productivity rule1858for the customer HIJ1814specifies that a productivity weight for a sequence of interactions with Service5is equal to a degree of match of the interactions to a defined sequence. For instance, if a sequence of interactions matches the defined sequence with a degree of match of 70%, the sequence of interactions can be considered 70% productive.

In some cases, a sequence specified in a productivity rule as productive can mean that a time duration of the sequence is considered to be productive (with other use of the service considered non-productive). A sequence can include a start interaction and an end interaction and possibly other interactions, and productive time can include time spent from the start action to the end interaction. In some cases, a rule can specify that if one or more particular sequences occur during use of a service, the entire use of the service is productive (e.g., including time spent on interactions with the service that may occur before or after the particular sequence).

FIG. 19is a screen shot showing an example of a true utilization page1900for determining true utilization based on productive behaviors. In contrast to the true utilization page600described above with respect toFIG. 6A, which displays a true utilization calculated based on active time, the true utilization page1900displays a true utilization metric1901which, as described by a note1902, is calculated by dividing productive time active on case work by total time ready for work. A graph1904displays true utilization values over a seven day period, for different teams. As described above, productive time may be based on sequences of interactions that are determined to be productive.

FIG. 20is a flowchart of a method2000for determining true utilization based on productive behaviors of operators in a digital environment. For example, the true utilization engine1702can be used to perform the method2000. For clarity of presentation, the description that follows generally describes method2000in the context of the other figures in this description. However, it will be understood that method2000can be performed, for example, by any suitable system, environment, software, and hardware, or a combination of systems, environments, software, and hardware, as appropriate. In some implementations, various steps of method2000can be run in parallel, in combination, in loops, or in any order.

The method2000can be performed to automatically determine productivity for each of multiple users of an organization. For example, productivity can be automatically determined for each customer support representative employed in the organization, for various periods (e.g., daily, weekly, monthly, annually). The method2000can be repeated for each respective user.

At2002, timing data is received for a user that indicates available work time for the user. The user can be, for example, a customer support representative providing support for customers of the organization.

At2004, interaction data for the user is received. The interactions can be performed in a digital workspace of the user. The interactions can be interactions occurring in multiple software services used by the user during the available work time for the user. The interactions can be part of sequences of interactions with a respective service.

At2006, productivity rules are received for an organization of the user that include conditions that define productivity of interactions by users of the organization. Different organizations can have different productivity rules. The productivity rules can include sequence productivity rules that each include conditions defining productivity of a respective sequence of interactions with a respective service. The productivity rules can be configured by the organization or an administrator of a true utilization engine. As another example, the productivity rules can be part of a learned model learned by a machine learning engine. The learned model can be trained based on known productive interactions and known unproductive interactions. The learned model can be updated based on feedback on productivity data determined by the machine learning engine.

At2008, productivity of interactions of the user is determined based on the interaction data and the productivity rules. For example, the interaction data can be matched to conditions specified in one or more productivity rules. As another example, the machine learning engine can determine productivity based on the learned model. Productivity can indicate whether time spent on certain interactions is productive or a degree of productivity.

At2010, true utilization for the user is determined based on the productivity of the interactions and the timing data. True utilization can be determined by dividing a sum of time spent performing productive interactions divided by a sum of time available within the digital workspace of the user.

At2012, action data is generated based on the determined true utilization.

At2014, action is taken based on the action data. Taking action can include generating and providing a warning in response to the true utilization being less than a threshold. As another example, taking action can include performing one or more of a personnel action, a training, action, or a tool configuration action based on the action data. Action can be taken based on action data for a particular user or aggregate action data gathered for multiple users.

FIG. 21is a block diagram of an example computer system2100used to provide computational functionalities associated with described algorithms, methods, functions, processes, flows, and procedures described in the present disclosure, according to some implementations of the present disclosure. The illustrated computer2102is intended to encompass any computing device such as a server, a desktop computer, a laptop/notebook computer, a wireless data port, a smart phone, a personal data assistant (PDA), a tablet computing device, or one or more processors within these devices, including physical instances, virtual instances, or both. The computer2102can include input devices such as keypads, keyboards, and touch screens that can accept user information. Also, the computer2102can include output devices that can convey information associated with the operation of the computer2102. The information can include digital data, visual data, audio information, or a combination of information. The information can be presented in a graphical user interface (UI) (or GUI).

The computer2102can serve in a role as a client, a network component, a server, a database, a persistency, or components of a computer system for performing the subject matter described in the present disclosure. The illustrated computer2102is communicably coupled with a network2130. In some implementations, one or more components of the computer2102can be configured to operate within different environments, including cloud-computing-based environments, local environments, global environments, and combinations of environments.

The computer2102can receive requests over network2130from a client application (for example, executing on another computer2102). The computer2102can respond to the received requests by processing the received requests using software applications. Requests can also be sent to the computer2102from internal users (for example, from a command console), external (or third) parties, automated applications, entities, individuals, systems, and computers.

Each of the components of the computer2102can communicate using a system bus2103. In some implementations, any or all of the components of the computer2102, including hardware or software components, can interface with each other or the interface2104(or a combination of both) over the system bus2103. Interfaces can use an application programming interface (API)2112, a service layer2113, or a combination of the API2112and service layer2113. The API2112can include specifications for routines, data structures, and object classes. The API2112can be either computer-language independent or dependent. The API2112can refer to a complete interface, a single function, or a set of APIs.

The service layer2113can provide software services to the computer2102and other components (whether illustrated or not) that are communicably coupled to the computer2102. The functionality of the computer2102can be accessible for all service consumers using this service layer. Software services, such as those provided by the service layer2113, can provide reusable, defined functionalities through a defined interface. For example, the interface can be software written in JAVA, C++, or a language providing data in extensible markup language (XML) format. While illustrated as an integrated component of the computer2102, in alternative implementations, the API2112or the service layer2113can be stand-alone components in relation to other components of the computer2102and other components communicably coupled to the computer2102. Moreover, any or all parts of the API2112or the service layer2113can be implemented as child or sub-modules of another software module, enterprise application, or hardware module without departing from the scope of the present disclosure.

The computer2102includes an interface2104. Although illustrated as a single interface2104inFIG. 21, two or more interfaces2104can be used according to particular needs, desires, or particular implementations of the computer2102and the described functionality. The interface2104can be used by the computer2102for communicating with other systems that are connected to the network2130(whether illustrated or not) in a distributed environment. Generally, the interface2104can include, or be implemented using, logic encoded in software or hardware (or a combination of software and hardware) operable to communicate with the network2130. More specifically, the interface2104can include software supporting one or more communication protocols associated with communications. As such, the network2130or the interface's hardware can be operable to communicate physical signals within and outside of the illustrated computer2102.

The computer2102includes a processor2105. Although illustrated as a single processor2105inFIG. 21, two or more processors2105can be used according to particular needs, desires, or particular implementations of the computer2102and the described functionality. Generally, the processor2105can execute instructions and can manipulate data to perform the operations of the computer2102, including operations using algorithms, methods, functions, processes, flows, and procedures as described in the present disclosure.

The computer2102also includes a database2106that can hold data for the computer2102and other components connected to the network2130(whether illustrated or not). For example, database2106can be an in-memory, conventional, or a database storing data consistent with the present disclosure. In some implementations, database2106can be a combination of two or more different database types (for example, hybrid in-memory and conventional databases) according to particular needs, desires, or particular implementations of the computer2102and the described functionality. Although illustrated as a single database2106inFIG. 21, two or more databases (of the same, different, or combination of types) can be used according to particular needs, desires, or particular implementations of the computer2102and the described functionality. While database2106is illustrated as an internal component of the computer2102, in alternative implementations, database2106can be external to the computer2102.

The computer2102also includes a memory2107that can hold data for the computer2102or a combination of components connected to the network2130(whether illustrated or not). Memory2107can store any data consistent with the present disclosure. In some implementations, memory2107can be a combination of two or more different types of memory (for example, a combination of semiconductor and magnetic storage) according to particular needs, desires, or particular implementations of the computer2102and the described functionality. Although illustrated as a single memory2107inFIG. 21, two or more memories2107(of the same, different, or combination of types) can be used according to particular needs, desires, or particular implementations of the computer2102and the described functionality. While memory2107is illustrated as an internal component of the computer2102, in alternative implementations, memory2107can be external to the computer2102.

The application2108can be an algorithmic software engine providing functionality according to particular needs, desires, or particular implementations of the computer2102and the described functionality. For example, application2108can serve as one or more components, modules, or applications. Further, although illustrated as a single application2108, the application2108can be implemented as multiple applications2108on the computer2102. In addition, although illustrated as internal to the computer2102, in alternative implementations, the application2108can be external to the computer2102.

The computer2102can also include a power supply2114. The power supply2114can include a rechargeable or non-rechargeable battery that can be configured to be either user- or non-user-replaceable. In some implementations, the power supply2114can include power-conversion and management circuits, including recharging, standby, and power management functionalities. In some implementations, the power-supply2114can include a power plug to allow the computer2102l to be plugged into a wall socket or a power source to, for example, power the computer2102or recharge a rechargeable battery.

There can be any number of computers2102associated with, or external to, a computer system containing computer2102, with each computer2102communicating over network2130. Further, the terms “client,” “user,” and other appropriate terminology can be used interchangeably, as appropriate, without departing from the scope of the present disclosure. Moreover, the present disclosure contemplates that many users can use one computer2102and one user can use multiple computers2102.

Described implementations of the subject matter can include one or more features, alone or in combination. For example, in a first implementation, a computer-implemented method includes the actions of: automatically determining productivity for each of multiple users of an organization, wherein determining productivity for a respective user comprises: receiving timing data for the user indicating available work time for the user; receiving interaction data for the user for interactions occurring in multiple software services used by the user during the available work time for the user; receiving productivity rules for an organization of the user that include conditions defining productivity of interactions by users of the organization; determining productivity of interactions of the user based on the interaction data and the productivity rules; determining true utilization for the user based on the productivity of the interactions and the timing data; and generating action data based on the determined true utilization; and taking action based on the action data.

In a second implementation, a non-transitory, computer-readable medium stores one or more instructions executable by a computer system to perform operations including: automatically determining productivity for each of multiple users of an organization, wherein determining productivity for a respective user comprises: receiving timing data for the user indicating available work time for the user; receiving interaction data for the user for interactions occurring in multiple software services used by the user during the available work time for the user; receiving productivity rules for an organization of the user that include conditions defining productivity of interactions by users of the organization; determining productivity of interactions of the user based on the interaction data and the productivity rules; determining true utilization for the user based on the productivity of the interactions and the timing data; and generating action data based on the determined true utilization; and taking action based on the action data.

In a fourth implementation, a system comprises one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations. The operations include: automatically determining productivity for each of multiple users of an organization, wherein determining productivity for a respective user comprises: receiving timing data for the user indicating available work time for the user; receiving interaction data for the user for interactions occurring in multiple software services used by the user during the available work time for the user; receiving productivity rules for an organization of the user that include conditions defining productivity of interactions by users of the organization; determining productivity of interactions of the user based on the interaction data and the productivity rules; determining true utilization for the user based on the productivity of the interactions and the timing data; and generating action data based on the determined true utilization; and taking action based on the action data.

A first feature, combinable with any of the following features, wherein the interactions are performed in a digital workspace of the user.

A second feature, combinable with any of the previous or following features, wherein determining the productivity of the interactions of the user based on the interaction data and the productivity rules comprises determining a degree of match between interactions in the interaction data and one or more conditions in the productivity rules that define criteria for productive or nonproductive work.

A third feature, combinable with any of the previous or following features, wherein true utilization is determined by dividing a sum of time spent performing productive interactions divided by a sum of time available within the digital workspace of the user.

A fourth feature, combinable with any of the previous or following features, wherein the user is a customer support representative providing support for customers of the organization.

A fifth feature, combinable with any of the previous or following features, wherein different organizations have different productivity rules.

A sixth feature, combinable with any of the previous or following features, wherein: the interactions comprise sequences of interactions with a respective service; and the productivity rules include sequence productivity rules that each include conditions defining productivity of a respective sequence of interactions with a respective service.

A seventh feature, combinable with any of the previous or following features, wherein the productivity rules are configured by the organization.

An eighth feature, combinable with any of the previous or following features, wherein the productivity rules are part of a learned model learned by a machine learning engine.

A ninth feature, combinable with any of the previous or following features, wherein the learned model is trained based on known productive interactions and known unproductive interactions.

A tenth feature, combinable with any of the previous or following features, wherein the learned model is updated based on feedback on productivity data determined by the machine learning engine.

An eleventh feature, combinable with any of the previous or following features, wherein taking action comprises generating and providing a warning in response to the true utilization being less than a threshold.

A twelfth feature, combinable with any of the previous or following features, wherein taking action comprises performing one or more of a personnel action, a training, action, or a tool configuration action based on the action data.