Source: http://www.google.com/patents/US20080189632?dq=5,581,513
Timestamp: 2016-07-26 17:00:42
Document Index: 237444228

Matched Legal Cases: ['art 410', 'art 410', 'art 420', 'art 430', 'art 440', 'art 1170', 'art 1178']

Patent US20080189632 - Severity Assessment For Performance Metrics Using Quantitative Model - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inPatentsPerformance metric scores are computed and aggregated by determining status bands based on boundary definitions and relative position of an input value within the status bands. A behavior of the score within a score threshold in response to a behavior of the input is defined based on a status indication...http://www.google.com/patents/US20080189632?utm_source=gb-gplus-sharePatent US20080189632 - Severity Assessment For Performance Metrics Using Quantitative ModelAdvanced Patent SearchPublication numberUS20080189632 A1Publication typeApplicationApplication numberUS 11/670,444Publication dateAug 7, 2008Filing dateFeb 2, 2007Priority dateFeb 2, 2007Publication number11670444, 670444, US 2008/0189632 A1, US 2008/189632 A1, US 20080189632 A1, US 20080189632A1, US 2008189632 A1, US 2008189632A1, US-A1-20080189632, US-A1-2008189632, US2008/0189632A1, US2008/189632A1, US20080189632 A1, US20080189632A1, US2008189632 A1, US2008189632A1InventorsIan Tien, Corey J. Hulen, Chen-I LimOriginal AssigneeMicrosoft CorporationExport CitationBiBTeX, EndNote, RefManPatent Citations (105), Non-Patent Citations (1), Referenced by (33), Classifications (8), Legal Events (2) External Links: USPTO, USPTO Assignment, EspacenetSeverity Assessment For Performance Metrics Using Quantitative Model
[0002] The core to scorecarding is the calculation of a score that represents performance across KPIs, their actual data, their target settings, their thresholds and other constraints. All metrics are, however, not equal. In most practical scenarios, different KPIs reporting to higher level ones have different severity levels. Ultimately most performance analysis comes down to a quantitative decision about resource allocation based on metrics such as budget, compensation, time, future investment, and the like. Since each of the metrics feeding into the decision process may have a different severity level, a confidently and accurately made decision requires assessment of metrics considering their severity levels among other aspects.
[0004] Embodiments are directed to computing scores of performance metrics by determining status bands based on boundary definitions and a relative position of an input value within the status bands. The scores may then be aggregated to obtain scores for higher level metrics utilizing predetermined aggregation rules.
[0008] FIG. 3 is a diagram illustrating scorecard calculations for two example metrics using normalized banding;
[0009] FIG. 4 illustrates four examples of determination of scores by setting boundary values and associated input and score thresholds;
[0010] FIG. 5 illustrates different aggregation methods for reporting metrics in an example scorecard;
[0011] FIG. 6 is a screenshot of a performance metric definition user interface for performing scorecard computations according to embodiments;
[0012] FIG. 7 is a screenshot of a performance metric definition user interface for defining an input value according to one method;
[0013] FIG. 8 is a screenshot of a performance metric definition user interface for defining an input value according to another method;
[0014] FIG. 9 is a screenshot of a performance metric definition user interface for defining input thresholds;
[0015] FIG. 10 is a screenshot of a performance metric definition user interface for defining score thresholds;
[0016] FIG. 11 is a screenshot of a performance metric definition user interface for testing the effects of proximity of a test input value to other input values;
[0017] FIG. 12 is a diagram of a networked environment where embodiments may be implemented;
[0018] FIG. 13 is a block diagram of an example computing operating environment, where embodiments may be implemented; and
[0019] FIG. 14 illustrates a logic flow diagram for a process of severity assessment for performance metrics using a quantitative model.
[0020] As briefly described above, performance metric scores may be computed based on comparison of actuals and targets of performance metrics by determining status bands from boundary definitions and determining a relative position of an input value within the status band. In the following detailed description, references are made to the accompanying drawings that form a part hereof, and in which are shown by way of illustrations specific embodiments or examples. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the spirit or scope of the present disclosure. The following detailed description is therefore not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims and their equivalents.
[0021] While the embodiments will be described in the general context of program modules that execute in conjunction with an application program that runs on an operating system on a personal computer, those skilled in the art will recognize that aspects may also be implemented in combination with other program modules.
[0022] Generally, program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that embodiments may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. Embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
[0023] Embodiments may be implemented as a computer process (method), a computing system, or as an article of manufacture, such as a computer program product or computer readable media. The computer program product may be a computer storage media readable by a computer system and encoding a computer program of instructions for executing a computer process. The computer program product may also be a propagated signal on a carrier readable by a computing system and encoding a computer program of instructions for executing a computer process.
[0024] Referring to FIG. 1, an example scorecard architecture is illustrated. The scorecard architecture may comprise any topology of processing systems, storage systems, source systems, and configuration systems. The scorecard architecture may also have a static or dynamic topology.
[0025] Scorecards are an easy method of evaluating organizational performance. The performance measures may vary from financial data such as sales growth to service information such as customer complaints. In a non-business environment, student performances and teacher assessments may be another example of performance measures that can employ scorecards for evaluating organizational performance. In the exemplary scorecard architecture, a core of the system is scorecard engine 108. Scorecard engine 108 may be an application software that is arranged to evaluate performance metrics. Scorecard engine 108 may be loaded into a server, executed over a distributed network, executed in a client device, and the like.
[0026] Data for evaluating various measures may be provided by a data source. The data source may include source systems 112, which provide data to a scorecard cube 114. Source systems 112 may include multi-dimensional databases such OLAP, other databases, individual files, and the like, that provide raw data for generation of scorecards. Scorecard cube 114 is a multi-dimensional database for storing data to be used in determining Key Performance Indicators (KPIs) as well as generated scorecards themselves. As discussed above, the multi-dimensional nature of scorecard cube 114 enables storage, use, and presentation of data over multiple dimensions such as compound performance indicators for different geographic areas, organizational groups, or even for different time intervals. Scorecard cube 114 has a bi-directional interaction with scorecard engine 108 providing and receiving raw data as well as generated scorecards.
[0027] Scorecard database 116 is arranged to operate in a similar manner to scorecard cube 114. In one embodiment, scorecard database 116 may be an external database providing redundant back-up database service.
[0028] Scorecard builder 102 may be a separate application or a part of a business logic application such as the performance evaluation application, and the like. Scorecard builder 102 is employed to configure various parameters of scorecard engine 108 such as scorecard elements, default values for actuals, targets, and the like. Scorecard builder 102 may include a user interface such as a web service, a GUI, and the like.
[0029] Strategy map builder 104 is employed for a later stage in scorecard generation process. As explained below, scores for KPIs and other metrics may be presented to a user in form of a strategy map. Strategy map builder 104 may include a user interface for selecting graphical formats, indicator elements, and other graphical parameters of the presentation.
[0030] Data Sources 106 may be another source for providing raw data to scorecard engine 108. Data sources 106 may also define KPI mappings and other associated data.
[0031] Additionally, the scorecard architecture may include scorecard presentation 110. This may be an application to deploy scorecards, customize views, coordinate distribution of scorecard data, and process web-specific applications associated with the performance evaluation process. For example, scorecard presentation 110 may include a web-based printing system, an email distribution system, and the like. In some embodiments, scorecard presentation 110 may be an interface that is used as part of the scorecard engine to export data for generating presentations or other forms of scorecard-related documents in an external application. For example, metrics, reports, and other elements (e.g. commentary) may be provided with metadata to a presentation application (e.g. PowerPoint� of MICROSOFT CORPORATION of Redmond, Wash.), a word processing application, or a graphics application to generate slides, documents, images, and the like, based on the selected scorecard data.
[0032] FIG. 2 illustrates a screenshot of an example scorecard with status indicators 230. As explained before, Key Performance Indicators (KPIs) are specific indicators of organizational performance that measure a current state in relation to meeting the targeted objectives. Decision makers may utilize these indicators to manage the organization more effectively.
[0033] When creating a KPI, the KPI definition may be used across several scorecards. This is useful when different scorecard managers might have a shared KPI in common. This may ensure a standard definition is used for that KPI. Despite the shared definition, each individual scorecard may utilize a different data source and data mappings for the actual KPI.
[0034] Each KPI may include a number of attributes. Some of these attributes include frequency of data, unit of measure, trend type, weight, and other attributes.
[0035] The frequency of data identifies how often the data is updated in the source database (cube). The frequency of data may include: Daily, Weekly, Monthly, Quarterly, and Annually.
[0036] The unit of measure provides an interpretation for the KPI. Some of the units of measure are: Integer, Decimal, Percent, Days, and Currency. These examples are not exhaustive, and other elements may be added without departing from the scope of the invention.
[0037] A trend type may be set according to whether an increasing trend is desirable or not. For example, increasing profit is a desirable trend, while increasing defect rates is not. The trend type may be used in determining the KPI status to display and in setting and interpreting the KPI banding boundary values. The arrows displayed in the scorecard of FIG. 2 indicate how the numbers are moving this period compared to last. If in this period the number is greater than last period, the trend is up regardless of the trend type. Possible trend types may include: Increasing Is Better, Decreasing Is Better, and On-Target Is Better.
[0038] Weight is a positive integer used to qualify the relative value of a KPI in relation to other KPIs. It is used to calculate the aggregated scorecard value. For example, if an Objective in a scorecard has two KPIs, the first KPI has a weight of 1, and the second has a weight of 3 the second KPI is essentially three times more important than the first, and this weighted relationship is part of the calculation when the KPIs' values are rolled up to derive the values of their parent metric.
[0039] Other attributes may contain pointers to custom attributes that may be created for documentation purposes or used for various other aspects of the scorecard system such as creating different views in different graphical representations of the finished scorecard. Custom attributes may be created for any scorecard element and may be extended or customized by application developers or users for use in their own applications. They may be any of a number of types including text, numbers, percentages, dates, and hyperlinks.
[0040] One of the benefits of defining a scorecard is the ability to easily quantify and visualize performance in meeting organizational strategy. By providing a status at an overall scorecard level, and for each perspective, each objective or each KPI rollup, one may quickly identify where one might be off target. By utilizing the hierarchical scorecard definition along with KPI weightings, a status value is calculated at each level of the scorecard.
[0041] First column of the scorecard shows example top level metric 236 “Manufacturing” with its reporting KPIs 238 and 242 “Inventory” and “Assembly”. Second column 222 in the scorecard shows results for each measure from a previous measurement period. Third column 224 shows results for the same measures for the current measurement period. In one embodiment, the measurement period may include a month, a quarter, a tax year, a calendar year, and the like.
[0042] Fourth column 226 includes target values for specified KPIs on the scorecard. Target values may be retrieved from a database, entered by a user, and the like. Column 228 of the scorecard shows status indicators 230.
[0043] Status indicators 230 convey the state of the KPI. An indicator may have a predetermined number of levels. A traffic light is one of the most commonly used indicators. It represents a KPI with three-levels of results—Good, Neutral, and Bad. Traffic light indicators may be colored red, yellow, or green. In addition, each colored indicator may have its own unique shape. A KPI may have one stoplight indicator visible at any given time. Other types of indicators may also be employed to provide status feedback. For example, indicators with more than three levels may appear as a bar divided into sections, or bands. Column 232 includes trend type arrows as explained above under KPI attributes. Column 234 shows another KPI attribute, frequency.
[0044] FIG. 3 is a diagram illustrating scorecard calculations for two example metrics using normalized banding. According to a typical normalized banding calculation, metrics such as KPI A (352) are evaluated based on a set of criteria such as “Increasing is better” (356), “Decreasing is better” (358), or “On target is better” (360). Depending on a result of the evaluation of the metric an initial score is determined on a status band 368 where the thresholds and band regions are determined based on their absolute values. The band regions for each criterion may be assigned a visual presentation scheme such as coloring (red, yellow, green), traffic lights, smiley icons, and the like.
[0045] A similar process is applied to a second metric KPI B (354), where the initial score is in the red band region on status band 370 as a result of applying the “Increasing is better” (362), “Decreasing is better” (364), or “On target is better” (366) criteria.
[0046] Then, the initial scores for both metrics are carried over to a normalized status band 372, where the boundaries and regions are normalized according to their relative position within the status band. The scores can only be compared and aggregated after normalization because their original status bands are not compatible (e.g. different boundaries, band region lengths, etc.). The normalization not only adds another layer of computations, but is also in some cases difficult to comprehend for users.
[0047] Once the normalized scores are determined, they can be aggregated on the normalized status band providing the aggregated score for the top level metric or the scorecard. The performance metrics computations in a typical scorecard system may include relatively diverse and complex rules such as:
some allow for performance bonus and some do not the performance of some may be considered more important than others some may be missing data [0056] The ability to express these complex rules may become more convoluted in a system using normalized status bands. At least, it is harder to visually perceive the flow of computations.
[0057] FIG. 4 illustrates four examples of determination of scores by setting boundary values and associated input and score thresholds. Score can then be computed based on the relationship between the input and score thresholds. Providing a straight forward visually adapted model for computing performance metric scores may enable greater objectivity, transparency, and consistency in reporting systems, reduce the risk of multiple interpretations of the same metric, and enhance the ability to enforce accountability throughout an organization. Thus, powerful and yet easy-to-understand quantitative models for assessing performance across an array of complex scenarios may be implemented.
[0058] As shown in chart 410, input ranges may be defined along an input axis 412. The regions defined by the input ranges do not have to normalized or equal. Next, the score ranges are defined along the score axis. Each score range corresponds to an input range. From the correspondence of the input and score ranges, boundary values may be set on the chart forming the performance contour 416. The performance contour shows the relationship between input values across the input axis and scores across the score axis. In a user interface presentation, the performance contour may be color coded based on the background color of each band within a given input range. In the example chart 410, the performance contour 416 reflects an increasing is better type trend. By using the performance contour, however, an analysis of applicable trend is no longer needed. Based on the definition of input and score thresholds, the trend type is automatically provided.
[0059] Example chart 420 includes input ranges along input axis 422 and score ranges along score axis 424. The performance contour 426 for this example matches a decreasing is better type trend. Example chart 430 includes input ranges along input axis 432 and score ranges along score axis 434. The performance contour 436 for this example matches an on target is better type trend.
[0060] Example chart 440 illustrates the ability to use discontinuous ranges according to embodiments. Input ranges are shown along input axis 422 and score ranges along score axis 424 again. The boundary values in this example are provided in a discontinuous manner. For example, there are two score boundary values corresponding to the input boundary value “20” and similarly two score boundary values corresponding to input boundary value “50”. Thus, a saw tooth style performance contour 446 is obtained.
[0061] As will be discussed later, a graphics based status band determination according to embodiments enables a subscriber to modify the bands and the performance contour easily and intuitively. In an authoring user interface, the subscriber can simply move the boundary values around on the chart modifying the performance contour, and thereby, a relationship between the input values and the scores.
[0062] FIG. 5 illustrates different aggregation methods for reporting metrics in an example scorecard. An important part of scorecard computations after calculating the scores for each metric is aggregating the scores for higher level metrics and/or for the overall scorecard.
[0063] The example scorecard in FIG. 5 includes a top level metric KPI 1 and three reporting metrics KPI 1.1-1.3 in metric column 552. Example actuals and targets for each metric are shown in columns 554 and 556. Upon determining status bands and input values for each metric status indicators may be shown in status column 558. These may be according to visualization scheme selected by the subscriber or by default. In the example scorecard a traffic light scheme is shown. The scores, computed using the performance contour method described above, are shown in column 560. The percentage scores of the example scorecard are not a result of accurate calculation. They are for illustration purposes only. Furthermore, a scorecard may include metrics in a much more complex hierarchical structure with multiple layers of child and parent metrics, multiple targets for each metric, and so on. The status determination and score computation principle remain the same, however.
[0064] Once the scores for lower level metrics are computed, the scores for higher level metrics or for the whole scorecard may be computed by aggregation or by comparison. For example, a relatively simple comparison method of determining the score for top level KPI 1 may include comparing the aggregated actual and target values of KPI 1.
[0065] Another method may involve aggregating the scores of KPI 1's descendants or children (depending on the hierarchical structure) by applying a subscriber defined or default rule. The rules may include, but are not limited to, sum of child scores, mean average of child scores, maximum of child scores, minimum of child scores, sum of descendant scores, mean average of descendant scores, maximum of descendant scores, minimum of descendant scores, and the like.
[0066] Yet another method may include comparison of child or descendant actual and target values applying rules such as: a variance between an aggregated actual and an aggregated target, and a standard deviation between an aggregated actual and an aggregated target, and the like. According to further methods, a comparison to an external value may also be performed.
[0067] FIG. 6 is a screenshot of a performance metric definition user interface for performing scorecard computations according to embodiments. As described above in more detail, performance metric operations begin with collection of metric data from multiple sources, which may include retrieval of data from local and remote data stores. Collected data is then aggregated and interpreted according to default and subscriber defined configuration parameters of a business service. For example, various metric hierarchies, attributes, aggregation methods, and interpretation rules may be selected by a subscriber from available sets.
[0068] The core to scorecarding is the calculation of a score that represents performance across KPIs, their actual data, their target settings, their thresholds and other constraints. According to some embodiments, the scoring process may be executed as follows:
[0069] 1) Input value for a KPI target is determined
a. Input can come from a variety of data sources or be user-entered [0071] 2) Status band is determined
a. A KPI target has status bands defined by boundaries. Based on those boundaries and the input value a status band is selected b. This determines the status icon, text and other properties to be shown in the visualization of a KPI [0074] 3) Relative position of input value within status band is determined
a. The relative distance between boundary values within a status band is determined [0076] 4) A score is computed
a. Based on the relative position of the input value to the status band boundaries and the range of scores available within the status band [0078] 5) The score can then be used to determine performance downstream
a. The score of one KPI can thus be used to inform higher levels of performance based on summaries of the base KPI. [0080] Once the aggregation and interpretation is accomplished per the above process, the service can provide a variety of presentations based on the results. In some cases, the raw data itself may also be presented along with the analysis results. Presentations may be configured and rendered employing a native application user interface or an embeddable user interface that can be launched from any presentation application such as a graphics application, a word processing application, a spreadsheet application, and the like. Rendered presentations may be delivered to subscribers (e.g. by email, web publishing, file sharing, etc.), stored in various file formats, exported, and the like.
[0081] Returning to FIG. 6, side panel 610 titled “Workspace Browser” provides a selection of available scorecards and KPIs for authoring, as well as other elements of the scorecards such as indicators and reports. A selected element, “headcount”, from the workspace browser is shown on the main panel 620.
[0082] The main panel 620 includes a number of detailed aspects of performance metric computation associated with “headcount”. For example display formats, associated thresholds, and data mapping types for actuals and targets of “headcount” are displayed at the top. The indicator set (624) is described and a link provided for changing to another indicator set (in the example Smiley style indicators are used). A preview of the performance contour reflecting scores vs. input values (622) is provided as well. The bands as defined by the boundaries (e.g. 628) are color coded to show the visualization scheme for status. A test input value is displayed on the performance contour linked to the status preview (626), which illustrates the status, indicator, score and distances to the boundaries for the test input value.
[0083] Under the preview displays, an authoring user interface 629 is provided for displaying, defining, and modifying input value, input threshold, and score threshold parameters. These are explained in more detail below in conjunction with FIG. 7 through FIG. 10.
[0084] FIG. 7 is a screenshot of a performance metric definition user interface for defining an input value according to one method. A relationship between an input value and input thresholds determines the overall status of a given target.
[0085] The example user interface of FIG. 7 includes the previews of the performance contour (722) and status (726) for a test input value as explained above in conjunction with FIG. 6. The definition section 730 of the user interface may be in a tab, pane, or pop-up window format with a different user interface for each of the input values, input thresholds, and score thresholds. The input values may be based on an aggregated score (732) or a value from the selected metric. If the input value is an aggregated score, the aggregation may be performed applying a default or subscriber defined rule. In the example user interface, a list of available aggregation rules (734) are provided with an explanation (736) of each selected rule provided next to the list.
[0086] According to some embodiment, the previews (722 and 726) may be updated automatically in response to subscriber selection of the aggregation rule giving the subscriber an opportunity to go back and modify the boundary values or status indicators.
[0087] FIG. 8 is a screenshot of a performance metric definition user interface for defining an input value according to another method. The previews of the performance contour (822) and status (826) for a test input value are the same as in previous figures. Differently from FIG. 7, the input value is defined as a value for the selected KPI (832) in the example user interface 830 of FIG. 8. Based on this definition, different options for determining the input value are provided in the list 835, which includes actual or target values of the KPI, a variance between the target and the actual of the selected KPI or between different targets of the selected KPI, and a percentage variance between the actual and target(s) of the selected KPI. Depending on the selection in list 835, additional options for defining actuals and targets to be used in computation may be provided (838). An explanation (736) for each selection is also provided next to the list 825.
[0088] In other embodiments, the definition user interface may be configured to provide the option of selecting the input value based on an external value providing the subscriber options for defining the source for the external value.
[0089] FIG. 9 is a screenshot of a performance metric definition user interface for defining input thresholds. Input thresholds determine the boundaries between status bands for a given indicator set.
[0090] The previews of the performance contour (922) and status (926) for a test input value are the same as in previous figures. In the definition user interface 930, input threshold parameters are displayed and options for setting or modifying them are provided. The parameters include input threshold values 946 for highest and lowest boundaries with other boundaries in between those two. The number of boundaries is based on the selected indicator set and associated number of statuses (944) displayed next to the list of boundary values. The names of the boundaries (942) are also listed on the left of the boundary value list.
[0091] FIG. 10 is a screenshot of a performance metric definition user interface for defining score thresholds. Score thresholds determine the score produced when an input falls in a specific status band.
[0092] The previews of the performance contour (1022) and status (1026) for a test input value are functionally similar to those in previous figures. Differently in FIG. 10, score threshold preview displays bands between default boundary values along a score threshold axis with a test input value on one of the bands. The status preview 1026 also includes a gauge style indicator instead of a Smiley style indicator. Other indicator types may also be used according to embodiments.
[0093] The definition user interface includes a listing of thresholds 1054 (e.g. over budget, under budget, etc.), lower (1056) and upper (1058) boundary values, and the effect of what happens when the input increases within each threshold (1052). For example, as the input increases within the “over budget” threshold, the score decreases. On the other hand, in the “within budget” threshold the score may increase as the input increases. Thus, a behavior of the score within each threshold based on a behavior of the input value may be defined or modified at this stage and the performance contour adjusted accordingly.
[0094] According to some embodiments, a multiplicative weighting factor may be applied to the score output when the scores are aggregated. The weighting factor may be a default value or defined by the subscriber using definition user interface 1030 or another one.
[0095] FIG. 11 is a screenshot of a performance metric definition user interface for testing the effects of proximity of a test input value to other input values. The previews of the performance contour (1122) and status (1126) for a test input value are the same as in FIG. 7. In addition, an information tip is provided showing a distance of an input value from the test value.
[0096] As illustrated under the “Sensitivity” tab of the example definition user interface, the subscriber may be provided with feedback by previewing how a KPI performance can change when the test input value is changed. A preview chart 1170 with the performance contour 1176 and the test input value may be displayed. When the subscriber selects another point on the performance contour, a distance of the new selection to the test input value and the new score may be provided instantaneously enabling the subscriber to determine effects of changes without having to redo the whole computation. A score change versus input value change chart 1178 may also be provided for visualization of the effects.
[0097] According to some embodiments, statistical analysis for past performance and/or future forecast may also be carried out based on subscriber definition (selection) of the computation parameters. A next step in the scorecard process is generation of presentations based on the performance metric data and the analysis results. Reports comprising charts, grid presentations, graphs, three dimensional visualizations, and the like may be generated based on selected portions of available data.
[0098] The example user interfaces and computation parameters shown in the figures above are for illustration purposes only and do not constitute a limitation on embodiments. Other embodiments using different user interfaces, graphical elements and charts, status indication schemes, user interaction schemes, and so on, may be implemented without departing from a scope and spirit of the disclosure.
[0099] Referring now to the following figures, aspects and exemplary operating environments will be described. FIG. 12, FIG. 13, and the associated discussion are intended to provide a brief, general description of a suitable computing environment in which embodiments may be implemented.
[0100] FIG. 12 is a diagram of a networked environment where embodiments may be implemented. The system may comprise any topology of servers, clients, Internet service providers, and communication media. Also, the system may have a static or dynamic topology. The term “client” may refer to a client application or a client device employed by a user to perform operations associated with assessing severity of performance metrics using a quantitative model. While a networked business logic system may involve many more components, relevant ones are discussed in conjunction with this figure.
[0101] In a typical operation according to embodiments, business logic service may be provided centrally from server 1212 or in a distributed manner over several servers (e.g. servers 1212 and 1214) and/or client devices. Server 1212 may include implementation of a number of information systems such as performance measures, business scorecards, and exception reporting. A number of organization-specific applications including, but not limited to, financial reporting/analysis, booking, marketing analysis, customer service, and manufacturing planning applications may also be configured, deployed, and shared in the networked system.
[0102] Data sources 1201-1203 are examples of a number of data sources that may provide input to server 1212. Additional data sources may include SQL servers, databases, non multi-dimensional data sources such as text files or EXCEL� sheets, multi-dimensional data source such as data cubes, and the like.
[0103] Users may interact with server running the business logic service from client devices 1205-1207 over network 1210. In another embodiment, users may directly access the data from server 1212 and perform analysis on their own machines.
[0104] Client devices 1205-1207 or servers 1212 and 1214 may be in communications with additional client devices or additional servers over network 1210. Network 1210 may include a secure network such as an enterprise network, an unsecure network such as a wireless open network, or the Internet. Network 1210 provides communication between the nodes described herein. By way of example, and not limitation, network 1210 may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.
[0105] Many other configurations of computing devices, applications, data sources, data distribution and analysis systems may be employed to implement rendering of performance metric based presentations using geometric objects. Furthermore, the networked environments discussed in FIG. 12 are for illustration purposes only. Embodiments are not limited to the example applications, modules, or processes. A networked environment for may be provided in many other ways using the principles described herein.
[0106] With reference to FIG. 13, a block diagram of an example computing operating environment is illustrated, such as computing device 1300. In a basic configuration, the computing device 1300 typically includes at least one processing unit 1302 and system memory 1304. Computing device 1300 may include a plurality of processing units that cooperate in executing programs. Depending on the exact configuration and type of computing device, the system memory 1304 may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. System memory 1304 typically includes an operating system 1305 suitable for controlling the operation of a networked personal computer, such as the WINDOWS� operating systems from MICROSOFT CORPORATION of Redmond, Wash. The system memory 1304 may also include one or more software applications such as program modules 1306, business logic application 1322, scorecard engine 1324, and optional presentation application 1326.
[0107] Business logic application 1322 may be any application that processes and generates scorecards and associated data. Scorecard engine 1324 may be a module within business logic application 1322 that manages definition of scorecard metrics and computation parameters, as well as computation of scores and aggregations. Presentation application 1326 or business logic application 1322 itself may render the presentation(s) using the results of computations by scorecard engine 1324. Presentation application 1326 or business logic application 1322 may be executed in an operating system other than operating system 1305. This basic configuration is illustrated in FIG. 13 by those components within dashed line 1308.
[0108] The computing device 1300 may have additional features or functionality. For example, the computing device 1300 may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 13 by removable storage 1309 and non-removable storage 1310. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. System memory 1304, removable storage 1309 and non-removable storage 1310 are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 1300. Any such computer storage media may be part of device 1300. Computing device 1300 may also have input device(s) 1312 such as keyboard, mouse, pen, voice input device, touch input device, etc. Output device(s) 1314 such as a display, speakers, printer, etc. may also be included. These devices are well known in the art and need not be discussed at length here.
[0109] The computing device 1300 may also contain communication connections 1316 that allow the device to communicate with other computing devices 1318, such as over a network in a distributed computing environment, for example, an intranet or the Internet. Communication connection 1316 is one example of communication media. Communication media may typically be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. The term computer readable media as used herein includes both storage media and communication media.
[0110] The claimed subject matter also includes methods. These methods can be implemented in any number of ways, including the structures described in this document. One such way is by machine operations, of devices of the type described in this document.
[0111] Another optional way is for one or more of the individual operations of the methods to be performed in conjunction with one or more human operators performing some. These human operators need not be collocated with each other, but each can be only with a machine that performs a portion of the program.
[0112] FIG. 14 illustrates a logic flow diagram for a process of severity assessment for performance metrics using a quantitative model. Process 1400 may be implemented in a business logic service that processes and/or generates scorecards and scorecard-related reports.
[0113] Process 1400 begins with operation 1402, where an input value for a target of a performance metric is determined. The input may be provide by a subscriber or obtained from a variety of source such as other applications, scorecard data store, and the like. Processing advances from operation 1402 to operation 1404.
[0114] At operation 1404, a status band is determined. Each performance metric target has associated status bands defined by boundaries. The status band may be selected based on the boundaries and the input value. Determination of the status band also determines the status icon, text, or other properties to be used in presenting a visualization of the metric. Processing proceeds from operation 1404 to operation 1406.
[0115] At operation 1406, a relative position of the input value within the status band is determined. The relative position of the input value is determined by determining the relative distance between boundary values within the status band. Processing moves from operation 1406 to operation 1408.
[0116] At operation 1408, the score for the performance metric is computed. The score is computed based on the relative position of the input value within the status band and a range of scores available within the status band. Processing advances to optional operation 1410 from operation 1408.
[0117] At optional operation 1410, the score is used to perform aggregation calculations using other scores from other performance metrics. As described previously, scores may be aggregated according to a default or user defined rule and the hierarchical structure of performance metrics reporting to a higher metric. The aggregation result(s) may then be used with the scores of the performance metrics to render presentations based on user selection of a presentation type (e.g. trend charts, forecasts, and the like). After optional operation 1410, processing moves to a calling process for further actions.
[0118] The operations included in process 1400 are for illustration purposes. Assessing severity of performance metrics using a quantitative model may be implemented by similar processes with fewer or additional steps, as well as in different order of operations using the principles described herein.
[0119] The above specification, examples and data provide a complete description of the manufacture and use of the composition of the embodiments. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims and embodiments.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS5404295 *Jan 4, 1994Apr 4, 1995Katz; BorisMethod and apparatus for utilizing annotations to facilitate computer retrieval of database materialUS5646697 *Jan 17, 1995Jul 8, 1997Sony CorporationSpecial effects video processorUS5764890 *Jul 23, 1997Jun 9, 1998Microsoft CorporationMethod and system for adding a secure network server to an existing computer networkUS5779566 *Feb 22, 1995Jul 14, 1998Wilens; Peter S.Handheld golf reporting and statistical analysis apparatus and methodUS5911143 *Aug 14, 1995Jun 8, 1999International Business Machines CorporationMethod and system for advanced role-based access control in distributed and centralized computer systemsUS6012044 *May 25, 1999Jan 4, 2000Financial Engines, Inc.User interface for a financial advisory systemUS6020932 *Nov 10, 1997Feb 1, 2000Sony CorporationVideo signal processing device and its methodUS6023714 *Apr 24, 1997Feb 8, 2000Microsoft CorporationMethod and system for dynamically adapting the layout of a document to an output deviceUS6061692 *Nov 4, 1997May 9, 2000Microsoft CorporationSystem and method for administering a meta database as an integral component of an information serverUS6182022 *Jan 26, 1998Jan 30, 2001Hewlett-Packard CompanyAutomated adaptive baselining and thresholding method and systemUS6219459 *Jul 15, 1998Apr 17, 2001Sony CorporationImage transform device for transforming a picture image to a painting-type imageUS6226635 *Aug 14, 1998May 1, 2001Microsoft CorporationLayered query managementUS6230310 *Sep 29, 1998May 8, 2001Apple Computer, Inc.,Method and system for transparently transforming objects for application programsUS6341277 *Apr 9, 1999Jan 22, 2002International Business Machines CorporationSystem and method for performance complex heterogeneous database queries using a single SQL expressionUS6529215 *Dec 31, 1998Mar 4, 2003Fuji Xerox Co., Ltd.Method and apparatus for annotating widgetsUS6563514 *Apr 13, 2000May 13, 2003Extensio Software, Inc.System and method for providing contextual and dynamic information retrievalUS6601233 *Jul 30, 1999Jul 29, 2003Accenture LlpBusiness components frameworkUS6677963 *Nov 16, 1999Jan 13, 2004Verizon Laboratories Inc.Computer-executable method for improving understanding of business data by interactive rule manipulationUS6687735 *May 30, 2000Feb 3, 2004Tranceive Technologies, Inc.Method and apparatus for balancing distributed applicationsUS6728724 *Nov 13, 2000Apr 27, 2004Microsoft CorporationMethod for comparative visual rendering of dataUS6763134 *Sep 13, 2002Jul 13, 2004Avid Technology, Inc.Secondary color modification of a digital imageUS6782421 *Jul 9, 2002Aug 24, 2004Bellsouth Intellectual Property CorporationSystem and method for evaluating the performance of a computer applicationUS6854091 *Jul 28, 2000Feb 8, 2005Nortel Networks LimitedMethod of displaying nodes and linksUS6867764 *Mar 22, 2001Mar 15, 2005Sony CorporationData entry user interfaceUS6901426 *Jul 1, 1998May 31, 2005E-Talk CorporationSystem and method for providing access privileges for users in a performance evaluation systemUS7015911 *Mar 31, 2003Mar 21, 2006Sas Institute Inc.Computer-implemented system and method for report generationUS7027051 *Jun 29, 2001Apr 11, 2006International Business Machines CorporationGraphical user interface for visualization of sampled data compared to entitled or reference levelsUS7043524 *Nov 6, 2001May 9, 2006Omnishift Technologies, Inc.Network caching system for streamed applicationsUS7065784 *Feb 13, 2004Jun 20, 2006Microsoft CorporationSystems and methods for integrating access control with a namespaceUS7079010 *Apr 7, 2004Jul 18, 2006Jerry ChamplinSystem and method for monitoring processes of an information technology systemUS7181417 *Jan 21, 2000Feb 20, 2007Microstrategy, Inc.System and method for revenue generation in an automatic, real-time delivery of personalized informational and transactional dataUS7200595 *Jun 28, 2004Apr 3, 2007Microsoft CorporationSystems and methods for fine grained access control of data stored in relational databasesUS7249120 *Jun 27, 2003Jul 24, 2007Microsoft CorporationMethod and apparatus for selecting candidate statistics to estimate the selectivity value of the conditional selectivity expression in optimize queries based on a set of predicates that each reference a set of relational database tablesUS7340448 *Nov 13, 2003Mar 4, 2008International Business Machines CorporationMethod, apparatus, and computer program product for implementing enhanced query governor functionsUS7349862 *Feb 19, 2002Mar 25, 2008Cognos IncorporatedBusiness intelligence monitor method and systemUS7359865 *Nov 5, 2001Apr 15, 2008I2 Technologies Us, Inc.Generating a risk assessment regarding a software implementation projectUS7383247 *Aug 29, 2005Jun 3, 2008International Business Machines CorporationQuery routing of federated information systems for fast response time, load balance, availability, and reliabilityUS7409357 *Jul 16, 2004Aug 5, 2008Accenture Global Services, GmbhQuantification of operational risksUS7412402 *Mar 21, 2006Aug 12, 2008Kim A. CooperPerformance motivation systems and methods for contact centersUS7496852 *May 16, 2006Feb 24, 2009International Business Machines CorporationGraphically manipulating a databaseUS7509343 *Jun 9, 2004Mar 24, 2009Sprint Communications Company L.P.System and method of collecting and reporting system performance metricsUS7548912 *Nov 13, 2006Jun 16, 2009Microsoft CorporationSimplified search interface for querying a relational databaseUS7702779 *May 27, 2005Apr 20, 2010Symantec Operating CorporationSystem and method for metering of application services in utility computing environmentsUS7716571 *Apr 27, 2006May 11, 2010Microsoft CorporationMultidimensional scorecard header definitionUS7716592 *Mar 30, 2006May 11, 2010Microsoft CorporationAutomated generation of dashboards for scorecard metrics and subordinate reportingUS20010004256 *Dec 21, 2000Jun 21, 2001Satoshi IwataDisplay system, display control method and computer readable medium storing display control program codeUS20020029273 *Jun 5, 2001Mar 7, 2002Mark HaroldsonSystem and method for calculating concurrent network connectionsUS20020052862 *Mar 9, 2001May 2, 2002Powerway, Inc.Method and system for supply chain product and process development collaborationUS20020091737 *Nov 1, 2001Jul 11, 2002Markel Steven O.System and method for rules based media enhancementUS20020099678 *Jan 9, 2002Jul 25, 2002Brian AlbrightRetail price and promotion modeling system and methodUS20030014290 *Jun 17, 2002Jan 16, 2003Mclean Robert I.G.Data processing system and method for analysis of financial and non-financial value creation and value realization performance of a business enterpriseUS20030014488 *Jun 11, 2002Jan 16, 2003Siddhartha DalalSystem and method for enabling multimedia conferencing services on a real-time communications platformUS20030040936 *Jul 30, 2002Feb 27, 2003Worldcom, Inc.Systems and methods for generating reportsUS20030055731 *Mar 23, 2001Mar 20, 2003Restaurant Services Inc.System, method and computer program product for tracking performance of suppliers in a supply chain management frameworkUS20030055927 *Jun 5, 2002Mar 20, 2003Claudius FischerFramework for a device and a computer system needing synchronizationUS20030061132 *Sep 26, 2001Mar 27, 2003Yu, Mason K.System and method for categorizing, aggregating and analyzing payment transactions dataUS20030069773 *Oct 5, 2001Apr 10, 2003Hladik William J.Performance reportingUS20030069824 *Mar 23, 2001Apr 10, 2003Restaurant Services, Inc. ("RSI")System, method and computer program product for bid proposal processing using a graphical user interface in a supply chain management frameworkUS20030078830 *Oct 22, 2001Apr 24, 2003Wagner Todd R.Real-time collaboration and workflow management for a marketing campaignUS20030093423 *Aug 1, 2001May 15, 2003Larason John ToddDetermining a rating for a collection of documentsUS20030110249 *Jun 8, 2001Jun 12, 2003Bryan BuusSystem and method for monitoring key performance indicators in a businessUS20030144868 *Oct 11, 2002Jul 31, 2003Macintyre James W.System, method, and computer program product for processing and visualization of informationUS20040021695 *Jul 31, 2002Feb 5, 2004Volker SauermannSlider bar scaling in a graphical user interfaceUS20040044678 *Dec 18, 2002Mar 4, 2004International Business Machines CorporationMethod and apparatus for converting legacy programming language data structures to schema definitionsUS20040064293 *Sep 24, 2003Apr 1, 2004Hamilton David B.Method and system for storing and reporting network performance metrics using histogramsUS20040066782 *Sep 22, 2003Apr 8, 2004Nassar Ayman EsamSystem, method and apparatus for sharing and optimizing packet services nodesUS20040117731 *Sep 22, 2003Jun 17, 2004Sergey BlyashovAutomated report building systemUS20040119752 *Sep 8, 2003Jun 24, 2004Joerg BeringerGuided procedure frameworkUS20040135825 *Jan 14, 2003Jul 15, 2004Brosnan Michael J.Apparatus for controlling a screen pointer that distinguishes between ambient light and light from its light sourceUS20040230471 *Feb 19, 2004Nov 18, 2004Putnam Brookes Cyril HenryBusiness intelligence system and methodUS20050049831 *Jan 24, 2003Mar 3, 2005Leica Geosystems AgPerformance monitoring system and methodUS20050065925 *Sep 23, 2003Mar 24, 2005Salesforce.Com, Inc.Query optimization in a multi-tenant database systemUS20050071737 *Sep 30, 2003Mar 31, 2005Cognos IncorporatedBusiness performance presentation user interface and method for presenting business performanceUS20050097438 *Sep 24, 2004May 5, 2005Jacobson Mark D.Method and system for creating a digital document altered in response to at least one eventUS20050097517 *Nov 5, 2003May 5, 2005Hewlett-Packard CompanyMethod and system for adjusting the relative value of system configuration recommendationsUS20050154628 *Apr 23, 2004Jul 14, 2005Illumen, Inc.Automated management of business performance informationUS20050216831 *Mar 29, 2004Sep 29, 2005Grzegorz GuzikKey performance indicator system and methodUS20060010032 *Sep 14, 2005Jan 12, 2006Blake Morrow Partners LlcSystem, method and computer program product for evaluating an asset management business using experiential data, and applications thereofUS20060010164 *Feb 3, 2005Jan 12, 2006Microsoft CorporationCentralized KPI framework systems and methodsUS20060020531 *Jul 21, 2005Jan 26, 2006Veeneman David CRisk return presentation methodUS20060074789 *Jun 29, 2005Apr 6, 2006Thomas CapotostoClosed loop view of asset management informationUS20060085444 *Oct 19, 2004Apr 20, 2006Microsoft CorporationQuery consolidation for retrieving data from an OLAP cubeUS20060089868 *Oct 27, 2004Apr 27, 2006Gordy GrillerSystem, method and computer program product for analyzing and packaging information related to an organizationUS20060095915 *Oct 14, 2005May 4, 2006Gene ClaterSystem and method for process automation and enforcementUS20060111921 *Nov 23, 2004May 25, 2006Hung-Yang ChangMethod and apparatus of on demand business activity management using business performance management loopsUS20060112130 *Nov 23, 2005May 25, 2006Linda LowsonSystem and method for resource managementUS20060136830 *Nov 3, 2005Jun 22, 2006Martlage Aaron ESystem and user interface for creating and presenting formsUS20060161471 *Jan 19, 2005Jul 20, 2006Microsoft CorporationSystem and method for multi-dimensional average-weighted banding status and scoringUS20060161596 *Jan 14, 2005Jul 20, 2006Microsoft CorporationMethod and system for synchronizing multiple user revisions to a balanced scorecardUS20070021992 *Jul 19, 2005Jan 25, 2007Srinivas KonakallaMethod and system for generating a business intelligence system based on individual life cycles within a business processUS20070033129 *Aug 2, 2006Feb 8, 2007Coates Frank JAutomated system and method for monitoring, alerting and confirming resolution of critical business and regulatory metricsUS20070050237 *Aug 30, 2005Mar 1, 2007Microsoft CorporationVisual designer for multi-dimensional business logicUS20070055564 *Jun 21, 2004Mar 8, 2007Fourman Clive MSystem for facilitating management and organisational development processesUS20070055688 *Sep 8, 2005Mar 8, 2007International Business Machines CorporationAutomatic report generationUS20070112607 *Nov 16, 2005May 17, 2007Microsoft CorporationScore-based alerting in business logicUS20070143161 *Dec 21, 2005Jun 21, 2007Microsoft CorporationApplication independent rendering of scorecard metricsUS20070143174 *Dec 21, 2005Jun 21, 2007Microsoft CorporationRepeated inheritance of heterogeneous business metricsUS20070143175 *Dec 21, 2005Jun 21, 2007Microsoft CorporationCentralized model for coordinating update of multiple reportsUS20070156680 *Dec 21, 2005Jul 5, 2007Microsoft CorporationDisconnected authoring of business definitionsUS20070168323 *Jan 3, 2006Jul 19, 2007Microsoft CorporationQuery aggregationUS20070174330 *Jul 13, 2006Jul 26, 2007Zdk Interactive Inc.Mobile report generation for multiple device platformsUS20070239508 *Jun 30, 2006Oct 11, 2007Cognos IncorporatedReport management systemUS20080005064 *Jun 28, 2005Jan 3, 2008Yahoo! Inc.Apparatus and method for content annotation and conditional annotation retrieval in a search contextUS20080059441 *Aug 29, 2007Mar 6, 2008Lockheed Martin CorporationSystem and method for enterprise-wide dashboard reportingUS20080172287 *Jan 17, 2007Jul 17, 2008Ian TienAutomated Domain Determination in Business Logic Applications* Cited by examinerNon-Patent CitationsReference1 *Paul Calame, Ravi Nannapaneni, Scott Peterson, Jay Turpin, and James Yu. "Cockpit: Decision Support Tool for Factory Operations and Supply Chain Management," Intel Technology Journal, Q1, 2000, February 2000* Cited by examinerReferenced byCiting PatentFiling datePublication dateApplicantTitleUS7716571Apr 27, 2006May 11, 2010Microsoft CorporationMultidimensional scorecard header definitionUS7716592Mar 30, 2006May 11, 2010Microsoft CorporationAutomated generation of dashboards for scorecard metrics and subordinate reportingUS7836052 *Nov 16, 2010Microsoft CorporationSelection of attribute combination aggregationsUS7840896Nov 23, 2010Microsoft CorporationDefinition and instantiation of metric based business logic reportsUS8126750Apr 27, 2006Feb 28, 2012Microsoft CorporationConsolidating data source queries for multidimensional scorecardsUS8190992Apr 21, 2006May 29, 2012Microsoft CorporationGrouping and display of logically defined reportsUS8219917 *Jul 26, 2005Jul 10, 2012International Business Machines CorporationBubbling up task severity indicators within a hierarchical tree controlUS8261181Mar 30, 2006Sep 4, 2012Microsoft CorporationMultidimensional metrics-based annotationUS8321805Jan 30, 2007Nov 27, 2012Microsoft CorporationService architecture based metric viewsUS8376755 *Feb 19, 2013Location Inc. Group CorporationSystem for the normalization of school performance statisticsUS8495663Feb 2, 2007Jul 23, 2013Microsoft CorporationReal time collaboration using embedded data visualizationsUS8732603 *Dec 11, 2006May 20, 2014Microsoft CorporationVisual designer for non-linear domain logicUS8762874 *Oct 18, 2011Jun 24, 2014Patrick Pei-Jan HongMethod of quantitative analysisUS8799058 *Dec 16, 2010Aug 5, 2014Hartford Fire Insurance CompanySystem and method for administering an advisory rating systemUS9058307Jan 26, 2007Jun 16, 2015Microsoft Technology Licensing, LlcPresentation generation using scorecard elementsUS9392026Jul 23, 2013Jul 12, 2016Microsoft Technology Licensing, LlcReal time collaboration using embedded data visualizationsUS20070028188 *Jul 26, 2005Feb 1, 2007International Business Machines CorporationBubbling up task severity indicators within a hierarchical tree controlUS20070112956 *Oct 9, 2006May 17, 2007Chapman Matthew PResource optimisation componentUS20070239657 *Mar 28, 2006Oct 11, 2007Microsoft CorporationSelection of attribute combination aggregationsUS20080168376 *Dec 11, 2006Jul 10, 2008Microsoft CorporationVisual designer for non-linear domain logicUS20090099907 *Oct 14, 2008Apr 16, 2009Oculus Technologies CorporationPerformance managementUS20090280465 *Nov 12, 2009Andrew SchillerSystem for the normalization of school performance statisticsUS20120096382 *Apr 19, 2012Patrick Pei-Jan HongMethod of quantitative analysisUS20120158465 *Jun 21, 2012Hartford Fire Insurance CompanySystem and method for administering an advisory rating systemUS20120166239 *Feb 27, 2012Jun 28, 2012Accenture Global Services LimitedBalanced Scorecard And Reporting ToolUS20120254056 *Mar 31, 2011Oct 4, 2012Blackboard Inc.Institutional financial aid analysisUS20130110640 *May 2, 2013Connectedu, Inc.Apparatus and Methods for an Application Process and Data AnalysisUS20140244343 *Feb 22, 2013Aug 28, 2014Bank Of America CorporationMetric management tool for determining organizational healthUS20150347933 *May 27, 2014Dec 3, 2015International Business Machines CorporationBusiness forecasting using predictive metadataUS20150348066 *Mar 5, 2015Dec 3, 2015International Business Machines CorporationBusiness forecasting using predictive metadataUSD753167 *Jun 27, 2014Apr 5, 2016Opower, Inc.Display screen of a communications terminal with graphical user interfaceUSD760261 *Jun 27, 2014Jun 28, 2016Opower, Inc.Display screen of a communications terminal with graphical user interfaceWO2016032531A1 *Aug 29, 2014Mar 3, 2016Hewlett Packard Enterprise Development LpImprovement message based on element score* Cited by examinerClassifications U.S. Classification715/764, 702/179, 702/182International ClassificationG06F15/00, G06F3/048, G06F17/18Cooperative ClassificationG06Q30/02European ClassificationG06Q30/02Legal EventsDateCodeEventDescriptionFeb 2, 2007ASAssignmentOwner name: MICROSOFT CORPORATION, WASHINGTONFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:TIEN, IAN;HULEN, COREY J.;LIM, CHEN-I;REEL/FRAME:018843/0386;SIGNING DATES FROM 20070117 TO 20070123Jan 15, 2015ASAssignmentOwner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTONFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034766/0509Effective date: 20141014RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services