Source: https://patents.google.com/patent/US7072899B2/en
Timestamp: 2020-08-14 23:52:49
Document Index: 224518131

Matched Legal Cases: ['Application No. 60', 'arts 602', 'arts 602', 'arts 602', 'arts 602', 'arts 602']

US7072899B2 - Automatic monitoring and statistical analysis of dynamic process metrics to expose meaningful changes - Google Patents
Automatic monitoring and statistical analysis of dynamic process metrics to expose meaningful changes Download PDF
US7072899B2
US7072899B2 US11/015,719 US1571904A US7072899B2 US 7072899 B2 US7072899 B2 US 7072899B2 US 1571904 A US1571904 A US 1571904A US 7072899 B2 US7072899 B2 US 7072899B2
US11/015,719
US20050138020A1 (en
2003-12-19 Priority to US53134703P priority Critical
2004-12-17 Application filed by Proclarity Corp filed Critical Proclarity Corp
2004-12-17 Priority to US11/015,719 priority patent/US7072899B2/en
2004-12-17 Assigned to PROCLARITY, INC. reassignment PROCLARITY, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: LOKKEN, ROBERT C.
2005-06-23 Publication of US20050138020A1 publication Critical patent/US20050138020A1/en
2006-07-04 Publication of US7072899B2 publication Critical patent/US7072899B2/en
2006-07-07 Assigned to PROCLARITY CORPORATION reassignment PROCLARITY CORPORATION CORRECTED ASSIGNMENT TO CORRECT ASSIGNEE'S NAME ON ORIGNAL COVER SHEET ON REEL 016107, FRAME 0858 Assignors: LOKKEN, ROBERT C.
2010-02-02 Assigned to MICROSOFT CORPORATION reassignment MICROSOFT CORPORATION MERGER (SEE DOCUMENT FOR DETAILS). Assignors: PROCLARITY CORPORATION
238000007619 statistical methods Methods 0.000 title description 4
238000003070 Statistical process control Methods 0.000 claims abstract description 27
238000002716 delivery method Methods 0.000 claims description 6
A selection module allows a user to specify at least one measure to be monitored in at least one dimension of a dimensional hierarchy. A control limit calculator extracts, for each specified measure and for each specified dimension, a time series from a multidimensional database for the specified measure in the specified dimension and automatically calculates one or more control limits for the specified measure in the specified dimension based on the extracted time series using a Statistical Process Control (SPC) technique. Thereafter, a monitoring module monitors newly acquired data including each specified measure in each specified dimension for an out-of-limits condition based on one or more automatically-calculated control limits. An alert module triggers an alert in response to an out-of-limits condition being detected.
This application claims the benefit of U.S. Provisional Application No. 60/531,347, filed Dec. 19, 2003, with inventor Robert C. Lokken, which application is incorporated herein by reference in its entirety.
The present invention relates generally to the field of data processing. More specifically, the present invention relates to techniques for analyzing multidimensional data.
Business people tend to either manage processes or projects. Processes lend themselves to repeatable, systematic measures that allow the manager of the process to determine if that process is performing as usual or if something about the process has changed. At the highest level, a business itself is a “process”—taking input resources (time, capital, people, materials, etc.) and producing an output (sales and profits).
Increasingly, business processes are being represented in multidimensional databases. Conceptually, a multidimensional database uses the idea of a data cube to represent the dimensions of data available to a user. For example, “sales” could be viewed in the dimensions of product model, geography, time, or some additional dimension. In this case, “sales” is known as the measure attribute of the data cube, and the other dimensions are seen as feature attributes. Additionally, a database creator can define hierarchies and levels within a dimension (for example, state and city levels within a regional hierarchy).
FIG. 1 is a schema for a multidimensional database.
Statistical Process Control (SPC) techniques are applied to business metrics, represented as measures in multidimensional data. These SPC techniques allow the system to filter out the normal day-to-day random variation in the metrics and test for underlying, fundamental changes in a business process. The system applies these techniques to metrics automatically, and determines the correct threshold that would determine the difference between normal random variation and fundamental changes. Thus, alerts may be triggered to notify the user that something has changed, without that user ever having to determine the specific thresholds. Moreover, users can monitor hundreds of metrics across dozens of processes for changing conditions without having to set specific alerts.
FIG. 1 is a schema for an On-Line Analytical Processing (OLAP) database 100. OLAP refers to a type of database that facilitates analysis of data or measures 102 that has been aggregated into various categories or dimensions 104. For example, in the database 100 of FIG. 1, the measures 102 may include “Sales,” “Sales Growth,” “Gross Margin,” “Return Ratio,” “Average discount,” etc. The dimensions 104 may include “Time,” “Products,” “Sales Teams,” “Customers,” etc. The dimensions 104, themselves, may include further dimensions 104, often referred to as levels. For instance, the “Customers” dimension 104 may include “Region,” “Country,” “State,” and “Customer” levels.
As shown in FIG. 2, an OLAP database 100 with three dimensions 104 may be conceptualized as a cube, with each axis representing a dimension 104 of a business (e.g., “Time,” “Product,” “Customers”) and each cell representing a measure 102 (e.g., “Sales Value”). Business processes may be easily represented within OLAP databases 100, which fuels their popularity.
FIG. 3 illustrates the above problem with a graph of a “Gross Margin” measure 102 over a “Months” dimension 104. A manager might, for example, manually set an alert threshold 300 at +/−6%. However, this will result in a number of false alarms 302 for values that are within the historically normal variation for the metric. Moreover, given the complexities of multidimensional models, a manager may need to configure hundreds or thousands of alerts and for each must determine the correct threshold value or expression.
The “metric” to be monitored corresponds to a dimension 104 in the OLAP database 100. The “area and depth” to be monitored represents one or more dimensions 104 from the dimensional hierarchy illustrated in FIG. 1. In other embodiments, the user could have specified various levels within the products dimension 104, such as the “PC product line” or the “XC-15 product.” The “frequency of detection” generally refers to one of the levels of the “Time” dimension 104, e.g., yearly, quarterly, monthly, daily, etc. The “tolerance” may be expressed as a confidence value, e.g., “95% confident that an alert is not a false alarm,” or as some other indicator of how “tightly” the automatically-generated control limits 412 should be construed by the monitoring module 416.
Products (all levels)
Sales teams (all levels)
Customers Gust the “all,” region, country, and state levels).
Frequency of detection: Daily
1. Extract a time series (measures 102 over a time dimension 104) from the OLAP database 100;
2. Automatically calculate control limits 412 for the selected measure 102 in the selected dimension 104;
3. Store the control limits 412 in control limit storage 414.
2. Monitor, for newly acquired data, each selected measure 102 across each selected dimension 104 for an out-of-limits condition based on the stored control limits.
3. If an out-of-limit condition is detected, trigger an alert.
4. Group alerts by recipient and deliver via specified method(s).
FIG. 6 illustrates one method for calculating control limits 412 using Statistical Process Control (SPC) techniques. SPC uses statistical methods to measure and analyze the variation in processes. Most often used in manufacturing, the intent of SPC is to monitor process quality and maintain processes within fixed tolerances.
SPC relies on a number of different types of control charts 602 that are applicable to different types of data, which, in turn, result in different calculations. There are at least four major types (with dozens of variations) of control charts 602. The first type, the x-chart. (and related x-bar, r, and s-charts), is depicted in FIG. 6. The x-chart is designed to be used primarily with “variables” data, which are usually measurements, such as the length of an object, the time it takes to complete a process, or the number of objects produced per period.
In addition to the x-chart, there are three specialized types of control charts 602, i.e., the p-chart, c-chart, and u-chart. These charts are used when the data being measured meet certain conditions (or attributes). For example, the p-chart is used with “binomial” data. P-charts are used for results of go-no go tests, such as percent of work orders completed within budgeted cost. In this case, a work order is either completed within budget or not (“go-no go”). P-charts have the advantage of taking into account the sample size (the number of work orders completed) and accounting for the high level of random fluctuations when the sample size is small (very few work orders completed).
The c-chart is used for “Poisson” processes. These are used with random arrival models, or when “counting” attributes. This type of chart, for example, can monitor the number of “defects” in each of many equal samples (constant sample size). Occurrence Reporting data (number of reports per month) empirically appear to fit the “Poisson” model, and the c-chart is recommended when charting occurrence report counts.
The u-chart is used when counting “defects” per sample when the sample size varies for each “inspection.” The number of cases is counted for fixed time intervals, such as monthly or yearly, but the sample size (number of man-hours worked during each time interval) changes.
In one embodiment, the control limit calculator 406 selects the appropriate “chart” (and corresponding calculations) based on the type of measures 102 being analyzed. This may be determined automatically from the data or may be specified by the user. Of course, a wide variety of other charts 602 and associated SPC calculations may be used as known to those of skill in the art.
UCL={overscore (X)}+Zσ Eq. 1
LCL={overscore (X)}−Zσ Eq. 2
X _ = ∑ i = 1 n ⁢ X i n Eq . ⁢ 3
where Xi is a measure 102 in the time series 600, and
σ = ∑ ( X i - X _ ) 2 n - 1 Eq . ⁢ 4
The second method is to multiply the average range value for pairs of data points by 0.887.
σ = p _ ⁡ ( 1 - p _ ) N Eq . ⁢ 6 where ⁢ ⁢ p _ = total ⁢ # ⁢ successes total ⁢ # ⁢ trials ⁢ ⁢ and ⁢ ⁢ N = # ⁢ ⁢ Trials ⁢ ⁢ in ⁢ ⁢ this ⁢ ⁢ time ⁢ ⁢ period .
For a u-chart, the standard deviation is likewise calculated for each datum value. The formula is:
σ = u _ N Std . sample . size Eq . ⁢ 7 where ⁢ ⁢ u = ( total . defects total . sample . size ) ⁢ ( std . sample . size ) ⁢ ⁢ and N = sample ⁢ ⁢ size ⁢ ⁢ for ⁢ ⁢ current ⁢ ⁢ time period ⁢ ⁢ ( i . e . , lot ⁢ ⁢ sample ⁢ ⁢ size ) .
Other ways for calculating the standard deviation may be used for different types of control charts 602 as will be known to those of skill in the art.
Individual points above the Upper Control Limit.
Individual points below the Lower Control Limit.
Seven points in a row all above average or all below average.
Seven points in a row increasing.
Seven points in a row decreasing.
Ten out of eleven points in a row all above average or all below average.
Cycles or other non-random patterns in the data.
Two out of three points in a row outside of two standard deviations above the average, or two out of three points in a row outside of two standard deviations below the average.
Four out of five points in a row outside of one standard deviation above the average, or four out of five points in a row outside of one standard deviation below the average.
After recalculating the average and control limits 412, further points may have become outliers. In severe cases, this can lead to an endless series of throwing out “outliers” until very little data are left. In such cases, it may be best to revert back to the original average and control limits 412.
In some cases, a number of points (e.g., seven or more) may be found in the time series 600 that are all increasing/decreasing. This condition indicates that a continuous change (ramp) may be occurring in the data. In one embodiment, the control limit calculator 406 may add the average and control limits. This Statistical Process Control (SPC) technique verifies that the data are undergoing continuous, significant change. SPC can be thought of as a formal “test” for the existence of significant change(s). In this case, the test shows that significant change is occurring.
If the regions become too short, the control limit calculator 406 may simply leave one longer time interval region with points going from below the LCL to above the UCL (or vice-versa). Again, SPC can be thought of as “testing” for the existence of the trend.
UCL = X _ + Z ⁢ ⁢ σ where X _ = ∑ i = 1 n ⁢ X i n σ = ∑ ( X i - X _ ) 2 n - 1
Xi is a measure in the time series,
LCL = X _ - Z ⁢ ⁢ σ where X _ = ∑ i = 1 n ⁢ X i n σ = ∑ ( X i - X _ ) 2 n - 1
US11/015,719 2003-12-19 2004-12-17 Automatic monitoring and statistical analysis of dynamic process metrics to expose meaningful changes Expired - Fee Related US7072899B2 (en)
US53134703P true 2003-12-19 2003-12-19
US11/015,719 US7072899B2 (en) 2003-12-19 2004-12-17 Automatic monitoring and statistical analysis of dynamic process metrics to expose meaningful changes
CN2004800379686A CN1894652B (en) 2003-12-19 2004-12-17 Automatic monitoring and statistical analysis of dynamic process metrics to expose meaningful changes
DK04814830T DK1695192T3 (en) 2003-12-19 2004-12-17 Automatic monitoring and statistical analysis of dynamic process metrics to expose meaningful changes
AT04814830T AT404917T (en) 2003-12-19 2004-12-17 Automatic monitoring and statistical analysis of dynamic process methods for the disclosure of significant changes
KR1020067014512A KR100841876B1 (en) 2003-12-19 2004-12-17 Automatic monitoring and statistical analysis of dynamic process metrics to expose meaningful changes
JP2006545537A JP4541364B2 (en) 2003-12-19 2004-12-17 Statistical analysis of automatic monitoring and dynamic process metrics to reveal meaningful variations
DE602004015836T DE602004015836D1 (en) 2003-12-19 2004-12-17 Automatic monitoring and statistical analysis of dynamic process methods for the disclosure of significant changes
EP04814830A EP1695192B1 (en) 2003-12-19 2004-12-17 Automatic monitoring and statistical analysis of dynamic process metrics to expose meaningful changes
PCT/US2004/042692 WO2005060687A2 (en) 2003-12-19 2004-12-17 Automatic monitoring and statistical analysis of dynamic process metrics to expose meaningful changes
US20050138020A1 US20050138020A1 (en) 2005-06-23
US7072899B2 true US7072899B2 (en) 2006-07-04
ID=34680956
US11/015,719 Expired - Fee Related US7072899B2 (en) 2003-12-19 2004-12-17 Automatic monitoring and statistical analysis of dynamic process metrics to expose meaningful changes
US (1) US7072899B2 (en)
EP (1) EP1695192B1 (en)
JP (1) JP4541364B2 (en)
KR (1) KR100841876B1 (en)
CN (1) CN1894652B (en)
AT (1) AT404917T (en)
DE (1) DE602004015836D1 (en)
DK (1) DK1695192T3 (en)
WO (1) WO2005060687A2 (en)
US10599979B2 (en) 2015-09-23 2020-03-24 International Business Machines Corporation Candidate visualization techniques for use with genetic algorithms
US10698393B2 (en) * 2014-12-05 2020-06-30 Safran Aircraft Engines Method of manufacturing parts based on analysis of statistical indicators in a situation of diminished control
KR100937512B1 (en) * 2007-09-20 2010-01-19 한양대학교 산학협력단 Statistical Process Control Method Using the Distribution of the Coefficient of Variation and Noncentral t Distribution
KR101044348B1 (en) * 2009-02-20 2011-06-29 한양대학교 산학협력단 Method and Apparatus for Process Control Using the Coefficient of Variation
EP2701020A1 (en) * 2012-08-22 2014-02-26 Siemens Aktiengesellschaft Monitoring of the initial equipment of a technical system for fabricating a product
KR101896157B1 (en) * 2016-11-23 2018-09-07 한양대학교 산학협력단 Method for controlling sensor based on statistical process control
US20020198919A1 (en) * 2001-06-14 2002-12-26 International Business Machines Corporation Employing a combined function for exception exploration in multidimensional data
JPH04222252A (en) * 1990-12-21 1992-08-12 Tsudakoma Corp Detection of abnormal machine and examination of set value
JPH07244694A (en) * 1994-03-04 1995-09-19 Fujitsu Lsi Technol Kk Quality control device
JP3355849B2 (en) * 1995-01-23 2002-12-09 株式会社デンソー Manufacturing process quality abnormality treatment system and quality control value updating method
CN1099060C (en) * 1999-04-14 2003-01-15 袁璞 Universal multi-variable quantity model pre-estimating coordinating control method
EP1195694A3 (en) * 2000-10-06 2006-01-11 International Business Machines Corporation Automatic determination of OLAP Cube dimensions
JP2002202806A (en) * 2000-12-28 2002-07-19 Mitsubishi Electric Corp Process control system and manufacturing method for article
JP2003316797A (en) * 2002-04-26 2003-11-07 Hitachi Ltd Information analysis system and method
2004-12-17 EP EP04814830A patent/EP1695192B1/en not_active Expired - Fee Related
2004-12-17 WO PCT/US2004/042692 patent/WO2005060687A2/en active Application Filing
2004-12-17 US US11/015,719 patent/US7072899B2/en not_active Expired - Fee Related
2004-12-17 CN CN2004800379686A patent/CN1894652B/en not_active IP Right Cessation
2004-12-17 DE DE602004015836T patent/DE602004015836D1/en active Active
2004-12-17 KR KR1020067014512A patent/KR100841876B1/en not_active IP Right Cessation
2004-12-17 AT AT04814830T patent/AT404917T/en not_active IP Right Cessation
2004-12-17 DK DK04814830T patent/DK1695192T3/en active
2004-12-17 JP JP2006545537A patent/JP4541364B2/en not_active Expired - Fee Related
US10467203B2 (en) * 2014-09-30 2019-11-05 International Business Machines Corporation Data de-duplication
EP1695192A2 (en) 2006-08-30
DE602004015836D1 (en) 2008-09-25
WO2005060687A2 (en) 2005-07-07
CN1894652B (en) 2010-06-16
WO2005060687A3 (en) 2006-03-30
JP4541364B2 (en) 2010-09-08
EP1695192A4 (en) 2006-12-27
CN1894652A (en) 2007-01-10
KR100841876B1 (en) 2008-06-27
EP1695192B1 (en) 2008-08-13
DK1695192T3 (en) 2008-11-24
US20050138020A1 (en) 2005-06-23
KR20070006696A (en) 2007-01-11
AT404917T (en) 2008-08-15
JP2007515020A (en) 2007-06-07
US10628292B2 (en) 2020-04-21 Methods and systems for predicting estimation of project factors in software development
US20150067160A1 (en) 2015-03-05 Systems and methods of specifying service level criteria
JP6613329B2 (en) 2019-11-27 Computer mounting method, computer system, and computer apparatus
Klein et al. 2009 Representing data quality in sensor data streaming environments
Candelon et al. 2011 Backtesting value-at-risk: a GMM duration-based test
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LOKKEN, ROBERT C.;REEL/FRAME:016107/0858
Free format text: CORRECTED ASSIGNMENT TO CORRECT ASSIGNEE'S NAME ON ORIGNAL COVER SHEET ON REEL 016107, FRAME 0858;ASSIGNOR:LOKKEN, ROBERT C.;REEL/FRAME:017968/0217