Source: http://www.google.de/patents/US9208209
Timestamp: 2018-01-19 07:47:41
Document Index: 64528975

Matched Legal Cases: ['art.\n1', 'art.\n6', 'art.\n10', '§119', 'Application No. 62', 'art 800', 'art 800', 'art 800', 'art 800', 'art 800', 'art 800', 'art 800', 'art 800', 'art 800', 'art 800', 'art 800', 'art 800']

Patent US9208209 - Techniques for monitoring transformation techniques using control charts - Google Patentsuche
Techniques for applying transformation techniques and using transformation outputs to implement remedial actions are included. A system may include a receiver that may receive time series data including data points and one or more specifications for a transformation technique applicable to the time series...http://www.google.de/patents/US9208209?utm_source=gb-gplus-sharePatent US9208209 - Techniques for monitoring transformation techniques using control charts
Veröffentlichungsnummer US9208209 B1
Anmeldenummer US 14/668,854
Veröffentlichungsdatum 8. Dez. 2015
Eingetragen 25. März 2015
Prioritätsdatum 2. Okt. 2014
Veröffentlichungsnummer 14668854, 668854, US 9208209 B1, US 9208209B1, US-B1-9208209, US9208209 B1, US9208209B1
Erfinder Joseph H. Katz
Patentzitate (166), Nichtpatentzitate (65), Referenziert von (1), Klassifizierungen (5), Juristische Ereignisse (1)
Techniques for monitoring transformation techniques using control charts
US 9208209 B1
Techniques for applying transformation techniques and using transformation outputs to implement remedial actions are included. A system may include a receiver that may receive time series data including data points and one or more specifications for a transformation technique applicable to the time series data. The system may include a transformation engine that may apply the transformation technique to the time series data to produce outputs. The system may include a control engine that may determine a set of control limits using, for example, a control chart and a set of residual values. The system may include an anomaly detection engine that may generate the set of residual values using the current data points and the outputs. The anomaly detection engine may further identify one or more anomalies of the set of residual values based on the set of control limits and the control chart.
1. A forecasting system for applying forecasts and using outputs to implement remedial actions, comprising:
time series data including a plurality of current data points; and
a forecast model for a forecast applicable to the time series data;
a transformation engine that applies the forecast model to the time series data to produce a plurality of outputs;
a control engine that determines a set of control limits using a control chart and a set of residual values;
an anomaly detection engine that:
generates the set of residual values using the plurality of current data points and the plurality of outputs, wherein each residual value in the set of residual values indicates a difference between a current data point of the plurality of current data points and an output of the plurality of outputs; and
identifies one or more anomalies of the set of residual values based on the set of control limits and the control chart; and
a remedial-action engine that:
determines a remedial action list including a plurality of types of remedial actions;
assigns a prioritization to the types of remedial actions in the remedial action list, wherein prioritizations are based on computational costs of the remedial actions;
determines a remedial action for the forecast model based on the prioritizations of the types of remedial actions in the remedial action list and the one or more anomalies; and
perform the remedial action related to the forecast model.
2. The system of claim 1, wherein an anomaly indicates that a residual value of the set of residual values occurs outside the set of control limits.
3. The system of claim 1, wherein an anomaly indicates that a subset of sequentially residual values incrementally increase.
4. The system of claim 1, wherein identifying one or more anomalies includes determining a vintage date,
wherein a vintage date indicates a date at which the forecast model was selected or modified, and
wherein the one or more anomalies are identified for dates occurring on or after the vintage date.
5. The system of claim 1, wherein identifying one or more anomalies includes analyzing the set of residual values using a set of rules, and
wherein analyzing the set of residual values using the set of rules includes determining a non-random condition in a control chart.
6. The system of claim 1, wherein determining a remedial action for the forecast model further comprises:
determining a classification for each of the one or more anomalies,
wherein a classification is associated with a remedial action;
sorting the classifications using a predetermined prioritization scheme; and
retaining a classification of the sorted classifications,
wherein the determined remedial action is associated with the retained classification.
7. The system of claim 1, wherein remedial actions include estimating parameters for the forecast model, applying the forecast model to the time series data to produce a plurality of outputs, or selecting a different forecast model.
8. The system of claim 7, wherein identifying one or more anomalies includes determining that the tracking signal equation has produced an output that exceeds one or more error threshold values.
9. The system of claim 1, wherein identifying one or more anomalies includes analyzing the set of residual values using a tracking signal equation,
wherein the tracking signal equation uses a sum of the set of residual values and a mean absolute deviation value, and
wherein analyzing the set of residual values using the tracking signal equation includes determining a non-random condition in a control chart.
10. The system of claim 8, wherein identifying one or more anomalies includes analyzing the set of residual values using the tracking signal equation further includes converting a product of the tracking signal equation using a standard deviation value.
11. A non-transitory computer program product, tangible embodied in a non-transitory machine readable storage medium, including instructions operable to cause a data processing apparatus to:
receive time series data including a plurality of current data points;
receive a forecast model for a forecast applicable to the time series data;
apply the forecast model to the time series data to produce a plurality of outputs;
generate a set of residual values using the plurality of current data points and the plurality of outputs, wherein a residual value indicates a difference between a current data point of the plurality of data points and an output of the plurality of outputs;
determine a set of control limits using a control chart and the set of residual values;
identify one or more anomalies of the set of residual values, wherein anomalies are identified using the set of control limits and the control chart;
determine a remedial action list including a plurality of types of remedial actions;
assign a prioritization to the types of remedial actions in the remedial action list, wherein prioritizations are based on computational costs of the remedial actions;
determine a remedial action for the forecast model based on the prioritizations of the types of remedial actions in the remedial action list and the one or more anomalies; and
12. The computer program product of claim 11, wherein an anomaly indicates that a residual value of the set of residual values occurs outside the set of control limits.
13. The computer program product of claim 11, wherein an anomaly indicates that a subset of sequentially residual values incrementally increase.
14. The computer program product of claim 11, wherein identifying one or more anomalies includes determining a vintage date,
15. The computer program product of claim 11, wherein identifying one or more anomalies includes analyzing the set of residual values using a set of rules, and
16. The computer program product of claim 11, wherein instructions for determining a remedial action includes further instructions operable to cause a data processing apparatus to:
determine a classification for each of the one or more anomalies,
sort the classifications using a predetermined prioritization scheme; and
retain a classification of the sorted classifications,
17. The computer program product of claim 11, wherein remedial actions estimating parameters for the forecast model, applying the forecast model to the time series data to produce a plurality of outputs, or selecting a different forecast model.
18. The computer program product of claim 11, wherein identifying one or more anomalies includes analyzing the set of residual values using a tracking signal equation,
19. The computer program product of claim 18, wherein identifying one or more anomalies includes analyzing the set of residual values using the tracking signal equation further includes converting a product of the tracking signal equation using a standard deviation value.
20. The computer program product of claim 18, wherein identifying one or more anomalies includes determining that the tracking signal equation has produced an output that exceeds one or more error threshold values.
21. A computer-implemented method for applying forecast models and using outputs to implement remedial actions, comprising:
receiving time series data including a plurality of current data points;
receiving a forecast model for a forecast applicable to the time series data;
applying the forecast model to the time series data to produce a plurality of outputs;
generating, using an anomaly detection engine, a set of residual values using the plurality of current data points and the plurality of outputs, wherein a residual value indicates a difference between a current data point of the plurality of data points and an output of the plurality of outputs;
determining a set of control limits using a control chart and the set of residual values;
identifying, using the anomaly detection engine, one or more anomalies of the set of residual values, wherein anomalies are identified using the set of control limits and the control chart;
determining, using a remedial-action engine, a remedial action list including a plurality of types of remedial actions;
assigning, using the remedial-action engine, a prioritization to the types of remedial actions in the remedial action list, wherein prioritizations are based on computational costs of the remedial actions;
determining, using the remedial-action engine, a remedial action for the forecast model based on the prioritizations of the types of remedial actions in the remedial action list and the one or more anomalies; and
performing, using the remedial-action engine, the remedial action related to the forecast model.
22. The computer-implemented method of claim 21, wherein an anomaly indicates that a residual value of the set of residual values occurs outside the set of control limits.
23. The computer-implemented method of claim 21, wherein an anomaly indicates that a subset of sequentially residual values incrementally increase.
24. The computer-implemented method of claim 21, wherein identifying one or more anomalies includes determining a vintage date,
25. The computer-implemented method of claim 21, wherein identifying one or more anomalies includes analyzing the set of residual values using a set of rules, and
26. The computer-implemented method of claim 21, wherein determining a remedial action further comprises:
27. The computer-implemented method of claim 21, wherein remedial actions include estimating parameters for the forecast model, applying the forecast model to the time series data to produce a plurality of outputs, or selecting a different forecast model.
28. The computer-implemented method of claim 21, wherein identifying one or more anomalies includes analyzing the set of residual values using a tracking signal equation,
29. The computer-implemented method of claim 28, wherein identifying one or more anomalies includes analyzing the set of residual values using the tracking signal equation further includes converting a product of the tracking signal equation using a standard deviation value.
30. The computer-implemented method of claim 28, wherein identifying one or more anomalies includes determining that the tracking signal equation has produced an output that exceeds one or more error threshold values.
The present disclosure claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Application No. 62/058,848, filed Oct. 2, 2014, the entirety of which is incorporated herein by reference.
The present disclosure generally relates to systems and methods for automated transformation technique monitoring and process control methodology using control charts.
Transformation technique specifications are a vital tool in determining subsequent data points from historical time series data. Maintaining a large number of these specifications may present computational challenges. For example, as further data points are received, remedial actions (e.g., specification selection, specification re-parameterization, and generation of output) related to these specifications may occur at inappropriate times. For example, performing a remedial action with respect to the specification every time a data point is received, is impractical and computationally extensive. As a further example, performing a remedial action with respect to the specification at periodic intervals may result in these remedial actions being unnecessarily performed.
In accordance with the teachings provided herein, systems and methods for applying transformation techniques and using transformation outputs to implement remedial actions are provided.
For example, a system is provided for applying transformation techniques and using transformation outputs to implement remedial actions. The system may include a receiver that receives time series data including a plurality of current data points and one or more specifications for a transformation technique applicable to the time series data. The system may further include a transformation engine that applies the transformation technique to the time series data to produce a plurality of outputs. The system may further include a control engine that determines a set of control limits using a control chart and a set of residual values. The system may further include an anomaly detection engine that generates the set of residual values using the plurality of current data points and the plurality of outputs, wherein each residual value in the set of residual values indicates a difference between a current data point of the plurality of current data points and an output of the plurality of outputs. The anomaly detection engine may additionally identify one or more anomalies of the set of residual values based on the set of control limits and the control chart. The system may further include a remedial-action engine that determines a remedial action for the transformation technique based on the one or more anomalies and performs the remedial action related to the one or more specifications for the transformation technique.
In another example, a non-transitory computer program product, tangibly embodied in a non-transitory machine readable storage medium is provided, the non-transitory computer program product including instructions operable to receive time series data including a plurality of current data points. The computer program product may include further instructions operable to receive one or more specifications for a transformation technique applicable to the time series data. The computer program product may include further instructions operable to apply the transformation technique to the time series data to produce a plurality of outputs. The computer program product may include further instructions operable to generate a set of residual values using the plurality of current data points and the plurality of outputs, wherein a residual value indicates a difference between a current data point of the plurality of data points and an output of the plurality of outputs. The computer program product may include further instructions operable to determine a set of control limits using a control chart and the set of residual values. The computer program product may include further instructions operable to identify one or more anomalies of the set of residual values, wherein anomalies are identified using the set of control limits and the control chart. The computer program product may include further instructions operable to determine a remedial action for the transformation technique based on the one or more anomalies. The computer program product may include further instructions operable to perform the remedial action related to the one or more specifications for the transformation technique.
In still one further example, a computer-implemented method is provided for applying transformation techniques and using transformation outputs to implement remedial actions. The method may include receiving time series data including a plurality of current data points. The computer-implemented method may further include receiving one or more specifications for a transformation technique applicable to the time series data. The computer-implemented method may further include applying the transformation technique to the time series data to produce a plurality of outputs. The computer-implemented method may further include generating a set of residual values using the plurality of current data points and the plurality of outputs, wherein a residual value indicates a difference between a current data point of the plurality of data points and an output of the plurality of outputs. The method may further include determining a set of control limits using a control chart and the set of residual values The method may further include identifying one or more anomalies of the set of residual values, wherein anomalies are identified using the set of control limits and the control chart. The method may further include determining a remedial action for the transformation technique based on the one or more anomalies. The method may further include performing the remedial action related to the one or more specifications for the transformation technique.
FIG. 1 illustrates a block diagram of an example of a computer-implemented environment for applying transformation techniques and using transformation outputs to implement remedial actions.
FIG. 2 illustrates a block diagram of an example of a processing system of FIG. 1 for applying transformation techniques and using transformation outputs to implement remedial actions.
FIG. 3 illustrates a block diagram of an example transformation technique specification development process.
FIG. 4 illustrates an example architecture of the transformation technique monitoring engine, in accordance with at least one embodiment.
FIG. 5 illustrates an example of a flow diagram for determining control limits for a set of residuals using a Shewhart control chart.
FIG. 6 illustrates an example of a flow diagram for processing a set of residuals using a tracking signal.
FIG. 7 illustrates an example of a flow diagram for performing remedial actions on a number of transformation technique specifications, in accordance with one embodiment.
FIG. 8 illustrates an example Shewhart control chart including a plot of a set of residuals.
FIG. 9 illustrates an example of a set of Western Electric tests for determining exceptions in a set of residuals using a control chart.
FIG. 10 illustrates an example of a set of custom tests for determining one or more anomalies in a set of residuals using a Shewhart control chart.
FIG. 11 illustrates an example of a set of custom tests for determining anomalies in a set of residuals using a tracking signal and a Shewhart control chart.
FIG. 12 illustrates an example of a flow diagram for applying transformation techniques and using transformation outputs to implement remedial actions.
Certain aspects of the disclosed subject matter relate to system and methods for automated transformation technique monitoring and process control methodology using control charts. As used herein, a “transformation technique” is intended to refer to forecasting a future data point from historical time series data. Forecasting, or forecast generation, is intended to include the process of making statements about events whose actual outcomes have not yet been observed. Examples of “a transformation technique specification,” may include a statistical and/or predictive model such as, a generalized linear model, logistic regression model, ordinary least square regression model, classification model, regression tree model, multivariate adaptive regression spline model, and the like. A “control chart” (e.g., a Shewhart control chart) is intended to refer to a graph used to study how a process changes over time. Data points may be plotted in chronological order on a control chart. A control chart can include a central line for the average, at least one upper control limit and at least one lower control limit, though multiple upper and lower control limits may be utilized. The control limits may be used to determine whether a process is in a state of statistical control. A “tracking signal,” as used herein, is a measurement of error that is used to determine whether bias is present in a transformation technique specification. The present disclosure relates to process control methods for automated monitoring of transformation technique specifications using control charts to detect scenarios in which performance of a remedial action may be appropriate.
Predictive modeling is a process used in predictive analytics to create a transformation technique specification (e.g., a predictive model). Predictive analytics is an area of data mining concerned with determining subsequent data points in a time series by applying a transformation technique according to a transformation technique specification. A transformation technique specification may be made up of a number of predictors, which are variable factors that are likely to influence subsequent data points. In predictive modeling, data is collected for the relevant predictors, a transformation technique specification is formulated, predictions are made and the specification is validated (or revised) as additional data becomes available. The specification may employ a simple linear equation or a complex neural network, for example. A “time series,” as used herein, is intended to refer to a sequence of data points, typically consisting of successive measurements made over a time interval. References to a “time series” is intended to refer to one or more individual time series unless otherwise specified.
Some transformation technique software tools and technologies may include capabilities for automated specification development, specification selection/comparison, parameter optimization, hierarchical reconciliation, user overrides, exception handling, accuracy determinations, and output generation. Specification families considered may typically include, for example, Exponential Smoothing Models (ESM), Auto-Regressive Integrated Moving Average Models either with or without exogenous variables (ARIMA[X]), Unobserved Components Models (UCM), and Intermittent Demand Models (IDM). Custom or user-defined specifications can be considered as well.
Once specifications are selected and output data points are generated, the resultant specifications may become inaccurate as new data points are added to the historical time series data. Some transformation techniques enable the user to specify a number of remedial actions be performed on the specification(s). For example, a user may cause different a set of specification(s) to be diagnosed and selected (SELECT). Additionally, or alternatively, the user may cause a fit/parameterization (FIT) of a set of current specification(s). Additionally, or alternatively, the user may cause a set of output data points to be updated using the currently selected specification(s) (GENERATE). Current techniques require that each time series must be treated in the same manner. Thus, using current techniques, the specified remedial action (e.g., SELECT, FIT, or GENERATE) is applied to each time series.
Executing such a transformation technique specification development and selection process each time a new data point is added to a time series is impractical and inadvisable as it results in a lack of specification stability and may cause output to fluctuate, in some cases, dramatically. To circumvent this issue, current techniques typically redevelop and reselect specifications periodically. As new data points are added to the time series, such specifications may become inaccurate. Thus, aspects of the current disclosure for managing specifications related to transformation techniques are more efficient, effective and/or avoid undesirable side effects or other drawbacks with respect to at least one significant use case.
FIG. 1 illustrates a block diagram of an example of a computer-implemented environment 100 for applying transformation techniques and using transformation outputs to implement remedial actions. The users 102 can interact with a system 104 hosted on one or more servers 106 through one or more networks 108. The system 104 can contain software operations or routines. The users 102 can interact with the system 104 through a number of ways, such as over the networks 108. The servers 106, accessible through the networks 108, can host the system 104. The system 104 can also be provided on a stand-alone computer for access by a user.
In one example, the environment 100 may include a stand-alone computer architecture where a processing system 110 (e.g., one or more computer processors) includes the system 104 being executed on it. The processing system 110 has access to a computer-readable memory 112.
In one example, the environment 100 may include a client-server architecture. The users 102 may utilize a PC to access the servers 106 running the system 104 on the processing system 110 via the networks 108. The servers 106 may access the computer-readable memory 112.
FIG. 2 illustrates a block diagram of an example of a processing system 110 of FIG. 1 for applying transformation techniques and using transformation outputs to implement remedial actions. A bus 202 may interconnect the other illustrated components of the processing system 110. A central processing unit (CPU) 204 (e.g., one or more computer processors) may perform calculations and logic operations used to execute a program. A processor-readable storage medium, such as a read-only memory (ROM) 206 and a random access memory (RAM) 208, may be in communication with the CPU 204 and may contain one or more programming instructions. Optionally, program instructions may be stored on a computer-readable storage medium, such as a magnetic disk, optical disk, recordable memory device, flash memory, or other physical storage medium. Computer instructions may also be communicated via a communications transmission, data stream, or a modulated carrier wave. In one example, program instructions implementing transformation technique monitoring engine 209, as described further in this description, may be stored on a storage drive 212, a hard drive 216, a read-only memory (ROM) 206, a random-access memory (RAM) 208, or may exist as a stand-alone service external to the stand-alone computer architecture.
Some or all of the process described in relation to transformation technique monitoring engine 209 may be performed under the control of one or more computer systems configured with specific computer-executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs or one or more applications) executing collectively on one or more processors, by hardware or combinations thereof. The code may be stored on a non-transitory computer-readable storage medium, for example, in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable storage medium may be non-transitory.
A disk controller 210 can interface with one or more optional disk drives to the bus 202. These disk drives may be external or internal floppy disk drives such as a storage drive 212, external or internal CD-ROM, CD-R, CD-RW, or DVD drives 214, or external or internal hard drive 216. As indicated previously, these various disk drives and disk controllers are optional devices.
A display interface 218 may permit information from the bus 202 to be displayed on a display 220 in audio, graphic, or alphanumeric format. Communication with external devices may optionally occur using various communication ports 222. In addition to the standard computer-type components, the hardware may also include data input devices, such as a keyboard 224, or other input/output devices 226, such as a microphone, remote control, touchpad, keypad, stylus, motion, or gesture sensor, location sensor, still or video camera, pointer, mouse or joystick, which can obtain information from bus 202 via interface 228.
FIG. 3 illustrates a block diagram of an example transformation technique specification development process 300. In accordance with at least one embodiment, a transformation technique specification may refer to a predictive model. The example transformation technique specification development process 300 illustrates, for example, a development process for a predictive model. For example, during a data acquisition phase of the development process, causal variables 302, historical data 304, and event data 306 may be acquired at block 308. A portion of the data acquired at 308 may be loaded in a data mart and/or a master control file at 310.
In accordance with at least one embodiment, data preparation, analysis, and pre-processing techniques may be performed on the loaded data at block 312. For example, such techniques may include data visualization, outlier/anomaly detection, data cleansing, and segmentation/clustering. Visualization may include tasks that provide a visual representation and/or report generation related to the data loaded at 310. Outlier/Anomaly detection may include tasks related to the identification of unusual data records or data errors within the loaded data. Data cleansing may include tasks related to the removal or correction of identified outliers and/or anomalies from the loaded data. Segmentation and/or clustering may include tasks related to discovering groups and structures that indicate relationships between variables and/or data points included in the loaded data.
In accordance with at least one embodiment, a number of outputs (e.g., forecasted data points) may be generated at block 314. During a process for generating the outputs, hierarchy and specification options, for example, may be identified. Additionally, transformation technique specifications (e.g., predictive models) may be created, selected, and/or fitted prior to or subsequent to producing the number of outputs.
In accordance with at least one embodiment, post-processing and analysis techniques may be performed on a selected transformation technique specification at block 316. For example, such techniques may include visualization of results, specification evaluation, exception identification, reporting, and specification performance monitoring and tracking. Performance monitoring and tracking may include evaluating performance of the outputs produced by the applying the transformation technique associated with the transformation technique specification to time series data.
At decision block 318, a determination may be made as to whether the output is acceptable (e.g., falls within quality threshold amount). If the output is not acceptable (e.g., the specification is inaccurate based on the evaluations of block 316), then the flow may proceed to 320 where the transformation technique specification (e.g., the predictive model) may be refined.
In accordance with at least one embodiment, if a determination is made at decision block 318 that the output is acceptable (e.g., based on the evaluations of block 316), then the flow may proceed to 322 where the transformation technique specification(s) and outputs may be “locked,” or otherwise assigned, to a time series data set.
In accordance with at least one embodiment, the transformation technique specification may be promoted to production and corresponding results published at block 324. At block 326, the data mart and/or master control file may be updated and the process may repeat from block 310.
In accordance with at least one embodiment, if a determination is made at decision block 318 that the output is not acceptable (e.g., based on the evaluations of block 316), then the flow may proceed to 328 where the transformation technique specification(s) may be refined (e.g., a new specification selected, parameterize the current specification, and/or generate output using the current specification).
FIG. 4 illustrates an example architecture 400 of a transformation technique monitoring engine 402 (e.g., the transformation technique monitoring engine 209 of FIG. 2), in accordance with at least one embodiment. The transformation technique monitoring engine 402 may include a plurality of modules that may carry out various embodiments. The modules 404 may be software modules, hardware modules, or a combination thereof. If the modules 404 are software modules, the modules 404 can be embodied on a computer-readable medium and processed by a processor in any of the computer systems described herein. The modules 404 may be configured in the manner suggested in FIG. 4 or the modules 404 may exist as separate modules or services external to the transformation technique monitoring engine 402. These modules are directed to performing actions of the transformation technique monitoring engine 402 with respect to providing remedial actions to transformation technique specifications. The overall goal of the transformation technique monitoring engine is to improve computational performance of CPU 204 of FIG. 2 during operations utilizing transformation techniques and time series data.
In the embodiment shown in the drawings, rule data store 408, a control chart data store 410, a time series data store 412, and a transformation technique data store 414 are shown, although data can be maintained, derived, or otherwise accessed from various data stores, either remotely or locally, to achieve the functions described herein. The transformation technique monitoring engine 402, shown in FIG. 4, includes various modules such as a graphical user interface 420, an application programming interface 422, a receiver 424, a transformation engine 426, a control engine 428, an anomaly-detection engine 430, and a remedial-action engine 432. Some functions of these modules are described below. However, for the benefit of the reader, a brief, non-limiting description of each of the modules is provided in the following paragraphs.
In accordance with at least one embodiment, a process is enabled for utilizing the transformation technique monitoring engine 402. For example, a user (e.g., a user of users 102 in FIG. 1) may utilize device(s) 434 (e.g., processing system 110 of FIG. 1) to interact with server(s) 436 (e.g., the server(s) 106 of FIG. 1) to utilize the functionality of transformation technique monitoring engine 402. The device(s) 434 and the server(s) 436 may be configured to communicate with networks 438 (e.g., the networks 108 of FIG. 1). For example, transformation technique monitoring engine 402 may be executed on the servers 436 or on the device(s) 434.
In accordance with at least one embodiment, a user may utilize device(s) 434 to request initiation of a monitoring process for one or more transformation technique specification(s). Additionally, or alternatively, servers 436 may transmit a request to initiate a monitoring process of one or more transformation technique specification(s). In either case, such a request may be received via graphical user interface 420 and application programming interface 422, both components of the transformation technique monitoring engine 402. In some instances, the request may be generated automatically by a triggering event (e.g., upon elapse of a specified time period, on a specified date/time, etc.).
In accordance with at least one embodiment, receiver 424, a component of the transformation technique monitoring engine 402, may receive or obtain time series data. For example, the receiver 424 may interact with time series data store 412 to obtain time series data. The time series data store 412 may be a data store responsible for storing data associated with one or more time series. Additionally, the receiver 424 may receive or obtain transformation technique specifications from, for example, the transformation technique data store 414, a data store responsible for storing such specification(s). Upon receiving and/or obtaining times series data and one or more transformation technique specifications, the receiver 424 may communicate such information to the transformation engine 426, a component of transformation technique monitoring engine 402.
In accordance with at least one embodiment, the transformation engine 426 may apply a transformation technique defined by the transformation technique specification to a data to produce a set of outputs. The set of outputs may include a number of predicted data points for a time series that indicate one or more predicted data values. The transformation engine 426 may communicate the set of outputs to anomaly-detection engine 430, a component of transformation technique monitoring engine 402.
In accordance with at least one embodiment, anomaly-detection engine 430, may receive, or otherwise obtain, a set of outputs from transformation engine 426. The anomaly-detection engine 430 may generate a set of residual values using the time series data and the set of outputs. A “residual value” is intended to refer to a computed difference between the observed value of a data point and a predicted value of the data point. In at least one example, the set of residual values may be generated by determining a difference between each current data point of the time series and a corresponding predicted data point of the set of outputs. The anomaly-detection engine 430 may obtain a number of rules and/or tests from, for example, rule data store 408, a data store responsible for storing such rules/tests. Additionally, anomaly-detection engine 430 may obtain one or more control charts from control chart data store 410, a data store responsible for storing one or more control charts. In at least one example, the anomaly-detection engine 430 may utilize the set of residuals, the one or more control charts, and the one or more rules/tests to identify exceptions and/or anomalies within the set of outputs. The anomaly-detection engine 430 may interact with control engine 428, and/or remedial-action engine 432, both components of the transformation technique monitoring engine 402.
In accordance with at least one embodiment, the control engine 428 may receive, or otherwise obtain a control chart and a set of residual values. For example, control engine 428 may obtain a control chart from control chart data store 410. The control engine 428 may obtain a set of residual values from the anomaly-detection engine 430. The control engine 428 may utilize the control chart and the set of residuals to determine a set of control limits. In at least one example, control engine 428 may calculate a tracking signal. A “tracking signal” is a measurement of error that is used to determine whether bias is present in a transformation technique specification. In at least one example, the calculated tracking signal may be used to construct control limits. The tracking signal may be computed, for example, as follows:
T S = Running Sum of the Residuals ( RSR ) Mean Absolute Deviation ( MAD ) = ∑ i = 1 n ( F i - A i ) 1 n ∑ i = 1 n  F i - A i 
Where Fi represents the transformation/forecast value in period i, Ai represents an actual value (e.g., in the time series data) in period i, and n represents the number of periods considered. At any suitable time, the calculated tracking signal may be converted to units of standard deviation (1 MAD=0.798 standard deviations) for later use in a control chart.
Control limits may be communicated to the anomaly-detection engine 430 or any suitable component of the transformation technique monitoring engine 402.
In accordance with at least one embodiment, remedial-action engine 432 may receive information related to one or more identified anomalies. For example, remedial-action engine 432 may receive identification information related to one or more anomalies identified by anomaly-detection engine 430. The remedial-action engine 432 may determine one or more remedial actions (e.g., SELECT, FIT, GENERATE) for the transformation technique based on the one or more anomalies. For example, the remedial-action engine 432 may utilize one or more rules/tests from rule data store 408 to determine the one or more remedial actions. The remedial-action engine 432 may perform the determined remedial actions on one or more transformation technique specifications. The remedial-action engine 432 may cause modified transformation technique specifications to be stored in transformation technique data store 414.
The modules described above provides an efficient and robust capability to determine when it is advisable to select, fit/parameterize, or generate output from one or more transformation technique specifications on an individual specification basis. Information related to actions performed by modules 404 may, at any suitable time, be displayed to a user of device(s) 434 via graphical user interface 420.
FIG. 5 illustrates an example of a flow diagram 500 for determining control limits for a set of residuals using a Shewhart control chart. It should be understood, that if application of a transformation technique has produced a perfectly predicted set of outputs (E.g., identical values with respect to the time series data), all residuals would be zero. Residual values can be either positive (output>actual) or negative (output<actual). If the transformation technique specification is in a state of statistical control, the residuals should be centered on a mean of zero and should appear random with no specific patterns evident.
The flow diagram 500 may begin at block 502. At block 504, 3-sigma upper and lower control limits may be determined using a Shewhart control chart by, for example, the control engine 428 of FIG. 4. For example, the Shewhart control chart may be initialized with a mean (p) equal to 0. A set of residuals may be plotted on the Shewhart control chart. The plotted chart may be optionally displayed to a user via graphical user interface 420 of FIG. 4.
At block 506, the determined control limits from block 504 may be refined. For example, the 3-sigma upper and lower control limits may be refined using a set of rules/tests (e.g., Western Electric tests). Western Electric rules/tests may be used for detecting out-of-control or non-random conditions in a control chart. In at least one example, Western Electric tests may identify exceptions in the control chart by identifying one or more outliers of the set of residuals. In at least one example, residual data points identified as outliers may be replaced with a corresponding upper or lower control limit.
At block 508, the determined control limits from block 506 may be refined using a second control chart with μ=0. The plotted chart may be optionally displayed to a user.
At block 510, the control limits from block 508 may be even further refined using a third control chart. For example, the third control chart may utilize Western Electric tests and a set of custom tests over a region including a vintage date. A “vintage date” is intended to refer to a date at which the specification for the transformation technique was last selected, or fitted/parameterized.
At block 512, exceptions may be identified in the set of residuals using a combination of the Western Electric tests and custom tests created using the third control chart. Such exceptions may be identified for dates that are, for example, greater than or equal to the vintage date.
At block 514, one or more anomalies may be identified from the exceptions identified at block 512. An anomaly may be associated with a remedial action (e.g., SELECT, FIT, GENERATE). Remedial actions may be prioritized according to a computational cost. For example, SELECT may be prioritized above FIT, while FIT may be prioritized above GENERATE. A computational cost may correspond to the computational requirements for performing the remedial action. For example, reselecting a transformation technique specification may be computationally more extensive, and thus more computationally costly, than fitting a transformation technique specification. Similarly, fitting a transformation technique specification may be computationally more extensive, and thus more computationally costly, than generating a set of output data points using the currently selected transformation technique specification. The prioritization scheme described above is illustrative in nature; any suitable prioritization scheme may be utilized.
At block 516, the one or more anomalies may be sorted according to a computation cost associated with corresponding remedial actions. In at least one example, a most computational costly anomaly may be determined and retained. For example, it may be the case that two anomalies are identified at block 514. The two anomalies may individually correspond to a SELECT remedial action and a GENERATE remedial action. Accordingly, the anomalies may be sorted such that the anomaly corresponding to the SELECT remedial action is retained according to the prioritization scheme described above.
At block 518, a transformation technique specification may be identified as requiring a remedial action. For example, a list, or other parameter, may be updated to indicate that the transformation technique corresponding to the set of residuals may require the identified remedial action from block 516.
FIG. 6 illustrates an example of a flow diagram 600 for a process for analyzing a set of residuals using a tracking signal. A tracking signal may be used as an alternative to, or in conjunction with, a Shewhart control chart process discussed above in connection with FIG. 5.
The flow diagram 600 may begin at 602. At block 604, a period loop variable (i) may be initialized to the value of 1. At block 606, a number of counters (e.g., Running Sum of the Absolute Residuals (RSAR) and Running Sum of the Residuals RSR) may be initialized. A determination may be made at decision block 608 as to whether i is equal to the last period in the set of residuals.
If i is not the last time period in the set of residuals at decision block 608, the flow may proceed to block 610 where the tracking signal may be computed, for example, as follows:
Where Fi represents the transformation/forecast value in period i, Ai represents an actual value (e.g., in the time series data) in period i, and n represents the number of periods considered. At block 612, or at any suitable time, the computed tracking signal from block 610 may be converted to units of standard deviation (1 MAD=0.798 standard deviations) for later use in a control chart.
At decision block 614, a determination may be made as to whether or not the absolute value of the tracking signal in MAD units (abs(TS_MAD)) is greater than or equal to 4. If abs(TS_MAD) is not greater than or equal to 4, then no bias has been detected in the transformation technique specification, and the flow may proceed to block 616, where i may be set to the next period in the set of residuals in order to be reevaluated at decision block 608.
If the abs(TS_MAD) is determined to be greater than or equal to 4 at block 620, then the flow may proceed to block 618 where the residual data point may be flagged, or otherwise identified, as a tracking signal exception. Additionally, period counters such as RSR and RSAR may be set to 0 at block 618. The flow may proceed to block 616 where i may be set to the next period in the data set in order to be reevaluated at decision block 608.
If i is determined to be the last time period in the set of residuals at decision block 608, the flow may proceed to block 620 where SIGMA levels (e.g., control limits) for a tracking signal control chart may be set. For example, sigma levels for a control chart may be initialized with an upper control limit of 3.192 (4 MAD converted to standard deviation units) and a lower control limit of −3.192 (−4 MAD converted to standard deviation units). At block 622, a tracking signal may be plotted on a Shewhart control chart using the sigma control levels from block 620 and a mean(mu) value of 0. At block 624, one or more tests (e.g., Western Electric tests and/or custom tests) may be applied over the entire region. At block 626, tracking signal exceptions may be identified for dates occurring on or after the vintage date. One or more anomalies may be identified for each tracking signal exception in a similar manner as described in connection with FIG. 6.
FIG. 7 illustrates an example of a flow diagram 700 for a process for performing remedial actions on a number of transformation technique specifications, in accordance with one embodiment. The flow diagram 700 may begin at block 702 where a main loop variable i is declared and initialized to a start value (e.g., the start value may be a number of periods in the past, calculated from the end date, where the start value indicate a date from which to begin the analysis). The flow may proceed to block 704, where a determination may be made as to whether or not i is greater than or equal to an end value. If i is greater than or equal to the end value, the flow may proceed to decision block 706. If i equals the start value at decision block 706, the flow may proceed to block 708. At block 708, a number of lists and variables may be initialized. For example, a revisit_master set of variables may be initialized. A number of variables of the revisit_master set may correspond to an individual time series. In at least one example, the revisit_master set may include a variable for each time series that may be used to indicate that each time series requires a transformation technique specification selection. In at least one example, the revisit_master set may include additional variables for each time series including, but not limited to, a data set specification selection date, a specification date, a specification fit/parameratized date, and/or a run date.
In at least one example, one or more remedial action lists may be initialized at block 708. For example, a remedial action list corresponding to time series in need of transformation technique specification selection (e.g., a “SELECT” list) may be initialized. Similarly, a remedial action list corresponding to time series associated with a transformation technique specification in need of fit/parameterization (e.g., a “FIT” list) may be initialized. Further, a remedial action list corresponding to time series associated with a transformation technique specification that needs to be executed to generate a new set of outputs (e.g., a “GENERATE” list) may be initialized. In one example, the “SELECT” list may be initialized with indicators for all the time series. The “FIT” list and the “GENERATE” list may each be initialized to NULL (e.g., an empty value). Thus, at block 708, the “SELECT” list may indicate that each time series requires a transformation technique specification to be selected.
The flow may proceed to block 710, where a time series input data set may be created for each time series. For example, data points for a time series may be included in a time series input data set for dates that occur after an end date. The end date may be utilized to structure the time series input data set. An end date may be used when, for example, causal/independent variables are provided but actual values for a target forecast variable for future period is unknown.
The flow may proceed to decision block 712, where “SELECT,” “FIT,” and “GENERATE” time series data sets may be created from the corresponding lists. Thus, in the current example, each time series may have “SELECT” time series data sets created.
The flow may proceed to block 714 where a determination as to whether the “SELECT” time series set is empty. Continuing with the current example, the “SELECT” set includes time series data sets for each time series, thus, the flow may proceed to block 716 where a transformation technique specification may be selected for each time series in the “SELECT” set. In at least one example, a predefined procedure for transformation technique specification selection may be utilized. For instance, procedure that diagnose statistical characteristics of a time series to identify an appropriate transformation technique specification may be used. An example of one such procedure includes “HPFDiagnose”, an algorithm included in SAS® Forecast Server Procedures (part of SAS® Forecast Server). HPFDiagnose, or similar procedures, may consider numerous types of transformation technique specifications capable of being applied to each time series. HPFDiagnose, or similar procedures, may determine autoregressive and moving average orders, detect outliers, and select best input variables. The HPFDiagnose procedure results in output such as transformation technique specifications for multiple types of specifications and a selection list of transformation technique specifications. Such output may be used as input for execution with a selection procedure, for example, HPFEngine, utilizing a task parameter “SELECT,” in order to select a particular transformation technique specification for each time series.
Upon selection of a transformation technique specification for each time series, the flow may proceed to block 718 where various lists and variables may be updated. For example, the revisit_master set variables may be updated.
The flow may proceed to block 720, where a determination as to whether or not the “FIT” set is empty. Given the ongoing example, the “FIT” set may be determined to be empty and the flow may proceed to block 722, where the “GENERATE” set may also be determined to be empty. The flow may proceed to block 724, where output for the “SELECT,” “FIT,” and “GENERATE” sets may be appended to an output master file, for example. The flow may proceed to block 726, where i may be decremented to the next period and the flow may proceed back to block 704.
Continuing with the ongoing example, a determination may be made at block 704 that i is greater than the end date. Thus, the flow may proceed to decision block 706 where it may be determined that i is no longer equal to the start date. Given such a determination, the flow may proceed to block 728 where a vintage set including, but not limited to, unique vintage dates and vintage macro variables may be created. The flow may proceed to 730, where a vintage loop variable j may be set to the first vintage date in the vintage set.
The flow may proceed to block 732, where a determination may be made as to whether j is equal to the last vintage date in the vintage set. If j is not equal to the last vintage date in the vintage set, the flow may proceed to block 734 where a residual set processing may be performed using one or more control charts. Residual set processing performed at block 734 may include either the process described above in connection with FIG. 5 (e.g., using a Shewhart control chart) or the process described above in connection with FIG. 6 (e.g., using a converted or modified tracking signal with a Shewhart control chart). One or more anomalies may be identified in a set of residuals at block 734. Accordingly, a remedial action for time series associated with the one or more anomalies may be determined.
The flow may proceed to block 736, where the revisit_master set may be updated with information indicating one or more times series for which remedial actions have been identified. The flow may proceed to block 738, where the “SELECT” list, the “FIT” list, and the “GENERATE” list may be updated to indicate time series associated with transformation technique specifications that need to be selected, fit/parameterized, or generated, respectively.
The flow may proceed to block 740, where j may be set to the next vintage date. The flow may then repeat block 732, block 734, block 736, block 738, and block 740 until j equals the last vintage date in the vintage set. At such time, the flow may proceed back to block 710, where, once again, a time series input data set may be created for each time series data set.
The flow may proceed though decision block 712, block 714, block 716, and block 718 as described above. The process may proceed to block 720, where a determination may be made as to whether the “FIT” set is empty. If the “FIT” set is not empty, the flow may proceed to block 742, where transformation technique specifications included in the “FIT” set may be fit. Fitting a transformation technique specification includes, but is not limited to, adjusting parameters and/or variables associated with the transformation technique specification in order to produce a fitted transformation technique specification. A fitted transformation technique specification may produce a set of outputs that fall within an error threshold amount of a set of data points in a time series. A selection procedure, for example, HPFEngine, utilizing a task parameter “FIT”, may produce one or more fitted transformation technique specifications for a time series indicated in the “FIT” set. The HPFEngine procedure may select a particular fitted transformation technique specification for each time series indicated in the “FIT” set. Upon selection of a transformation technique specification for each time series, the flow may proceed to block 744 where various lists and variables may be updated. For example, the revisit_master set variables may be updated.
The flow may proceed to block 722, where a determination may be made as to whether or not the “GENERATE” set is empty. If the “GENERATE” set is not empty, the flow may proceed to block 746, where a set of output data points may be generated using the currently selected transformation technique specification for a corresponding time series included in the “GENERATE” set. Upon generating the set of output data points, the flow may proceed to block 748 where various lists and variables may be updated. For example, the revisit_master set variables may be updated.
The flow may proceed through block 724 and block 726 as described above. The process may repeat in the same manner as described above until i equals the end date at block 704. When i equals the end date at block 704, the process may end at block 750.
The flow above enables each time series to be individually monitored and custom remedial actions to be employed, rather than a “one size fits all” manner that exists today in current techniques.
FIG. 8 illustrates an example Shewhart control chart 800 including a plot of a set of residuals. The Shewhart control chart 800 may include a central line 802, an upper control limit 804, a lower control limit 806, and a set of zones 808. The set of zones 808 may include 3 total zones as indicated in FIG. 8 (e.g., zone A, zone B, and zone C), though any number of zones may be utilized. A set of residual data points 810 may be plotted on the Shewhart control chart 800 according to a time associated with each residual data point. The x-axis of the Shewhart control chart 800 may include time measurements, while the y-axis may indicate residual data point values. A vintage date 812 may be indicated on the Shewhart control chart 800 by a vertical line as depicted in FIG. 8, or by any suitable means. The Shewhart control chart 800 is illustrative of a Shewhart control chart generated on Dec. 28, 2003, using time series data points posted up to, and including, Dec. 21, 2003. Additionally, the Shewhart control chart 800 includes a vintage date of Nov. 30, 2003. Accordingly, only residual data points occurring on or after Nov. 30, 2003 through Dec. 21, 2003 may be considered, in accordance with the processes described herein. Exceptions (indicated in FIG. 8 using dashed lines) may be determined using any suitable combination of Western Electric tests and custom tests. Such tests will be discussed further in connection with FIGS. 9-11. As an illustrative example, data points 814 may test positive as an exception due to multiple consecutive data points falling outside the lower control limit 806 of Shewhart control chart 800. Classification of data points 814 as an exception may cause a remedial action to be performed with respect to a transformation technique specification used to produce outputs that were, in turn, used to produce the residual data points graphed in Shewhart control chart 800.
FIG. 9 illustrates an example of a set of Western Electric tests 900 for determining exceptions in a set of residuals using a control chart (e.g., the Shewhart control chart 800 of FIG. 8). For example, Western Electric tests 900 may include one or more non-random conditions that, if present, indicate an exception in a control chart. The set illustrated in FIG. 9 includes multiple tests. For example, if one data point (e.g., a residual data point) is beyond zone A (e.g., zone A of FIG. 8), the condition for test 1 of the Western Electric tests may be met. If nine sequential data points occur in zone C (e.g., zone C of FIG. 8) or beyond, on one side of the central line (e.g., the central line 802 of FIG. 8), the conditions for test 2 may be met. In at least one example, the number of data points included in test 2 may vary (e.g., 7, 8, 9, 11, 14, 20, etc.). If six sequential data points (or 7 sequential data points, or 8 sequential data points) steadily increase or steadily decrease, the conditions for test 3 may be met. If fourteen sequential data points alternate up and down, the conditions for test 4 may be met. If two out of three sequential data points are in zone A (e.g., zone A of FIG. 8) or beyond, the conditions for test 5 may be met. If four out of five sequential data points occur in zone B (e.g., zone B of FIG. 8), the conditions for test 6 may be met. If fifteen sequential data points occur in zone C (e.g., zone C of FIG. 8) on either, or both sides, of the central line (e.g., the central line 802 of FIG. 8), the conditions for test 7 may be met. If eight sequential data points occur on either or both sides of the central line 802, the conditions for test 8 may be met. FIG. 9 is illustrative in nature, a subset of the tests included in FIG. 9, additional tests not described in FIG. 9, or any suitable combination thereof, may be applied in the processes described above.
FIG. 10 illustrates an example of a set of custom tests 1000 for determining one or more anomalies in a set of residuals using a Shewhart control chart (e.g., the Shewhart control chart 800 of FIG. 8). For example, custom tests 1000 may include one or more anomalies that, if present in a control chart, indicate a necessity to perform a remedial action on a transformation technique specification. Custom tests 1000 may be used alone or in combination with Western Electric tests (e.g., the Western Electric tests 900 of FIG. 9). As discussed above, an anomaly may be associated with a remedial action to be performed on a transformation technique specification. In FIG. 10, anomaly 1 specifies that if 2 out of 2 data points are beyond 3-sigma limits, a transformation technique specification should be selected that better models the time series. Anomaly 2 specifies that if 7 out of 7 data points occur between the 1-sigma and 3-sigma limits, a transformation technique specification should be selected that better models the time series. Anomaly 3 specifies that if 3 out of 3 data points beyond 2-sigma limit, a transformation technique specification should be fit (e.g., re-parameterized) to better model the time series. Anomaly 4 specifies that if 5 data points sequentially increase, and Western Electric test 1 (e.g., test 1 of Western Electric tests 900 of FIG. 9) is met, a transformation technique specification should be fit to better model the time series. Anomaly 5 specifies that if 5 data points sequentially decrease, and Western Electric test 1 is met, a transformation technique specification should be fit to better model the time series. FIG. 10 is illustrative in nature, a subset of the custom tests 1000, additional custom tests not described in FIG. 10, or any suitable combination thereof, may be utilized in the processes described above.
FIG. 11 illustrates an example of a set of custom tests 1100 for determining anomalies in a set of residuals using a tracking signal and a control chart. FIG. 11 illustrates an example of a set of the custom tests 1100 for determining one or more anomalies in a set of residuals using a converted tracking signal and, for example, a Shewhart control chart. For example, custom tests 1100 may include one or more anomalies that, if present in the Shewhart control chart 800, indicate a necessity to perform a remedial action with respect to a transformation technique specification. Custom tests 1100 may be used alone or in combination with Western Electric tests (e.g., the Western Electric tests 900 of FIG. 9). Anomaly 1 specifies that if 3 out of 3 data points are beyond the 3-sigma limits (e.g., in units of standard deviation), a transformation technique specification should be selected that better models the time series. Anomaly 2 specifies that if 5 data points sequentially increase, a transformation technique specification should be selected that better models the time series. Anomaly 3 specifies that if 5 data points sequentially decrease, a transformation technique specification should be selected that better models the time series. Anomaly 4 specifies that if 5 out of 5 data points are beyond a 2-sigma limit, a transformation technique specification should be selected that better models the time series. FIG. 11 is illustrative in nature, a subset of the custom tests 1100, additional custom tests not described in FIG. 11, or any suitable combination thereof, may be utilized in the processes described above.
FIG. 12 illustrates an example of a flow diagram 1200 for applying transformation techniques and using transformation outputs to implement remedial actions. The flow may begin at block 1202 where time series data including a plurality of current data points may be received (e.g., by receiver 424 of FIG. 4). Receiver 424 may receive one or more specifications for a transformation technique applicable to the time series data at block 1204.
At block 1206, a plurality of outputs may be determined. For example, a transformation technique may be applied to the time series data (e.g., by transformation engine 426 of FIG. 4) to produce a plurality of outputs.
At block 1208, a set of residual values may be generated using the plurality of current data points and the plurality of output (e.g., by the anomaly-detection engine 430 of FIG. 4). Each residual value in the set of residual values may indicate a difference between a current data point of the plurality of current data points and an output of the plurality of outputs.
At block 1210, a set of control limits may be determined (e.g., by control engine 428 of FIG. 4) using a control chart (e.g., a Shewhart control chart) and the set of residual values determined at block 1208. The set of control limits may be determined, for example, by using either or both processes described in connection with FIGS. 5 and 6.
At block 1212, one or more anomalies of the set of residual values may be identified (e.g., by the anomaly-detection engine 430 of FIG. 4). The identification of the one or more anomalies may be based on the set of control limits determined at block 1208 and a control chart.
At block 1214, a remedial action may be determined for the transformation technique (e.g., by the remedial-action engine 432 of FIG. 4). As discussed above, a remedial action may include, but is not limited to, actions that cause a transformation technique specification to be selected, cause a transformation technique specification to be fit/parameterized, or cause a transformation technique specification to be applied to the time series to generate a plurality of outputs. The determined remedial action may be based on, for example, standard and/or custom tests including, but not limited to, the Western Electric tests 900 of FIG. 9, the custom tests 1000 of FIG. 10, the custom tests 1100 of FIG. 11, or any suitable combination thereof.
At block 1216, the remedial action related to the transformation technique may be performed (e.g., by the remedial-action engine 432 of FIG. 4). Thus, a different transformation technique specification may be selected, a current transformation technique specification may be fit/parameterized, or a transformation technique specification to be applied to the time series to generate a plurality of outputs.
Systems and methods according to some examples may include data transmissions conveyed via networks (e.g., local area network, wide area network, Internet, or combinations thereof, etc.), fiber optic medium, carrier waves, wireless networks, etc. for communication with one or more data processing devices. The data transmissions can carry any or all of the data disclosed herein that is provided to, or from, a device.
The data (e.g., associations, mappings, data input, data output, intermediate data results, final data results, etc.) may be stored and implemented in one or more different types of computer-implemented data stores, such as different types of storage devices and programming constructs (e.g., RAM, ROM, Flash memory, removable memory, flat files, temporary memory, databases, programming data structures, programming variables, IF-THEN (or similar type) statement constructs, etc.). It is noted that data structures may describe formats for use in organizing and storing data in databases, programs, memory, or other computer-readable media for use by a computer program.
A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. The processes and logic flows and figures described and shown in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output.
Generally, a computer can also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data (e.g., magnetic, magneto optical disks, or optical disks). However, a computer need not have such devices. Moreover, a computer can be embedded in another device, (e.g., a mobile telephone, a personal digital assistant (PDA), a tablet, a mobile viewing device, a mobile audio player, a Global Positioning System (GPS) receiver), to name just a few. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks). The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
The computer components, software modules, functions, data stores and data structures described herein may be connected directly or indirectly to each other in order to allow the flow of data needed for their operations. It is also noted that a module or processor includes, but is not limited to, a unit of code that performs a software operation, and can be implemented, for example, as a subroutine unit of code, or as a software function unit of code, or as an object (as in an object-oriented paradigm), or as an applet, or in a computer script language, or as another type of computer code. The software components or functionality may be located on a single computer or distributed across multiple computers depending upon the situation at hand.
The computer may include a programmable machine that performs high-speed processing of numbers, as well as of text, graphics, symbols, and sound. The computer can process, generate, or transform data. The computer includes a central processing unit that interprets and executes instructions; input devices, such as a keyboard, keypad, or a mouse, through which data and commands enter the computer; memory that enables the computer to store programs and data; and output devices, such as printers and display screens, that show the results after the computer has processed, generated, or transformed data.
Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer program products (i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, data processing apparatus). The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated, processed communication, or a combination of one or more of them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question (e.g., code that constitutes processor firmware, a protocol stack, a graphical system, a database management system, an operating system, or a combination of one or more of them).
While this disclosure may contain many specifics, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features specific to particular implementations. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be utilized. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software or hardware product or packaged into multiple software or hardware products.
Some systems may use Hadoop®, an open-source framework for storing and analyzing big data in a distributed computing environment. Some systems may use cloud computing, which can enable ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Some grid systems may be implemented as a multi-node Hadoop® cluster, as understood by a person of skill in the art. Apache™ Hadoop® is an open-source software framework for distributed computing. Some systems may use the SAS® LASR™ Analytic Server in order to deliver statistical modeling and machine learning capabilities in a highly interactive programming environment, which may enable multiple users to concurrently manage data, transform variables, perform exploratory analysis, build and compare models and score. Some systems may use SAS In-Memory Statistics for Hadoop® to read big data once and analyze it several times by persisting it in-memory for the entire session.
It should be understood that as used in the description herein and throughout the claims that follow, the meaning of “a,” “an,” and “the” includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise. Finally, as used in the description herein and throughout the claims that follow, the meanings of “and” and “or” include both the conjunctive and disjunctive and may be used interchangeably unless the context expressly dictates otherwise; the phrase “exclusive or” may be used to indicate situations where only the disjunctive meaning may apply.
US5461699 25. Okt. 1993 24. Okt. 1995 International Business Machines Corporation Forecasting using a neural network and a statistical forecast
US5870746 31. Okt. 1996 9. Febr. 1999 Ncr Corporation System and method for segmenting a database based upon data attributes
US6052481 2. Sept. 1994 18. Apr. 2000 Apple Computers, Inc. Automatic method for scoring and clustering prototypes of handwritten stroke-based data
US6189029 20. Sept. 1996 13. Febr. 2001 Silicon Graphics, Inc. Web survey tool builder and result compiler
US6223173 8. Febr. 2000 24. Apr. 2001 Fujitsu Limited Database system with original and public databases and data exploitation support apparatus for displaying response to inquiry of database system
US6230064 29. Juni 1998 8. Mai 2001 Kabushiki Kaisha Toshiba Apparatus and a method for analyzing time series data for a plurality of items
US6317731 11. Mai 2000 13. Nov. 2001 Joanne Sylvia Luciano Method for predicting the therapeutic outcome of a treatment
US6356842 18. Apr. 2000 12. März 2002 Carmel Systems, Llc Space weather prediction system and method
US6397166 6. Nov. 1998 28. Mai 2002 International Business Machines Corporation Method and system for model-based clustering and signal-bearing medium for storing program of same
US6400853 17. März 1998 4. Juni 2002 Canon Kabushiki Kaisha Image retrieval apparatus and method
US6526405 17. Dez. 1999 25. Febr. 2003 Microsoft Corporation Determining similarity between event types in sequences
US6542869 11. Mai 2000 1. Apr. 2003 Fuji Xerox Co., Ltd. Method for automatic analysis of audio including music and speech
US6564190 30. Dez. 1999 13. Mai 2003 General Electric Capital Corporaton Method and formulating an investment strategy for real estate investment
US6591255 5. Apr. 2000 8. Juli 2003 Netuitive, Inc. Automatic data extraction, error correction and forecasting system
US6609085 18. Jan. 1999 19. Aug. 2003 Asahi Glass Company, Ltd. Method for storing time series data and time series database system, method and system for processing time series data, time series data display system, and recording medium
US6640227 5. Sept. 2000 28. Okt. 2003 Leonid Andreev Unsupervised automated hierarchical data clustering based on simulation of a similarity matrix evolution
US6735738 20. Okt. 1999 11. Mai 2004 Fujitsu Limited Method and device for reconstructing acoustic data and animation data in synchronization
US6775646 23. Febr. 2000 10. Aug. 2004 Agilent Technologies, Inc. Excitation signal and radial basis function methods for use in extraction of nonlinear black-box behavioral models
US6850871 18. Okt. 1999 1. Febr. 2005 Agilent Technologies, Inc. Method and apparatus for extraction of nonlinear black-box behavioral models from embeddings of the time-domain measurements
US6876988 16. März 2001 5. Apr. 2005 Netuitive, Inc. Enhanced computer performance forecasting system
US6878891 3. Nov. 2003 12. Apr. 2005 Siemens Energy & Automation, Inc. Switchgear enclosure
US6928398 9. Nov. 2000 9. Aug. 2005 Spss, Inc. System and method for building a time series model
US6978249 28. Juli 2000 20. Dez. 2005 Hewlett-Packard Development Company, L.P. Profile-based product demand forecasting
US7072863 8. Sept. 1999 4. Juli 2006 C4Cast.Com, Inc. Forecasting using interpolation modeling
US7103222 1. Nov. 2002 5. Sept. 2006 Mitsubishi Electric Research Laboratories, Inc. Pattern discovery in multi-dimensional time series using multi-resolution matching
US7130822 31. Juli 2000 31. Okt. 2006 Cognos Incorporated Budget planning
US7152068 21. Dez. 2001 19. Dez. 2006 Honeywell International Inc. Method and apparatus for retrieving time series data related to an activity
US7171340 2. Mai 2005 30. Jan. 2007 Sas Institute Inc. Computer-implemented regression systems and methods for time series data analysis
US7222082 28. Aug. 2000 22. Mai 2007 Kronos Technology Systems Limited Partnership Business volume and workforce requirements forecaster
US7251589 9. Mai 2006 31. Juli 2007 Sas Institute Inc. Computer-implemented system and method for generating forecasts
US7260550 17. Sept. 1999 21. Aug. 2007 I2 Technologies Us, Inc. System and method for multi-enterprise supply chain optimization
US7280986 9. Febr. 2004 9. Okt. 2007 The Board Of Trustees Of The University Of Illinois Methods and program products for optimizing problem clustering
US7433809 25. Jan. 2006 7. Okt. 2008 Sas Institute Inc. System and method for non-linear modeling
US7433834 11. März 2002 7. Okt. 2008 Raymond Anthony Joao Apparatus and method for facilitating transactions
US7454420 8. Nov. 2004 18. Nov. 2008 Sas Institute Inc. Data sorting method and system
US7523048 19. Jan. 2001 21. Apr. 2009 Bluefire Systems, Inc. Multipurpose presentation demand calendar for integrated management decision support
US7530025 9. Mai 2005 5. Mai 2009 Sas Institute Inc. Systems and methods for handling time-stamped data
US7565417 5. Mai 2005 21. Juli 2009 Rowady Jr E Paul Event-driven financial analysis interface and system
US7570262 8. Aug. 2002 4. Aug. 2009 Reuters Limited Method and system for displaying time-series data and correlated events derived from text mining
US7617167 8. Apr. 2004 10. Nov. 2009 Avisere, Inc. Machine vision system for enterprise management
US7624114 5. Juni 2006 24. Nov. 2009 Microsoft Corporation Automatically generating web forms from database schema
US7660823 30. Dez. 2004 9. Febr. 2010 Sas Institute Inc. Computer-implemented system and method for visualizing OLAP and multidimensional data in a calendar format
US7664618 17. Dez. 2004 16. Febr. 2010 Korea Research Institute Of Standards And Science Trend monitoring and diagnostic analysis method and system for failure protection and for predictive maintenance of a vacuum pump
US7693737 31. Okt. 2006 6. Apr. 2010 International Business Machines Corporation Enterprise planning
US7702482 30. Dez. 2004 20. Apr. 2010 Microsoft Corporation Dependency structure from temporal data
US7711734 5. Apr. 2007 4. Mai 2010 Sas Institute Inc. Systems and methods for mining transactional and time series data
US7716022 9. Mai 2006 11. Mai 2010 Sas Institute Inc. Computer-implemented systems and methods for processing time series data
US7774179 30. Sept. 2008 10. Aug. 2010 Sas Institute Inc. System and method for non-linear modeling
US7941413 29. Jan. 2007 10. Mai 2011 Hitachi, Ltd. Index processing method and computer systems
US7966322 15. Apr. 2008 21. Juni 2011 Sas Institute Inc. Computer-implemented system and method for visualizing OLAP and multidimensional data in a calendar format
US8014983 22. Juni 2010 6. Sept. 2011 Sas Institute Inc. Computer-implemented system and method for storing data analysis models
US8015133 * 6. Sept. 2007 6. Sept. 2011 Sas Institute Inc. Computer-implemented modeling systems and methods for analyzing and predicting computer network intrusions
US8087001 29. Juni 2007 27. Dez. 2011 Sas Institute Inc. Computer-implemented systems and methods for software application testing
US8321479 16. Juli 2010 27. Nov. 2012 Ca, Inc. Efficient processing of time series data
US8326677 9. Dez. 2010 4. Dez. 2012 Jianqing Fan System and method for selecting an optimal forecasting hierarchy
US8374903 20. Juni 2008 12. Febr. 2013 Sas Institute Inc. Information criterion-based systems and methods for constructing combining weights for multimodel forecasting and prediction
US8489622 12. Dez. 2008 16. Juli 2013 Sas Institute Inc. Computer-implemented systems and methods for providing paginated search results from a database
US8645421 30. Sept. 2008 4. Febr. 2014 Sas Institute Inc. Attribute based hierarchy management for estimation and forecasting
US8676629 4. Dez. 2007 18. März 2014 Sas Institute Inc. System and methods for retail forecasting utilizing forecast model accuracy criteria, holdout samples and marketing mix data
US8768866 21. Okt. 2011 1. Juli 2014 Sas Institute Inc. Computer-implemented systems and methods for forecasting and estimation using grid regression
US20020169657 29. Okt. 2001 14. Nov. 2002 Manugistics, Inc. Supply chain demand forecasting and planning
US20030101009 30. Okt. 2001 29. Mai 2003 Johnson Controls Technology Company Apparatus and method for determining days of the week with similar utility consumption profiles
US20030105660 20. Febr. 2002 5. Juni 2003 Walsh Kenneth Peter Method of relating multiple independent databases
US20030110016 29. Juni 2001 12. Juni 2003 Daniel Stefek Integrative method for modeling multiple asset classes
US20030154144 27. Nov. 2002 14. Aug. 2003 Kimberly-Clark Worldwide, Inc. Integrating event-based production information with financial and purchasing systems in product manufacturing
US20030200134 28. März 2003 23. Okt. 2003 Leonard Michael James System and method for large-scale automatic forecasting
US20030212590 13. Mai 2002 13. Nov. 2003 Klingler Gregory L. Process for forecasting product demand
US20040030667 2. Aug. 2002 12. Febr. 2004 Capital One Financial Corporation Automated systems and methods for generating statistical models
US20040041727 31. März 2003 4. März 2004 Fujitsu Limited Crossover detection method, radar apparatus and crossover detection program
US20040172225 30. Juli 2003 2. Sept. 2004 Prosanos Corp. Information processing method and system for synchronization of biomedical data
US20040230470 29. Jan. 2004 18. Nov. 2004 Accenture Global Services Gmbh Marketing forecasting tool using econometric modeling
US20050102107 7. Nov. 2003 12. Mai 2005 Porikli Fatih M. Method for determining similarities between data sequences using cross-correlation matrices and deformation functions
US20050114391 20. Nov. 2003 26. Mai 2005 Corcoran Brian J. Apparatus, system, and method for communicating a binary code image
US20050159997 17. Dez. 2004 21. Juli 2005 Thomas John Systems and methods for planning demand for configurable products
US20050177351 9. Febr. 2004 11. Aug. 2005 The Board Of Trustees Of The University Of Illinois Methods and program products for optimizing problem clustering
US20050209732 16. Mai 2003 22. Sept. 2005 Srinivasaragavan Audimoolam Decision support system for supply chain management
US20050249412 7. Mai 2004 10. Nov. 2005 Regunathan Radhakrishnan Multimedia event detection and summarization
US20050271156 6. Juni 2005 8. Dez. 2005 Nahava Inc. Method and apparatus for cached adaptive transforms for compressing data streams, computing similarity, and recognizing patterns
US20060063156 5. Dez. 2003 23. März 2006 Willman Cheryl L Outcome prediction and risk classification in childhood leukemia
US20060064181 1. Juli 2003 23. März 2006 Seiji Kato Controlled-object model generation method program thereof and control parameter adjustment method and program thereof
US20060085380 19. Okt. 2004 20. Apr. 2006 Microsoft Corporation Reentrant database object wizard
US20060112028 24. Nov. 2004 25. Mai 2006 Weimin Xiao Neural Network and Method of Training
US20060143081 19. Nov. 2005 29. Juni 2006 International Business Machines Corporation Method and system for managing customer network value
US20060164997 30. Dez. 2004 27. Juli 2006 Microsoft Corporation Dependency structure from temporal data
US20060241923 2. Mai 2006 26. Okt. 2006 Capital One Financial Corporation Automated systems and methods for generating statistical models
US20060247900 2. Mai 2005 2. Nov. 2006 Sas Institute Inc. Computer-implemented regression systems and methods for time series data analysis
US20070011175 5. Juli 2005 11. Jan. 2007 Justin Langseth Schema and ETL tools for structured and unstructured data
US20070055604 31. Okt. 2006 8. März 2007 Cognos Corporation Enterprise planning
US20070094168 12. Aug. 2005 26. Apr. 2007 The Florida International University Board Of Trustees Artificial neural network design and evaluation tool
US20070162301 5. Febr. 2007 12. Juli 2007 Adam Sussman Computer-implemented systems and methods for resource allocation
US20070208492 3. März 2006 6. Sept. 2007 Inrix, Inc. Dynamic time series prediction of future traffic conditions
US20070208608 9. Nov. 2001 6. Sept. 2007 Prasanna Amerasinghe Forecasting and revenue management system
US20070291958 15. Juni 2007 20. Dez. 2007 Tristan Jehan Creating Music by Listening
US20080040202 19. Okt. 2007 14. Febr. 2008 Walser Joachim P Generating an Optimized Price Schedule for a Product
US20080208832 26. Febr. 2007 28. Aug. 2008 Friedlander Robert R System and method for deriving a hierarchical event based database optimized for pharmaceutical analysis
US20080288537 16. Mai 2007 20. Nov. 2008 Fuji Xerox Co., Ltd. System and method for slide stream indexing based on multi-dimensional content similarity
US20090018996 28. Jan. 2008 15. Jan. 2009 Herbert Dennis Hunt Cross-category view of a dataset using an analytic platform
US20090099988 * 12. Okt. 2007 16. Apr. 2009 Microsoft Corporation Active learning using a discriminative classifier and a generative model to detect and/or prevent malicious behavior
US20090172035 19. Dez. 2008 2. Juli 2009 Pieter Lessing System and method for capturing and storing casino information in a relational database system
US20090216611 25. Febr. 2008 27. Aug. 2009 Leonard Michael J Computer-Implemented Systems And Methods Of Product Forecasting For New Products
US20090319310 20. Juni 2008 24. Dez. 2009 Sas Institute Inc. Information Criterion-Based Systems And Methods For Constructing Combining Weights For Multimodel Forecasting And Prediction
US20100030521 13. Febr. 2008 4. Febr. 2010 Murad Akhrarov Method for analyzing and classifying process data
US20100063974 31. Aug. 2009 11. März 2010 International Business Machines Identifying optimal multi-scale patterns in time-series streams
US20100106561 28. Okt. 2008 29. Apr. 2010 Sergiy Peredriy Forecasting Using Share Models And Hierarchies
US20100114899 7. Okt. 2009 6. Mai 2010 Aloke Guha Method and system for business intelligence analytics on unstructured data
US20100121868 7. Nov. 2008 13. Mai 2010 Yann Le Biannic Converting a database query to a multi-dimensional expression query
US20100257133 22. Juni 2010 7. Okt. 2010 Crowe Keith E Computer-Implemented System And Method For Storing Data Analysis Models
US20110106723 3. Nov. 2009 5. Mai 2011 Michael Ryan Chipley Computer-Implemented Systems And Methods For Scenario Analysis
US20110119374 19. Okt. 2010 19. Mai 2011 Jan Matthias Ruhl Method and System for Detecting Anomalies in Time Series Data
US20110145223 11. Dez. 2009 16. Juni 2011 Graham Cormode Methods and apparatus for representing probabilistic data using a probabilistic histogram
US20110208701 22. Febr. 2011 25. Aug. 2011 Wilma Stainback Jackson Computer-Implemented Systems And Methods For Flexible Definition Of Time Intervals
US20110307503 10. Nov. 2010 15. Dez. 2011 Paul Dlugosch Analyzing data using a hierarchical structure
US20120053989 26. Aug. 2010 1. März 2012 Victor Harold Richard Systems and methods for propagating changes in a demand planning hierarchy
US20120123994 * 17. März 2011 17. Mai 2012 Bank Of America Corporation Analyzing data quality
US20120310939 6. Juni 2011 6. Dez. 2012 Taiyeong Lee Systems And Methods For Clustering Time Series Data Based On Forecast Distributions
US20130024167 22. Juli 2011 24. Jan. 2013 Edward Tilden Blair Computer-Implemented Systems And Methods For Large Scale Automatic Forecast Combinations
US20130024173 5. Apr. 2012 24. Jan. 2013 Jerzy Michal Brzezicki Computer-Implemented Systems and Methods for Testing Large Scale Automatic Forecast Combinations
US20130238399 20. Febr. 2013 12. Sept. 2013 Sas Institute Inc. Computer-Implemented Systems and Methods for Scenario Analysis
US20130268318 21. Dez. 2012 10. Okt. 2013 Sas Institute Inc. Systems and Methods for Temporal Reconciliation of Forecast Results
US20140019088 13. Juli 2012 16. Jan. 2014 Michael James Leonard Computer-Implemented Systems and Methods for Time Series Exploration
US20140019448 13. Juli 2012 16. Jan. 2014 Michael James Leonard Computer-Implemented Systems and Methods for Efficient Structuring of Time Series Data
US20140019909 18. Juli 2012 16. Jan. 2014 Michael James Leonard Computer-Implemented Systems And Methods For Time Series Exploration Using Structured Judgment
US20140257778 6. März 2013 11. Sept. 2014 Sas Institute Inc. Devices for Forecasting Ratios in Hierarchies
US20150052173 15. Aug. 2014 19. Febr. 2015 Sas Institute Inc. In-memory time series database and processing in a distributed environment
EP2624171A2 29. Jan. 2013 7. Aug. 2013 Sics, Swedish Institute Of Computer Science AB Method and node for analysing periodic movements
WO2002017125A1 21. Aug. 2000 28. Febr. 2002 Neil Johnson Method for processing time series data
WO2005124718A2 19. Juni 2005 29. Dez. 2005 Cvidya Networks Ltd. Methods, systems and computer readable code for forecasting time series and for forecasting commodity consumption
1 Agarwal, Deepak et al., "Efficient and Effective Explanation of Change in Hierarchical Summaries", The 13th International Conference on Knowledge Discovery and Data Mining 2007, Aug. 12-15, 2007 (10 pages).
2 Aiolfi, Marco et al., "Forecast Combinations," Creates Research Paper 2010-21, School of Economics and Management, Aarhus University, 35 pp. (May 6, 2010).
3 Atuk, Oguz et al., "Seasonal Adjustment in Economic Time Series," Statistics Department, Discussion Paper No. 2002/1, Central Bank of the Republic of Turkey, Central Bank Review, 15 pp. (2002).
4 Automatic Forecasting Systems Inc., Autobox 5.0 for Windows User's Guide, 82 pp. (1999).
5 Babu, G., "Clustering in non-stationary environments using a clan-based evolutionary approach," Biological Cybernetics, Sep. 7, 1995, Springer Berlin I Heidelberg, pp. 367-374, vol. 73, Issue: 4.
6 Bradley, D.C. et al., "Quantitation of measurement error with Optimal Segments: basis for adaptive time course smoothing," Am J Physiol Endocrinol Metab Jun. 1, 1993 264:(6) E902-E911.
7 Bruno, Giancarlo et al., "The Choice of Time Intervals in Seasonal Adjustment: A Heuristic Approach," Institute for Studies and Economic Analysis, Rome Italy, 14 pp. (2004).
8 Bruno, Giancarlo et al., "The Choice of Time Intervals in Seasonal Adjustment: Characterization and Tools," Institute for Studies and Economic Analysis, Rome, Italy, 21 pp. (Jul. 2001).
9 Cecil Bozarth, Ph.D., Measuring Forecast Accuracy: Approaches to Forecasting : A Tutorial, Published Jan. 25, 2011, 3 pages.
10 * Cecil Bozarth, Ph.D., Measuring Forecast Accuracy: Approaches to Forecasting: A Tutorial, Published Jan. 25, 2011, 3 Pages.
11 Choudhury, J. Paul et al., "Forecasting of Engineering Manpower Through Fuzzy Associative Memory Neural Network with ARIMA: A Comparative Study", Neurocomputing, vol. 47, Iss. 1-4, pp. 241-257 (Aug. 2002).
12 Costantini, Mauro et al., "Forecast Combination Based on Multiple Encompassing Tests in a Macroeconomic DSGE System," Reihe Okonomie/ Economics Series 251, 24 pp. (May 2010).
13 Data Mining Group, available at http://www.dmg.org, printed May 9, 2005, 3 pp.
14 * DataNet Quality Systems, What are the Western Electric Rules, http://www.winspc.com/14-datanet-quality-systems/support-a-resources/179-what-are-the-western-electric-rules, (available online Apr. 14, 2014).
15 * Davis Aquilano Chase, Fundamentals of Operations Management, Chapter 9 Forecasting, The McGraw-Hill Companies, Inc. 2003, 42 pages.
16 Funnel Web, Web site Analysis Report, Funnel Web Demonstration, Clients History, http://www/quest.com/funnel.sub.--web/analyzer/sample.ClientHist- .html (2 pp.), Mar. 2002.
17 Funnel Web, Web site Analysis. Report, Funnel Web Demonstration, Authenticated Users History, http://www.quest.com/funnel.sub.--web/analyzer/sample/UserHist.html (1 pg.), Mar. 2002.
18 Garavaglia, Susan et al., "A Smart Guide to Dummy Variables: Four Applications and a Macro," accessed from: http://web.archive.org/web/20040728083413/http://www.ats.ucla.edu/stat/sa- s/library/nesug98/p046.pdf, (2004).
19 Green, Kesten C. et al., "Structured analogies for forecasting" International Journal of Forecasting, vol. 23, pp. 365-376 (2007).
20 Guerard John B. Jr., Automatic Time Series Modeling, Intervention Analysis, and Effective Forecasting. (1989) Journal of Statistical Computation and Simulation, 1563-5163, vol. 34, Issue 1, pp. 43-49.
21 Guralnik, V. and Srivastava, J., Event Detection from Time Series Data (1999), Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 33-42.
22 Harrison, H.C. et al., "An Intelligent Business Forecasting System", ACM Annual Computer Science Conference, pp. 229-236 (1993).
23 Harvey, Andrew, "Forecasting with Unobserved Components Time Series Models," Faculty of Economics, University of Cambridge, Prepared for Handbook of Economic Forecasting, pp. 1-89 (Jul. 2004).
24 Huang, N. E. et al.,"Applications of Hilbert-Huang transform to non-stationary financial time series analysis." Appl. Stochastic Models Bus. Ind., 19: 245-268 (2003).
25 Hyndman, Rob J. et al., "Optimal combination forecasts for hierarchical time series", Monash University, Department of Econometrics and Business Statistics, http://www.buseco.monash.edu.au/de)lts/ebs/pubs/w)lapers/ (2007) 23 pages.
26 IBM, "IBM Systems, IBM PowerExecutive Installation and User's Guide," Version 2.10, 62 pp. (Aug. 2007).
27 Jacobsen, Erik et al., "Assigning Confidence to Conditional Branch Predictions", IEEE, Proceedings of the 29th Annual International Symposium on Microarchitecture, 12 pp. (Dec. 2-4, 1996).
28 Kalpakis, K. et al., "Distance measures for effective clustering of ARIMA time-series,"Data Mining, 2001. ICDM 2001, Proceedings IEEE International Conference on, vol., No., pp. 273-280, 2001.
29 Keogh, E. et al., "An online algorithm for segmenting time series," Data Mining, 2001. ICDM 2001, Proceedings IEEE International Conference on , vol., No., pp. 289-296, 2001.
30 Keogh, Eamonn et al., "Segmenting Time Series: A Survey and Novel Approach," Department of Information and Computer Science, University of California, Irvine, California 92697, 15 pp. (2004).
31 Keogh, Eamonn J. et al., "Derivative Dynamic Time Warping", In First SIAM International Conference on Data Mining (SDM'2001), Chicago, USA, pp. 1-11 (2001).
32 Kobbacy, Khairy A.H., et al., Abstract, "Towards the development of an intelligent inventory management system," Integrated Manufacturing Systems, vol. 10, Issue 6, (1999) 11 pp.
33 Kumar, Mahesh, "Combining Forecasts Using Clustering", Rutcor Research Report 40-2005, cover page and pp. 1-16 (Dec. 2005).
34 Leonard, Michael et al., "Mining Transactional and Time Series Data", abstract and presentation, International Symposium of Forecasting, 23 pp. (2003).
35 Leonard, Michael et al., "Mining Transactional and Time Series Data", abstract, presentation and paper, SUGI, 142 pp. (Apr. 10-13, 2005).
36 Leonard, Michael, "Large-Scale Automatic Forecasting Using Inputs and Calendar Events", abstract and presentation, International Symposium on Forecasting Conference, 56 pp. (Jul. 4-7, 2004).
37 Leonard, Michael, "Large-Scale Automatic Forecasting Using Inputs and Calendar Events", White Paper, pp. 1-27 (2005).
38 Leonard, Michael, "Large-Scale Automatic Forecasting: Millions of Forecasts", abstract and presentation, International Symposium of Forecasting, 156 pp. (2002).
39 Leonard, Michael, "Predictive Modeling Markup Language for Time Series Models", abstract and presentation, International Symposium on Forecasting Conference, 35 pp. (Jul. 4-7, 2004).
40 Leonard, Michael, "Promotional Analysis and Forecasting for Demand Planning: A Practical Time Series Approach", with exhibits 1 and 2, SAS Institute Inc., Cary, North Carolina, 50 pp. (2000).
41 Lu, Sheng et al., "A New Algorithm for Linear and Nonlinear ARMA Model Parameter Estimation Using Affine Geometry", IEEE Transactions on Biomedical Engineering, vol. 48, No. 10, pp. 1116-1124 (Oct. 2001).
42 Malhotra, Manoj K. et al., "Decision making using multiple models", European Journal of Operational Research, 114, pp. 1-14 (1999).
43 McQuarrie, Allan D.R. et al., "Regression and Time Series Model Selection", World Scientific Publishing Co. Pte. Ltd., 40 pp. (1998).
44 Montgomery, Douglas C. Introduction to Statistical Quality Control. 6th ed. Hoboken, N.J.: Wiley, 2009.
45 Oates, Tim et al., "Clustering Time Series with Hidden Markov Models and Dynamic Time Warping", Computer Science Department, LGRC University of Massachusetts, In Proceedings of the IJCAI-99, 5 pp. (1999).
46 Palpanas, T. et al, "Online amnesic approximation of streaming time series," Data Engineering, 2004. Proceedings. 20th International Conference on , vol., No., pp. 339-349, Mar. 30-Apr. 2, 2004.
47 Park, Kwan Hee, Abstract "Development and evaluation of a prototype expert system for forecasting models", Mississippi State University, 1990, 1 pg.
48 Product Brochure, Forecast PRO, 2000, 12 pp.
49 Quest Software, "Funnel Web Analyzer: Analyzing the Way Visitors Interact with Your Web Site", http://www.quest.com/funnel.sub.--web/analyzer (2 pp.), Mar. 2002.
50 Safavi, Alex "Choosing the right forecasting software and system." The Journal of Business Forecasting Methods & Systems 19.3 (2000): 6-10. ABI/INFORM Global, ProQuest.
51 SAS Institute Inc., SAS/ETS User's Guide, Version 8, Cary NC; SAS Institute Inc., (1999) 1543 pages.
52 SAS Institute Inc., SAS/QC 13.2 User's Guide. Cary, NC: SAS Publications, (2014).
53 SAS Institute Inc., SAS/QC 9.1: User's Guide. Cary, NC: SAS Publications, (2004).
54 Seasonal Dummy Variables, Mar. 2004, http://shazam.econ.ubc.ca/intro/dumseas.htm, Accessed from: http://web.archive.org/web/20040321055948/http://shazam.econ.ubc.ca/in tro-/dumseas.htm.
55 Simoncelli, Eero, "Least Squares Optimization," Center for Neural Science, and Courant Institute of Mathematical Sciences, pp. 1-8 (Mar. 9, 2005).
56 Tashman, Leonard J. et al., Abstract "Automatic Forecasting Software: A Survey and Evaluation", International Journal of Forecasting, vol. 7, Issue 2, Aug. 1991, 1 pg.
57 Using Predictor Variables, (1999) SAS OnlineDoc: Version 8, pp. 1325-1349, Accessed from: http://www.okstate.edu/sas/v8/saspdf/ets/chap27.pdf.
58 van Wijk, Jarke J. et al., "Cluster and Calendar based Visualization of Time Series Data", IEEE Symposium on Information Visualization (INFOVIS '99), San Francisco, pp. 1-6 (Oct. 25-26, 1999).
59 Vanderplaats, Garret N., "Numerical Optimization Techniques for Engineering Design", Vanderplaats Research & Development (publisher), Third Edition, 18 pp. (1999).
60 Wang Xiao-Ye; Wang Zheng-Ou; "A structure-adaptive piece-wise linear segments representation for time series," Information Reuse and Integration, 2004. IR I 2004. Proceedings of the 2004 IEEE International Conference on , vol., No., pp. 433-437, Nov. 8-10, 2004.
61 Wang, Liang et al., "An Expert System for Forecasting Model Selection", IEEE, pp. 704-709 (1992).
62 Wang, Ming-Yeu et al., "Combined forecast process: Combining scenario analysis with the technological substitution model," Technological Forecasting and Social Change, vol. 74, pp. 357-378 (2007).
63 Wheeler, Donald J. Advanced Topics in Statistical Process Control. Knoxville, Tenn.: SPC Press, 1995.
64 Wheeler, Donald J., and David S. Chambers. Understanding Statistical Process Control. 2nd ed. Knoxville, Tenn.: SPC Press, 1992.
65 Yu, Lean et al., "Time Series Forecasting with Multiple Candidate Models: Selecting or Combining?" Journal of System Science and Complexity, vol. 18, No. 1, pp. 1-18 (Jan. 2005).
Internationale Klassifikation G06F17/30, G06F9/44
Unternehmensklassifikation G06N5/025, G06F17/30598, G06F17/30563
Owner name: SAS INSTITUTE INC, NORTH CAROLINA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KATZ, JOSEPH H.;REEL/FRAME:035257/0357