Abstract:
A method ( 200 ) and system ( 10 ) for analysing data is disclosed. One or more data records ( 30 ) are passed to a data analysis system ( 20 ). The data records ( 30 ) comprised a plurality of data items ( 35 ) and a first one of the data items is selected from the data items ( 35 ) in the data record ( 30 ). A statistical model ( 50 ) can be retrieved from a store ( 40 ) in a computer system ( 10 ) and the statistical model ( 50 ) used to detect abnormal results from the selected data item ( 35 ) and produce a data model ( 55 ). This statistical model ( 50 ) is stored with the data record in the data base ( 60 ).

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    The present application claims the benefit of the filing date of U.S. Provisional Patent Application Ser. No. 62/000,609 filed by the present inventors on May 20, 2014 and entitled “Method and system for analysing data.” 
         [0002]    The aforementioned provisional patent application is hereby incorporated by reference in its entirety. 
     
    
     BACKGROUND OF THE INVENTION 
       [0003]    1. Field of the Invention 
         [0004]    The field of the invention relates to a system and method for analysing data. 
         [0005]    2. Brief Description of the Related Art 
         [0006]    Many companies in a large variety of industries store large volumes of data, which are increasing in volume over time. These large volumes of data may include, but are not limited to, financial transaction data, computer network, infrastructure data, environmental data, operational data and social statistics. The data can be analysed or mined to identify trends, anomalies, and/or patterns in the data. The identified trends, anomalies or patterns can be used to understand and address particular problems. Current methods of data analysis are good at retrieving specific content about the data. For example, they are very efficient at retrieving all transactions for a specific user or host. 
         [0007]    Known data analysis systems, such as Splunk, are able to store, index and utilize the large volumes of data. Other systems are known that create complex data models from the data, for example a system supplied by Prelert, Inc. Framingham, Mass. These data models enable users to glean insights such as anomalies or trends into historical and newly ingested data. 
         [0008]    The known systems generate the information by a complete re-analysis of the incoming data. This re-analysis of the data can take a large amount of time. For example, it is common to analyse the data overnight in a batch process, when a processor is not heavily used. This (re)-analysis of the data is suitable for the identification of trends in the data, for which no immediate action has to be made. On the other hand, if the results are required in real-time, for example because of a change in a trend or a series of anomalous results, then these prior art methods do not enable the provision of the information sufficiently quickly. 
         [0009]    One example of a prior art method for the analysis of data is disclosed in the U.S. Patent Application Publication No. 2013/0198206 [Jones, assigned to Master Card International]. This patent document teaches a computer-based method of determining a so-called weirdness score for variables within a large data set. 
       SUMMARY OF THE INVENTION 
       [0010]    A method for analysing data is disclosed. One or more data records are passed to a data analysis system. The data records comprised a plurality of data items. In the case of anomaly detection, the value of one or more of the data items in one or more of the data records is compared with an expected value derived from a statistical model. The statistical model is derived from previous data records. On identifying an abnormal value, i.e. a value that falls substantially outside of the range of expected values, then the data model can be updated to indicate an anomaly. The statistical model is updated using the passed data records using the earlier statistical model. The statistical models are persisted for use as more data items are analysed, and the data models are persisted as a database of insights into the original data records. 
         [0011]    The method of this disclosure enables statistical models to be quickly and efficiently updated by using the previously calculated statistical model and updating the statistical models with the new data records. The resulting data model (containing insights such as anomalies) can be stored with the associated data records in the data base, which allows the data records to be readily accessed if required. For example, the user or supervisor might receive the message about the anomaly and wish to review the associated data records. 
         [0012]    In the case of anomaly detection, the expected values could be described by a normal distribution parameterised by the mean and variance of some of the values of data items, an indication of the class of membership of one or more of the data items or membership of a cluster, or a periodic trend. These examples are not limiting of the invention. In one aspect of the invention, the data model is indexed to allow rapid retrieval. 
         [0013]    The disclosure also teaches a system for analysing of the data that comprises at least one data entry device for the ingestion of at least one data record. A data analysis system accepts at least one of the plurality of data items from the data records and compares the value of the accepted one of the plurality of data items with an expected value. The data analysis system also updates a statistical model using the accepted one of the plurality of data items. An entry can be written to the data model if the value is abnormal or extraneous, i.e. lies outside a range of expected values. This entry can be reviewed by a user or administrator and the incident investigated. This forensic investigation can also use entries from the data model. The updated statistical model is stored in a statistical model. 
         [0014]    Still other aspects, features, and advantages of the present invention are readily apparent from the following detailed description, simply by illustrating a preferable embodiments and implementations. The present invention is also capable of other and different embodiments and its several details can be modified in various obvious respects, all without departing from the spirit and scope of the present invention. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature, and not as restrictive. Additional objects and advantages of the invention will be set forth in part in the description which follows and in part will be obvious from the description, or may be learned by practice of the invention. 
     
    
     
       BRIEF DESCRIPTION OF THE FIGURES 
         [0015]    For a more complete understanding of the present invention and the advantages thereof, reference is now made to the following description and the accompanying drawings, in which: 
           [0016]      FIG. 1  shows an example of the system of the analysis of the data. 
           [0017]      FIG. 2  shows the method for analysing data. 
           [0018]      FIG. 3  shows clustering of data points. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0019]    The invention will now be described on the basis of the drawings. It will be understood that the embodiments and aspects of the invention described herein are merely examples and do not limit the protective scope of the claims in any way. The invention is defined by the claims and their equivalents. It will be understood that features of one aspect or embodiment of the invention can be combined with a feature of a different aspect or aspects and/or embodiments of the invention. 
         [0020]      FIG. 1  shows an example of the system  10  for the analysis of data, according to one aspect of this invention. The system comprises a data analysis system  20 , which is connected to a store  40  and a data base  60 . The data base  60  stores data records  30  and is managed by a data base management system  65 . The store  40  stores a statistical model  50 , which is created from the data records  30 , and one or more data models  55 . The construction of the statistical model  50  and the data models  55  will be discussed later. The data base  60  is adapted to store in a non-transient manner a plurality of data records  30 , which are input into the data analysis system  10 . A terminal with a viewer  90  is connected to the system and receives messages from the data analysis system  20  for review by a user or administrator. The user or administrator can use the terminal  90  to investigate the messages and then review the data records  30  in the data base  60   
         [0021]    The input of the data records  30  is made by means of a data entry device  15 . The data entry device  15  collects the data records  30  from a number of sources, including but not limited to, user entry devices, such as terminal  90 , sensors measuring physical quantities, the internet, HTTP requests, IP (or similar) addresses and an intranet. The data entry device  15  passes the data records  30  to the data analysis system  20 . It will be seen that the data records  30  comprise a plurality of data input items, which are collectively labelled with the reference numeral  35 . Each one of the data items  35  can be individually processed. Examples of the data items  35  include, but are not limited to, timestamps and values of data. 
         [0022]    In one aspect of the invention it is possible to aggregate the values of the data over time and instead of storing multiple data records  30  only a single data record with an average value is stored, as will be explained later. 
         [0023]      FIG. 2  shows an example of the method of this disclosure. The method uses a Bayesian framework in the data analysis system  30  to develop the statistical model  50  from the input of the data records  30 . The data records  30  with the plurality of data items  35  are entered using the data entry device  15  from a plurality of sources in step  200 . As noted above, it is possible to pre-process the data by creating an aggregate value over a period of time. The data analysis system  20  uses the newly input data items  35  to compare the newly input data items  35  with the statistical model  50  in the statistical model store in comparison step  210 . 
         [0024]    This comparison step  210  will highlight any insights in the newly input data items  35 . 
         [0025]    The statistical model initially starts off with an initial model that can be non-informative or can incorporate expert knowledge, such as CPU ranges from 0-100%. The aim of the method is to develop the statistical model  50  such that the statistical model  50  identifies relationships between different ones of the data items  35  and the comparison step  210  can identify insights, such as anomalies in the data items  35  because one or more values of the data items  35  are different than the expected values or the relationships between the data items  35  are different. It will be appreciated that initially the variance in values of the data items  35  may be large. Over a short period of time, it will be expected that the average range of values of each of the data items  35  is established and that the variance of the values decreases. For example, any daily variations in the values of the data items  35  should be identified within a few days, whereas monthly variations in the values of the data items  35  will take a few months. The relationships will be multi-dimensional and clustering of the data items  35  will be established, as shown in connection with  FIG. 3 . 
         [0026]    Should the comparison step  210  identify an “abnormal” value, i.e. a value lying substantially outside the range of expected values of one or more of the data items  35 , then this abnormal value can be highlighted in one of the message to the user or the supervisor in step  220 . The user or supervisor uses this highlighted value to investigate the reasons for the abnormal value using the terminal. The user or supervisor can gain an insight into the data in which the abnormal value was identified. 
         [0027]    The statistical model  50  is updated in update step  230  using the newly inputted data items  35  from input step  210 . The updated statistical model  50  can be stored in the data base  60  together with the data records  30  with the data items  35  in storage step  240 . This updating of the stored statistical model  50  in update step  230  happens in real time or could be initiated in a batch process, for example overnight when the system  10  has available processing capacity. 
         [0028]    As noted above, not all of the data items  30  needs to be stored in the data base  60  or processed by the data analysis system  20 . Indeed it is possible that the data items  30  are not stored in the system  10  at all, but are accessible from the system. 
         [0029]    A data aggregator  70  can be used to aggregate or bucket together several of the data items  35  from different ones of the data records  30 . For example, all of the values of one data item  35  could be averaged over a period of time. The data aggregator  70  would then provide the average value of the one data item  35  for use in the comparison step  210  and the update step  240  as well as storage in the data base  60 . This saves in processing time and storage space. Additionally, the data items  35  coming from the same one of the data entry devices  15  could also be stored together or averaged, depending on the requirements of the system  10 . 
         [0030]    The statistical model  50  is self-learning. It is not created using a set of ‘training’ data that has been labelled. The data analysis system  20  using the data items  35  to create a series of relationships between the various data items  35 . The relationships could be temporal relationships, i.e. that one of the data items  35  takes particular values after a certain amount of time, could be averages or means with standard deviations or could be examples of variances in the data. The statistical model  50  is continually updated by the newly ingested data records  30 . So, for example, as a configuration of the system  10  changes then new data items  35  are entered and the statistical model  50  does not remain static, but is able to adjust its calculation based on the newly ingested data records  30 . 
         [0031]    In one aspect of the invention, the statistical model  50  can be used to predict future values and forecast events. It is also possible to calculate the probability of a particular event happening and then make a comparison after the event has happened. 
       Example 1 
       [0032]    The system and method of the disclosure can be used to determine changes in the occurrence of events and values of the data. Suppose that the incoming data record  30  has data items  35  which are measured at a time (t) and have a value (V) of the data for the occurrence of a particular event E. The data items  35  have a timestamp associated with them, which has the value t. 
         [0033]    The values V of the data are used to develop and update the statistical model  50  in the store  40 . In this example, the user is interested in the number of events E over time as well as the average V over time. The rate of change of the value of the data as well as the change in the number of events E is recorded in the statistical model having been calculated from previous values of the data records  30 . The mean of the values of the data, the running total average of the values of the data and/or the standard deviation of the value are stored in the statistical model  50 . The direct storage of these values in the data base  60  means that these values no longer need to be re-calculated if the supervisor  80  wishes to review the patterns of the data. The supervisor  80  can merely interrogate the data base  60  to obtain the values of interest. The raw values of the data, i.e. the values V and the timestamp can also be stored as part of the data record  30 . 
         [0034]    A baseline for normal behaviour is calculated, which is reflected in the statistical model  50 . So, if the rate of change of the data, the mean of the data or the standard deviation is within the baseline calculated by the statistical model  50  then the data analysis system  20  will merely store these values in the data base  60 . The data analysis system  20  can generate a new data model or update an existing data model to reflect any abnormal behaviour and issue an message (such as an alert) if the data analysis system  20  detects that any one of the values deviates abnormally from the baseline. This can be indicated to the supervisor  80  at the terminal  90 . This abnormal deviation is stored in the data model  55 . 
         [0035]    The supervisor  80  can use the data model  55  and directly access the data base  60  to see the stored data items  30  and review the previous updated statistical model and otherwise view of manipulate the data items  30 . 
       Example 2 
       [0036]    A further use of the system a method of this disclosure is shown with respect to  FIG. 3 , which shows a series of clustered data points  310   a ,  310   b . The clustered data points  310   a ,  310   b  are calculated in real time as the data records  30  are ingested into the data analysis system  20 . The cluster  310   a ,  310   b  of which the data records  30  is a member is stored with the data record  30  in the data base  60 . Suppose, however, that the calculation of the membership of the data cluster  310   a ,  310   b  using the statistical model  50  indicates that there is a deviation from the expected membership. This will be notified to the supervisor  80 , who is then able to investigate. 
         [0037]    One particular example which may be of interest to the supervisor  80  is shown by the arrow  320  on  FIG. 3 , which shows that a series of data points representing the data record  30  is moving from membership of one data cluster  310   a  to another one of the data cluster  310   n . This movement will be documented and recorded in the data store  60  and can then be investigated by the supervisor  80  together with the associated data record  30 . The supervisor  80  can query the data records  30  as well as the statistical model. 
       Example 3 
       [0038]    A further use of the system and method would be in the use of financial trading. The statistical model  50  represent in this case the profile of financial traders and the transactions will be recorded in the data base  60  together with statistical model  50  representing the trades. One indication of whether a trader is carrying out the trades or whether this has been done by an automated bot is the rate of trades. The “normal” trading rate is a statistic that can be calculated by the data analysis system  20  and forms part of the statistical model. Should the trading rate increase rapidly, or come from a different IP address than expected, then these are abnormal values that are identified and recorded in the data model  55  to allow subsequent investigation. 
       Example 4 
       [0039]    The system and method could be used by a retail store to monitor purchases, stocks, revenue etc. In this example, the data aggregator  70  will be used to at least reduce the amount of storage required in the data base  60 . The system and method are used to forecast and/or predict sales. Factors, such as holiday periods or weather patterns can be further stored and the statistical model  50  used to establish relationships. For example, the relationship between summer weather and purchase of barbecue sets. Any abnormalities in the sales are stored in the data model  55  and can be analysed for insights into the sales. 
         [0040]    The type of relationships that can be established is dependent on the data ingested by the system in step  220 . This will depend on the available data as well as the administrator&#39;s interests. To take Example 4, a relationship between summer weather and the purchase of barbecue sets is only possible if data relating to the summer weather (temperature, rainfall, etc.) is ingested as well as details of the sale of the barbecue sets. 
         [0041]    The system and method can also enable forecasting or prediction of trends. 
         [0042]    The foregoing description of the preferred embodiment of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention. The embodiment was chosen and described in order to explain the principles of the invention and its practical application to enable one skilled in the art to utilize the invention in various embodiments as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims appended hereto, and their equivalents. The entirety of each of the aforementioned documents is incorporated by reference herein. 
       REFERENCE NUMERALS 
       [0000]    
       
           10  System 
           15  Data entry device 
           20  Data analysis system 
           30  Data records 
           35  Data items 
           40  Store 
           50  Statistical model 
           55  Data model 
           60  Data base 
           65  Data base management system 
           70  Data aggregator 
           80  Investigator 
           90  Terminal