System and method of reducing data in a storage system

The system and method of the present disclosure relates to technology for reducing the amount of data stored in a storage system by processing subsets of data stored in data sources using advanced analytics. The process generally includes extracting data from data sources for analysis by ranking the data, marking the data, identifying pattern changes in the data, comparing pattern changes in the data and purging and/or masking the data for storage. The system also includes databases for storing and defining rules, patterns, policies and classification data to be applied to the data from the data sources and analytics to apply the rules, patterns, policies and classification information on the data. As a result, the data stored in the data sources is reduced, and processing efficiency is increased.

BACKGROUND

The amount of data in the world has been increasing over time, and analyzing large data sets (i.e., big data) has increasingly become a basis of competition, supporting productivity growth, innovation, and consumer surplus, according to recent research statistics. For example, different market sectors such as healthcare, retail, manufacturing and personal-location data generate enormous amounts of data. As organizations create, store and analyze more data, performance can improve on everything from product inventories to employee productivity and ultimately into their bottom line. Intelligent data collection and advanced analysis can facilitate better management decisions and forecasting. However, big data can be so voluminous and unstructured that organizing it for meaningful analysis is complex and time consuming, not to mention that amount of storage space that is required to maintain the data. Moreover, analyzing big data can suffer from various problems, including missing relevant data, inaccurate algorithms, incorrect assumptions, etc. Thus, many challenges exists in the collection and processing of big data. Such challenges include analysis, capture, curation, search, sharing, storage, transfer, visualization, and privacy violations. Determining how to organize and analyze big data to meet these challenges is a time consuming and daunting task.

BRIEF SUMMARY

The present disclosure, generally described, relates to technology for reducing the amount of data stored in a storage system, and in particular, to analyzing and processing subsets of the data using advanced analytics to reduce the amount of data being stored in the storage system. Intelligently reducing the amount of data will allow for more efficient processing.

The system and method of the present disclosure relates to a data analytics system to process big data having been collected from a wide variety of sources, such as businesses, governments and individuals. Advanced (or big) data analytics, which is the process of examining large data sets having different data types, are used to uncover hidden patterns, unknown correlations, market trends, customer preferences and other useful business information. The process of the present disclosure generally includes extracting data from data sources across networks for analysis by ranking the data, marking the data, identifying pattern changes in the data, comparing pattern changes in the data and purging and/or masking the data for storage. The system also includes databases for storing and defining rules, patterns, policies and classification data to be applied to the data from the data sources and analytics to apply the rules, patterns, policies and classification information on the data. The analytics applying the rules may also apply a pre-defined rule or set of rules.

DETAILED DESCRIPTION

The system and method of the present disclosure relates to a data analytics system to process big data having been collected from a wide variety of sources, such as businesses, governments and individuals. Advanced (or big) data analytics, which is the process of examining large data sets having different data types, are used to uncover hidden patterns, unknown correlations, market trends, customer preferences and other useful business information. The process of the present disclosure generally includes extracting data from data sources across networks for analysis by ranking the data, marking the data, identifying pattern changes in the data, comparing pattern changes in the data and purging and/or masking the data for storage. The system also includes databases for storing and defining rules, patterns, policies and classification data to be applied to the data from the data sources and analytics to apply the rules, patterns, policies and classification information on the data. The analytics applying the rules may also apply a pre-defined rule or set of rules.

FIG. 1is an exemplary diagram illustrating a data analytics system for processing big data. The data analytics system10includes for example, one or more data sources40, data analytics20and one or more data collectors45, communicating, for example, via a network101. The network101(described in more detail below) can be a local area network (LAN), a wireless network, a mobile communications network, a wide area network (WAN), such as the Internet, or similar communication system. The data sources40can be any type of storage for storing big data collected from any wide variety of sources, such as businesses and government agencies. A few examples of big data stored in the data sources40can include, but are not limited to, personal data such as data including salary information, shopping statistics, banking methods, map routes, medical records or more general data such as census statistics, market trends, scientific data, network activity data, social media content and data, etc.

More specifically, the data analytics system10relates to big data analytics used to uncover hidden patterns, unknown correlations, market trends, customer preferences and other useful business information. The data sources40and the data analytics20can be hosted by any type of computing device including server computers, gateway computers, desktop computers, laptop computers, tablet computer, notebook computer, PDA (personal digital assistant), mobile communications devices, cell phones, smart phones, hand-held computers, or similar computing device. Alternatively, any combination of the data sources40and the data analytics20can be hosted on a single computing device including server computers, gateway computers, desktop computers, laptop computers, mobile communications devices, cell phones, smart phones, hand-held computers, or similar computing device.

Data collectors45include, for example, any sensor capable of collecting data, web crawlers, pluggable software modules, servers, computers, mobile devices, scanners, etc. The data collectors may reside at independent locations throughout the network101or together with the data sources40. Data collectors may also be directly connected to the network101or through any component that is connected or connectable to the network101. Additionally, the Internet of Things (IoT) allows data to be collected using a different types of data collectors. Objects, animals or people are provided with unique identifiers and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction. IoT has evolved from the convergence of wireless technologies, micro-electromechanical systems (MEMS) and the Internet. For example, a “thing”, in the Internet of Things, can be a person with a heart monitor implant, a farm animal with a biochip transponder, an automobile that has built-in sensors to alert the driver when tire pressure is low—or any other natural or man-made object that can be assigned an IP address and provided with the ability to transfer data over a network.

During operation of the data analytics system10(which will be described in more detail below), the data analytics20can receive real-time data (or non-real time data) to be collected from one or more of the data sources40. The data analytics20can include a processor25, analytics30, and a memory35. In one embodiment, data collected by the data analytics20is stored in storage35A of memory35such that real-time data not requiring processing may be stored for later use. The processor25, as described below, may process the data according to, for example, rules or rule sets or instructions stored storage35A of memory35. The rules can be defined by a user (e.g., system engineer, process engineer, industrial engineer, system administrator, etc.), or generated by the system automatically based on patterns and analysis of data. The rules may also be stored in rule set35B of memory35. Processor25can also perform processing of the data in storage35A of memory35. For example, processor25can perform processing, such as massive parallel processing (MPP) of the data, map-reduce processing, on-line transaction processing, extreme transaction processing, etc. The processor25can store the results of the processing in storage, such as storage35A, data sources40, or any other storage component connected or connectable to the data analytics system10. Real-time streaming of data can be processed, for example using stream processing designed to analyze and act on the real-time data, using “continuous queries” (i.e., SQL-type queries that operate over time and buffer windows). Essential to stream processing is streaming analytics, or the ability to continuously calculate mathematical or statistical analytics on the fly within the stream. As appreciated, analytics30may include stream analytics as part of its processing analytics (see below). Stream processing solutions are designed to handle high volumes of data in real time with a scalable, highly available and fault tolerant architecture. This enables analysis of data in motion. In one embodiment, a data mart provides end users, ad-hoc continuous query access to the streaming data that is aggregated in memory. The data mart analytics is capable of slicing, dicing and aggregating data dynamically in response to input from an end user as it occurs in real time.

The analytics30is responsible for analyzing the data in conjunction with the processor25, as described below in more detail. As appreciated, the amount of data received is extremely large (i.e., big data) and can affect processing of the data. Thus, data analytics20is essential to ensuring that the volumes of data being received may be processed and analyzed for more efficient use and storage. The analytics30can receive a real-time data stream from the one or more data sources40. The real-time data stream includes data to be collected by the data analytics20. The analytics30can identify real-time data from the data sources40to store in storage35A in the memory35, which is resident in the analytics system20. The analytics30can identify the real-time data that does not satisfy one or more rules in the rule set35B as real-time data, and similarly may identify the real-time data that does satisfy one or more rules in the rule set35B as real-time data. The analytics30may also store data in a format suitable for processing by the processor25.

In one embodiment, analytics30applies analytics on the data in the storage35A in memory35and updates the data based on the applied analytics. In another embodiment, analytics30provides the data to a server50(or other processing entity) located remotely from data analytics20for analysis. The analytics30can continuously apply the rules from rule set35B to the data stream associated with the data sources40. As the rules are updated or new rules are added, the analytics30may apply the updated rules and/or new rules to the data stored in storage35A. Big data can be analyzed with commonly used software as part of advanced analytics, such as predictive analytics, data mining, text analytics, pattern detection, artificial intelligence, rule based association of patterns, natural language processing and statistical analysis. Business intelligence software and data visualization tools can also be used in the analysis process. However, analytics are not limited to the above described methods, and any type of analytics and data processing may be used, as understood by the skilled artisan. Data analytics are further described with reference toFIGS. 2 and 3below.

FIG. 2is an exemplary diagram illustrating a pattern processing system and database for use with the data analytics system ofFIG. 1. In one embodiment, pattern processor55could be implemented using processor25ofFIG. 1. In the diagrams that follow, like reference numerals correspond to like elements. The pattern processing system200includes a pattern processor55, data sources40, server50, pattern rule set database60and pattern database65each of which are connected via network101. The pattern processor55may include various elements or components to perform various tasks. For example, the pattern processor55may include a pattern detector55A, a pattern correlator55B, and correlation analytics55C. Although these elements55A,55B and55C are illustrated and described with regard to a single pattern processor55, the functionality of these components may be provided by individual applications, by more than one application, within the same component, or other arrangement.

As part of the process in analyzing big data in a network environment, data or a data set from the big data stored in data sources40is extracted, a pattern in the data set is detected or identified by pattern detector55A, and the detected patterns are correlated and labeled by pattern correlator55B, such that each label indicates a specific condition associated with the big data. The labels may then be searched using the correlation analytics55C to answer a query regarding the big data stored in data sources40. In specific embodiments, detecting the pattern includes capturing gradients between each consecutive adjacent data points in the data set, aggregating the gradients into a gradient data set, dividing the gradient data set into windows, calculating a statistical parameter of interest for each window, aggregating the statistical parameters into a derived data set, and repeating the dividing, the calculating and the aggregating on derived data sets over windows of successively larger sizes until a pattern is detected.

The pattern processor55is connected to the network101, such as one or more of a local area network, the Internet, or other network. Also connected to the network are data sources40, server50, pattern rule set database60and pattern database65. As part of the pattern processor55, the pattern detector55A executes to retrieve patterns and pattern rules sets from the pattern database and pattern rule set database65and60, respectively, over the network101. Additionally, reference models may be stored in the pattern and pattern rule set databases60,65. The pattern detector then applies the retrieved patterns and pattern rule sets to the pattern correlator55B and correlation analytics55C to correlate and determine patterns in big data stored in data sources40. For example, the pattern correlator55B generates a relationship sequence with respect to data or data sets retrieved form the data sources40and correlates each relationship.

In the example that follows, extracted data includes extracted process data. While the example is described using process data, the analysis is not limited only to process data. Rather, as noted above, the data may be in any form and be collected from a wide variety of sources. As briefly explained above, pattern analysis can be performed by the pattern processor55using reference models, pattern rules and pattern rule sets present in the pattern database65and pattern rule set database60. Pattern detector55A then identifies usage patterns in the extracted data from data sources40. The extracted data is correlated with data in the pattern and pattern rule set databases60,65, which may include previously analyzed data. The data can also be used for adaptive learning, that is, adopting an efficient real time process as a standard reference model in the databases60,65.

Analysis of the extracted data by pattern processor55in relation to other extracted information from the pattern and pattern databases60,65can provide insight into existing processes. Correlation analytics55C of pattern processor55can promote an understanding of the relationships and interrelationships among applications and processes, and can offer details regarding integration of different applications and process re-engineering including conversion of multiple parallel processes performed by different departments, to a single enterprise-level standard process. Pattern processor55and correlation analytics55C can combine or compare the process data with data from the database60to establish what process is where, what belongs, and how it relates to data from database65. In accordance with one embodiment, pattern processor55can determine if a similar process has been encountered using data in databases60,65. Upon completion of the analysis, processor55may provide the analyzed process data to data sources40for storage. Results from applying the analytics may be stored as new or modified patterns or pattern rule sets in the respective databases. Pattern detection is described in more detail with reference toFIGS. 5 and 8.

FIG. 3is an exemplary diagram illustrating a ranking and policy analytics system and databases for use with the data analytics system ofFIG. 1. In one embodiment, ranking processor70could be implemented using processor25ofFIG. 1. The rank processing system300includes a ranking processor70, data sources40, server50, policy rule set database90and ranking rule set database95each of which are connected via network101. The ranking processor70may include various elements or components to perform various tasks. For example, the ranking processor70may include a ranking analyzer70A, a policy analyzer70B, and correlation analytics70C. Although these elements70A,70B and70C are illustrated and described with regard to a single ranking processor70, the functionality of these components may be provided by individual applications, by more than one application, within the same component, or other arrangement. The ranking and processing of policies against the data will be used by the system ofFIG. 1to purge (i.e., reduce) and mask data stored in data sources40, as further described below with reference to the various figures.

Ranking by the ranking processor70may begin once the data in data sources40has been analyzed by the processor25and analytics30of data analytics20, including completion of pattern processor55and any other analytics being executed to analyze the data (for example, the analysis of data described with reference toFIG. 6). In the example that follows, the data being retrieved and analyzed from data sources40is document data. Ranking analyzer70A of ranking processor70may analyze the retrieved document data to identify document data satisfying a particular criteria or rule, such as a rule from ranking rule set database85. For example, document data may be identified that contains one or more terms or phrases, or identifying document data that satisfies the particular criteria or rule and is within a predetermined threshold (for example, a term must appear more than 100 times before the predetermined threshold is met). Yet other techniques are known to those skilled in the art. After identifying the document data, the ranking analyzer70A may then score the documents based on a ranking rule set stored in ranking rule set database85. For example, ranking analyzer70A and correlation analytics70C may determine which rule sets in the ranking rule set database85apply, as well as correlate sets of document data to one another based on the various rule sets. Finally, the scored and correlated document data may then be ranked or prioritized based on the results.

In addition to scoring and ranking the data, policies can also be applied to the data using policy analyzer70B and policy rules stored in policy rule set database80. The policies stored in policy rule set database80may include, but are not limited to, sensitive data policies, backup and restore policies, access policies, privacy policies and others that may be used to analyze the data. For example, in the case of sensitive or privacy data policies, the policy analyzer70B may select a set of sensitive data types that need to be protected. Examples include, but are not limited to, credit card numbers, social security numbers, medical record numbers, addresses, names of patients, names high net-worth individuals, driver's license numbers, and bank account numbers. There can also be policies stored in the policy rule set database80controlling how the sensitive data, once identified, is handled by the policy analyzer70B. For example, one policy might state that credit card numbers should be masked with a format-preserving masking. The same policy may also indicated that social security numbers need to be encrypted with a particular encryption key. A different policy might indicate that telephone numbers need to be masked consistently. The same policy may also state that any file containing particular email addresses needs to be quarantined, i.e., access to it should be restricted. Once the policies relating to data privacy are defined, the policy analyzer70B upon detecting the sensitive will ensure that the policies are properly adhered to by the system. Additionally, a policy may state that data should not be included in scoring or ranking, or should not be correlated with other data by the correlation analytics70C.

FIG. 4is an illustrative embodiment of a general computer system to implement the systems depicted inFIGS. 1, 2 and 3. The general computer system which is shown and is designated100may be used to implement devices such as data analytics20and processors25,55and70(and/or any other system components) illustrated inFIGS. 1-3. The computer system100can include a set of instructions that can be executed to cause the computer system100to perform any one or more of the methods or computer based functions disclosed herein. The computer system100may operate as a standalone device or may be connected, for example, using a network101, to other computer systems or peripheral devices.

In a networked deployment, the computer system100may operate in the capacity of a server or as a client user computer in a server-client user network environment, or as a peer computer system in a peer-to-peer (or distributed) network environment. The computer system100can also be implemented as or incorporated into various devices, such as an call interceptor, an IVR, a context manager, an enrichment sub-system, a message generator, a message distributor, a rule engine, an IVR server, an interface server, a record generator, a data interface, a filter/enhancer, a script engine, a PBX, stationary computer, a mobile computer, a personal computer (PC), a laptop computer, a tablet computer, a wireless smart phone, a personal digital assistant (PDA), a global positioning satellite (GPS) device, a communication device, a control system, a web appliance, a network router, switch or bridge, a web server, or any other machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. The computer system100can be incorporated as or in a particular device that in turn is in an integrated system that includes additional devices. In a particular embodiment, the computer system100can be implemented using electronic devices that provide voice, video or data communication. Further, while a single computer system100is illustrated, the term “system” shall also be taken to include any collection of systems or sub-systems that individually or jointly execute a set, or multiple sets, of instructions to perform one or more computer functions.

As illustrated inFIG. 4, the computer system100includes a processor110. A processor for a computer system100is tangible and non-transitory. As used herein, the term “non-transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period of time. The term “non-transitory” specifically disavows fleeting characteristics such as characteristics of a particular carrier wave or signal or other forms that exist only transitorily in any place at any time. A processor is an article of manufacture and/or a machine component. A processor for a computer system100is configured to execute software instructions in order to perform functions as described in the various embodiments herein. A processor for a computer system100may be a general purpose processor or may be part of an application specific integrated circuit (ASIC). A processor for a computer system100may also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device. A processor for a computer system100may also be a logical circuit, including a programmable gate array (PGA) such as a field programmable gate array (FPGA), or another type of circuit that includes discrete gate and/or transistor logic. A processor for a computer system100may be a central processing unit (CPU), a graphics processing unit (GPU), or both. Additionally, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in, or coupled to, a single device or multiple devices.

Moreover, the computer system100includes a main memory120and a static memory130that can communicate with each, and processor110, other via a bus108. Memories described herein are tangible storage mediums that can store data and executable instructions, and are non-transitory during the time instructions are stored therein. As used herein, the term “non-transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period of time. The term “non-transitory” specifically disavows fleeting characteristics such as characteristics of a particular carrier wave or signal or other forms that exist only transitorily in any place at any time. A memory describe herein is an article of manufacture and/or machine component. Memories described herein are computer-readable mediums from which data and executable instructions can be read by a computer. Memories as described herein may be random access memory (RAM), read only memory (ROM), flash memory, electrically programmable read only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, tape, compact disk read only memory (CD-ROM), digital versatile disk (DVD), floppy disk, blu-ray disk, or any other form of storage medium known in the art. Memories may be volatile or non-volatile, secure and/or encrypted, unsecure and/or unencrypted.

As shown, the computer system100may further include a video display unit150, such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT). Additionally, the computer system100may include an input device160, such as a keyboard/virtual keyboard or touch-sensitive input screen or speech input with speech recognition, and a cursor control device170, such as a mouse or touch-sensitive input screen or pad. The computer system100can also include a disk drive unit180, a signal generation device190, such as a speaker or remote control, and a network interface device140.

In a particular embodiment, as depicted inFIG. 4, the disk drive unit180may include a computer-readable medium182in which one or more sets of instructions184, e.g. software, can be embedded. Sets of instructions184can be read from the computer-readable medium182. Further, the instructions184, when executed by a processor, can be used to perform one or more of the methods and processes as described herein. In a particular embodiment, the instructions184may reside completely, or at least partially, within the main memory120, the static memory130, and/or within the processor110during execution by the computer system100.

In an alternative embodiment, dedicated hardware implementations, such as application-specific integrated circuits (ASICs), programmable logic arrays and other hardware components, can be constructed to implement one or more of the methods described herein. One or more embodiments described herein may implement functions using two or more specific interconnected hardware modules or devices with related control and data signals that can be communicated between and through the modules. Accordingly, the present disclosure encompasses software, firmware, and hardware implementations. Nothing in the present application should be interpreted as being implemented or implementable solely with software and not hardware such as a tangible non-transitory processor and/or memory.

FIG. 5is an exemplary flowchart of processing big data for reduction in accordance with the systems ofFIGS. 1-3. That is, the process described below may be implemented by any of the systems and components described and illustrated inFIGS. 1-3. The process generally includes extracting data from the data sources for analysis by ranking the data, marking the data, identifying pattern changes in the data, comparing pattern changes in the data and purging and/or masking the data for storage. The system also includes databases storing and defining rules, patterns, policies and classification data to be applied to the data from the data sources and analytics to apply the rules, patterns, policies and classification information on the data. The analytics applying the rules may apply a pre-defined rule or set of rules. With reference toFIG. 5, data is extracted from data sources40at205. The data sources40may include, for example, structured data sources, unstructured data sources, email servers, enterprises data, log files, etc. The data sources40may also include batch and real-time data sources. Real-time data sources include, for example, streaming data as described above. While the disclosed embodiment refers to data extraction from data sources40, the data may be collected and extracted from any element or component connected to the system, such as server50. At210, using for example the data analytics20, the extracted data from the data sources40is analyzed. The analysis of data includes used of analytical tools to determine various features and characteristics of the data, such as relevancy of the data as determined by pattern analysis, age or persistence of the data, sensitivity and privacy of the data and any other input or query condition (manual or automatic) that may be entered into the analytics for use during the analysis. Each of the analytical tools may be based on rules or sets of rules stored in corresponding databases, such as databases60,65,80and85described above. A more detailed discussion of the analytical tools will be described below with reference toFIG. 6.

At215, the analyzed data is then ranked or prioritized, using for example ranking processor70, to determine whether data should be marked for purging (i.e., reduction) and/or masking. Data may be scored using ranking analyzer70A based on a ranking rule set stored in ranking rule set database85. For example, ranking analyzer70A (along with correlation analytics70C) may determine which rule sets in the ranking rule set database85apply, as well as correlate sets of data to one another based on the various rule sets. The scored and correlated data may then be ranked for purging based on the results. For example, if a rule in the ranking rule set database85is “purge data greater than 5 years old”, any data that is determined by the ranking analyzer70A to be five or more years old will be ranked to meet a threshold condition for purging. Similarly, data may be analyzed to determine whether policies or policy rules stored in the policy rule set database80apply to the data. For example, a policy may exist that any data including credit card information be identified as private and sensitive, such that the data is masked from any other data. When the policy analyzer708analyzes the data, it will reference the policy rule set database80to determine whether the data satisfies the policy, and mask the data accordingly. Data that is identified in the ranking215to be marked for purging or masking is marked at220. Marking the data may include setting a flag or tagging the data for purging or masking in any manner well known in the art. A further description of the ranking and masking will be described with reference toFIG. 7below.

Once the data has been marked at220, using for example processor25,55or70, the extracted data without the marked data is analyzed to determine whether any pattern changes exists. That is, the extracted data without the marked changes (hereinafter, unmarked data) is analyzed by the pattern detector55A which applies the patterns and pattern rule sets retrieved from pattern and pattern rule set databases60,65. The pattern detector55A, in conjunction with the pattern correlator55B and correlation analytics55C, then determines patterns in the unmarked data at225. The analysis can include, for example, identifying usage patterns, comparing extracted data with pre-defined processes or known patterns in the pattern and pattern rule set databases60,65, and other techniques known in the art. Additionally or alternatively, the analysis may be the same or similar to the analysis described above with respect to210. Once patterns have been determined (i.e., found) in the unmarked data by the pattern detector55A at230, the identified patterns in the unmarked data are compared to the patterns identified in the extracted data (i.e., the analyzed data prior to marking) at235using for example, processor25. For example, processor25of data analytics20can compare the analyzed data prior to marking with the analyzed data after marking. Analysis of the data may include, for example, mining the data for natural language text by transposing words and phrases in unstructured data into numerical values or scores that can be linked with structured data (e.g., rules or rule sets) stored in the various rule databases. The mined data in the analyzed data prior to marking could be compared to the mined data in the data after marking (i.e., marked data) to determine whether any changes exist, and if so, whether the changes are significant. That is, the comparison may determine whether the changes fall within a range or meet a threshold value (determine whether the changes are acceptable) at240. If patterns are not detected at225, then the data is re-analyzed by applying different criteria, such as different rules or rule sets. A more detailed description of determining pattern changes is detailed below with reference toFIG. 8.

At240, if the processor25determines that the comparison results in satisfying the range or threshold value, then the results are deemed acceptable. The marked data is then purged or masked at250for example by processor25. Specifically, data that has been previously marked to be purged at220will be deleted from the data, and data that has been previously marked to be masked at220will be hidden from the data. The purged and masked data will then be stored permanently in the data sources40. As will be appreciated, the purging and masking of data provides a reduction in the amount of data to be stored and processed. This purging and masking of big data is key in the ability to continually acquire and collect new data, while at the same time providing an affordable and reliable way to maintain data for processing at reasonable speeds and cost. If, on the other hand, the results of the comparison at240are not acceptable, then the criteria used to analyze the data is changed at245using, for example, processor25. For example, the rules, rule sets, policies, classification, relevancy, sensitivity, privacy, etc. may be modified by altering the parameters defining the analytics. Also, the ranking scores may be changed to define how analyzed data is ranked for purging and masking, and the range or threshold criteria of marking the data for purging or masking may be changed. Once the changes have been made, the extracted data (prior to analysis) is analyzed again at210.

FIG. 6is an exemplary flowchart of the data analysis described with reference toFIG. 5. In particular,FIG. 6discloses a more detailed description of the analysis of data in210ofFIG. 5. Data analysis and analytics includes a wide range of existing technologies and mechanisms, many of which may be employed in the current system to analyze data. In the embodiments that follow, the technologies and mechanisms focus on analysis of big data using advanced analytics. It is appreciated that the analytics disclosed herein are non-limiting and exemplary in nature. Advanced analytics may include, for example, predictive analysis, data mining, big data analytics and location intelligence. Any of these mechanisms may be used together or individually to analyze big data. In the example ofFIG. 6, the analysis of data405includes parsing the data into various categories such as relevancy410, age415, privacy data420and manual query data425. Each category of data is processed using a corresponding set of analytics, as described above. Moreover, it is appreciated that the depicted categories are non-limiting examples of parsing dating, and that other categories of data may equally be applied. As explained above, pattern analysis430includes, for example, the pattern detector55A executing to retrieve patterns and pattern rules sets from the pattern database and pattern rule set database65and60, respectively, over the network101. The pattern detector then applies the retrieved patterns and pattern rule sets to the pattern correlator55B and correlation analytics55C to correlate and determine patterns in big data stored in data sources40. For example, the pattern correlator55B generates a relationship sequence with respect to data or data sets retrieved form the data sources40and correlates each relationship. The determined relationships and patterns are then used to determine the relevancy of the data.

A threshold analysis (or range analysis)435may be used to determine the persistence of data or whether the data satisfies a particular range. That is, data is analyzed to determine whether it falls within a specified range or meets a certain criteria or threshold as part of the analysis of data405. If the data meets the specific requirements, then it may be used as part of the analysis, otherwise the data will be marked for purging or masking as not falling within the “age” requirements.

Privacy analysis420includes the classification of data using classification analysis440. Once data is classified as private, using for example rules stored in policy rule set database80, a classification analysis440on the private data the policy analyzer70B may be executed. For example, in the case of sensitive or privacy data policies, the policy analyzer70B may select a set of sensitive data types that need to be protected. Examples include, but are not limited to, credit card numbers, social security numbers, medical record numbers, addresses, names of patients, names high net-worth individuals, driver's license numbers, and bank account numbers. In one embodiment, the privacy analysis of the data identified as sensitive traces events related to privacy, summarizes activity relating to the access to personal data, and flags any suspected breaches of privacy rules.

Manual query425includes the analysis of data based on rules, parameters, characteristics, etc. that have been manually input into the system. A manual input analysis445may include, for example, defining the type of data sources available, locations of available data sources, access rights of the available data sources, data formats of the available data sources, etc. The manually input information can be a predefined set of rules or defined at the time of data extraction. The set of predefined rules may include definitions for how the raw data and/or information sources has to be cleansed, parsed, and transformed at the time of data extracting and processing.

FIG. 7is an exemplary flowchart of ranking data for purging and masking in accordance with the flowchart ofFIG. 5. In particular,FIG. 7discloses a more detailed description of the rank data in215ofFIG. 5. The data analyzed inFIG. 6is ranked to determine whether the data should be purged or masked to reduce the overall amount of data being stored and processed. The reduction or masking of the data is key to efficiently process the data. Moreover, the reduction of data allows for less utilization of storage space, which is a significant factor when storing big data. The ranking of data (rank data505) is executed using two sub-processes, namely purge data510and mask data530. A ranking processor70may be used to perform the analysis. Ranking processor70may include, for example, a ranking analyzer70A, a policy analyzer70B, and correlation analytics70C. Ranking by the ranking processor70may begin once the data in data sources40has been analyzed by the processor25and analytics30of data analytics20, including completion of pattern processor55and any other analytics being executed to analyze the data, as described with reference toFIG. 6. Ranking of the data has been previously described with reference toFIG. 3and will therefore not be detailed again. However, in determining whether to mark and process data for purging520,525, the ranked data is analyzed to determine whether rule sets, instructions, criteria, ranges and/or threshold conditions have been satisfied. For example, if a range is set to be “less than 50 mph and greater than 20 mph”, the analyzed data must satisfy the range in order to be marked for purging. Thus, if a sensor measures the speed of a vehicle as 30 mph, the range is satisfied and the data will be marked for purging. In this particular instance, the data marked for purging is not relevant and need not be processed, but data falling outside of the range is relevant for further analysis. Imagine data being stored by a government unit such as a police force. The unit collects enormous amounts of data and requires a means to process the data. In this example, one means is to ignore (or mask) the data. For example, vehicles that travel within a specified speed range satisfy the speed limit condition and do not require further analysis (vehicles are not speeding), while vehicles traveling outside of the speed range do not fall within the speed range and require additional analysis (vehicles may be speeding, but further analysis required).

Similarly, data may be detected to satisfy policies to mark the data for masking. Data masking is a process whereby the information in a database is masked or de-identified to ensure the protection of sensitive information stored in the database, while allowing the analysis of non-sensitive data that is part of the same information. The data masking process enforces “need to know access”, minimizing the risks associated with real-time data analysis. Policies can also be applied to the data using policy analyzer70B and policy rules stored in policy rule set database80. For example, in the case of sensitive or privacy data policies, the policy analyzer70B may select a set of sensitive data types that need to be protected. Examples include, but are not limited to, credit card numbers, social security numbers, medical record numbers, addresses, names of patients, names high net-worth individuals, driver's license numbers, and bank account numbers. There can also be policies stored in the policy rule set database80controlling how the sensitive data, once identified, is handled by the policy analyzer70B. For example, one policy might state that credit card numbers should be masked with a format-preserving masking. Thus, if the policy analyzer70B detects data to be credit card data, the policy relating to sensitive data types is satisfied (535), and the data will be marked for masking at540. Marking of the data may include, for example, processing the data to add a flag or tag or other identifying features or characteristics to indicate the data is ready to be purged or masked (525,545).

FIG. 8is an exemplary flowchart of determining patterns in accordance with the flowchart ofFIG. 5. In particular,FIG. 8discloses a more detailed description of the determination of pattern changes in225ofFIG. 5. To determine patterns (605), a query is made by the pattern processor55to the pattern database65and pattern rule set database60(FIG. 2) at610. The query to the databases is to access the rules and rules sets stored therein to identify patterns in the data being analyzed, as detailed above. In one embodiment, rule based pattern correlation analytics70C can include predetermined (e.g., preconfigured) rules. In other embodiments, rule based pattern correlation analytics70C may include rules that are defined based on various patterns identified in system ofFIG. 1. In yet other embodiments, rules may be specified to predict various results and/or scenarios, for example, using predictive analytics. Moreover, in other embodiments, the pattern database65and pattern rule set database60may be used to improve the pattern detection accuracy. For example, the databases may be provided with various learning patterns associated with complex and non-linear data sets. Sample derived data for learning may be extracted from actual data and patterns and provided as a result of the analysis. Other techniques such as trend-changing mechanisms and pattern matching algorithms may also be used.

At615, rules from the pattern and pattern ruleset databases are matched based upon the storage variety of factors including, for example, whether the data is graphical, textual, mathematical, the format of the data, the genre of the data, etc. At620, the matched rules are sent to the pattern processor55and/or ranking processor70for rule analysis, such as rule prioritization and ranking. Alternatively, the matched rules may be sent to server50for processing, or any other remote location capable of processing. The matched rules are prioritized at625based on criteria stored, for example, in memory35, server50, policy rule set database80or ranking rule set database85. At630, the prioritized rules are ranked by the ranking processor70, as described above with reference toFIG. 2.

In another embodiment, and as part of the pattern analysis (605-630), the pattern processor55is responsible for the rule selection based on input data in the pattern rule set database60and pattern database65, as well as providing matched rules to ranking processor70. Matched rules includes, for example, rule based data matching to determine if data or subsets of data are similar such that the data may be merged into a single set of data. In one embodiment, rules based matching executes a rule or data mines a set of target objects to match against the data or subset of data. The rule may be, for example, a predetermined or default rule that compares the fields of a source and target object. The ranking processor70then prioritizes the matched rules based on rules stored in the ranking rule set database85, policy rule set database80and other predetermined criteria. Once the matched rules have been prioritized, the prioritized rules may be processed against the data by processor55. Processed data may then be stored in the data sources40for storage. Using pattern and rule-based data, users of the system are afforded the opportunity to define rules and facts based on business requirements and may instruct the system to prioritize the matched rules based upon a variety of factors and criteria. Thus, the pattern analysis provides a procedure by which data may be reduced for improved storage and processing.

In one embodiment, there is a method of reducing data in a storage system, including accessing the data stored in the storage system by a processor; parsing the data accessed from the storage system into subsets of data using the processor, the parsing comprising categorizing the subsets of data using key identifiers, each of the categorical subsets of data analyzed based on a rule set associated with a respective category for each of the subsets of data; for each of the analyzed subsets of data, using the processor to detect the subsets of data to be purged based on a threshold condition having been satisfied, and ranking the subsets of data for which the threshold condition has been satisfied, and detect the subsets of data to be masked based on a policy having been satisfied, and ranking the subsets of data for which the policy has been satisfied; individually marking the subsets of data based on the ranking for purging using the processor when the threshold condition has been satisfied, and individually marking the subsets of data for masking based on the ranking using the processor when the policy has been satisfied; identifying pattern changes using the processor between the subsets of data prior to analysis and the marked subsets of data for purging and between the subsets of data prior to analysis and the marked subsets of data for masking; and processing the subsets of data for permanent change by reducing the amount of data using the processor when pattern changes satisfying a predetermined criteria have been identified, and providing the permanently changed subsets of data with the reduced amount of data to the storage system for storage.

In another embodiment, there is an apparatus to reduce storage of data, including a data source to store data for processing; a processor configured to categorize and analyze subsets of the data accessed from the data source; the processor configured to, for each of the analyzed subsets of data, detect and rank the subsets of data to be one of purged or masked data based on range; and the processor configured to mark the subsets of data based on the ranking for purging when the range is satisfied, and mark the subsets of data for masking based on the ranking of private data; the processor configured to identify pattern changes between the subsets of data prior to analysis and the marked subsets of data for purging and between the subsets of data prior to analysis and the marked subsets of data for masking; and the processor configured to process the subsets of data for updating by reducing the amount of data using the processor when pattern changes satisfying a predetermined criteria have been identified, and providing the updated subsets of data with the reduced amount of data to the storage system for storage.

In yet another embodiment, there is a computer program product, including a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising: computer readable program code configured to access subsets of data from a data source, comprising categorizing and analyzing the subsets of data using relationships based on correlated subsets of data; computer readable program code configured to, for each of the analyzed subsets of data, detect and rank the subsets of data to be one of purged or masked based on a threshold condition; and computer readable program code configured to mark the subsets of data based on the ranking for purging when a threshold condition has been satisfied, and mark the subsets of data for masking based on the ranking when a policy has been satisfied; computer readable program code configured to identify pattern changes between the subsets of data prior to analysis and the marked subsets of data for purging and between the subsets of data prior to analysis and the marked subsets of data for masking; and computer readable program code configured to process the subsets of data for permanent change by reducing the amount of data using the processor when pattern changes satisfying the relationship between the correlated subsets of data have been identified, and providing the permanently changed subsets of data with the reduced amount of data to the storage system for storage.

For purposes of this document, each process associated with the disclosed technology may be performed continuously and by one or more computing devices. Each step in a process may be performed by the same or different computing devices as those used in other steps, and each step need not necessarily be performed by a single computing device.