Modifying data cleansing techniques for training and validating an artificial neural network model

Disclosed is a method and system for modifying data cleansing techniques for training and validating an Artificial Neural Network (ANN) model. The method comprises normalizing and validating data of different formats, obtained from different sources. The ANN model is trained using the normalized and validated data. Alternatively, the ANN model could be trained using data of a common format obtained from a deterministic data model. The trained ANN model is used to generate ANN stochastic data. Data validation component from the deterministic data model is reused for the normalizing and the validating of the data, for validating the ANN stochastic data.

TECHNICAL FIELD

The present subject matter described herein, in general, relates to Artificial Neural Networks, and more particularly to modifying data cleansing techniques for training and validating an Artificial Neural Network model.

BACKGROUND

Data processing techniques are used to generate test data, such as One Test Data. Such test data is generated intelligently based on the previous data, using Artificial Neural Networks (ANN). In such scenarios, the test data is stochastic in nature i.e. the data has a random probability distribution or pattern that could be analysed statistically but may not be predicted precisely. Therefore, such test data may require parsing to clean the test data from bad data. Parsing generally involves breaking data blocks of the test data into smaller chunks by following a set of rules, so that the test data could be interpreted easily.

Building data cleansing solution for ANN generated data requires data modelling to match cleansing requirements and the ANN stochastically generated data. Deterministic data validation works for algorithmic generated deterministic data, and generally does not work for non-algorithmic stochastic data generated by the ANN. Further, manual human validation of data is time consuming and is also prone to errors. Therefore, there is a need of a system and a method for modifying data cleansing techniques for training and validating an Artificial Neural Network (ANN) model

SUMMARY

Before the present systems and methods for modifying data cleansing techniques for training and validating an Artificial Neural Network (ANN) model, are described, it is to be understood that this application is not limited to the particular systems, and methodologies described, as there can be multiple possible embodiments which are not expressly illustrated in the present disclosures. It is also to be understood that the terminology used in the description is for the purpose of describing the particular implementations or versions or embodiments only, and is not intended to limit the scope of the present application.

This summary is provided to introduce aspects related to a system and a method for modifying data cleansing techniques for training and validating an ANN model. This summary is not intended to identify essential features of the claimed subject matter nor is it intended for use in determining or limiting the scope of the claimed subject matter.

In one implementation, a system for modifying data cleansing techniques for training and validating an ANN model is disclosed. In one aspect, the system comprises a memory and a processor coupled to the memory. Further, the processor may be capable of executing instructions in the memory to perform one or more steps. In the aspect, the system may comprise a memory and a processor coupled to the memory. The processor is capable of executing instructions to perform several steps described now. The processor may normalize and validate data collected in different formats from different data sources. Specifically, the data present in different formats may be normalized to a common format by mapping the different data sources to a common data source. The normalized data may then be parsed. Successively, format validation and restriction validation may be performed on the parsed data. Parsing operation may be restarted upon occurrence of an error till data of a required format is obtained.

The ANN model may be trained using the normalized and validated data. Alternatively, a deterministic data model such as OneTest Data may be used to obtain the data of a common format, and the ANN model may be trained upon such data. ANN stochastic data may be generated using the trained ANN model. Data validation component of the deterministic data model, used for the normalizing and the validating of the data, may be used for validating the ANN stochastic data.

In one implementation, a method for modifying data cleansing techniques for training and validating an ANN model is disclosed. In one aspect, the method may comprise normalizing and validating data collected in different formats from different data sources. Specifically, the data present in different formats may be normalized to a common format by mapping the different data sources to a common data source. The normalized data may then be parsed. Successively, format validation and restriction validation may be performed on the parsed data. Parsing operation may be restarted upon occurrence of an error till data of a required format is obtained.

The ANN model may be trained using the normalized and validated data. Alternatively, a deterministic data model such as OneTest Data may be used to obtain the data of a common format, and the ANN model may be trained upon such data. ANN stochastic data may be generated using the trained ANN model. Data validation component of the deterministic data model, used for the normalizing and the validating of the data, may be used for validating the ANN stochastic data.

In yet another implementation, non-transitory computer readable medium embodying a program executable in a computing device for modifying data cleansing techniques for training and validating an ANN model is disclosed. In one aspect, the program may comprise a program code for normalizing and validating data collected in different formats from different data sources. Specifically, the data present in different formats may be normalized to a common format by mapping the different data sources to a common data source. The program may further comprise a program code for parsing the normalized data. The program may further comprise a program code for performing format validation and restriction validation on the parsed data. The program may further comprise a program code for restarting parsing operation upon occurrence of an error till data of a required format is obtained.

The program may further comprise a program code for training the ANN model using the normalized and validated data. Alternatively or additionally, program code may be present for using a deterministic data model such as OneTest Data for obtaining the data of a common format and training the ANN model upon such data. The program may further comprise a program code for generating ANN stochastic data using the trained ANN model. The program may further comprise a program code for using a data validation component of the deterministic data model, used for the normalizing and the validating of the data, for validating the ANN stochastic data.

DETAILED DESCRIPTION

Some embodiments of this disclosure, illustrating all its features, will now be discussed in detail. The words “comprising,” “having,” “containing,” and “including,” and other forms thereof, are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items. It must also be noted that as used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Although any systems and methods for modifying data cleansing techniques for training and validating an Artificial Neural Network (ANN) model, similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present disclosure, the exemplary, systems and methods for modifying data cleansing techniques for training and validating an ANN model are now described. The disclosed embodiments for modifying data cleansing techniques for training and validating an ANN model are merely examples of the disclosure, which may be embodied in various forms.

Various modifications to the embodiment will be readily apparent to those skilled in the art and the generic principles herein may be applied to other embodiments for modifying data cleansing techniques for training and validating an ANN model. However, one of ordinary skill in the art will readily recognize that the present disclosure for modifying data cleansing techniques for training and validating an ANN model is not intended to be limited to the embodiments described, but is to be accorded the widest scope consistent with the principles and features described herein.

Referring now toFIG.1, a network implementation diagram100of a system102for modifying data cleansing techniques for training and validating an Artificial Neural Network (ANN) model, in accordance with an embodiment of the present subject matter may be described. In one example, the system102may be connected with data sources104-1through104-N (collectively referred as104) through a communication network106.

It should be understood that the system102and the data sources104correspond to computing devices. It may be understood that the system102and the data sources104may also be implemented in a variety of computing systems, such as a laptop computer, a desktop computer, a notebook, a workstation, a mainframe computer, a server, a network server, a cloud-based computing environment, or a mobile and the like.

In one implementation, the communication network106may be a wireless network, a wired network, or a combination thereof. The communication network106can be implemented as one of the different types of networks, such as intranet, Local Area Network (LAN), Wireless Personal Area Network (WPAN), Wireless Local Area Network (WLAN), wide area network (WAN), the internet, and the like. The communication network106may either be a dedicated network or a shared network. The shared network represents an association of the different types of networks that use a variety of protocols, for example, MQ Telemetry Transport (MQTT), Extensible Messaging and Presence Protocol (XMPP), Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), Wireless Application Protocol (WAP), and the like, to communicate with one another. Further, the communication network106may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, and the like. The communication network106may be organized as a peer to peer (P2P) network, where the system102and the data sources104represent peer nodes with equivalent roles in the network.

Referring now toFIG.2, a block diagram200of the system102is illustrated in accordance with an embodiment of the present subject matter. In one embodiment, the system102may include at least one processor202, an input/output (I/O) interface204, and a memory206. The at least one processor202may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the at least one processor202may be configured to fetch and execute computer-readable instructions stored in the memory206.

The memory206, amongst other things, serves as a repository for storing data processed, received, and generated by one or more of modules208. The memory206may include any computer-readable medium or computer program product known in the art including, for example, volatile memory, such as Static Random Access Memory (SRAM) and Dynamic Random Access Memory (DRAM), and/or non-volatile memory, such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable and Programmable ROM (EEPROM), flash memories, hard disks, optical disks, and magnetic tapes.

The memory206may include data generated as a result of the execution of one or more of the modules208. In one implementation, the memory206may include data210. The modules208include routines, programs, objects, components, data structures, etc., which perform particular tasks or implement particular abstract data types. In one implementation, the modules208may include a normalizing and validating module212, a training module214, a generating module216, and a reusing module218. The modules208described herein may be implemented as software modules that may be executed in the cloud-based computing environment of the system102.

The data210may include a repository220for storing data processed, computed, received, and generated by one or more of the modules208. Furthermore, the data210may include other data222for storing data generated as a result of the execution of modules than the ones mentioned above.

In one implementation, to modify existing data cleansing techniques for training and validating an Artificial Neural Network (ANN) model, at first, data may be obtained from one or more data sources. The data may be present in a single format or different formats. While the data is present in different formats, the data may be normalized and validated by the normalizing and validating module212. In order to normalize and validate the data, the data present in different data formats may be to a common format by mapping the different data sources to a common data source. The normalized data may then be parsed using a suitable parsing technique. Format validation and restriction validation may be performed on the parsed data to determine presence of the data in a common suitable format. Parsing operation may be restarted upon occurrence of an error till data of a required format is obtained.

In an alternate embodiment, instead of obtaining the data present in different formats and processing such data to obtain data in a common format, a deterministic data model may be utilized for generating data of a common format. OneTest Data is one such deterministic data model that could be used for generating data of a common format. Such common format may be selected amongst a range of available formats, such as (Comma-Separated Values) CSV, JavaScript Object Notation (JSON), and Extensible Markup Language (XML). Further, a custom format may be defined by a user, and the data may be obtained in such custom format.

After the data of a common format is obtained, the ANN model may be trained using the data by the training module214. Upon such training, a trained ANN model may be obtained. The ANN may be implemented using one of Recurrent Neural Network (RNN) classes i.e. a Long Short-Term Memory (LSTM) and a Gated Recurrent Unit (GRU). Alternatively, the ANN may be implemented based on Restricted Boltzmann Machines (RBM) and a Deep Belief Net (DBN).

The trained ANN model may be used for generating ANN stochastic data by the generating module216. A data validation component of a deterministic data model used for normalizing and validating of the data may be used by the reusing module218for validating the ANN stochastic data.

In one implementation, referring toFIG.3illustrating an exemplary process flow for modifying data cleansing techniques for training and validating an Artificial Neural Network (ANN) model, a practical implementation of the proposed method is described.

At first, data was obtained from two different data sources, namely data source 1 at block302and data source 2 at block304. In one scenario, data source 1 included address of different users staying in New York city. Further, the data source 1 was present as a CSV format and included 500 thousand records. Further, the data source 2 included address of different users staying in Palm Beach FL state. Further, the data source 2 was present as a CSV format and included 500 thousand records.

Such data obtained from the data source 1 and the data source 2 was normalized to obtain a total data set in a common format, at block306. The total data set was provided to an ANN model to allow the ANN model to understand the total data set, at block308. Thereupon, the trained ANN model was used to generate ANN stochastic data, at block310.

Format validation and restriction validation were performed on the ANN stochastic data, at block312. Further, the validation operations were restarted upon occurrence of an error till data of a required format was obtained. Restrictions used during the restriction validation included instructions for identifying allowed data. For example, state must be FL or NY, must be a letter, and must be combination of letters and numbers, ZIP code has to be the 5 digit integer number starting with 3. Further, the allowed data must satisfy plurality of different parsing rules defined internally or externally via lookups. Instructions for identifying disallowed data included that it cannot be a number, cannot be a letter, and should be standard defined types i.e. must be a float number, date, time, or must be a custom type.

Finally, upon successful validations, valid data was obtained at block314and rejected data was obtained at block316. The valid data included addresses that matched with normalized type trees. The rejected data included addresses that did not match with the normalized type trees.

Below provided is a sample of the data obtained from the above implementation, where rejected generated addresses are marked with Bad prefix, where a bad record has 7 fields instead of 8.

Therefore, above described embodiments provide for reusing existing data cleansing techniques and models for validation with restrictions and restart of Artificial Intelligence stochastically generated data.

Referring now toFIG.4, a method400for modifying data cleansing techniques for training and validating an Artificial Neural Network (ANN) model is described, in accordance with an embodiment of the present subject matter. The method400may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, etc., that perform particular functions or implement particular abstract data types.

The order in which the method400for modifying data cleansing techniques for training and validating an Artificial Neural Network (ANN) model is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method400or alternate methods. Additionally, individual blocks may be deleted from the method400without departing from the spirit and scope of the subject matter described herein. Furthermore, the method can be implemented in any suitable hardware, software, firmware, or combination thereof. However, for ease of explanation, in the embodiments described below, the method400may be considered to be implemented in the above described system102.

At block402, data obtained from different data sources may be normalized and validated.

At block404, the ANN model may be trained using the normalized and validated data.

At block406, ANN stochastic data may be generated using the trained ANN model.

At block408, data validation component of a deterministic data model used for the normalizing and the validating of the data may be reused for validating the ANN stochastic data.

Although implementations for methods and systems for modifying data cleansing techniques for training and validating an Artificial Neural Network (ANN) model have been described in language specific to structural features and/or methods, it is to be understood that the appended claims are not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as examples of implementations for modifying data cleansing techniques for training and validating an Artificial Neural Network (ANN) model.