Systems and methods for machine learning based rule discovery for data migration

Systems and methods for deriving classification rules from documents and a database using rule-based machine learning. The method includes extracting first variables from documents corresponding to an organization. The method further includes extracting second variables from a database corresponding to the organization. The method also includes filtering the extracted second variables based on at least one of null values, repeat variables, location variables, ID variables, or data variables. The method further includes deriving first classification rules based on the first variables using a rule-based machine learning algorithm. The method also includes calculating an accuracy of the derived first classification rules. The method also includes deriving second classification rules based on the first variables and the filtered second variables. The method further includes determining a suggested additional variable based on the derived second classification rules and the calculated accuracy.

FIELD OF THE INVENTION

The present invention relates generally to systems and methods for extracting data from documents and databases, including systems and methods for extracting data from documents using natural language processing and extracting data from databases to discover rules using machine learning-based algorithms.

BACKGROUND OF THE INVENTION

Employers often use human resources (HR) software to manage benefits provided to their employees. Eligibility criteria determines whether an employee is eligible for a particular benefit. Generally, employers determine the eligibility criteria based on the benefits provided, and use the eligibility criteria to determine the benefits their employees are eligible for. The HR software assists employers to manage the complexities of determining eligibility criteria.

Most HR software uses Plan Administration Manual (PAM) documents that contains specific eligibility criteria for each benefit. PAM documents are typically converted to HTML pages and made available for internal business review. The finalized criteria are then configured in to the system, which allows the participants to select the benefits based on their eligibility. However, PAM documents are typically updated regularly, altering the specific eligibility criteria for each benefit. Manually analyzing the updated PAM documents and comparing them to current eligibility criteria is time consuming and often leads to inconsistencies.

SUMMARY OF THE INVENTION

Accordingly, an object of the invention is to provide administrators with systems and methods for analyzing eligibility criteria from documents. It is an object of the invention to provide administrators with systems and methods for deriving classification rules for eligibility criteria from documents and databases. It is an object of the invention to provide administrators with systems and methods for deriving classification rules from documents using rule-based machine learning. It is an object of the invention to provide administrators with systems and methods for deriving classification rules from documents and a database using rule-based machine learning. It is an object of the invention to provide administrators with systems and methods for deriving classification rules from documents and databases using rule-based machine learning.

In some aspects, a method for deriving classification rules from documents and a database using rule-based machine learning includes extracting, by a server computing device, first variables from documents corresponding to an organization. The method further includes extracting, by the server computing device, second variables from a database corresponding to the organization. The method also includes filtering, by the server computing device, the extracted second variables based on at least one of null values, repeat variables, location variables, ID variables, or date variables. The method also includes deriving, by the server computing device, first classification rules based on the first variables using a rule-based machine learning algorithm.

Further, the method includes calculating, by the server computing device, an accuracy of the derived first classification rules. The method also includes deriving, by the server computing device, second classification rules based on the first variables and the filtered second variables. Further, the method includes generating, by the server computing device, for display the derived first classification rules, the derived second classification rules, the calculated accuracy, and the suggested additional variable.

In some embodiments, the server computing device is configured to calculate the accuracy of the derived first classification rules based on known classification rules corresponding to the organization.

In some embodiments, the server computing device is further configured to extract the first variables using natural language processing. In other embodiments, the database comprises demographic data, employment data, and benefit plan data.

In some embodiments, the server computing device is further configured to map the extracted first variables to corresponding entries of the database. In other embodiments, the server computing device is further configured to classify each of the extracted first variables and second variables as character-based or numeric. In some embodiments, the first classification rules are derived sequentially using the rule-based machine learning algorithm.

In some embodiments, the server computing device is further configured to derive the second classification rules based on the first variables and the filtered second variables. In other embodiments, the server computing device is further configured to calculate an accuracy of the derived second classification rules based on known classification rules corresponding to the organization. For example. in some embodiments, the server computing device is further configured to generate for display the derived first plurality of classification rules, the derived second plurality of classification rules, the calculated accuracy, and the suggested additional variable.

In some aspects, a system for deriving classification rules from documents and a database using rule-based machine learning includes a server computing device communicatively coupled to a database corresponding to an organization and a display device. The server computing device is configured to extract first variables from documents corresponding to an organization. The server computing device is also configured to extract second variables from the database corresponding to the organization. Further, the server computing device is configured to filter the extracted second variables based on at least one of null values, repeat variables, location variables, ID variables, or date variables. The server computing device is also configured to derive first classification rules based on the first variables using a rule-based machine learning algorithm.

Further, the server computing device is configured to calculate an accuracy of the derived first classification rules. The server computing device is further configured to derive a second plurality of classification rules based on the first variables and the filtered second variables. The server computing device is also configured to generate for display the derived first classification rules, the derived second classification rules, the calculated accuracy, and the suggested additional variable on the display device.

In some embodiments, the server computing device is configured to calculate the accuracy of the derived first classification rules based on known classification rules corresponding to the organization.

In some embodiments, the server computing device is further configured to extract the first variables using natural language processing. In other embodiments, the database comprises demographic data, employment data, and benefit plan data.

In some embodiments, the server computing device is further configured to map the extracted first variables to corresponding entries of the database. In other embodiments, the server computing device is further configured to classify each of the extracted first variables and second variables as character-based or numeric. In some embodiments, the first classification rules are derived sequentially using the rule-based machine learning algorithm.

In some embodiments, the server computing device is further configured to derive the second classification rules based on the first variables and the filtered second variables. In other embodiments, the server computing device is further configured to calculate an accuracy of the derived second classification rules based on known classification rules corresponding to the organization. For example. in some embodiments, the server computing device is further configured to generate for display the derived first plurality of classification rules, the derived second plurality of classification rules, the calculated accuracy, and the suggested additional variable.

Other aspects and advantages of the invention can become apparent from the following drawings and description, all of which illustrate the principles of the invention, by way of example only.

DETAILED DESCRIPTION OF THE INVENTION

Typically, Plan Administration Manual (PAM) documents are finalized by employers prior to each year's enrollment period. During the year, employers could consider tweaking the eligibility criteria. Once approved, the configurations in HR systems are updated along with the PAM documents to maintain the consistency. Sometimes, PAM documents are not updated, leading to discrepancies between the eligibility criteria reflected in the PAM documents and the actual eligibility criteria selected by the employer. At certain times, the criteria provided by employers, correctly finds some participants not eligible for some benefits. However, employers want to provide those benefits to a few selective participants, and so requests to alter participant's eligibility status. Currently, HR systems do not have an option to force the change of eligibility status from the front-end. Hence, this change request is often sent to IT teams to change the eligibility status of the participant in the back-end database. The inconsistency for participant data and the forced status change of eligibility flag also leads to discrepancy.

Migrating from one HR system to another is often time consuming, taking months or even years to accomplish. Migration requires manually discovering eligibility rules for each plan by combining information from PAM documents, participant data and business knowledge. Since there is no single point of truth, the chances of missing critical information are quite high. Hence, the accuracy of the eligibility rules discovered is not guaranteed. Therefore, there is a need to improve the accuracy of migration from one HR system to another, and to decrease the time it takes to do so. The systems and methods described herein overcome these challenges by employing feature engineering techniques and rule-based machine learning.

In some aspects, the systems and methods described herein can include one or more mechanisms or methods for providing administrators with systems and methods for analyzing eligibility criteria from documents. The system and methods can include mechanisms or methods for deriving classification rules for eligibility criteria from documents and databases. The systems and methods described herein can provide administrators with systems and methods for deriving classification rules from documents using rule-based machine learning. The systems and methods described herein can include one or more mechanisms or methods for deriving classification rules from documents and a database using rule-based machine learning. The systems and methods described herein can facilitate administrators with systems and methods for deriving classification rules from documents and databases using rule-based machine learning.

Referring toFIGS.1and2, an exemplary communications system100includes data communications network150, exemplary server computing devices200, and exemplary mobile devices250. In some embodiments, the system100includes one or more server computing devices200and one or more mobile devices250. Each server computing device200can include a processor202, memory204, storage206, and communication circuitry208. Each mobile device250can include a processor252, memory254, storage256, and communication circuitry258. In some embodiments, communication circuitry208of the server computing devices200is communicatively coupled to the communication circuitry258of the mobile devices250via data communications network150. Communication circuitry208and communication circuitry258can use Bluetooth, Wi-Fi, or any comparable data transfer connection. The mobile devices250can include personal workstations, laptops, tablets, mobile devices, or any other comparable device.

An exemplary process300for deriving classification rules from documents and databases is illustrated inFIG.3. As shown, the process300starts with initial variable extraction302from PAM documents350. For example, process300can be implemented on a web self-service application that allows the authorized users to access the system through any web browser like Chrome, Firefox, Safari and Internet Explorer on their local machine. In some embodiments, the User Interface (UI) of the tool is developed in Angular JS and Node.js, and allows the user to select one of the employers (e.g. Office Depot) from a drop-down list. In some embodiments, a user can click a “Generate Report” button, initiating the front-end to send a request to the business logic to automate the process of document exploration from the PAM documents350.

Initial variable extraction302allows for replication of the manual effort of information extraction from PAM documents. In some embodiments, the document exploration and information extraction are performed using Natural Language Processing (NLP). For example, BeautifulSoup is one of the popular libraries provided by Python to scrape data from HTML pages. Every tag in HTML can have attribute information (i.e., class, id, href, and other useful information) that helps in identifying the element uniquely. By analyzing the HTML tags and their attributes in the PAM document350, the benefit plan types and the plan names offered by the employers to its participants can be identified and extracted. Similarly, under the eligibility section of each benefit plan types, the variables and the eligibility criteria rule provided by the employer can be extracted.

Process300continues through initial variable mapping304. For example, the finite set of variables (initial findings) extracted from PAM documents350can be mapped to its corresponding database table and column name. In some embodiments, there are a set of variables which are more frequently used than others. These variables, for example, can be manually mapped and stored to be utilized in the mapping of variables for other employers.

Process300continues through data extraction306from database360. In some embodiments, the PAM documents350have inconsistencies when compared with actual business criteria. Analyzing benefits data from database360can provide critical variables for deriving eligibility, which could be missing in the PAM documents350. In some embodiments, the process of querying and scanning the database360is performed automatically and in parallel to initial variable extraction302. In some embodiments, the database360contains participants demographic data, employment data and selected benefit plan data across 10 tables comprising of total 906 variables. In other embodiments, the variables of the tables include data type, distinct values, maximum and minimum values, etc.

Process300continues through feature engineering308. As shown in relation toFIGS.4and5, the sequence of logical operations performed on the variables to filter them can be based on their data type. On a high level, the variables are classified to either character400or numeric500. The classifications of character variables400include Null values402, only one value404, and greater than one categories406. In some embodiments, the greater than one categories406can include location variables412, identifier variables414, greater than 90% missing variables416, skewed variables418, and non-skewed variables420. The classification of numeric variables500include continues variables502, date variables504, and ID variables506. In some embodiments, the continuous variables502include system variables512and attribute variables514. In some embodiments, the date variables504include notification status variables522and info update variables524.

Generally, not all variables will be utilized by all/some employers. So, many variables among them go unused and contain Null values402. Analyzing the character variables further, many variables such as client name, country (if all participants are from the same region e.g. United States of America) will hold same value for all participants. In some embodiments, such variables do not play any role for deriving eligibility rules and are removed. While location variables such as state, zip code are generally important, their significance varies from one benefit plan to another. In some embodiments, ID variables506such as person_id, payroll_id and system variables512such as object_version_number are insignificant and are removed. Date variables504such as date_of_birth, date_of_joining often play an important role towards the benefit plan eligibility of participants. Through feature engineering308, data exploration, data clean-up and data transformation are performed, which results with filtered set of variables for the machine learning process310.

Process300continues by deriving rules through machine learning310. By applying technologies such as machine learning (ML) to the data, classification rules can be learned and derived. To solve data migration challenges, in some embodiments, tree and rule-based algorithms techniques can be implemented. For example, when implementing decision trees in Python, algorithms require the non-numerical labels to be converted to number labels using label encoder. However, the generated rules from the decision tree may not make business sense. On the other hand, in some embodiments, sequential covering is a general procedure that repeatedly learns a single rule to create a decision list (or set) that covers the entire dataset rule by rule. Many rule-based algorithms are variants of the sequential covering and are effective. In some embodiments, sequential covering algorithms are implemented in R.

In some aspects, by applying ML on the initial findings, eligibility rules for each plan can be generated. Further, the baseline accuracy and misclassifications can be computed. As discussed above, there could be some significant variables driving the eligibility which may have missed out when performing manual analysis. In some embodiments, by applying ML on the filtered variables, eligibility rules can be derived by iteratively adding one variable at a time to initial discovered variables. The new accuracy and misclassification can then be computed and compared with the initial accuracy and misclassifications.

Process300continues by displaying the ML results312. For example, as shown inFIG.6, display results600show the benefit plan names provided by the employer, followed by the initial discovered variables from the PAM documents350and baseline accuracy. In some embodiments, the best ML performance with the initial variables and one additional variable is displayed as well as the suggested list of variables. In some embodiments, display results600also shows the misclassified count (shown in brackets). Display results600allows an administrator the opportunity of having a snapshot view of all the other possible variables that could help improve the rule with higher performance. In some embodiments, display results600can be generated on a mobile device250.

Process300continues through variable reduction314. For example, in some embodiments, administrators with business knowledge can review the list of variables and the derived rules to evaluate if the variable is valid and accepted from a business perspective. In some embodiments, the variable reduction process314can be automated based on historical data and a minimum accuracy required. After variable reduction314, process300finishes by displaying the new results312. For example, in some embodiments, the variables and the values with the conditions that participants should have to be eligible for a particular plan can be displayed. In some embodiments, the results can be exported to be stored or transferred.

The systems and methods described herein use machine learning to facilitate analysis of eligibility rules. Generally, migration efforts from one HR system to another are preceded by a manual analysis of understanding the data before deriving eligibility criteria. In case of eligibility rules migration, an analyst reviews the PAM documents350and maps them to the database360table columns, and then performs the analysis to derive eligibility rules. Due to frequent inconsistencies between the PAM documents350and the actual configured rules in the current system, the manually derived eligibility rules cannot be guaranteed. The analyst requires some business knowledge and understanding of the database360, to iterative select a group of variables to analyze, derive rules and evaluate the performance. In some embodiments, manually deriving eligibility rules for a single employer can take a significant amount of time; months or even years.

Process300can perform the variable extraction from PAM documents350using NLP and query to extract data from database360. Process300can iteratively go through significant variables and apply the ML algorithm to identify the few variables which provides improved eligibility rules. For example, process300can assist administrators by finding hidden eligibility/ineligibility patterns. Further, process300provides improved accuracy and confidence of the results. The output of process300also provides the misclassified count along with accuracy. This allows administrators to focus their energy only on the misclassified participants and their data. Using process300, the migration time can be shortened to weeks or even days.

Referring toFIG.7, a process700for deriving classification rules from documents and a database using rule-based machine learning is illustrated. The process700begins by extracting, by a server computing device200, first variables from documents corresponding to an organization in step702. For example, the server computing device200can be configured to extract first variables from PAM documents350corresponding to an organization. In some embodiments, the first variables are extracted from the PAM documents350using natural language processing.

Process700continues by extracting, by the server computing device200, second variables from a database corresponding to the organization in step704. For example, the server computing device200can be configured to extract second variables from database360corresponding to the organization. In some embodiments, the database360includes demographic data, employment data, and benefit plan data. In some embodiments, the server computing device200is further configured to map the extracted first variables to corresponding entries of the database360. In some embodiments, the server computing device200is further configured to classify each of the extracted first variables and extracted second variables as character variables400or numeric variables500.

Process700continues by filtering, by the server computing device200, the extracted second variables based on at least one of null values, repeat variables, location variables, ID variables, or date variables in step706. As discussed in relation toFIGS.4and5, in some embodiments, some of the extracted second variables are insignificant and are removed. For example, in some embodiments, some of the extracted second variables contain Null values402. In other embodiments, some of the extracted second variables contain ID variables506and system variables512.

Process700continues by deriving, by the server computing device200, first classification rules based on the first variables using a rule-based machine learning algorithm in step708. For example, as discussed in relation toFIG.3, the server computing device200can be configured to implement tree and/or rule-based machine learning algorithms to derive the first classification rules. In some embodiments, the first classification rules are derived sequentially using the rule-based machine learning algorithm.

Process700continues by calculating, by the server computing device200, an accuracy of the derived first classification rules in step710. For example, in some embodiments, the server computing device200is configured to calculate the accuracy of the derived first classification rules based on known classification rules corresponding to the organization.

Process700continues by deriving, by the server computing device200, second classification rules based on first variables and the filtered second variables using a rule-based machine learning algorithm in step712. In some embodiments, the server computing device200can be configured to derive the second classification rules using a rule-based machine learning algorithm. Process700continues by determining, by the server computing device200, a suggested additional variable based on the derived second classification rule and the calculated accuracy in step714. In some embodiments, the server computing device200determines the suggested additional variable by iteratively calculating accuracies when adding variables to the first variables to derive second classification rules. The server computing device200can determine the suggested additional variable based on the variable that resulted in the highest accuracy. Process700finishes by generating, by the server computing device200, for display the derived first classification rules, the derived second classification rules, the calculated accuracy, and the suggested additional variable in step716. For example, in some embodiments, the server computing device200can be configured to generate display results600on a mobile device250.

In some embodiments, the server computing device200is further configured to derive the second classification rules based on the first variables and the filtered second variables. For example, in some embodiments, the server computing device200is further configured to calculate an accuracy of the derived second classification rules based on known classification rules corresponding to the organization. In some embodiments, the server computing device200is further configured to generate for display the derived second classification rules, the calculated accuracy of the derived second classification rules, and the suggested additional variable.

In some aspects, process700can be implemented on a system for deriving classification rules from documents and a database using rule-based machine learning. The system includes a server computing device200communicatively coupled to a database360corresponding to an organization and a display device250. The server computing device200is configured to extract first variables from documents350corresponding to an organization. The server computing device200is also configured to extract second variables from the database360corresponding to the organization. Further, the server computing device200is configured to filter the extracted second variables based on at least one of null values, repeat variables, location variables, ID variables, or date variables. The server computing device200is also configured to derive first classification rules based on the first variables using a rule-based machine learning algorithm.

Further, the server computing device200is configured to calculate an accuracy of the derived first classification rules. The server computing device200is further configured to derive a second plurality of classification rules based on the first plurality of variables and the suggested additional variable. The server computing device200is configured to calculate an accuracy of the derived second classification rules and the suggested additional variable. The server computing device200is also configured to generate for display the derived first classification rules, the derived second classification rules, the calculated accuracy, and the suggested additional variable on the display device250.

To provide for interaction with a user, the above described techniques can be implemented on a computing device in communication with a display device, e.g., a CRT (cathode ray tube), plasma, or LCD (liquid crystal display) monitor, a mobile device display or screen, a holographic device and/or projector, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse, a trackball, a touchpad, or a motion sensor, by which the user can provide input to the computer (e.g., interact with a user interface element). Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, and/or tactile input.

The above-described techniques can be implemented using supervised learning and/or machine learning algorithms. Supervised learning is the machine learning task of learning a function that maps an input to an output based on example input-output pairs. It infers a function from labeled training data consisting of a set of training examples. Each example is a pair consisting of an input object and a desired output value. A supervised learning algorithm or machine learning algorithm analyzes the training data and produces an inferred function, which can be used for mapping new examples.