Systems and methods for expert driven document identification

Systems and methods for identifying data strings in electronic documents using pattern recognition. The method includes receiving a first data string corresponding to an electronic reference document from a first database and a second data string corresponding to an electronic legal document from a second database. The method also includes processing the first data string into a first processed data string and processing the second data string into a second processed data string. The method also includes calculating a cosine similarity between the first processed data string and the second processed data string. The method also includes receiving a feedback score from a user which corresponds to an accuracy of the calculated cosine similarity. The method also includes calculating an adjusted cosine similarity between the first processed data string and the second processed data string based on the calculated cosine similarity and the feedback score.

FIELD OF THE INVENTION

The present invention relates generally to systems and methods for identifying data strings in electronic documents, including systems and methods for identifying data strings in electronic documents using pattern recognition.

BACKGROUND OF THE INVENTION

Generally, organizations create and store electronic documents that contain client data corresponding to the relationships between the organization and the client. For example, contracts are the legally binding agreements, amendments, and other related documents established with clients which helps organizations and institutions stay compliant with existing regulations while safeguarding mutual interests during any unforeseen event. The process of creating and reviewing a contract is completely manual; typically a legal expert (“expert”) will use their overall experience on contract execution to decide on the contents of the contract in such a way as to protect the interests of the involved parties in the best possible manner.

However, many institutions have a significant number of clients, and for each client there can be multiple revisions and amendments within the client's existing electronic documents. For example, if there is a need to identify a specific contract clause from within this vast collection of electronic documents, it may take a substantial amount of time and effort to manually go through all the available electronic documents and manually flag the ones that have the specific clause. Therefore, there is a need for improving the electronic document creation and review process such that less time and effort is required of experts during electronic document creation and review.

SUMMARY OF THE INVENTION

Accordingly, an object of the invention is to provide systems and methods for identifying data strings in electronic documents. It is an object of the invention to provide systems and methods for identifying data strings in electronic documents using pattern recognition. It is an object of the invention to provide systems and methods for searching, identifying, reviewing, and tagging data strings within electronic documents using pattern recognition. It is an object of the invention to provide systems and methods for determining feedback from users in order to automate the improvement of identifying data strings in electronic documents.

In some aspects, a method for identifying data strings in electronic documents using pattern recognition includes receiving, by a server computing device, a first data string corresponding to a first sentence of an electronic reference document from a first database. The method further includes receiving, by the server computing device, a second data string corresponding to a second sentence of an electronic legal document from a second database. The method also includes processing, by the server computing device, the first data string corresponding to the first sentence into a first processed data string. Processing the first data string includes at least one of removing stop words, removing punctuation, removing digits, converting all characters to lower-case, or lemmatization. Further, the method includes processing, by the server computing device, the second data string corresponding to the second sentence into a second processed data string. Processing the second data string includes at least one of removing stop words, removing punctuation, removing digits, converting all characters to lower-case, or lemmatization.

Further, the method also includes calculating, by the server computing device, a cosine similarity between the first processed data string and the second processed data string. The cosine similarity is based on a term frequency-inverse document frequency algorithm and a centroid-based algorithm. The method also includes receiving, by the server computing device, a feedback score from a user. The feedback score corresponds to an accuracy of the calculated cosine similarity between the first processed data string and the second processed data string. The method further includes calculating, by the server computing device, an adjusted cosine similarity between the first processed data string and the second processed data string based on the calculated cosine similarity and the feedback score.

In some embodiments, the server computing device is configured to process the first data string and the second data string by removing stop words. In other embodiments, the server computing device is configured to process the first data string and the second data string by removing punctuation. In some embodiments, the server computing device is configured to process the first data string and the second data string by removing digits. In other embodiments, the server computing device is configured to process the first data string and the second data string by converting all characters to lower-case. In some embodiments, the server computing device is configured to process the first data string and the second data string through lemmatization.

In some embodiments, the term frequency-inverse document frequency algorithm includes comparing words of the first processed data string with words of the second processed data string one word at a time. In other embodiments, the centroid-based algorithm includes representing words of the first processed data string and words of the second processed data string as real-valued vectors.

In some embodiments, the server computing device is configured to calculate the adjusted cosine similarity based on a random forest machine learning algorithm. In other embodiments, the server computing device is configured to generate for display the first sentence of the first plurality of sentences, the second sentence of the second plurality of sentences, and at least one of the calculated cosine similarity, the feedback score, or the calculated adjusted cosine similarity.

In some aspects, a system for identifying data strings in electronic documents using pattern recognition includes a server computing device communicatively coupled to a first database, a second database, and a display device. The server computing device is configured to receive a first data string corresponding to a first sentence of an electronic reference document from the first database. The server computing device is also configured to receive a second data string corresponding to a second sentence of an electronic legal document from the second database. Further, the server computing device is configured to process the first data string corresponding to the first sentence into a first processed data string. Processing the first data string includes at least one of removing stop words, removing punctuation, removing digits, converting all characters to lower-case, or lemmatization. The server computing device is also configured to process the second data string corresponding to the second sentence into a second processed data string. Processing the second data string includes at least one of removing stop words, removing punctuation, removing digits, converting all characters to lower-case, or lemmatization.

Further, the server computing device is configured to calculate a cosine similarity between the first processed data string and the second processed data string. The cosine similarity is based on a term frequency-inverse document frequency algorithm and a centroid-based algorithm. The server computing device is also configured to receive a feedback score from a user. The feedback score corresponds to an accuracy of the calculated cosine similarity between the first processed data string and the second processed data string. Further, the server computing device is configured to calculate an adjusted cosine similarity between the first processed data string and the second processed data string based on the calculated cosine similarity and the feedback score.

In some embodiments, the server computing device is configured to process the first data string and the second data string by removing stop words. In other embodiments, the server computing device is configured to process the first data string and the second data string by removing punctuation. In some embodiments, the server computing device is configured to process the first data string and the second data string by removing digits. In other embodiments, the server computing device is configured to process the first data string and the second data string by converting all characters to lower-case. In some embodiments, the server computing device is configured to process the first data string and the second data string through lemmatization.

In some embodiments, the term frequency-inverse document frequency algorithm includes comparing words of the first processed data string with words of the second processed data string one word at a time. In other embodiments, the centroid-based algorithm includes representing words of the first processed data string and words of the second processed data string as real-valued vectors.

In some embodiments, the server computing device is configured to calculate the adjusted cosine similarity based on a random forest machine learning algorithm. In other embodiments, the server computing device is configured to generate for display the first sentence of the first plurality of sentences, the second sentence of the second plurality of sentences, and at least one of the calculated cosine similarity, the feedback score, or the calculated adjusted cosine similarity.

Other aspects and advantages of the invention can become apparent from the following drawings and description, all of which illustrate the principles of the invention, by way of example only.

DETAILED DESCRIPTION OF THE INVENTION

In some aspects, the systems and methods described herein can include one or more mechanisms or methods for identifying data strings in electronic documents. Specifically, the system and methods described herein can include mechanisms or methods for identifying data strings in electronic documents using pattern recognition. The systems and methods described herein can include one or more mechanisms or methods for searching, identifying, reviewing, and tagging data strings within electronic documents using pattern recognition. The systems and methods described herein can include one or more mechanisms or methods for determining feedback from users in order to automate the improvement of identifying data strings in electronic documents.

The systems and methods described herein can include one or more mechanisms or methods for leveraging the pre-identified legal clauses provided by the experts and finding the probabilistic match within each electronic document available, in order to avoid manual searching. The systems and methods described herein can include one or more mechanisms or methods for creating a legal language search engine which provides a set of electronic documents matching the input legal clauses. The systems and methods described herein can include one or more mechanisms or methods for an expert feedback collection mechanism to improve the results on an ongoing basis.

Referring toFIGS. 1 and 2, an exemplary communications system100includes data communications network150, exemplary server computing devices200, and exemplary client devices250. In some embodiments, the system100includes one or more server computing devices200and one or more client devices250. Each server computing device200can include a processor202, memory204, storage206, and communication circuitry208. Each client device250can include a processor252, memory254, storage256, and communication circuitry258. In some embodiments, communication circuitry208of the server computing devices200is communicatively coupled to the communication circuitry258of the client devices250via data communications network150. Communication circuitry208and communication circuitry258can use Bluetooth, Wi-Fi, or any comparable data transfer connection. The client devices250can include personal workstations, laptops, tablets, mobile devices, or any other comparable device.

The systems and methods described herein can be implemented using communications system100. For example, referring toFIG. 3, an exemplary process300for identifying data strings in electronic documents using the exemplary data communications network100is illustrated. As shown, the process300includes electronic documents310and electronic reference documents320undergoing text processing330and similarity determination340, before being presented to a user using user interface350. In some embodiments, the electronic documents310correspond to contract documents stored on a database implemented by a server computing device200. Similarly, in some embodiments, electronic reference documents320correspond to reference contract clauses stored on a database implemented by a server computing device200. The electronic documents310and electronic reference documents320can be stored on the same database or on different databases. As described further below with respect toFIGS. 4-6, the electronic documents310and electronic reference documents320can be read and fed into the text processing engine330, after which the similarity determination340is applied and the results are populated in the user interface350. The user interface350can be used to determine feedback about the performance of the similarity determination340, which can be utilized in fine-tuning the results presented in the user interface350.

Referring toFIG. 4, an exemplary process400for processing the electronic documents shown inFIG. 3using text processing engine330is illustrated. The text processing engine330processes every incoming sentence of the electronic documents310and320such that they are in a format that can be fed to the similarity determination models340. For example, in some embodiments, the text processing engine330can remove stop-words, remove punctuation, remove digits, convert to lower-case, or lemmatize. Stop words are words which are filtered out before processing of natural language data or text. Stop words are generally the most common words in a language, e.g., the, is, at, which, and on but are not critical for the context in the sentence. For example, removing stop-words from the following sentence: “this is a sample sentence, showing off the stop words filtration,” becomes a processed sentence: “this sample sentence, showing stop words filtration.”

Other examples involve removing punctuation or digits from the pre-processed sentence. For example, removing punctuation from the following sentence: “this is a sample sentence, to show stripping punctuation,” becomes a processed sentence: “this is a sample sentence to show stripping punctuation.” Similarly, removing digits from the following sentence: “this is 1 sample sentence that contains 1 numeric character,” becomes a processed sentence: “this is sample sentence that contains numeric character.” Lemmatization, on the other hand, is the process of converting a word to its base form by considering the context. For example, lemmatizing the following sentence: “sample sentences that shows lemmatization,” results in a processed sentence: “sample sentence that show lemmatization.”

Every sentence in an electronic document310and reference electronic document320is processed by the text processing engine330. The resulting output is processed electronic document410and processed reference document420. Referring toFIG. 5, an exemplary process500for calculating a similarity of data strings in the processed electronic documents410and420using similarity determination models340is illustrated. The similarity determination models340include cosine similarity models using a term frequency-inverse document frequency (“TF-IDF”) algorithm and a centroid-based algorithm.

A TF-IDF cosine similarity algorithm uses “local” context—the words that are used within the clauses from the processed reference documents420are compared with the clauses from the processed electronic documents410one at a time and tagged if found similar based on the co-occurrence of the word alone. This algorithm does not bother with “global” context—in general usage how any word is used and how its occurrences affect the context of the statement is not considered in this approach. On the other hand, a centroid-based cosine similarity algorithm uses Word Embeddings, which provide a “global” context. In some embodiments, combining both algorithms improves the performance of the system and accuracy of the results.

TF-IDF is a measure of originality of a word by comparing the number of times a word appears in a single document with the number of documents the word appears in. For a term i in document j:

where tfi,jis the number of occurrences of i in j, dfiis the number of documents containing i, and N is the total number of documents. As shown inFIG. 5, a TF-IDF matrix510is computed for every processed electronic document410and processed reference document420. For example, when comparing the following sentences: “this is sample sentence” and “this sample sentence is to understand tfidf,” N is 2 and the TF-IDF matrix510is:

Cosine similarity is a metric used to measure how similar the documents are irrespective of their size. Mathematically, it measures the cosine of the angle between two vectors projected in a multi-dimensional space. The cosine similarity is advantageous because even if the two similar documents are far apart by the Euclidean distance (due to the size of the document), chances are they may still be oriented closer together. The smaller the angle, higher the cosine similarity with max value being 1. For two vectors, a and b, the cosine similarity is:

cos⁢θ=a→·b→a→⁢b→=∑1n⁢ai⁢bi∑1n⁢ai2⁢∑1n⁢bi2,
where {right arrow over (a)}·{right arrow over (b)} is the dot product of the two vectors. For the above example, the two sentences have a cosine similarity score of 0.63. The cosine similarity score is calculated for every sentence in the processed electronic document410against every sentences in the processed reference document420. The calculated cosine similarity scores520are arranged in descending order, with highest scores of similarities being on top.

The centroid-based cosine similarity algorithm uses Word Embeddings, which provide a “global” context. Word Embeddings are a type of word representation that allows words with similar meaning to have a similar representation. Using this algorithm, individual words are represented as real-valued vectors in a predefined vector space. Each word is mapped to one vector and the vector values are learned in a way that resembles a neural network. Each word is represented by a real-valued vector, often tens or hundreds of dimensions. The distributed representation is learned based on the usage of words. This allows words that are used in similar ways to result in having similar representations, naturally capturing their meaning. Examples of pre-trained Word Embeddings are word2vec and GloVe. The centroid similarity equates to:

Centroid=∑wi*TF-IDFiTF-IDFi,
a sum of the products of the embedding vectors and TF-IDFs. For the above example, the centroid-based similarity score is 0.65. The centroid-based cosine similarity score is calculated for every sentence in the processed electronic document410against every sentences in the processed reference document420. Similarly, the calculated centroid-based cosine similarity scores520are arranged in descending order, with highest scores of similarities being on top.

As shown inFIG. 6, the output of the similarity determinations models340is represented in user interface350, and includes the respective sentence from the electronic document310, the respective sentence from the electronic reference document320, and their cosine similarity score. In some embodiments, user interface350also includes IDs for the electronic documents310. The user interface350can be used to determine a feedback for each of the presented similarity scores. For example, in some embodiments a user can use the user interface350to provide the feedback. In some embodiments, the user interface350includes a sliding scale that the user can use to provide the feedback.

Using the originally presented similarity scores as independent variables and the feedback provided by the users as the target, a Random Forrest algorithm can be applied to further refine the results presented using the user interface350. A random forest algorithm is a classification algorithm consisting of many decisions trees. It uses bagging and feature randomness when building each individual tree to try to create an uncorrelated forest of trees whose prediction by committee is more accurate than that of any individual tree. For example, the system can use the feedback to create a probability score. The probability score acts as a weight to the similarity scores in the subsequent searches, thereby increasing the model efficiency.

Referring toFIG. 7, a process700for identifying data strings in electronic documents using pattern recognition is illustrated. The process700begins by receiving, by a server computing device200, a first data string corresponding to a first sentence of an electronic reference document320from a first database in step702. Process700continues by receiving, by the server computing device200, a second data string corresponding to a second sentence of an electronic legal document310from a second database in step704.

Process700continues by processing, by the server computing device200, the first data string corresponding to the first sentence into a first processed data string in step706. Process700continues by processing, by the server computing device200, the second data string corresponding to the second sentence into a second processed data string in step708. As described in relation toFIG. 4, processing the first data string and the second data string can include at least one of removing stop words, removing punctuation, removing digits, converting all characters to lower-case, or lemmatization. In some embodiments, the server computing device200is configured to generate a processed electronic document410and a processed reference document420in response to processing the first data string and the second data string.

Process700continues by calculating, by the server computing device200, a cosine similarity between the first processed data string and the second processed data string in step710. As described in relation toFIG. 5, the cosine similarity is based on a term frequency-inverse document frequency algorithm and a centroid-based algorithm. In some embodiments, the term frequency-inverse document frequency algorithm includes comparing words of the first processed data string with words of the second processed data string one word at a time. In some embodiments, the centroid-based algorithm includes representing words of the first processed data string and words of the second processed data string as real-valued vectors.

Process700continues by receiving, by the server computing device200, a feedback score from a user in step712. The feedback score corresponds to an accuracy of the calculated cosine similarity between the first processed data string and the second processed data string. For example, as described in relation toFIG. 6, a user interface350can be configured to determine a feedback from a user. Process700finishes by calculating, by the server computing device200, an adjusted cosine similarity between the first processed data string and the second processed data string based on the calculated cosine similarity and the feedback score in step714. In some embodiments, the server computing device200is configured to calculate the adjusted cosine similarity based on a random forest machine learning algorithm. In some embodiments, the server computing device200is configured to generate for display the first sentence, the second sentence, and at least one of the calculated cosine similarity, the feedback score, or the calculated adjusted cosine similarity using user interface350.

In some aspects, the above-described techniques can be implemented on a system for identifying data strings in electronic documents using pattern recognition. The system includes a server computing device200communicatively coupled to a first database, a second database, and a display device. The server computing device200is configured to receive a first data string corresponding to a first sentence of an electronic reference document320from the first database. The server computing device200is also configured to receive a second data string corresponding to a second sentence of an electronic legal document310from the second database. Further, the server computing device200is configured to process the first data string corresponding to the first sentence into a first processed data string. The server computing device200is also configured to process the second data string corresponding to the second sentence into a second processed data string. Processing the first data string and the second data string includes at least one of removing stop words, removing punctuation, removing digits, converting all characters to lower-case, or lemmatization. The server computing device200is also configured to calculate a cosine similarity between the first processed data string and the second processed data string. The cosine similarity is based on a term frequency-inverse document frequency algorithm and a centroid-based algorithm. Further, the server computing device200is configured to receive a feedback score from a user. The feedback score corresponds to an accuracy of the calculated cosine similarity between the first processed data string and the second processed data string. The server computing device200is also configured to calculate an adjusted cosine similarity between the first processed data string and the second processed data string based on the calculated cosine similarity and the feedback score.

The systems and methods described herein address the concerns related to current methods of identifying similarities between data strings in electronic documents. The systems and methods described herein combines local and global context to provide end users with improved performance and results For example, instead of using supervised machine learning which requires annotated datasets, the systems and methods described herein uses unsupervised machine learning from the outset. The systems and methods described herein provides end users with interim results using a unique and easy to use user interface which also allows for feedback determination. The systems and methods described herein also uses feedback to train a supervised learning model. Further, the systems and methods described herein provides a continuous learning enablement which improves the effectiveness of data string identification in electronic documents.

To provide for interaction with a user, the above described techniques can be implemented on a computing device in communication with a display device, e.g., a CRT (cathode ray tube), plasma, or LCD (liquid crystal display) monitor, a mobile device display or screen, a holographic device and/or projector, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse, a trackball, a touchpad, or a motion sensor, by which the user can provide input to the computer (e.g., interact with a user interface element). Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, and/or tactile input.

The above-described techniques can be implemented using supervised learning and/or machine learning algorithms. Supervised learning is the machine learning task of learning a function that maps an input to an output, based on example input-output, pairs. It infers a function from labeled training data consisting of a set of training examples. Each example is a pair consisting of an input object and a desired output value. A supervised learning algorithm or machine learning algorithm analyzes the training data and produces an inferred function, which can be used for mapping new examples.