METHODS AND SYSTEMS FOR INTELLIGENT TEXT CLASSIFICATION WITH LIMITED OR NO TRAINING DATA

Methods and apparatuses are described for intelligent text classification with limited or no training data. A server computing device receives one or more of structured text or unstructured text corresponding to compliance text data from a database. The server computing device executes a trained few-shot natural language inference (NLI) classification model on one or more sentences in the received compliance text data to identify whether the one or more sentences comprise a compliance violation. The server computing device transmits the results of the model execution to a remote computing device.

TECHNICAL FIELD

This application relates generally to methods and apparatuses, including computer program products, for intelligent text classification with limited or no training data.

BACKGROUND

Many industries, including financial services, must follow strict regulatory rules in their communications with the public. For example, regulatory content standards require that communications are fair and balanced and are not promissory, exaggerated, unwarranted, or misleading to investors. There is significant expense to comply with regulatory rules due to the cost of hiring highly qualified staff to review and remediate materials, and file and address comments with respective regulatory agencies. With recent advances in Natural Language Processing (NLP) technologies, it is increasingly becoming possible to automatically flag high-risk language and thus reduce the cost of compliance reviews.

Current technology has applied text classification techniques to legal texts. For example, there are methods based on counting the words in the text and then classifying using methods such as support vector machines (as described in Cortes and Vapnik, 1995, Support-vector networks, Machine learning, 20(3):273-297 (incorporated herein by reference)) for example by Sulea et al., 2017, Exploring the use of text classification in the legal domain. In Proceedings of the Second Workshop on Automated Semantic Analysis of Information in Legal Texts co-located with the 16th International Conference on Artificial Intelligence and Law (ICAIL 2017), London, UK, Jun. 16, 2017, volume 2143 of CEURWorkshop Proceedings, CEUR-WS.org. (incorporated herein by reference), where they applied this method to the classification of texts according to the legal area, ruling and time span of the text. Deep learning methods such as Convolutional Neural Networks (CNNs) have been shown to further improve the performance of such systems (as described in Wei et al., 2018, Empirical study of deep learning for text classification in legal document review, In IEEE International Conference on Big Data, Big Data 2018, Seattle, Wash., USA, Dec. 10-13, 2018, pages 3317-3320, IEEE (incorporated herein by reference)). More recently, the emergence of large pretrained language models such as BERT (as described in Devlin et al., 2019, BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume1(Long and Short Papers), pages 4171-4186, Minneapolis, Minn. Association for Computational Linguistics (incorporated herein by reference)) has further increased the performance and Shaheen et al., 2020, Large scale legal text classification using transformer models. CoRR, available at arxiv.org/abs/2010.12871 (incorporated herein by reference) showed that these models could be used to classify legal texts according to thousands of labels and even on multiple languages if sufficient training data exists.

A criticism of such NLP-based approaches to predictive coding, especially with the emergence of more sophisticated deep learning methods, is that they can appear to be ‘black boxes,’ and thus there has been work in providing explicable systems (as in Mahoney et al., 2019, A framework for explainable text classification in legal document review. In 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, Calif., USA, Dec. 9-12, 2019, pages 1858-1867, IEEE (incorporated herein by reference)) that can identify snippets and provides explanations for why they make certain predictions. Similarly, some work has gone into the investigation of specific complexities of legal texts, such as in Nallapati and Manning, 2008, Legal docket classification: Where machine learning stumbles. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 438-446, Honolulu, Hi., Association for Computational Linguistics (incorporated herein by reference), who showed that for some legal texts the complex combination of negative and positive statements can confuse machine learning approaches. Nallapati and Manning showed that by combining these machine learning approaches with propositional logic, text classification systems could handle intricate legal wording.

However, artificial intelligence (Al) models such as NLP models generally require a large amount of relevant, high-quality data. The more labeled data, the more accurate the machine learning model predictions can be. A lack of professional expertise in labeling data can lead to unreliable Al software. As can be appreciated, creating that data is manual and expensive, especially in regulatory technology regarding the cost of legal and compliance experts for manual review and quality assurance.

In addition, existing technology for compliance detection of content standards violations is primarily lexicon-based. For example, these systems generally flag any content that includes certain keywords such as “guarantee.” This approach is inefficient and can lead to missed violations that don't include specific keywords and incorrectly flagging content that is not problematic (in this example, incorrectly flagging content like ‘guarantee period’ in disclosure or an annuity guarantee).

Finally, most existing methods of text classification only consider the local features of the samples, and their experimental results show better performance than traditional non-deep learning methods. However, in these methods, the global features of the sample are usually ignored, and these ignored global features will affect the classification accuracy.

SUMMARY

Therefore, what is needed are computerized methods and systems to overcome the above-described challenges and provide for the utilization of Natural Language Inference (NLI) to allow Al models to classify sentences in certain violation categories for regulatory content standards with limited labeled data by experts. The NLI techniques described herein are capable of understanding the compositional semantic and language context of documents (text, transcripts of audio/video input, etc.) in order to predict the classes of violations. This NLI approach takes the text to be predicted as the premise and the classifications, i.e. promissory/other, as hypothesis. When the NLI model predicts that the premise “entails” the hypothesis, the model takes the label to be true-turning NLI into classification for compliant classes of content standards.

The technology described herein advantageously uses a triple capsule network architecture for classifying compliance-related text. This architecture classifies if a given sentence is in a particular class of compliance violation to a regulatory content standard. Triplet loss enables the network to distinguish between positive and negative examples of a class. This distinction allows end users to understand the specific reason/risk as it relates to the regulatory content standards to enable them to understand the problem and to remediate accordingly.

The network learns sentence representations where examples of the same class are close together. The closeness of two sentences can be measured by calculating the euclidean distance between their representation. For the final classification, the methods and systems use Support Vector Machine (SVM) with Radial Basis Function (RBF) kernel. SVM is used for classification as it learns by minimizing the hinge loss which is like the loss used for training the triplet network.

The invention, in one aspect, features a computerized method of intelligent text classification with limited or no training data. A server computing device receives one or more of structured text or unstructured text corresponding to compliance text data from a database.

The server computing device executes a trained few-shot natural language inference (NLI) classification model on one or more sentences in the received compliance text data to identify whether the one or more sentences comprise a compliance violation. The server computing device transmits output from the model execution to a remote computing device.

The invention, in another aspect, features a system for intelligent text classification with limited or no training data. The system comprises a server computing device that receives one or more of structured text or unstructured text corresponding to compliance text data from a database. The server computing device executes a trained few-shot natural language inference (NLI) classification model on one or more sentences in the received compliance text data to identify whether the one or more sentences comprise a compliance violation. The server computing device transmits output from the model execution to a remote computing device.

Any of the above aspects can include one or more of the following features. In some embodiments, the trained few-shot NLI classification model comprises a plurality of instances of a same neural network with shared parameters. In some embodiments, each neural network instance of the trained few-shot NLI classification model receives a different text sample from the received text. In some embodiments, a first neural network instance receives a positive text sample, a second neural network instance receives an anchor text sample, and a third neural network instance receives a negative text sample. In some embodiments, the anchor text sample and the positive text sample correspond to a first class and the negative text sample corresponds to a second class.

In some embodiments, each neural network instance comprises an encoder layer, a perceptron layer comprising a first fully connected layer and a rectified linear activation function (ReLU) layer, and a second fully connected layer. In some embodiments, the trained few-shot NLI classification model generates a first output comprising (i) a first distance between the positive text sample processed by the first neural network instance and the anchor text sample processed by the second neural network instance and (ii) a second distance between the anchor text sample processed by the second neural network instance and the negative text sample processed by the third neural network instance. In some embodiments, the first distance comprises a Euclidian distance and the second distance comprises a Euclidian distance.

In some embodiments, the server computing device applies a triplet loss function to the first distance and the second distance to retrain the few-shot natural language inference (NLI) classification model. In some embodiments, the server computing device classifies output from the trained few-shot NLI classification model using a support vector machine (SVM) with radial basis function (RBF) kernel. In some embodiments, when the SVM with RBF kernel classifies the output from the trained few-shot NLI classification model as comprising a compliance violation, the remote computing device transmits an alert message to a client computing device for remediation of the compliance violation.

DETAILED DESCRIPTION

FIG.1is a block diagram of a system100for intelligent text classification with limited or no training data. The system100includes a client computing device102, a communications network104, a server computing device106that includes a text classification module108, and a database114that includes text data.

The client computing device102connects to the communications network104in order to communicate with the server computing device106to provide input and receive output relating to the process of intelligent text classification with limited or no training data as described herein. Exemplary client computing devices102include but are not limited to computing devices such as smartphones, tablets, laptops, desktops, or other similar devices. It should be appreciated that other types of devices that are capable of connecting to the components of the system100can be used without departing from the scope of invention.

The communications network104enables the client computing device102to communicate with the server computing device106. The network104is typically a wide area network, such as the Internet and/or a cellular network. In some embodiments, the network104is comprised of several discrete networks and/or sub-networks (e.g., cellular to Internet, PSTN to Internet, PSTN to cellular, etc.).

The server computing device106is a device including specialized hardware and/or software modules that execute on a processor and interact with memory modules of the server computing device106, to receive data from other components of the system100, transmit data to other components of the system100, and perform functions for intelligent text classification with limited or no training data as described herein. The server computing device106includes a text classification module108that executes on one or more processors of the server computing device106. In some embodiments, the module108is a specialized set of computer software instructions programmed onto one or more dedicated processors in the server computing device106and can include specifically-designated memory locations and/or registers for executing the specialized computer software instructions.

It should be appreciated that any number of computing devices, arranged in a variety of architectures, resources, and configurations (e.g., cluster computing, virtual computing, cloud computing) can be used without departing from the scope of the invention. The exemplary functionality of the text classification module108is described in detail throughout this specification.

In some embodiments, the text classification module108can comprise a software program that receives text data (e.g., compliance related text data/documents in the form of structured or unstructured text) from database114and processes the text data as described herein to classify the text (e.g. according to compliance violation parameters) and provide the classified text to a remote user.

The database114is a computing device (or in some embodiments, a set of computing devices) coupled to the server computing device106and is configured to receive, generate, and store specific segments of data relating to the process of intelligent text classification with limited or no training data as described herein. In some embodiments, all or a portion of the database114can be integrated with the server computing device106or be located on a separate computing device or devices. The database114can comprise one or more databases configured to store portions of data used by the other components of the system100, as will be described in greater detail below.

FIG.2is a block diagram of a triple capsule network architecture200for intelligent text classification, used by the text classification module108of server computing device106ofFIG.1. As shown inFIG.2, the triple capsule network200comprises three instances (202a,202b,220c) of the same neural network with shared parameters. The network takes as input three examples in each sample. The three samples consist of the anchor204a(s), positive204b(s+) and negative 204c (s-) example. The anchor204aand positive204bexample belong to the same class, while the negative 204c example belongs to a different class. The network200outputs two values, the distance206abetween the anchor and the positive example and the distance206bbetween the anchor and the negative example.

FIi.3is a flow diagram of a computerized method300of intelligent text classification with limited or no training data, using system100ofFIG.1. The text classification module108receives (step302) a corpus of structured and/or unsmuictured text from the database114for classification. The text classification module108executes (step304) a trained few-shot NLI classification model on sentences from the received compliance text data to identify whether the received text includes a compliance violation. During execution of the model, the network200in text classification Module108encodes each incoming sentence using a Sentence-Bert (S-Bert) Encoder (e.g.,208) (as described in Reimers and Gurevych, 2019. Sentence-bert: Sentence embeddings using siamese hert-networks. In Proceedings of the2019Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Naiaral Language Processing, EMN LP-lJCNLP2019, Hong Kong, China, Nov. 3-7, 2019, pages3980---3990, Association for Computational Linguistics (incorporated herein by reference)). The Sentence-Bert Encoder captures the contextual information in a sentence in a fixed size vector representation. The contextual sentence embedding is then fed to a two-layer perceptron: fully connected layer (FC) and rectified linear activation function layer (ReLU) (layer210) and another FC layer (layer212). The hidden layer210has ReLL activation for introducing non-linearity in the perceptron.

Exemplary algorithms used by the layers of the neural network are below:

and the parameter matrices to be learned during training., Triplet loss (as described in Hoffer and Ailon, 2015, Deep metric learning using triplet network. In Similarity-Based Pattern Recognition-Third International Workshop, SIMBAD 2015, Copenhagen, Denmark, Oct. 12-14, 2015, Proceedings, volume9370of Lecture Notes in Computer Science, pages 84-92, Springer (incorporated herein by reference)) has been used in few-shot classification methods. Although introduced for images, it has been successfully adapted in natural language processing. Triplet loss () enables the network to distinguish between positive and negative examples of a class. It is defined in the equation below:

where α is the anchor sentence, p is a sentence drawn from the same class as a and n is a sentence drawn from a class different from that of a. The function d computes the distance between two sentences and a is the margin enforced between the positive and negative examples.

The function d is defined below:

where II.2denotes the 12 norm. The triplet loss is leveraged to train the network model.

The network learns sentence representations where examples of the same class are close together. The closeness of two sentences can be measured by calculating the euclidean distance between their representation. For the final classification, the system uses a Support Vector Machine (SVM) with Radial Basis Function (RBF) kernel. An exemplary SVM with RBF kernel is described in K. Thurnhofer-Hemsi et al., “Radial basis function kernel optimization for Support Vector Machine classifiers,” arXiv:2007.08233[cs.LG], 17 Jul. 2020, which is incorporated herein by reference. The system uses an SVM for classification as it learns by minimizing the hinge loss which is similar to the loss used for training the triplet network.

As can be appreciated, the systems and methods described herein can be applied to structured or unstructured text in any of a variety of different subject matter areas or domains.

Exemplary domains include but are not limited to financial services, compliance, governmental regulation, pharmaceutical, and legal. In one example use case, the systems and methods described herien can be applied for regulatory compliance in the financial domain under, e.g., the U.S. regulation FINRA22101(described at www.finra.org/rules-guidance/rulebooks/finra-rules/2210, which states that “no member may make any false, exaggerated, unwarranted, promissory or misleading statement or claim in any communication.”). An exemplary text classification generated by the system100under this regulation is provided inFIG.4. As shown inFIG.4, the first example 402 displays a contradiction in that the hypothesis statement (“77% of Americans anxious over financial situation”) contradicts the premise (“Stop worrying, the best returns are yet to come.”) and would not be labeled as a promissory compliance violation. The second example 404 displays entailment in that the hypothesis statement (“You'll never have to worry, this will take the worry out of your retirement.”) confirms the premise (“Stop worrying, the best is yet to come.”) and thus the hypothesis statement is labeled as a promissory violation.

In some embodiments, when the text classification module108classifies the received text as either containing a compliance violation or not containing a compliance violation, server computing device106transmits (step306) output from the module108to a remote computing device (e.g., client computing device102). For example, server computing device106can transmit one or more data packets to client computing device102that include data (e.g., a flag, a text string, etc.) that indicates whether the received text comprises a compliance violation or not.

Upon receiving the output, client computing device102can take one or more actions based upon the content of the output. In one example, client computing device102can transmit an alert message to one or more other computing devices that indicates the text includes a compliance violation and requests that the violation is remediated. In some embodiments, the alert message can include the specific text (e.g., one or more sentences) that were analyzed by text classification module108and determined to contain a violation along with a reference to a location of the text (e.g., document name, document number, version, etc.).

The systems and methods were trained and tested in several different settings: first, in a traditional data-heavy supervised setting, where a large number of existing examples have been classified; second, in a zero-shot training situation, where an expert was to provide only rough guidelines for what is not compliant with the legal code; and third, combining this in a few-shot setting where with comparatively little training data, the system achieves performance that is equivalent with the data-heavy supervised setting and thus enables text classification systems for regulatory compliance to be constructed quickly and with little effort allowing them to cover a wide range of industries and national regulatory frameworks.

In the experimental setup, the dataset was split into training, development, and test datasets. These datasets comprise varying numbers of promissory and non-promissory sentences. For the zero-shot learning model, the system samples40promissory and190non-promissory example sentences from the training set and trains the model on this subset.

The classification performance of the few-shot learning model described herein was compared against existing supervised learning methods:Naive Bayes: We train a Naive Bayes classification model using tf-idf scores of the tokens in the sentence.Multi Layer Perceptron (MLP): We train a two layer perceptron with ReLu activation in the hidden layer using the tf-idf scores of the sentence tokens as input features to the model.SVM: Similar to the MLP model, we train a SVM model for the classification task. We set the regularization parameter C and gamma to1.0and0.1respectively.Sentence-Bert: This setting is similar to our proposed approach. We encode each sentence into a fixed sized vector using its Sentence-Bert embedding. The sentence embedding is then fed into a 3 layer fully connected neural network with ReLu activation in the first two layers. The model is trained by minimizing the CrossEntropy Loss of classification using Adam optimizer.Laser: In this setting, we encode each sentence using its Laser embeddings. The remaining architecture remains the same as that using in the Sentence-Bert model.

In addition to the supervised approaches, we compare the few-shot learning approach against a zero-shot learning approach. Yin et al., 2019, Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach, In Proceedings of the2019Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, Nov. 3-7, 2019, pages 3912-3921, Association for Computational Linguistics (incorporated herein by reference) suggested a method for using pre-trained natural language inference models as sequence classifiers. To this end, the text classification module108uses the BART model (described in Lewis et al., 2020, BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, A C L2020, Online, Jul. 5-10, 2020, pages 7871-7880, Association for Computational Linguistics (incorporated herein by reference)) as the zero-shot learning model. The text classification module108considers the sentences tagged as ‘promissory’ as hypothesis. The probability of a sentence being the premise for these tagged sentence is calculated using the BART model. The module108then considers the maximum of those scores, and if the maximum score is greater than 0.7, the module108classifies the sentence as a promissory sentence.

For the task, the module108uses the Sentence-Bert base model. It encodes an sentence into a fixed size vector of length768. The module108sets de1, de2and de3to768,300and10respectively. For every positive sentence, the module108supplies three negative sentences for the anchor sentence. The value of a is set to 1:0. The batch size is set to 16 for the triplet network and is trained with Adam optimizer (as described in Kingma and Ba, 2015, “Adam: A method for stochastic optimization,” In 3rd International Conference on Learning Representations, ICLR2015, San Diego, Calif., USA, May 7-9, 2015, Conference Track Proceedings (incorporated herein by reference)) with a learning rate of le-5for 10 epochs. The module108sets the cost parameter C and gamma of the SVM to0.03and0.1respectively.

As shown in Table 1 below, the few-shot method with very limited sentences provides solid results for precision, recall, and accuracy in comparison to the other supervised learning models:

Appendix A attached hereto provides further experimental test results that show the benefits of the few-shot text classification architecture described herein.

To provide for interaction with a user, the above described techniques can be implemented on a computing device in communication with a display device, e.g., a CRT (cathode ray tube), plasma, or LCD (liquid crystal display) monitor, a mobile device display or screen, a holographic device and/or projector, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse, a trackball, a touchpad, or a motion sensor, by which the user can provide input to the computer (e.g., interact with a user interface element).

Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, and/or tactile input.

APPENDIX A

TF-IDF vectorization with different baseline models.PG-1T

Deep Learning Experiments Results:

Classification using LASER embeddings:

AUC Curve

Classification Result on Label2

For label2, we consider the two classes to be “promissory” and “rest”

Zero Shot Learning

Method: The 40 sentences are treated as classes and the probability of a sentence lying in those classes is calculated. We take the max of those scores, and if the max is greater than 0.7, we classify it as promissory.

Few Shot Siamese Network

We sample40examples from the promissory cases and190examples from the non-promissory cases. We then learn a compact representation of the sentences using SBert and triplet loss. For final classification we use SVM since triplet loss draws a margin between examples.