Patent ID: 12229644

DETAILED DESCRIPTION

Overview

A supervised text classification dataset often includes highly imbalanced data which has a few classes that contain a very large number of text records (i.e., large classes, which are also known as major classes) and a few classes that contain a very small number of text records (i.e., small classes, which are also known as minor classes). New records in a dataset to be classified can confuse a supervised machine learning algorithm if the new records are to be placed in small classes because the algorithm is biased towards placing records in large classes. Thus, the highly imbalanced data decreases performance of a text classification machine learning model (i.e., supervised text classification model), in terms of decreased accuracy, decreased F1 score, and similar effects on other parameters. Current approaches to balancing a dataset include (1) text record level methods of oversampling of small class text records and under-sampling large class text records; (2) word level methods of (i) randomly replacing any word by its synonym or antonym, per a language dictionary, (ii) randomly replacing any word by an equivalent word generated by a static word embeddings model based on cosine similarity, (iii) randomly replacing any word by an equivalent word generated by a contextual language model based on surrounding words in a text record, (iv) randomly inserting any word at any position in the text record, and (v) randomly deleting any word at any position in a text record; and (3) character level methods of randomly inserting any character at any position in any word in a text record and randomly deleting any character at any position in any word in a text record. Using the current approaches for balancing the dataset, the text classification machine learning model remains deficient in terms of accuracy and F1 score. The aforementioned static word embeddings model and the contextual language model do not consider class-specific word structure. Furthermore, contextual language models generate different embeddings for the same word in different contexts whereas static word embeddings models do not consider the context of the word. In the aforementioned word level methods, synonyms may not consider word context and class-specific word structure and some words may not have synonyms.

As used herein, an F1 score is defined as the harmonic mean of a supervised machine learning model's precision and recall and is a measure of the model's accuracy on a dataset.

Embodiments of the present invention address the aforementioned unique challenges of traditional text classification techniques by providing an approach of text augmentation of a small class to balance a dataset to improve the performance of a text classification machine learning model in terms of accuracy, F1 score, and generalizability. In one embodiment, text augmentation of a small class balances the dataset from a class point of view rather than at a record level. In one embodiment, text augmentation of a small class in text classification problems uses a combination of word importance statistics, natural language processing (i.e., lexical natural language features and syntactic natural language features), and natural language generation (i.e., word context). In one embodiment, the text augmentation approach includes selecting words to be replaced in the text record by using lexical features, syntactic features, and word importance statistics. In one embodiment, the text augmentation approach includes generating replacement word(s) by using lexical features, contextual relevance, and word importance statistics.

System for Augmenting Text of a Small Class

FIG.1is a block diagram of a system100for augmenting text of a small class, in accordance with embodiments of the present invention. System100includes a computer102that includes a software-based small class text augmentation system104, which includes a main module106, a text augmentation module108, a word priority module110, a suitable word generation module112, and a class word score module114.

Small class text augmentation system104receives an imbalanced dataset116, which is a supervised text classification dataset. As used herein, an imbalanced dataset is a dataset that includes (1) a relatively small number of classes (i.e., large class) that include a substantially large number of text records and (2) a relatively small number of classes (i.e., small class) that include a substantially small number of text records. As an example, imbalanced dataset116may include a single large class that includes 65% of the total number of text records in imbalanced dataset116and a single small class that includes 1% of the total number of text records. As used herein, “relatively small number of classes” means a number of classes that is substantially less than the total number of classes that categorize text records in the dataset.

Main module106receives the imbalanced dataset116and reads the text records and their classes in the imbalanced dataset116. Main module106identifies a small class among the classes in the imbalanced dataset116and sends the small class to text augmentation module108, which augments the text records in the small class and sends back to main module106(i) the old text records (i.e., initial text records) that were initially in the small class in the imbalanced dataset116and (ii) new augmented text records. Main module106creates a balanced dataset118that includes the old text records and the new augmented text records. Although not shown inFIG.1, small class text augmentation system104sends the balanced dataset118to a text classification model (i.e., supervised machine learning model) and trains the text classification model on the balanced dataset118. Furthermore, small class text augmentation system104uses the trained text classification model to perform a text classification on a new dataset whose domain matches the domain of the imbalanced dataset116. The performance of the text classification in terms of accuracy and F1 score is enhanced because the text classification model was trained on balanced dataset118instead of imbalanced dataset116.

Text augmentation module108sends text records of the small class to word priority module110, which selects the words in the text records that can be replaced and determines in what order the selected words can be replaced. Word priority module110selects the words to be replaced by considering lexical features, syntactic features, and word importance statistics (i.e., word frequency statistics). Word priority module110creates a word priority list that includes the selected words in a descending order based on respective word scores calculated by class word score module114. The word scores indicate a measure of importance of the corresponding words from a class point of view.

Text augmentation module108sends each word in the aforementioned word priority list and the corresponding text record to suitable word generation module112, which generates a suitable word list for a given word. The suitable word list includes word(s) that text augmentation module108uses to replace the given word, thereby creating a new augmented text record. Suitable word generation module112selects the word(s) in the suitable word list based on suitable word scores that are based on cosine similarity scores with class-based word importance statistics, contextual probability scores with class-based word importance statistics, and synonym-based class word scores. Class word score module114calculates class word scores that are used in conjunction with the aforementioned cosine similarity scores, contextual probability scores, and synonym-based class word scores to generate suitable word scores.

In one or more embodiments, system100includes the components of system200inFIG.2and the components of system300inFIG.3.

The functionality of the components shown inFIG.1is described in more detail in the discussion ofFIG.2,FIG.3,FIG.4,FIG.5,FIGS.6A-6B,FIGS.7A-7B,FIGS.8A-8B, andFIG.9presented below.

FIG.2is a block diagram of a system200for determining words in a text record that can be replaced, where the system is implemented within the system ofFIG.1, in accordance with embodiments of the present invention. System200includes word priority module110, class word score module114, and the following software-based modules: a part of speech (POS) module202, a stop word module204, and a dependency module206.

Word priority module110identifies words in the text record that can be replaced and in what order the identified words can be replaced. Word priority module110receives a text record from text augmentation module108(seeFIG.1) and sends the text record to class word score module114, POS module202, stop word module204, and dependency module206. Class word score module114returns class word scores to word priority module110. POS module returns POS scores to word priority module110. Stop word module returns stop word scores to word priority module110. Dependency module206returns dependency scores to word priority module110.

For a given word in the received text record, word priority module110weights the class word score, the POS score, the stop word score, and the dependency score with different respective weights, and then calculates a word priority score by adding the aforementioned weighted scores. Word priority module110determines whether the word priority score exceeds a defined threshold score. If the word priority score exceeds the threshold score, word priority module110adds the word corresponding to the word priority score to a word priority list, which is a list of words that can be replaced to create an augmented text record. After similarly processing the other words in the received text record, word priority module110sends the resulting word priority list to text augmentation module108(seeFIG.1).

POS module202uses a POS tagger model (not shown) to generate respective POS scores of the words in the text record. The POS tagger model determines respective lexical categories of the words in the text record.

Stop word module204uses a stop word list (not shown) to generate respective stop word scores of the words in the text record. The stop word list includes stop words, which are words that are commonly used words in a natural language (e.g., “a,” “an,” “the,” etc.) and which add little or no value to text classification.

Dependency module206uses a dependency parser model (not shown) to generate respective dependency scores for the words in the text record. The dependency parser model determines a syntactic dependency relationship between the words of the text record by analyzing the grammatical structure of the sentences that include the words.

The functionality of the components shown inFIG.2is described in more detail in the discussion ofFIG.4,FIG.5,FIGS.6A-6B,FIGS.7A-7B,FIGS.8A-8B, andFIG.9presented below.

FIG.3is a block diagram of a system300for generating word(s) that are suitable replacements for a given word in the word priority list generated by system200(seeFIG.2), where system300is implemented within the system ofFIG.1, in accordance with embodiments of the present invention. System300includes suitable word generation module112, which includes a synonym based sub-module302, a static embeddings similarity based sub-module304, and a contextual language based sub-module306. System300also includes text augmentation module108, class word score module114, imbalanced dataset116, and the following software-based modules: a static embeddings module308, a static embeddings similarity module310, and a contextual language module312.

Suitable word generation module112generates a suitable word list that includes word(s) that are suitable replacements for a given word by considering cosine similar word embeddings with class-based word importance statistics, synonyms with class-based word importance statistics, and contextual words with class-based word importance statistics. Suitable word generation module112receives from text augmentation module108the word priority list and the corresponding text record. Suitable word generation module112sends the word priority list to synonym based sub-module302, which returns suitable word scores corresponding to the words in the word priority list. Suitable word generation module112sends the word priority list to static embeddings similarity module310, and in response receives similar words (i.e., words similar to the words in the word priority list) and cosine similarity scores of the similar words. Suitable word generation module112sends the word priority list and the corresponding text record to contextual language module312, and in response receives likely words (i.e., words that are contextually relevant to words in the word priority list) and probability scores of the likely words.

Static embeddings module308generates top k similar tokens for a given token in the vocabulary of a transfer learning based static embeddings model. Static embeddings module308sends a static embeddings similarity list to static embeddings similarity module310. The static embeddings similarity list includes the top k similar tokens and their cosine similarity scores.

Static embeddings similarity module generates top k similar words for a given word in a word priority list using the static embeddings model. Static embeddings similarity module310sends the aforementioned similar words and cosine similarity scores of the similar words to static embeddings similarity based sub-module304. Class word score module114sends class word scores to static embeddings similarity based sub-module304. Using the cosine similarity scores and the class word scores, static embeddings similarity based sub-module304generates suitable word scores.

Contextual language module312generates top j likely words for a given word by using a contextual language model. Contextual language module312sends likely words and contextual probability scores of the likely words to contextual language based sub-module306. Class word score module114sends class word scores to contextual language based sub-module306. Using the contextual probability scores and the class word scores, contextual language based sub-module306generates suitable word scores.

Class word score module114sends class word scores to synonym based sub-module302. Using the class word scores, synonym based sub-module302generates suitable word scores.

The functionality of the components shown inFIG.3is described in more detail in the discussion ofFIG.4,FIG.5,FIGS.6A-6B,FIGS.7A-7B,FIGS.8A-8B, andFIG.9presented below.

Process for Augmenting Text of a Small Class

FIG.4is a flowchart of a process of augmenting text of a small class, where the process is implemented in the system ofFIG.1, in accordance with embodiments of the present invention. The process ofFIG.4begins at a start node400. In step402, small class text augmentation system104(seeFIG.1) receives imbalanced dataset116(seeFIG.1) for supervised machine learning text classification.

In step404, small class text augmentation system104(seeFIG.1) (1) obtains text records in imbalanced dataset116(seeFIG.1) and classes of the text records by reading imbalanced dataset116(seeFIG.1); (2) obtains counts of text records in each of the classes; and (3) identifies a class having the lowest count of text records (i.e., identifies a small class). Hereinafter, text records in the small class identified in step404are referred to as initial text records. In one embodiment, main module106performs step404.

In step406, main module106(seeFIG.1) sends the initial text records in the identified small class to text augmentation module108(seeFIG.1).

In step408, text augmentation module108(seeFIG.1) augments the initial text records in the small class (also known as text augmentation of the small class). The augmentation of the initial text records includes generating new augmented text records and adding the new augmented text records to the initial text records.

Text augmentation module108(seeFIG.1) generates the new augmented text records by using (1) class word score module114to determine word importance statistics, (2) word priority module110(seeFIG.1) to select words in the initial text records, where the selected words are to be replaced to generate new augmented text records, and (3) suitable word generation module112(seeFIG.1) to determine respective sets of word(s) that are suitable replacement word(s) for the aforementioned selected words.

Step408also includes text augmentation module108(seeFIG.1) sending the new augmented initial text records to main module106(seeFIG.1).

Additional details of step408are described below in the discussions ofFIG.5,FIGS.6A-6B,FIGS.7A-7B, andFIGS.8A-8B.

In step410, small class text augmentation system104(seeFIG.1) generates balanced dataset118(seeFIG.1) by adding the augmented initial text records to the original text records that had been in imbalanced dataset116(seeFIG.1). In one embodiment, main module106(seeFIG.1) performs step410.

In step412, small class text augmentation system104(seeFIG.1) sends the balanced dataset118(seeFIG.1) to a supervised machine learning model. In one embodiment, main module106(seeFIG.1) performs step412. In step414, small class text augmentation system104(seeFIG.1) trains the supervised machine learning model on balanced dataset118(seeFIG.1).

In step416, using the supervised machine learning model, which is employing the augmented initial text records, small class text augmentation system104(seeFIG.1) performs a text classification of a new dataset whose domain matches a domain of the imbalanced dataset116(seeFIG.1).

The process ofFIG.4ends at an end node418.

FIG.5is a flowchart of a process performed by a text augmentation module included in the systems ofFIG.1andFIG.3, where the process is included in the process ofFIG.4, in accordance with embodiments of the present invention. The process ofFIG.5begins at a start node500. In step502, text augmentation module108(seeFIG.1) receives the initial text records in the small class from main module106(seeFIG.1).

In step504, text augmentation module108(seeFIG.1) sends each of the initial text records to word priority module110(seeFIG.1) (i.e., sends each text record of the small class to word priority module110(seeFIG.1)). Details of the processing of the initial text records by word priority module110(seeFIG.1) are included in the discussion ofFIGS.6A-6B, presented below.

In step506, text augmentation module108(seeFIG.1) receives word priority lists generated by word priority module110(seeFIG.1), where the word priority lists are associated with the initial text records in a one-to-one correspondence.

In step508, text augmentation module108(seeFIG.1) sends each word in a given word priority list and the corresponding text record to suitable word generation module112(seeFIG.1). Details of the processing of the word priority list and corresponding text record by suitable word generation module112(seeFIG.1) are included in the discussion ofFIGS.8A-8B, presented below.

In step510, text augmentation module108(seeFIG.1) receives suitable word lists from suitable word generation module112(seeFIG.1), where the suitable word lists are associated with the words in the word priority lists in a one-to-one correspondence. A given suitable word list includes word(s) that are suitable replacement word(s) for a given word in a word priority list generated by word priority module110(seeFIG.1).

In step512text augmentation module108(seeFIG.1) identifies the words in the initial text records that are in the word priority lists generated by word priority module110(seeFIG.1).

In step514and for given word(s) that are in an initial text record and are identified as being in a word priority list in step512, text augmentation module108(seeFIG.1) generates new text record(s) by replacing the given word(s) in the initial text record with suitable word(s) from the corresponding suitable word list(s). In one embodiment, the number of new text records generated may be more than the number of words in the corresponding suitable word list. In one embodiment, text augmentation module108(seeFIG.1) ensures that no combination of words from the corresponding word priority list is repeated while generating new text record(s) in step514.

In step516, text augmentation module108(seeFIG.1) generates augmented text records that include the initial text records and the new text record(s) generated in step514.

In step518, text augmentation module108(seeFIG.1) sends the augmented text records to main module106(seeFIG.1).

In one embodiment, step408(seeFIG.4) includes steps504,506,508,510,512,514, and516.

The process ofFIG.5ends at an end node520.

FIGS.6A-6Bdepict a flowchart of a process performed by a word priority module included in the systems ofFIG.1andFIG.2, where the process is included in the process ofFIG.4, in accordance with embodiments of the present invention. The process ofFIGS.6A-6Bincludes word priority module110(seeFIG.1) performing the following actions:

(1) determines which words in the text record are to be replaced;

(2) selects the right word and the right position in a sentence in a given text record by considering a weighted average of lexical features (i.e., parts of speech), syntactic features (i.e., dependency tag), word frequency statistics (i.e., class word score), and a stop word score; and

(3) generates a word priority list for the given text record.

The process ofFIGS.6A-6Bbegins a start node600inFIG.6A. In step602, word priority module110(seeFIG.1) defines a threshold score T1for determining word priority.

In step604, word priority module110(seeFIG.1) defines weights w1, w2, w3, and w4for indicating the relative importance of scores.

In step606, word priority module110(seeFIG.1) defines the following scales as ranges of values for respective scores: scale1, scale2, scale3, and scale4.

In step608, word priority module110(seeFIG.1) receives a text record from text augmentation module108(seeFIG.1).

In step610, word priority module110(seeFIG.1) sends the text record received in step608to class word score module114(seeFIG.1), stop word module204(seeFIG.2), dependency module206(seeFIG.2), and POS module202(seeFIG.2).

In response to receiving the text record sent by word priority module110(seeFIG.1), POS module202(seeFIG.2) performs the following actions:(1) obtain a POS tagger model, which provides lexical categories of words in the text record;(2) send entire text record to the POS tagger model;(3) for each word in the text record, (i) obtain the POS tag from the POS tagger model and (ii) determine a POS score by assigning respective numeric values in scale4to the POS tags, where the numeric values determine the importance of the different POS tags; and(4) return entire text record along with the POS scores in scale4for the words in the text record

In response to receiving the text record sent by word priority module110(seeFIG.1), dependency module206(seeFIG.2) performs the following actions:(1) obtain a dependency parser model, which provides syntactic dependency relationships between words in the text record;(2) send the entire text record to the dependency parser model;(3) for each word in the text record, (i) obtain a dependency tag from the dependency parser model and (ii) determine a dependency score by assigning numeric values in scale3to the dependency tags, where the numeric values determine the importance of the different dependency tags; and(4) return the entire text record along with the dependency scores in scale3for the words in the text record

In response to receiving the text record sent by word priority module110(seeFIG.1), stop word module204(seeFIG.2) performs the following actions:(1) define a stop word list of words that add little or no value to text classification;(2) for each word in the text record, assign a predefined low stop word score if the word is present in the stop word list or assign a predefined high stop word score if the word is not present in the stop word list; and(3) return the stop word scores in scale2

In one embodiment, stop word module204(seeFIG.2) assigns respective pairs of numeric values (low, high) in scale2to the classes of the text records, where a pair for a given class has a low value for assignment to a word that is present in the stop word list and a high value for assignment to a word that is not present in the stop word list.

In step612, for each word in the text record received in step608, word priority module110(seeFIG.1) receives a class word score in scale1, a stop word score in scale2, a dependency score in scale3, and a POS score in scale4.

After step612, the process ofFIGS.6A-6Bcontinues with step614inFIG.6B.

Prior to step614and for each word in the text record received in step608(seeFIG.6A), word priority module110(seeFIG.1) determines whether the word in the text record received in step608(seeFIG.6A) is a special character or an out of vocabular word. In step614, word priority module110(seeFIG.1) assigns a predefined low score in scale5to any word that was determined to be a special character or an out of vocabulary word.

In step616, word priority module110(seeFIG.1) converts the scores received in step612(seeFIG.6A) to scale5.

In step618and for each word in the text record received in step608(seeFIG.6A), word priority module110(seeFIG.1) calculates a corresponding word priority score as w1*class word score+w2*stop word score+w3*dependency score+w4*POS score.

In step620, word priority module110(seeFIG.1) determines whether each of the word priority scores calculated in step618is greater than threshold score T1. If word priority module110(seeFIG.1) determines in step620that a given word priority score calculated in step618is greater than threshold score T1, then the Yes branch of step620is taken and step622is performed. In step622, word priority module110(seeFIG.1) adds the word corresponding to the given word priority score calculated in step618to a word priority list. The repeated performance of step622generates a final word priority list to be sent to text augmentation module108(seeFIG.1). After step622, step624is performed.

Returning to step620, if word priority module110(seeFIG.1) determines that the word priority score is not greater than threshold score T1, then the No branch of step620is taken and step624is performed.

In step624, word priority module110(seeFIG.1) determines whether another text record remains to be processed by the process ofFIGS.6A-6B. If word priority module110(seeFIG.1) determines in step624that another text record remains to be processed, then the Yes branch of step624is taken and the process loops back to step608(seeFIG.6A).

Returning to step624, if word priority module110(seeFIG.1) determines that no other text record remains to be processed by the process ofFIGS.6A-6B, then the No branch of step624is taken and step626is performed.

In step626, word priority module110(seeFIG.1) arranges words in the word priority list in descending order according to corresponding word priority scores.

In step628, word priority module110(seeFIG.1) sends the word priority list to text augmentation module108(seeFIG.1).

The process ofFIGS.6A-6Bends at an end node630.

In one embodiment, the process ofFIGS.6A-6Bis performed between steps504and506inFIG.5.

FIGS.7A-7Bdepict a flowchart of a process performed by a class word score module included in the systems ofFIG.1,FIG.2, andFIG.3, where the process is included in the process ofFIG.4, in accordance with embodiments of the present invention. In the process ofFIGS.7A-7B, class word score module114(seeFIG.1) performs the following:(1) provides word importance statistics from a class point of view;(2) considers word frequency scores and a unique weighting mechanism to maintain class-specific word structure, while avoiding the introduction of words important for one class into another class;(3) groups word scored in terms of percentiles in descending order, generates weighted word scores, and calculates a final class word score for each unique word as an average of weighted word scores in scale1of each class;(4) penalizes words with high term frequency and high document frequency (i.e., the word appears in all classes)(5) replaces words that are not common to all classes and also not important to one class; and(6) avoids (i) the introduction of unnecessary bias into the supervised text classification model and (ii) a reduction of the performance of the supervised text classification model

The process ofFIGS.7A-7Bbegins at a start node700. In step702, class word score module114(seeFIG.1) obtains all classes and the text records of the classes, where the text records are included in imbalanced dataset116(seeFIG.1).

In step704and for each class in imbalanced dataset116(seeFIG.1), class word score module114(seeFIG.1) concatenates the text records in the class to create a single text record.

In step706, class word score module114(seeFIG.1) defines p, the number of groups into which the word scores in scale1are to be divided.

In step708, class word score module114(seeFIG.1) defines word weights ww1, ww2, . . . wwp, such that ww1<ww2< . . . <wwp, where the word weights determine the priority of the different groups of word scores.

In step710and for each unique word in each class, class word score module114(seeFIG.1) performs the following actions:(1) calculate a term frequency value as the number of times the word appears in the class/total number of words in the class;(2) calculate an inverse document frequency value as the logarithm (total number of classes/number of classes in which the word occurs); and(3) calculate a word score in scale1for the word as term frequency value*inverse document frequency value

In step712and for each class, class word score module114(seeFIG.1) performs the following actions:(1) arrange the word scores in descending order;(2) divide the word scores into p groups in terms of percentile in terms of descending order; and(3) for each word in the class, calculate weighted word score in scale1as word score*word weight assigned to the group that includes the word score (i.e., multiply word score of one group with the weights assigned to that same group) (e.g., the word score in the highest percentile group is multiplied with word weight ww1, which has the lowest word weight value)

After step712, the process ofFIGS.7A-7Bcontinues with step714inFIG.7B.

In step714, class word score module114(seeFIG.1) obtains a list of unique words across all the classes in the imbalanced dataset116(seeFIG.1).

In step716and for each unique word across all the classes, class word score module114(seeFIG.1) calculates a class word score in scale1for the word as an average of weighted word scores of each class.

In step718, class word score module114(seeFIG.1) defines a default value in case a word is not present in any of the classes and for out of dataset words, class word score module114(seeFIG.1) assigns the default value as the class word score in scale1.

In step720, class word score module114(seeFIG.1) sends a list of words and the respective class word scores to the module that called the class word score module114(seeFIG.1). For example, in step720. class word score module114(seeFIG.1) sends the list of words and the respective class word scores to word priority module110(seeFIG.1).

After step720, class word score module114(seeFIG.1) ends at an end node722.

In one embodiment, the process ofFIGS.7A-7Bis performed between steps610and612inFIGS.6A-6B.

FIGS.8A-8Bdepict a flowchart of a process performed by a suitable word generation module included in the systems ofFIG.1andFIG.3, where the process is included in the process ofFIG.4, in accordance with embodiments of the present invention. The process ofFIGS.8A-8Bincludes suitable word generation module112(seeFIG.1) performing the following actions:(1) determines the right replacement word(s); and(2) generates a suitable word list as a replacement(s) for a given word by considering a weighted average of (i) cosine similar word embeddings with class-based word importance statistics, (ii) context specific words with class-based word importance statistics, and (iii) synonyms from a language dictionary with class-based importance statistics.

The process ofFIGS.8A-8Bbegins at a start node800. In step802, suitable word generation module112(seeFIG.1) defines a threshold score T2for determining a suitable word score.

In step804, suitable word generation module112(seeFIG.1) defines weights w5, w6, w7, and w8for indicating a relative importance of scores.

In step806, suitable word generation module112(seeFIG.1) defines scale1, scale5, scale6, and scale7as ranges of values for respective scores.

In step808, suitable word generation module112(seeFIG.1) defines a language dictionary that includes synonyms of words.

In step810, suitable word generation module112(seeFIG.1) receives a word priority list and a text record corresponding to the word priority list from text augmentation module108(seeFIG.1).

In step812and for each word in the word priority list received in step810, suitable word generation module112(seeFIG.1) calculates suitable word score(s) in scale5using static embeddings similarity based sub-module304(seeFIG.3) to perform the following actions:(1) using static embeddings similarity module310(seeFIG.3), obtain top k similar words and calculate the cosine similarity scores in scale6for the top k similar words(2) using the class word score module114(seeFIG.1), calculate class word scores in scale1of the top k similar words;(3) convert cosine similarity scores and class word scores to scale5; and(4) for each of the top k similar words, calculate a suitable word score as w5*cosine similarity score+w6*class word score

In one embodiment, using the static embeddings similarity module310(seeFIG.3) generates top k similar words for a given word in the word priority list using a static embeddings model and includes performing the following actions:(1) obtain a pretrained transfer learning based static embeddings model;(2) obtain a pretrained transfer learning based static embeddings model tokenizer;(3) obtain the static embeddings similarity list from static embeddings module308(seeFIG.3), where the actions of the static embeddings module308are described below;(4) obtain the word priority list from suitable word generation module112(seeFIG.1);(5) for each word in the word priority list, (i) tokenize the word using the static embeddings model tokenizer to generate a token, (ii) using the token, obtain the top k similar tokens and their cosine similarity scores from the static embeddings similarity list, and (iii) convert the top k similar tokens to top k similar words using the in-built function of the static embeddings model;(6) return the word priority list and the corresponding top k similar words and their cosine similarity scores to the suitable word generation module112(seeFIG.1).

In one embodiment, static embeddings module308generates top k similar tokens for a given token in the vocabulary of a static embeddings model by performing the following actions:(1) obtain the pretrained transfer learning based static embeddings model;(2) obtain the vocabulary of the static embeddings model (i.e., tokens and their corresponding index identifiers (IDs));(3) define k, the number of similar words for a given word;(4) for each token in the vocabulary, (i) obtain the corresponding index ID, (ii) obtain static embeddings for each index ID from in-built functions of the static embeddings model, (iii) calculate cosine similarity between the current token's static embeddings and the static embeddings of all the other tokens, and (iv) generate a static embeddings similarity list of top k similar tokens along with their cosine similarity scores and arrange the list in descending order according to the cosine similarity score(5) save the static embeddings similarity list for all the tokens in the vocabulary; and(6) send the static embeddings similarity list to static embeddings similarity module310

After step812, the process ofFIGS.8A-8Bcontinues with step814inFIG.8B.

In step814and for each word in the word priority list received in step810(seeFIG.8A), suitable word generation module112(seeFIG.1) calculates suitable word score(s) in scale5using contextual language based sub-module306(seeFIG.3) to perform the following actions:(1) obtain the text record corresponding to the word priority list;(2) send the text record to contextual language module312(seeFIG.3) and in response, obtain top j likely words and contextual probability scores in scale7of the top j likely words;(3) using the class word score module114(seeFIG.1), suitable word generation module112(seeFIG.1) calculates class word scores in scale1of the top j likely words(4) convert contextual probability scores and class word scores to scale5;(5) for each of the top j likely words, calculate a suitable word score as w7*contextual probability score+w8*class word score

In one embodiment, class language module312(seeFIG.3) performs the following actions to generate the top j likely words for a given word by using a contextual language model;(1) obtain the pretrained transfer learning based contextual language model;(2) obtain all classes in imbalanced dataset116(seeFIG.1) and the text records in the classes;(3) fine tune the contextual language model on the given classes as a classification task;(4) define j, the number of probable words (i.e., likely words) for a given word;(5) obtain the word priority list and the corresponding text record from suitable word generation module112(seeFIG.3);(6) for each word in the word priority list, (i) send the remaining words of the text record in the same sequence to the contextual language model and (ii) receive in response a list of top j probable words and their probability scores; and(7) return the word priority list and the corresponding top j probable words and their probability scores to suitable sord generation module112(seeFIG.3).

In step816and for each word in the word priority list received in step810(seeFIG.8A), suitable word generation module112(seeFIG.1) calculates suitable word score(s) in scale5using synonym based sub-module302(seeFIG.3) to perform the following actions:(1) obtain the synonyms of the word from the language dictionary;(2) using class word score module114(seeFIG.1), calculate the class word scores in scale1of the synonyms;(3) convert the class word scores to scale5; and(4) for each of the synonyms, calculate a suitable word score as the converted class word score

In step818and for each word in the word priority list received in step810(seeFIG.8A), suitable word generation module112(seeFIG.1) identifies word(s) obtained in steps812,814, and816whose suitable word score >T2.

In step820and for each word in the word priority list received in step810(seeFIG.8A), suitable word generation module112(seeFIG.1) adds the words identified in step818to a suitable word list, thereby generating a final suitable word list to be sent to text augmentation module108(seeFIG.1). Suitable word generation module112(seeFIG.1) arranges the words in the suitable word list in descending order according to the respective suitable word scores of the words.

In step822, suitable word generation module112(seeFIG.1) sends the suitable word lists for the respective words in the word priority list to text augmentation module108(seeFIG.1).

Following step822, the process ofFIGS.8A-8Bends at an end node824.

In one embodiment, the process ofFIGS.8A-8Bis performed between steps508and510inFIG.5.

In one embodiment, the process ofFIGS.7A-7Bis performed in the portions of steps812,814, and816that use class word score module114(seeFIG.1).

In one embodiment, the conversions of scores from one scale to another scale (i.e., from an old scale to a new scale), as described in step616(seeFIG.6A), and in the actions described above performed by synonym based sub-module302, static embeddings similarity based sub-module304, and contextual language based sub-module306use the following scale converter formula.

Scale Converter Formula:

old⁢range⁢of⁢old⁢scale=old⁢maximum-old⁢minimum⁢new⁢range⁢of⁢new⁢scale=new⁢maximum-new⁢minimum⁢new⁢value⁢in⁢new⁢scale=((old⁢value-old⁢minimum)old⁢range*new⁢range)+new⁢minimum

Examples

FIG.9is an example900of augmenting text records of a small class in the process ofFIG.5, where the process is performed by a text augmentation module included in the system ofFIG.1, in accordance with embodiments of the present invention. Text record902is an example of the initial text records received by text augmentation module108(seeFIG.1) in step502(seeFIG.5). Word priority list904includes the words (i.e., Word2, Word4, and Word5) that word priority module110(seeFIG.1) identified as words in text record902that are to be replaced.

Text augmentation module108(seeFIG.1) receives a suitable word list906, which is an example of step510(seeFIG.5). Suitable word lists906includes a first suitable word list consisting of Word44, Word89, Word9045, and Word8645, which are identified as suitable replacement words for Word2. Suitable word list906also includes a second suitable word list consisting of Word4289and Word1022, which are identified as suitable replacement words for Word4. Suitable word list906also includes a third suitable word list consisting of Word900, Word764, and Word5239, which are identified as suitable replacement words for Word5.

Text augmentation module108(seeFIG.1) generates new augmented text records908of the small class, which is an example of step514(seeFIG.5). For example, text augmentation module108(seeFIG.1) generates the new text record that consists of Word1, Word44, Word3, Word4, and Word5(i.e., the first row in the new augmented text records908) by replacing a word from word priority list904(i.e., Word2) with a replacement word (i.e., Word44) selected from the suitable word list that corresponds to Word2, where the suitable word list is included in suitable word lists906). As another example, text augmentation module108(seeFIG.1) generates the new text record that consists of Word1, Word2, Word3, Word4289, and Word5(i.e., the fifth row in the new augmented text records908) by replacing Word4in word priority list904with a replacement word (i.e., Word4289) selected from the corresponding suitable word list included in suitable word lists906.

FIG.10is an example1000of determining class word scores in the process ofFIGS.7A-7B, where the process is performed by a class word score module in the system ofFIG.1, in accordance with embodiments of the present invention. Class word score module114(seeFIG.1) obtains text records and all classes1002from imbalanced dataset116, which is an example of step702(seeFIG.7A). In step710(seeFIG.7A), class word score module114(seeFIG.1) makes calculations1004, which include term frequency, inverse document frequency and word scores for each unique word in each class. In step712(seeFIG.7A), class word score module114(seeFIG.1) makes an arrangement of word scores1006in descending order for each class.

In step1008, class word score module114(seeFIG.1) groups the word scores in terms of percentile to generate grouped word scores1010, which is an example of a portion of step712(seeFIG.7A). Another portion of step712(seeFIG.7A) includes class word score module114(seeFIG.1) calculating weighted word scores1012for each word in a class. In step716(seeFIG.7B), class word score module114(seeFIG.1) calculates a class word score1014for each unique word across all the classes by calculating an average of the weighted word scores of each class.

FIG.11is an example1100of generating a word priority list in the process ofFIGS.6A-6B, where the process is performed by a word priority module in the system ofFIG.1, in accordance with embodiments of the present invention. In step608(seeFIG.6A), word priority module110(seeFIG.1) receives a text record1102from text augmentation module108(seeFIG.1). Text record1102consists of Word1, Word,2, Word3, Word4, and Word5. In step612, word priority module110(seeFIG.1) receives scores1104, which include a POS score, a class word score, a stop word score, and a dependency score for each word in text record1102.

In step618(seeFIG.6B), word priority module110(seeFIG.1) makes a word priority score calculation1106for each word in text record1102, which results in word priority scores1108for respective words in text record1102. The word priority score is based on the corresponding POS score, class word score, stop word score, and dependency score included in scores1104.

In step620(seeFIG.6B), word priority module110(seeFIG.1) makes a determination1110that word priority scores corresponding to Word2, Word4, and Word5are greater than a threshold score T1. In step622(seeFIG.6B) and based on the aforementioned word priority scores being greater than the threshold score T1, word priority module110(seeFIG.1) generates word priority list1112that consists of Word2, Word,4, and Word5. Word priority list1112indicates that Word2, Word4, and Word5are the words in text record1102that are to be replaced by suitable word replacements to generate new augmented text records.

FIG.12is an example1200of generating a suitable word list in the process ofFIGS.8A-8B, where the process is performed by a suitable word generation module in the system ofFIG.1, in accordance with embodiments of the present invention. In step810(seeFIG.8A), suitable word generation module112(seeFIG.1) receives word priority list1202, which is the same word priority list generated by the example1100(seeFIG.11).

In step812(seeFIG.8A) and steps814and816(seeFIG.8B), suitable word generation module112(seeFIG.1) generates initial (i.e., possible or tentative) suitable words1204for each word in word priority list1202. The initial suitable words1204are determined by synonym based sub-module302(seeFIG.3), static embeddings similarity based sub-module304(seeFIG.3), and contextual language based sub-module306(seeFIG.3).

In step812(seeFIG.8A) and steps814and816(seeFIG.8B), suitable word generation module112(seeFIG.1) calculates respective suitable word scores1206for the initial suitable words1204by using synonym based sub-module302(seeFIG.3), static embeddings similarity based sub-module304(seeFIG.3), and contextual language based sub-module306(seeFIG.3).

In step1208, suitable word generation module112(seeFIG.1) determines which of the suitable word scores1206is greater than a threshold score T2. If a suitable word score for a given word is greater than the threshold score T2, suitable word generation module112(seeFIG.1) designates the given word as a final suitable replacement word for the corresponding word in word priority list1202and adds the given word to suitable word lists1210. For example, suitable word generation module112(seeFIG.1) determines that the suitable word score for Word44is greater than the threshold score T2and in response, adds Word44to suitable word lists1210.

Computer System

FIG.13is a block diagram of a computer that is included in the system ofFIG.1and that implements the processes ofFIG.4,FIG.5,FIGS.6A-6B,FIGS.7A-7B, andFIGS.8A-8B, in accordance with embodiments of the present invention. Computer102is a computer system that generally includes a central processing unit (CPU)1302, a memory1304, an input/output (I/O) interface1306, and a bus1308. Further, computer102is coupled to I/O devices1310and a computer data storage unit1312. CPU1302performs computation and control functions of computer102, including executing instructions included in program code1314for small class text augmentation system104(seeFIG.1) to perform a method of augmenting text of a small class, where the instructions are executed by CPU1302via memory1304. CPU1302may include a single processing unit or processor or be distributed across one or more processing units or one or more processors in one or more locations (e.g., on a client and server).

Memory1304includes a known computer readable storage medium, which is described below. In one embodiment, cache memory elements of memory1304provide temporary storage of at least some program code (e.g., program code1314) in order to reduce the number of times code must be retrieved from bulk storage while instructions of the program code are executed. Moreover, similar to CPU1302, memory1304may reside at a single physical location, including one or more types of data storage, or be distributed across a plurality of physical systems or a plurality of computer readable storage media in various forms. Further, memory1304can include data distributed across, for example, a local area network (LAN) or a wide area network (WAN).

I/O interface1306includes any system for exchanging information to or from an external source. I/O devices1310include any known type of external device, including a display, keyboard, etc. Bus1308provides a communication link between each of the components in computer102, and may include any type of transmission link, including electrical, optical, wireless, etc.

I/O interface1306also allows computer102to store information (e.g., data or program instructions such as program code1314) on and retrieve the information from computer data storage unit1312or another computer data storage unit (not shown). Computer data storage unit1312includes one or more known computer readable storage media, where a computer readable storage medium is described below. In one embodiment, computer data storage unit1312is a non-volatile data storage device, such as, for example, a solid-state drive (SSD), a network-attached storage (NAS) array, a storage area network (SAN) array, a magnetic disk drive (i.e., hard disk drive), or an optical disc drive (e.g., a CD-ROM drive which receives a CD-ROM disk or a DVD drive which receives a DVD disc).

Memory1304and/or storage unit1312may store computer program code1314that includes instructions that are executed by CPU1302via memory1304to augment text of a small class. AlthoughFIG.13depicts memory1304as including program code, the present invention contemplates embodiments in which memory1304does not include all of code1314simultaneously, but instead at one time includes only a portion of code1314.

Further, memory1304may include an operating system (not shown) and may include other systems not shown inFIG.13.

As will be appreciated by one skilled in the art, in a first embodiment, the present invention may be a method; in a second embodiment, the present invention may be a system; and in a third embodiment, the present invention may be a computer program product.

Any of the components of an embodiment of the present invention can be deployed, managed, serviced, etc. by a service provider that offers to deploy or integrate computing infrastructure with respect to augmenting text of a small class. Thus, an embodiment of the present invention discloses a process for supporting computer infrastructure, where the process includes providing at least one support service for at least one of integrating, hosting, maintaining and deploying computer-readable code (e.g., program code1314) in a computer system (e.g., computer102) including one or more processors (e.g., CPU1302), wherein the processor(s) carry out instructions contained in the code causing the computer system to augment text of a small class. Another embodiment discloses a process for supporting computer infrastructure, where the process includes integrating computer-readable program code into a computer system including a processor. The step of integrating includes storing the program code in a computer-readable storage device of the computer system through use of the processor. The program code, upon being executed by the processor, implements a method of augmenting text of a small class.

While it is understood that program code1314for augmenting text of a small class may be deployed by manually loading directly in client, server and proxy computers (not shown) via loading a computer-readable storage medium (e.g., computer data storage unit1312), program code1314may also be automatically or semi-automatically deployed into computer102by sending program code1314to a central server or a group of central servers. Program code1314is then downloaded into client computers (e.g., computer102) that will execute program code1314. Alternatively, program code1314is sent directly to the client computer via e-mail. Program code1314is then either detached to a directory on the client computer or loaded into a directory on the client computer by a button on the e-mail that executes a program that detaches program code1314into a directory. Another alternative is to send program code1314directly to a directory on the client computer hard drive. In a case in which there are proxy servers, the process selects the proxy server code, determines on which computers to place the proxy servers' code, transmits the proxy server code, and then installs the proxy server code on the proxy computer. Program code1314is transmitted to the proxy server and then it is stored on the proxy server.

Another embodiment of the invention provides a method that performs the process steps on a subscription, advertising and/or fee basis. That is, a service provider can offer to create, maintain, support, etc. a process of augmenting text of a small class. In this case, the service provider can create, maintain, support, etc. a computer infrastructure that performs the process steps for one or more customers. In return, the service provider can receive payment from the customer(s) under a subscription and/or fee agreement, and/or the service provider can receive payment from the sale of advertising content to one or more third parties.

The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) (i.e., memory1304and computer data storage unit1312) having computer readable program instructions1314thereon for causing a processor (e.g., CPU1302) to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that can retain and store instructions (e.g., program code1314) for use by an instruction execution device (e.g., computer102). The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Computer readable program instructions (e.g., program code1314) described herein can be downloaded to respective computing/processing devices (e.g., computer102) from a computer readable storage medium or to an external computer or external storage device (e.g., computer data storage unit1312) via a network (not shown), for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card (not shown) or network interface (not shown) in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

Computer readable program instructions (e.g., program code1314) for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations (e.g.,FIG.4,FIG.5,FIGS.6A-6B,FIGS.7A-7B, andFIGS.8A-8B) and/or block diagrams (e.g.,FIG.1,FIG.2,FIG.3, andFIG.13) of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions (e.g., program code1314).

These computer readable program instructions may be provided to a processor (e.g., CPU1302) of a general purpose computer, special purpose computer, or other programmable data processing apparatus (e.g., computer102) to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium (e.g., computer data storage unit1312) that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions (e.g., program code1314) may also be loaded onto a computer (e.g. computer102), other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be accomplished as one step, executed concurrently, substantially concurrently, in a partially or wholly temporally overlapping manner, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

While embodiments of the present invention have been described herein for purposes of illustration, many modifications and changes will become apparent to those skilled in the art. Accordingly, the appended claims are intended to encompass all such modifications and changes as fall within the true spirit and scope of this invention.