Patent ID: 11934795
Assignee: ORACLE INTERNATIONAL CORPORATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 14:
15. One or more non-transitory computer-readable media comprising instructions, which when executed by one or more hardware processors, cause performance of operations, comprising:
generating a test set of texts for testing a first machine-learned classification model, wherein generating the test set of texts comprises:
generating a set of expanded texts,
wherein a first expanded text, of the set of expanded texts, corresponds to a first masked text of a set of masked texts,
wherein the first masked text, of the set of masked texts, comprises at least one mask,

wherein the first expanded text is generated from the first masked text at least by inserting one or more insertion words in place of the first masked text,
wherein the first masked text corresponds to a first base text of a set of base texts,
wherein the first masked text is generated from the first base text at least by inserting the at least one mask at a first position within the first base text;

applying a second machine-learned classification model to a second expanded text to determine a first predicted classification for the second expanded text;
comparing the first predicted classification to a first true classification of a second base text corresponding to the second expanded text;
determining that the first predicted classification corresponding to the second expanded text differs from the first true classification of the second base text;
applying a natural language model to the second expanded text to determine a probability of encountering the second expanded text in real-world data,
wherein the real-world data comprises grammatically correct text;

determining that the second expanded text meets a threshold probability of being encountered in the real-world data;
including the second expanded text in the test set of texts based at least on (a) the first predicted classification differing from the first true classification and (b) the second expanded text meeting the threshold probability of being encountered in the real-world data;

applying the first machine-learned classification model to the test set of texts to determine a performance score for the first machine-learned classification model.