Patent ID: 11922126
Assignee: INTUIT INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 12:
13. A system comprising:
a processor;
a machine learning model (MLM) executable by the processor, wherein the MLM comprises a language processing MLM;
a data repository in communication with the processor, and storing:
a user input,
a plurality of modified inputs that are based on the user input, wherein the plurality of modified inputs each are semantically related to the user input,
a plurality of model outputs of the MLM,
a plurality of sampled model outputs of the MLM,
a plurality of clusters of the sampled model outputs of the MLM, wherein each cluster of the plurality of clusters represents a distinct semantic meaning of the plurality of sampled model outputs,
a confidence metric for the user input, wherein the confidence metric comprises a predictive entropy of the plurality of clusters, and
a threshold value comparable to the confidence metric; and

a confidence estimation controller which, when executed by the processor:
executes the MLM to generate the plurality of model outputs, wherein the MLM takes as input a plurality of instances of each of the plurality of modified inputs,
samples the plurality of model outputs using a statistical sampling strategy to generate the plurality of sampled model outputs,
clusters the plurality of sampled model outputs into the plurality of clusters, and
generates the confidence metric; and

a machine learning controller configured to route, automatically, the user input based on whether the confidence metric satisfies or fails to satisfy the threshold value.