Patent ID: 11886892
Assignee: AUTOMATION ANYWHERE, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 8:
9. A computer system, comprising:
data storage having stored thereupon a plurality of image files, each image file containing one or more user interface objects of a user interface generated by an application program; and
a processor, programmed with instructions that cause the processor to train, test and retrain a machine learning model, the machine learning model being used to detect one or more user interface objects contained in a screen image of a user interface generated by an application program by way of invariance guided learning, the instructions when executed by the processor operate to at least:
selecting a dataset that comprises a plurality of object classes, each object class corresponding to a type of user interface object and comprising at least one variance parameter;
training a machine learning model with a portion of the dataset designated as a training portion of the dataset to cause the machine learning model to detect, in the training portion of the dataset, objects in each of the plurality of object classes;
testing the machine learning model with a portion of the dataset designated as a testing portion of the dataset to determine whether the machine learning model can detect objects in the plurality of object classes with an accuracy threshold;
when the machine learning model exhibits an accuracy that does not meet the accuracy threshold, the method further comprises the operations of:
processing the testing portion of the dataset to generate, for each object class in the testing portion of the dataset, a numerical value that represents failure of the machine learning model with respect to each variance parameter of a corresponding object class;
generating an updated dataset based on the numerical values for the variance parameters; and
retraining the machine learning model with a portion of the updated dataset designated as an updated training portion of the dataset to cause the machine learning model to detect, in the updated training portion of the dataset, objects in each of the plurality of object classes.