Patent ID: 8745521

Claim:
A method for creating an annotation for an unannotated graphical user interface (GUI) object in a GUI, the method comprising: extracting, from the GUI, a plurality of GUI objects and properties for each of the plurality of GUI objects, wherein: the properties for each of the plurality of GUI objects comprise a role describing a function of the GUI object in the GUI, a descriptor related to the role of the GUI object, a location of the GUI object within the GUI, and a bounding rectangle specifying the boundaries of the GUI object; and the plurality of GUI objects comprises the unannotated GUI object and a plurality of candidate GUI objects, wherein each of the plurality of candidate GUI objects has the descriptor comprising a text string; determining one or more nearby candidate GUI objects from the plurality of candidate GUI objects that are in the vicinity of the unannotated GUI object, based on the location and the bounding rectangle of the unannotated GUI object, the location and the bounding rectangle of each of the plurality of candidate GUI objects, the role of the unannotated GUI object, and a precedence criteria of the unannotated GUI object; selecting a best candidate GUI object from the one or more nearby candidate GUI objects, based on the precedence criteria of the unannotated GUI object; and assigning the text string of the best candidate GUI object as the annotation for the unannotated GUI object.