Patent Document ID: 10088969
Application ID: 14009580
Patent Status: 1

Claim One:
1. A non-transitory processor-readable medium storing instructions that when executed by a processor, cause the processor to: access an instruction describing an action on a graphical user interface displayed on a display; identify, in a first image of the graphical user interface, a graphical object displayed on the graphical user interface relative to which the action is to be performed, the graphical object having a first appearance in the first image; perform the action relative to the graphical object at the graphical user interface; identify, in a second image of the graphical user interface, the graphical object, wherein an appearance of the graphical object is changed in response to performance of the action, the graphical object having a second appearance in the second image; and generate semantic information related to the first appearance and the second appearance of the graphical object, the semantic information semantically describing an appearance of the graphical object both prior to and after performance of the action such that the action performed relative to the graphical object is identifiable from the semantic information.