Patent ID: 11954507
Assignee: NIPPON TELEGRAPH AND TELEPHONE CORPORATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A screen recognition device comprising:
a processor; and
a storage medium having computer program instructions stored thereon, when executed by the processor, perform to:
perform a marking process on a graphical user interface (GUI) component in an active display screen provided by an application software, wherein the marking process distinguishes a background of the display screen from the GUI component in the display screen of the application software;
extract the GUI component included in the display screen of application software from the display screen;
superimpose a mouse cursor on the GUI component based on at least one of a form and a position of the GUI component extracted;
identify a change in the displayed symbol image of the mouse cursor;
determine that a kind of the extracted GUI component cannot be identified based only on the change in the displayed symbol image of the mouse cursor;
in response to the determining, determine the kind of the GUI component extracted and an operation that can be performed to the GUI component, based on the change in the mouse cursor and a change in the GUI component at the same time the mouse cursor is superimposed on the GUI component; and
moving the mouse cursor sequentially to each of the plurality of graphical user interfaces in the display screen based on the form and the position of each GUI component, wherein the mouse cursor is moved to and superimposed on each GUI component to determine the kind and the operation of the graphical user interface.