Patent ID: 8401221

Claim:
A computer-implemented method of automatically controlling execution of an application program by a computer system, the application program having a graphical user interface for displaying information on a display coupled to the computer system comprising: learning an input action performed by a user while the user is using the application program and its graphical user interface during a recording phase of executing the application program by the computer system; capturing images displayed by the graphical user interface in response to the input action being learned during the recording phase; analyzing the user input action being learned and the resultant displayed images to generate an execution scenario script during the recording phase, wherein the learned input action and captured images are to be stored in a memory; generating simulated user input action based on the execution scenario script during a playback phase of executing the application program by the computer system and inputting the simulated user input action to the application program; performing image analysis on images displayed by the graphical user interface on the display as a result of processing the simulated user input action during the playback phase and captured displayed images from the recording phase; checking recorded time conditions in the execution scenario script to handle variations in playback speed, and automatically controlling execution of the application program by the computer system based at least in part on the image analysis.