Using graphical image analysis for identifying image objects

An image of a graphical user interface is captured. For example, a screen shot of a browser display is captured. Text syntax is executed that contains one or more parameters for identifying a graphical object. For example, the text syntax may identify a rectangle that contains the text “OK” where the text is red. Based on the text syntax, a graphical object is identified in the image of the graphical user interface. Information is returned that identifies how to access the graphical object in the graphical user interface. For example, coordinates of the graphical object are identified. This information can then be used in a test script using existing programming languages to test the graphical user interface. For example, the coordinates may be used to click on the OK button.

FIELD

The disclosure relates generally to software testing and particularly to software testing using image analysis.

BACKGROUND

Traditional software programming methods typically use information from the code base of an application for developing test software. For example, a developer of a test script may use a Document Object Model DOM of a web page to develop a set of test scripts to test the application. One problem with this approach is that the developer of the test script may not actually have access to the source code of the application under test. For example, the application under test is still being developed or the application under test is being tested by a third party.

An alternative is to use captured images of a user graphical user interface of the application under test (e.g., a screen shot of a browser) in order to identify graphical objects in the graphical user interface. However, when it comes to image comparison, the challenge is quite different. For example, in order to do image comparison, current systems use image comparison algorithms to match one image with another. While image comparison can be very useful in identifying graphical objects in a graphical user interface, there are still limitations in the current image comparison algorithms. There are times when the image comparison algorithm cannot completely parse and understand a target graphical object. For example, in most cases, the target graphical object contains unnecessary information such as additional image edges, unimportant colors, shapes etc. This type information can reduce the accuracy of identifying a graphical object in a user interface. What is needed is a simplified way of identifying specific graphical objects in the graphical user interface.

SUMMARY

These and other needs are addressed by the various embodiments and configurations of the present disclosure. An image of a graphical user interface is captured. For example, a screen shot of a browser display is captured. Text syntax is executed that contains one or more parameters for identifying a graphical object. For example, the text syntax may identify a rectangle that contains the text “OK” where the text is red. Based on the text syntax, a graphical object is identified in the image of the graphical user interface. Information is returned that identifies how to access the graphical object in the graphical user interface. For example, coordinates of the graphical object are identified. This information can then be used in a test script using existing programming languages to test the graphical user interface. For example, the coordinates may be used to click on the OK button.

The present disclosure can provide a number of advantages depending on the particular configuration. These and other advantages will be apparent from the disclosure contained herein.

The term “execute” as described herein and in the claims refers to the execution/interpretation of text syntax. For example, the text syntax may be executed during runtime by an interpreter. The text syntax may be compiled into a binary executable and executed during runtime.

The preceding is a simplified summary to provide an understanding of some aspects of the disclosure. This summary is neither an extensive nor exhaustive overview of the disclosure and its various embodiments. It is intended neither to identify key or critical elements of the disclosure nor to delineate the scope of the disclosure but to present selected concepts of the disclosure in a simplified form as an introduction to the more detailed description presented below. As will be appreciated, other embodiments of the disclosure are possible utilizing, alone or in combination, one or more of the features set forth above or described in detail below. Also, while the disclosure is presented in terms of exemplary embodiments, it should be appreciated that individual aspects of the disclosure can be separately claimed.

DETAILED DESCRIPTION

FIG.1is a block diagram of a first illustrative system100for identifying graphical object(s) in an image of a graphical user interface. The first illustrative system100comprises a test system101, a network110, and test server/device120.

The test system101comprises a processor102, an image capture module103, a test program104, a code execution module105and machine learning106. The test system101can be or may include any device used for testing the test server/device120, such as, a Personal Computer (PC), a telephone, a video system, a cellular telephone, a Personal Digital Assistant (PDA), a tablet device, a notebook device, a smartphone, a server, and/or the like. Although not shown inFIG.1, the test system101may comprise multiple test systems101. For example, the test system101may comprise a plurality of tests systems101that test the test server/device120in parallel and/or in series.

The processor102can be, or may include, any kind of processor that can process computer code, such as, a hardware processor, a microprocessor, a micro controller, a multi-core processor, an application specific processor, a virtual machine, and/or the like.

The image capture module103can be, or may include, any software/hardware that can capture and process an image. The image capture module103may capture an image using a camera or may directly capture the image that is generated by the test system101. For example, the image capture module103may capture an image that is generated directly from a headless browser.

The test program104can be, or may include, any software/hardware that can generate test(s) for testing the application under test121. The test program104can be written in various programming languages, such as, C, C++, Java, JavaScript, Hyper Text Markup Language (HTML), PERL, and/or the like. The test program104may include any of the test scripts/Application Programming Languages (APIs)/text syntax described herein in conjunction with any known programming languages.

The code execution module105can be, or may include, any hardware/software that can be used to execute the test program104. The code execution module105may run any developed test scripts/test programs104using the text syntax/APIs described herein. The code execution module105may be a code interpreter, may execute code that has been compiled into binary code, and/or the like.

The machine learning106can be or may include any software/hardware process that can learn based on an input, such as, supervised machine learning, unsupervised machine learning, reinforcement machine learning, and/or the like. The machine learning106can be used to learn how to identify graphical objects in a graphical user interface as described below inFIG.2.

The network110can be or may include any collection of communication equipment that can send and receive electronic communications, such as the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), a Voice over IP Network (VoIP), the Public Switched Telephone Network (PSTN), a packet switched network, a circuit switched network, a cellular network, a combination of these, and/or the like. The network110can use a variety of electronic protocols, such as Ethernet, Internet Protocol (IP), Session Initiation Protocol (SIP), Integrated Services Digital Network (ISDN), Hyper Text Markup Language (HTML), Web Real-Time Transport (Web RTC) protocol, and/or the like. Thus, the network110is an electronic communication network configured to carry messages via packets and/or circuit switched communications.

The test server/device120can be or may include any server/device that can support an application under test121. For example, the test server/device120may be a file server, a web server, an application server, a computer, an embedded device, or any device that generates a graphical user interface that can be tested. The test server/device120further comprises the application under test121and a processor122.

The application under test121can be or may include any application that has a graphical user interface that can be tested. The application under test121is tested by the test program104using the processes described herein in conjunction with known testing techniques.

The processor122can be similar to the processor102. For example, the processor122may be a microprocessor.

In one embodiment, the test system101may be part of the test server120. In this embodiment, the network110may not be used and the test system is executed on the test server/device120.

FIG.2is a diagram of images200that are used to train a machine learning algorithm for identifying graphical object(s) in an image of a graphical user interface (e.g. graphical user interface300). Illustratively, the test system101, the image capture module103, the test program104, the code execution module105, the test server/device120, and the application under test121are stored-program-controlled entities, such as a computer or microprocessor, which performs the method ofFIGS.2-4and the processes described herein by executing program instructions stored in a computer readable storage medium, such as a memory (i.e., a computer memory, a hard disk, and/or the like). Although the methods described inFIGS.2-4are shown in a specific order, one of skill in the art would recognize that the steps inFIGS.2-4may be implemented in different orders and/or be implemented in a multi-threaded environment. Moreover, various steps may be omitted or added based on implementation.

The diagram of images200comprises arrow images201A-201N, telephone images202A-202N, rectangle images203A-203N, and question mark images204A-204N. The images201-204are used as input into the machine learning106so that the machine learning106can better identify specific types of graphical objects in the graphical user interface.

For example, the machine learning106takes in the different arrow images201A-201N so that the machine learning106can better identify different types of arrows. The arrow images201A-201N can be different in size, orientation, shading, thickness, and/or the like. As indicated by arrow image201N, there can be any number of arrow images201that can be used as input to train the machine learning106. Although not shown, the arrow images201A-201N be in different colors or shades of colors. The machine learning106, based on the arrow images201A-201N can now better identify new arrow images201in the graphical user interface.

Likewise, the different telephone images202A-202N can be provided as an input to the machine learning106so that the machine learning106can be trained to better identify images of telephones in the graphical user interface. Although shown as black and white images, the telephone images may be in different colors and/or shades. Likewise, as indicated by telephone image201N, there can be any number of telephone images202that can be used as input to train the machine learning106.

The different rectangle images203A-203N can be provided as an input to the machine learning106so that the machine learning106can be trained to better identify images of rectangles in the graphical user interface. For example, the machine learning106may now be able to identify a rectangle that is embedded within a particular graphical object. Although shown as black, white, and grey images, the rectangle images203A-203N may be in different colors. Likewise, as indicated by rectangle image203N, there can be any number of rectangle images203that can be used as input to train the machine learning106.

Likewise, the different question mark images204A-204N can be provided as an input to the machine learning106so that the machine learning106can be trained to better identify images of question marks in the graphical user interface. Although shown as black, white, and grey images, the question mark images204A-204N may be in different colors. Likewise, as indicated by question mark image204N, there can be any number of question mark images204that can be used as input to train the machine learning106.

FIG.2shows four different types of images201-204that can be used to train the machine learning106. As one of skill in the art would understand, the machine learning106can be trained to identify any number of graphical objects by using one or more images for identifying a particular type of graphical object. For example, the machine learning106can be trained to identify any known graphical object, such as, buttons, radio buttons, text boxes, text areas, check boxes, menus, menu items, lists, icons, images, tab fields, scrollbars, circles, ovals, triangles, hexagons, boxes, star shapes, and/or the like based on being trained using similar graphical objects.

Once the machine learning106has completed the training, the machine learning106can now be applied to a captured image of the graphical user interface. The machine learning106is then tied to various kinds of text syntax that allows a developer to test the application under test121. The text syntax is a form of an Application Programming Interface (API) that the developer can incorporate into existing programs (e.g., a JavaScript program for testing the application under test121.

FIG.3is a diagram of an exemplary captured image of a graphical user interface300. The captured graphical user interface300comprises the graphical objects301A-301Z. WhileFIG.3shows various graphical objects301A-301Z, one of skill in the art would recognize that other types of graphical objects301may be captured in the image of the graphical user interface300. The captured image of the graphical user interface300may also comprise a series of captured images of the application under test121. For example, the series of captured images may be images of multiple web pages of the application under test121.

Graphical objects301A-301D are button objects. Graphical object301A contains the text “Click Me” that is black in color. Graphical object301B contains the text “Push Me” that is black in color. Graphical object301C contains the text “Push me” that is black in color. Graphical object301D contains the text “Push me” that is grey in color. Note that in the graphical objects301C-301E the m in “me” is lower case where in graphical object301B the M in “Me” is in upper case.

Graphical object301E is a text object that contains the text “Push me” in the color black. Graphical object301F is a check box object that has been checked and has an associated text of “Use Default Search” in the color black.

Graphical objects301G-301I are text field objects. Graphical object301G is a password field that contains the text “******” in the color black. Graphical object301H is a text field that has no text. Graphical object301I is a text field that contains the word “Hello” in the color black.

Graphical objects301J-301M comprise is a radio button object. The radio button object is a compound object that comprise four selectable radio button objects. The four selectable radio buttons (301J-301M) are labeled “One”, “Two”, “Three”, and “Four” respectively. The labels for the radio buttons301J-301M are black in color.

The graphical objects301N-301Q comprise a compound list object. The graphical objects301N-301P are individual list objects that are labeled “Item A1”, “Item A2”, and “Item AN” respectively. The list objects301N-301P are in the color black. The graphical object301Q is a scrollbar object that allows a user to scroll the list objects301N-301Q.

The graphical object301R is a text area object. The text area object301R contains four lines of text: 1) “TEXT AREA”, 2) “Multiline text 1”, 3) “Multiline text 1”, and 4) “Multiline text 2”. The text objects “TEXT AREA” and “Multiline text 2” are black in color. There are two text lines with the text “Multiline text 1” in the object301R where one is in the color grey and one is in the color black.

The graphical objects301S-301T are circular button objects. The graphical object301S is a circular button object that allows a user to scroll down in a window. The graphical object301S contains a down arrow object. The graphical object301S has an associated text that says “Down” and is the color black. The graphical object301T is a circular button object that allows the user to scroll up in a window. The graphical object301T contains an up arrow object. The graphical object301T has an associated text that says “Up” and is the color black.

The graphical objects301U-301V are tab pane objects. The graphical object301U is a first tab pane object that is labeled “Tab 1” and contains the text “Tab 1 Text”. The graphical object301V is a second tab pane object that is labeled “Tab 2”. Only the tab of the second tab pane object301V is shown because the rest of the second tab pane object301V is hidden. The color of the tab and the text of the graphical object301U is black. The color of the tab pane object301V is grey.

The graphical objects301W-301X are scrollbar objects. The graphical object301W is a vertical scrollbar object and the graphical object301X is a horizontal scrollbar object. The graphical objects301W-301X are compound objects that consist of a rectangle object and two square objects. The two square objects each contain a triangle object (a scrollbar pointer). The triangle objects point in the direction of scrolling for the scrollbar pointer.

The graphical objects301Y-301Z are icon objects. The graphical object301Y is for a cloud icon object. The graphical object301Z is a pen icon object. The graphical objects301Y-301Z may be button objects that cause an event to occur when selected.

In order to identify the graphical objects301a defined text syntax (an Application Programming Interface (API)) is used that has some similarities to existing programming languages (e.g., Cascading Style Sheets), but has been adopted specifically for identifying the graphical objects (e.g., graphical objects301A-301Z). While the format may have some similarities, the application of the text syntax is being applied in a new and novel format. An exemplary embodiment of the text syntax is show below:

The term “shape” refers to the type of shape. For example, the shape could be a rectangle, a circle, a triangle, a square, text, etc. The shape may also be a machine learned shape, such as, a telephone, a question mark, an arrow, a button, a menu, a menu item, a scrollbar, and/or the like. The attribute (e.g., attribute 0) is an attribute associated with the shape. For example, the shape may be a rectangle and the attribute may be a color of the shape.

The API set may use relational operators to further identify objects. Relational operations can be used to further refine which graphical object301is being searched for. An illustrative example of the relational operators is shown below. As one of skill in the art would understand, different symbols may be used for the relational operators. In addition, other relational operators may be envisioned depending upon the type of graphical object301being searched for.

!>does not contain operator

!˜not around operator

? unknown object operator

& overlap object operator

!& not overlap operator

The index is used to identify a specific graphical object301out of a group of graphical objects301that have been identified using the text syntax. For example, if two graphical objects301are identified, the index can be used to further refine the search parameters to identify one of the two graphical objects301.

To illustrate, consider the text syntax “text(“Click Me”)”. InFIG.3, this text syntax identifies the graphical object301A, which is a button that that contains the text “Click Me”. In this example, the graphical element301A is the only graphical object301that has the text “Click Me”. However, if the text syntax “text(“Push me”)” is used, graphical objects301C-301E (two button objects and a text object) are identified. If the user wanted to get information about an individual graphical object301of the graphical objects301C-301E, the syntax “text(“Push me”) [0]” would return the text element301C. The index for identifying a specific graphical object301can be based on top to bottom using a left to right flow if at the same level or any process where individual graphical objects301can be identified. In this example, graphical object301B would not be identified because the “M” in graphical object301B is capitalized. If graphical object301B was to be identified, the text syntax “text(“Push Me”)” may be used.

Individual graphical objects301can be identified in other ways. For example, using the contains relational operator (>), the graphical object301B can be identified as follows: “rect>text (“Click Me”)”. The text syntax is looking for a rectangle that contains the text “Click Me”, which resolves only to the graphical object301B. The text syntax “rect>text (“Push me”)” identifies graphical objects301C and301D because the graphical objects301C and301D are both rectangles that contain the text “Push Me”. InFIG.3, the text of graphical object301C is black and the text of graphical object301D is grey. To identify a specific graphical object301C-301D, further refinement is needed using the index field. This can be accomplished by using the following text syntax: “rect>text(“Push Me”) [color=“black”]”. In this example, only graphical object301C is identified. If the text “Push Me” in the graphical object301C was the color blue instead of black, then the text syntax would be “rect>text(“Push Me”) [color=“blue”]”.

If the user wanted to get information for all rectangles that contain the text “push me” regardless of capitalization, the expression “rect>text(“push me”) [cap=“ignore”]” would be used. In this example, information associated with graphical objects301B-301D would be returned. Likewise, if a button was a machine learned object, the syntax “button>text(“Push Me”) [color=grey]” would return information associated with graphical object301D. If the user wanted to get all button objects that don't contain the text “Push Me” the text syntax “button!>text(“Push Me”)” would return information associated with graphical objects301A and301C-301D.

To identify the graphical object301F (the check box) the text syntax “square>icon (“Check”)” would identify the checkbox301F because there is only one checked checkbox inFIG.3. If the user wanted to identify a checkbox that is not checked, the text syntax “square>icon (“empty”)” would identify any checkboxes that are not checked. InFIG.3, this would return null because there are no checkbox objects that are not checked. Alternatively, if a specific checkbox was needed to be identified, the following text syntax would identify the graphical element301F: “square>:right text (“Use Default Search”)”. The text syntax “:right” is a shape descriptor that identifies where to locate the checkbox object301F by locating the text “Use Default Search” to the right of the checkbox square.

The text syntax “rect>icon astrick:6” would be used to identify graphical object301G (a password object). This text syntax looks for six asterisk objects within any rectangle object. The text syntax “rect>text(“ ”)” would be used to identify graphical object301H (an empty text field). The text syntax “rect>text(“Hello”)” would be used to identify graphical object301I.

Likewise, any of the rectangle objects can be identified using the index field (assuming that the index is based on top to bottom using right to left). For example, the text syntax rect[1] will identify the graphical object301A, the text syntax rect[2] will identify the graphical object301B, the text syntax rect[3] will identify graphical object301C, the text syntax rect[4] will identify graphical object301D, the text syntax rect[5] will identify graphical object301G (not the checkbox301F because it is a square object), and so on.

The radio button is a compound object that consists of four graphical objects301J-301M. Graphical object301J can be identified in different ways. For example, the text syntax “circle>icon (“Circle”)” can be used to identify graphical object301J (the currently selected radio button). The graphical object301J can also be identified with the text syntax “circle˜:right text(“One”)”. To identify graphical object301K (radio button two), the text syntax “circle:right text(“Two”)” is used. To identify graphical object301L (radio button three), the text syntax “circle ˜:right text(“Three”)” is used. To identify graphical object301M (radio button four), the text syntax “circle ˜:right text(“Four”)” is used.

Graphical objects301N-301Q comprise a compound list object. The compound list object comprise list objects301N-301P and a scrollbar object301Q. The graphical objects301N-301Q can be identified in various ways. For example, the list item301N can be identified using the text syntax “text(“Item A1”)”, the list item301O can be identified using the text syntax “text(“Item A2”)”, and the list item301P can be identified using the text syntax “text(“Item AN”)”. The scrollbar object301Q can be identify using the text syntax “verticalscrollbar ˜:left text (“Item A2”)”. Alternatively, the scrollbar object301Q can be identified using the text syntax “rect>icon (“Up Triangle” and “Down Triangle”)”. In this example, the scrollbar object301D has two triangles, one that points up and one that points down.

The graphical object301R (text area object301R) can be identified in various ways, such as, “rect>text(“TEXT AREA”)”, “rect>text(“Multiline text 1”)”, “rect>text(“text 2”)”, and/or the like. In these examples, information associated with the graphical object301R is returned. However, a user may want to get a specific text location in the text area object301R. For example, the user may want to insert text into the text area object301R and needs a specific location to insert the text. The text syntax “rect>: end text(“Multiline text 1”)[color=“grey”]” will identify the line position that is right after the end of the grey “Multiline text 1” text string in the text area object301R. Likewise, the text syntax “rect>: start text(“TEXT AREA”)” identifies the starting position in the text area object301R. Another alternative is “rect>text(“TEXT AREA”) [position=20]”. In this example, the returned information includes a location in the text area object301R just after the 20thcharacter (assuming an index of 0 to indicate the beginning of the text area object301R) in the text area object301R. This process can also be used for any of the text field objects301G-301I.

The graphical objects301S-301T are two button objects that allows a user to scroll up and down in a window. The graphical objects301S-301T may be identified in various ways. For example, the text syntax “circle>icon (“Arrow”)” would identify both graphical objects301S-301T. If the user wanted to identify a specific one of the graphical objects301S-301T, additional information is needed, such as, “circle>icon (“Down Arrow”)” to identify graphical object301S and “circle>icon (“Up Arrow”)” to identify graphical object301T. Alternatively, the text syntax “circle˜:above text(“Down”)” would identify the graphical object301S.

The graphical objects301U-301V are tab pane objects. In this example, the tab pane objects301U-301V are machine learned objects. To identify the tab pane object301U, the text syntax “tabpane>tab text(“Tab 1”)” can be used; to identify tab pane object301V and text syntax “tabpane>tab text(“Tab 2”)” can be used. Since the text of tab pane301U is visible, the text syntax “tabpane>text(“Tab 1 Text”) can be also used to identify tab pane object301U.

The graphical objects301W-301X are scrollbar objects. The scrollbar object301W can be identified with the text syntax “verticalscrollbar: right”. Likewise, the scrollbar object301X can be identified with the text syntax “horizontalscrollbar: bottom”. The returned information in this example, may also include the position of the slider and locations of the of the two scrollbar pointers.

The graphical objects301Y-301Z are exemplary graphical objects301that are graphical objects301where machine learning106has not taken place and are unknown graphical objects301. In some cases, a user may want to identify unknown graphical objects301, such as, graphical objects301Y-301Z. To do this, the text syntax “object+” is used. This text syntax returns information associated with any unknown graphical objects301, which in this example are graphical objects301Y-301Z. If the user wants to identify a specific one of the graphical objects301Y-301Z, the text syntax “object+[0] will return information about graphical object301Y and the text syntax “object+[1] will return information about graphical object301Z.

Although not shown inFIG.3, the system can identify if a graphical object301overlaps. For example, if there were two overlapping graphical objects (e.g., the button objects301A and301B overlapped), the text syntax “object &” would return the two overlapping graphical objects301. The returned value may also include a location of the overlap. Alternatively, the returned value may be the top or bottom overlapping graphical object301depending on implementation. Likewise, the not overlap operator (!&) may also be used to identify any graphical objects301that don't overlap. For example, if the text syntax “object !& were used for the graphical user interface300, all the graphical objects301A-301Z would be returned.

The not around operator (!˜) can be used to identify any graphical objects301that are not around a specific graphical object301. The not around operator !˜may include a range parameter that identifies a distance from the graphical object301.

Each of the operators may have additional parameters and/or combinations that can identify relationships between the graphical objects301in the graphical user interface300. For example, the text syntax “object>[color=“yellow”]” may be used to identify all graphical objects that contain the color yellow. As one of skill in the art would recognize, all kinds of operators and parameters can be used to identify any kind of graphical object.

If a new graphical object is identified where machine learning can be used, the user can add a new graphical object type using the text syntax. Using the cloud button object301Y as an example, the user can define a new text name (e.g., “cloudbutton”) and then provide a series of examples of cloud buttons images to be associated with the new cloudbutton object (e.g., via a graphical user interface) as described inFIG.2. For example, the user may select the cloud button object301Y and then be asked provide the new text name (e.g., “cloudbutton”) and to provide images for the machine learning106.

Once trained, the machine learning106is then ready to now use the new text syntax. In one embodiment, the process could use unsupervised machine learning to dynamically (e.g. automatically) identify the cloud button object based on a history of machine learning. The system could dynamically prompt the user to provide the new cloud button text name and approve the addition of the new cloud button text syntax. At this point, the user can start writing new code for the test program104. For example, the text syntax “cloudbutton >text(“CB”)” could now be used to identify the cloudbutton graphical object301Y.

The returned information about a graphical object301can comprise various types of information associated with the graphical object301. For example, the returned information may include a center location of the graphical object301, coordinates of the graphical object301, coordinates of fields of compound graphical objects301, locations of scrollbar pointers (e.g., top pointer location, bottom pointer location, right pointer location, left pointer location, etc.), locations in a text area or field, and/or the like.

In addition, the return information can identify a plurality of graphical objects301in the graphical user interface300that match the text syntax. For example, the returned value may be an array of information, one for each identified graphical object301. The returned information may include a message indicating that a plurality of graphical objects301are in the graphical user interface300. If there are more than one graphical object301identified, the returned information may be a message indicating to refine the text syntax. Alternatively, if there are no identified graphical objects301a message indicating that no graphical objects301may be returned.

A user can then use the return information to programmatically perform actions. For example, the action may be to simulate a user action, such as, clicking on a graphical object301, entering text in the graphical object301, selecting a check box, selecting a radio button, selecting a menu selecting a menu item, moving a scrollbar, selecting a tab, doing a mouse hover, doing left mouse click, doing right mouse click, selecting a window, and/or the like.

To illustrate how a user can incorporate the text syntax into the test program104, consider the following illustrative example where the text syntax is highlighted in bold along with line numbers to identify a specific line of code.

In line number 1, the code launches the Chrome browser. Line number 2 causes the browser to navigate to “http://www.bing.com”. In line 3, the text syntax “red>text(“search”)” identifies a rectangle graphical object301that contains the text “search” (i.e., a search button). The return information (e.g., a coordinates of a center location of the search button) is used to perform a click event on the search button. The above example is a test script for testing the application under test121. One advantage to the text syntax is that the test script can be developed before and/or concurrently with the graphical user interface300based on a mockup model of the graphical user interface300.

FIG.4is a flow diagram of a process for identifying graphical object(s)301in an image of a graphical user interface300. The process starts in step400. The image capture module103captures, in step402, an image of a graphical user interface300(could be multiple images). The image of the graphical user interface300may be captured by intercepting a video image sent to a display or may be captured via a camera. The image of the graphical user interface300may be an image generated by a client/server application (application under test121). Alternatively, the captured image of the graphical user interface300may come from a browser that is running a web page provided by the application under test121. In another embodiment, the test system101may be on the test server/device120.

The code execution module105determines, in step404, if the text syntax has been reached in the test program104. If the text syntax has not been reached in step404, the process goes to step414. Otherwise, if the text syntax has been reached in step404, the code execution module105executes, in step406, the text syntax in order to identify graphical objects301as described above inFIGS.2-3.

The code execution module105determines, in step408, if any graphical objects301have been identified based on the text syntax. If no graphical objects301were found that match the text syntax, in step408, the code execution module105returns information to indicate that there were no graphical objects301that were found in step410and the process goes to step414. Otherwise if one or more graphical objects301we found in step408, the test program104returns, in step412, information that identifies how to access the graphical object(s)301that are in the user interface300.

The code execution module105determines, in step414, if the test is complete. If the test is not complete in step414, the process goes back to step404. Otherwise, if the testing is complete in step414, the process ends in step416.