Patent Publication Number: US-2023153126-A1

Title: Screen recognition apparatus, screen recognition method and program thereof

Description:
TECHNICAL FIELD 
     The present invention relates to a screen recognition device, a screen recognition method and a program. 
     BACKGROUND ART 
     There is a proposition for automatically generate a user operation scenario from a GUI program with no separate labor in a development process (for example, Patent Literature 1). 
     CITATION LIST 
     Patent Literature 
     Patent Literature 1: Japanese Patent Laid-Open No. 2010-102620 
     SUMMARY OF THE INVENTION 
     Technical Problem 
     For automating a work using an OSS (Operation Support System), it is necessary to extract the position and form of a GUI (Graphic User Interface) component on a screen of the OSS and the kind such as a character, a button and a text box, and to identify an operation that can be performed on the screen. 
     For this kind of extraction and identification, in the related art including the technology described in Patent Literature 1, it is necessary to analyze and determine a program code and an object in a terminal memory for a target application. Therefore, the extraction and the identification cannot be applied to an environment in which it is not possible to analyze the program code and the object in the terminal memory for the target application, for example, to the screen of the OSS in which an operator cannot see a source code. 
     The present invention has been made in view of the above circumstance, and an object thereof is to provide a screen recognition device, a screen recognition method and a program that make it possible to recognize the GUI component in the screen without analyzing the program code of application software, the object in the terminal memory for the target application, and the like. 
     Means for Solving the Problem 
     An aspect of the present invention includes: an extraction unit configured to extract a GUI component included in a display screen of application software, from the display screen; and a determination unit configured to determine a kind of the GUI component extracted by the extraction unit and an operation that can be performed to the GUI component, based on a change in a mouse cursor and a change in the GUI component when the mouse cursor is superimposed on the GUI component. 
     Effects of the Invention 
     According to an aspect of the present invention, it is possible to recognize the GUI component in the screen without analyzing the program code of application software, the object in the terminal memory for the target application, and the like. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG.  1    is a block diagram showing a GUI component analysis unit and a GUI component database (DB) unit according to an embodiment of the present invention. 
         FIG.  2    is a flowchart showing processing contents for identifying the form and position of a GUI component in a screen according to the embodiment. 
         FIG.  3    is a flowchart showing processing contents for identifying the kind and operation of the GUI component in the screen according to the embodiment. 
         FIG.  4    is a diagram exemplifying a screen on which a home page at a website is displayed according to the embodiment. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     An embodiment when the present invention is applied to a device that performs screen recognition will be described below. 
     [Configuration] 
       FIG.  1    is a block diagram showing the functional configuration of the device that performs screen recognition, and is a block diagram showing a GUI component analysis unit  10  and a GUI component database (DB) unit  20  in which many pieces of information about a GUI component that is a recognition target are recorded. The GUI component analysis unit  10  includes a GUI component form/position determination unit  11 , a GUI component kind identification unit  12  and a GUI component operation identification unit  13 . All results from sequential analysis processing by the units of the GUI component form/position determination unit  11 , the GUI component kind identification unit  12  and the GUI component operation identification unit  13  are recorded in the GUI component DB unit  20 . 
     The content shown in  FIG.  1    shows, as a block diagram, the functional concept configuration of a software program (hereinafter referred to as “analysis software”) that performs an analysis according to the embodiment, and the hardware configuration, which is the same as that of a general computer, includes a keyboard and a pointing device such as a mouse as an input unit, a monitor display to perform screen display as an output unit, a processor as a processing unit, various memories and storage devices, and others. 
     [Behavior] 
     Next, the behavior of the embodiment will be described. 
       FIG.  2    is a flowchart showing processing contents when the GUI component form/position determination unit  11  identifies the form and position of the GUI component in a screen provided by application software, with the analysis software in the embodiment. 
     In the beginning of the process, the GUI component form/position determination unit  11  performs a marker process to the GUI component in the screen, by accepting a key operation such as a simultaneous operation of a “Ctrl (control)” key and an “A” key or a continuous operation of a “Tab” key through the keyboard (step S 111 ). 
     A region range in which neither the display form of a mouse cursor nor the display form of the GUI component changes at the time when the marker process of the GUI component is performed and the mouse cursor is superimposed is regarded as a region that is a background other than the GUI component, and is eliminated from the target of the analysis process for the GUI component (step S 112 ). 
     In other words, a range in which the display form of at least one of the mouse cursor and the GUI component changes at the time when the marker process of the GUI component is performed and the mouse cursor is superimposed is regarded as the range of the GUI component, and is adopted as a target of the analysis process. 
     Then, the identification process for the form of the GUI component ends, and next, the GUI component form/position determination unit  11  transitions to an identification process for the position range of the GUI component. 
     First, the GUI component form/position determination unit  11  executes a process for fixing the operation range of the GUI component in the screen, by accepting a combined operation of a “Home” key and an “End” key or a combined operation of a “PgUp (page up)” key and a “PgDn (page down)” key and the like through the keyboard (step S 113 ). 
     Furthermore, the mouse cursor is superimposed on identified operations ranges of GUI components (mouse cursor over) sequentially and consecutively (step S 114 ), and acquires changes in the states of the operation ranges of the GUI components and the mouse cursor itself are acquired at the time when the mouse cursor is superimposed in the screen of the application software (step S 115 ). 
     Based on the change in each GUI component, the GUI component form/position determination unit  11  creates a coordinate range of the GUI component in the screen, and recognizes the form of the outline (step S 116 ). 
     From the results, the GUI component form/position determination unit  11  comprehensively determines the form and position range of each GUI component in the screen, and records the determination result in the GUI component DB unit  20  (step S 117 ). Then, the process in the GUI component form/position determination unit  11  ends. 
     As described above, the GUI component form/position determination unit  11  has the function to identify the form of the GUI component in steps S 111  and S 12  and the function to identify the position of the GUI component shown in steps S 113  to S 117 . 
     Next, processes by the GUI component kind identification unit  12  and the GUI component operation identification unit  13  after the form and position of the GUI component are identified will be described with use of  FIG.  3   . 
       FIG.  3    is a flowchart showing a sequence of processing contents for identifying the kind and operation of one GUI component with the analysis software in the embodiment. First, based on the determination result about the form and position range of each GUI component that is recorded in the GUI component DB unit  20 , the GUI component kind identification unit  12  superimpose the mouse cursor on the GUI component (mouse cursor over), and changes the display forms of the mouse cursor itself and the GUI component in the screen of the application software (step S 121 ). 
     The GUI component kind identification unit  12  determines whether the kind of the GUI component can be identified based on only the change in the mouse cursor (step S 122 ). 
     In the case where the GUI component kind identification unit  12  determines that the kind of the GUI component can be identified based on only the change in the mouse cursor (YES in step S 122 ), the GUI component kind identification unit  12  identifies the kind of the GUI component (step S 123 ). 
     In the case where the GUI component kind identification unit  12  determines that the kind of the GUI component can be identified based on only the change in the mouse cursor in step S 122  (NO in step S 122 ), the GUI component kind identification unit  12  identifies the kind of the GUI component based on both the change in the cursor and the change in the GUI component (step S 124 ). 
     Next, the GUI component operation identification unit  13  identifies the operation corresponding to the GUI component based on the kind of the GUI component identified in step S 123  or step S 124  (step S 131 ), and records the identified result in the GUI component DB unit  20 . Then, the process in  FIG.  3    ends. 
     The GUI component kind identification unit  12  and the GUI component operation identification unit  13  identify the kinds of all GUI components recorded in the GUI component DB unit  20  and the operations corresponding to the kinds, by repeatedly executing the process in  FIG.  3    to the number of the recorded GUI components, and record the identified results in the GUI component DB unit  20 . 
       FIG.  4    is a diagram exemplifying a screen on which a home page (Home) at a certain website is displayed by a web browser as an example of the application software. In the figure, a search guide portion SG for assisting a content search in the website is displayed at the center of the screen, and a search symbol SS for giving an instruction of the execution of the search, a text input box TB for inputting a search text and a microphone symbol MS for giving an instruction of the text input by voice are disposed in the search guide portion SG. Further, a search button SB for giving an instruction of the execution of the search is displayed on the lower side of the search guide portion SG. 
     As specific GUI components, the search symbol SS in a circular range, the text input box TB in a rectangular range, the microphone symbol MS in a rectangular range and the search button SB in a rectangular range are recognized differently from the screen background other than them. 
     In the whole of a peripheral region AA including the search guide portion SG, an arrow pointer AP is displayed as the mouse cursor. Meanwhile, when an instruction point at an arrow tip of the arrow pointer AP moves particularly to the search symbol SS, microphone symbol MS and search button SB recognized as GUI components and becomes the state of the mouse cursor over, the display of the mouse cursor changes from the arrow pointer AP to the hand pointer HP, and waits for the jump to respective preset link destinations by click operations in the GUI components. 
     Furthermore, when the instruction point at the arrow tip of the arrow pointer AP moves to the text input box TB and becomes the state of the mouse cursor over, the display of the mouse cursor changes from the arrow pointer AP to an I-beam pointer, and waits for the input of the text information into the text input box TB. 
     Here, the analysis of the screen based on the change in the mouse cursor, particularly, the analysis of the GUI component will be described afresh. 
     As described above, the symbol image of the mouse cursor that is displayed is determined by the setting of the property of the mouse cursor. The mouse cursor corresponding to the content to be operated is displayed depending on the kind of the GUI component at the point position. For example, the arrow pointer AP is displayed in the case of the normal mouse cursor, the I-beam pointer IP is displayed in the case of the input of a text, and the hand pointer HP is displayed in the case of the selection of a link. 
     Therefore, based on the characteristic of the change in the mouse cursor, it is possible to perform the drawing recognition relevant to the GUI component, from combinations of operations that are normally performed by a user and that do not influence a computer system and the like, as exemplified by combinations of the mouse cursor over and the key input operation for texts and the like. 
     Specific examples are shown below. Examples in the analysis of the screen at the website and the like are shown as follows. 
     The I-beam pointer IP corresponding to text input can be displayed at the time when the mouse cursor is input to an area to which a text can be input or an area to which a text can be copied. 
     The distinction between mere text data and a text link can be determined from the combination between a link selection cursor and an underline. 
     A mere character or figure can be determined from the simultaneous operation of the “Ctrl” key and the “A” key. 
     The entrance to an object region where a button, an icon or the like can be selected can be determined from the change in the symbol image between a normal mouse cursor for selection and a mouse cursor for objects and the display of a guide message. 
     The mouse cursor for the link selection has a characteristic of being not displayed when there is no link destination, and therefore can be used for the determination of an area for which the possibility of the link selection is low, as exemplified by a menu bar and a title bar. 
     In the case of the change from the normal arrow pointer AP to another mouse cursor, the function of an area where the arrow pointer AP just before the change is displayed can be determined from the display form of the arrow pointer AP. 
     A top range of a page can be grasped by the operation of the “Home” key or the “PgUp” key of the keyboard. 
     A bottom range of a page can be grasped by the operation of the “End” key or the “PgDn” key of the keyboard. 
     Therefore, since “some mouse cursors are displayed only in particular areas in the screen” and “some mouse cursors transition to particular operations”, by using the change characteristic of the mouse cursor, it is possible to recognize the GUI component in the screen. Particularly, by considering not only the mouse cursor but also the change in the GUI component, it is possible to enhance the recognition accuracy. 
     Effect of Embodiment 
     As described above in detail, it is not necessary to analyze the program code of the application software, the object in the terminal memory for the target application, or the like, and therefore, it is possible to recognize the GUI component even in the environment in which the program code cannot be analyzed, and the like. Therefore, the embodiment can contribute to RPA (Robotic Process Automation) that is a business process automation technology, for example. 
     Further, in the embodiment, before the identification of the operation corresponding to the kind of the GUI component, the mouse cursor is sequentially superimposed on the GUI component in the screen, or the preset combined key operation is accepted. Then, the range that is around the GUI component and in which the display form does not change is removed as the background, and the remaining range of the GUI component is extracted. Thereby, it is possible to efficiently perform the identification work after that. 
     Furthermore, in the embodiment, in the case where the kind of the GUI component cannot be identified based on only the change in the mouse cursor when the mouse cursor is superimposed on the GUI component, the kind of the GUI component is identified based on the change in the mouse cursor and the change in the GUI component, and therefore it is possible to more efficiently identify the kind of the GUI component. 
     The device in the present invention can be realized also by a computer and a program, and the program can be recorded in a recording medium, or can be provided through a network. 
     In addition, the invention in the present application is not limited to the embodiment, and various modifications can be made in the practical phase without departing from the spirit of the invention. Further, inventions in various phases are included in the embodiment, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even when some constituent elements are removed from all constituent elements shown in the embodiment, a configuration in which the constituent elements are removed can be extracted as an invention, as long as the problem described in the section “Technical Problem” can be solved and the effect described in the section “Effects of the Invention” can be obtained. 
     REFERENCE SIGNS LIST 
       10  GUI component analysis unit 
       11  GUI component form/position determination unit 
       12  GUI component kind identification unit 
       13  GUI component operation identification unit 
       20  GUI component database (DB) unit 
     AA Search box periphery 
     AP Arrow pointer 
     HP Hand pointer 
     IP I-beam pointer 
     MS Microphone symbol 
     SB Search button 
     SG Search guide portion 
     SS Search symbol 
     TB Text input box