English:
[0001] Test automation software may record and automatically replay user interactions with a graphical user interface ("GUI"). GUI developers may use automation software to record and replay mouse input (e.g., points and clicks) and web developers may use it to record and replay web page navigations. These tools alleviate manual testing, which is often laborious and time-consuming.
[0002] FIG. 1 is an example system in accordance with aspects of the disclosure.
[0003] FIG. 2 is an example flow diagram in accordance with aspects of the disclosure.
[0004] FIG. 3 is a working example in accordance with aspects of the disclosure.
[0005] FIG. 4 is a further working example in accordance with aspects of the disclosure.
[0006] As noted above, GUI developers may use test automation to record user operations and to replay those operations automatically as many times as needed. Such tools save a significant amount of time in quality assurance and regression testing. Some test automation tools use an agent- based approach. In this approach, an agent module may be inserted into an application under test ("AUT"). The agent may record information associated with graphical objects as users interact therewith and may also replay those user interactions when needed. One advantage of an agent-based approach is the precision with which graphical data can be captured during the recording phase. An agent may capture detailed data associated with complex graphical user interfaces, such as graphical tree views and calendars. However, the agent-based approach has some disadvantages. For example, if the AUT is executing remotely, security restrictions may prevent insertion of the agent into the remote computer. Furthermore, customized agents may have to be developed for each platform. For example, specific agents may need to be developed for each version of Internet Explorer®.
[0007] In view of the foregoing, various examples disclosed herein provide a system, non-transitory computer-readable medium, and method for recording and replaying user operations. In one aspect, an appearance of a graphical object may be recorded. In a further aspect, it may be determined whether the recorded appearance is different than the current appearance of the graphical object. In another example, particular images, text, and pixel measurements of the graphical object may be recorded. Thus, the system, non-transitory computer-readable medium, and method disclosed herein may capture details of complex graphical objects based on the user interface displayed on the screen without having to insert an agent into the AUT. The techniques disclosed herein capture precise details of the displayed graphical objects commensurate with the details gathered using an agent based approach. The aspects, features and advantages of the application will be appreciated when considered with reference to the following description of examples and accompanying figures. The following description does not limit the application; rather, the scope of the application is defined by the appended claims and equivalents.
[0008] FIG. 1 presents a schematic diagram of an example system that includes an illustrative computer apparatus 100. FIG. 1 depicts various components in accordance with aspects of the disclosure. The computer apparatus 100 may include all the components normally used in connection with a computer. For example, it may have a keyboard, mouse, and/or various other types of input devices such as pen-inputs, joysticks, buttons, touch screens, etc., as well as a display, which could include, for instance, a CRT, LCD, plasma screen monitor, TV, projector, etc. Computer apparatus 100 may also comprise a network interface (not shown) to communicate with other devices over a network using conventional protocols (e.g., Ethernet, Wi-Fi, Bluetooth, etc.). The computer apparatus 100 may also contain a processor 1 10, which may be any number of well known processors, such as processors from Intel ® Corporation. In another example, processor 110 may be an application specific integrated circuit ("ASIC").
[0009] Non-transitory computer readable medium ("CRM") 112 may store instructions that may be retrieved and executed by processor 110. As will be discussed in more detail below, the instructions may include recording module 114 and a replay module 1 16. Non-transitory CRM 112 may be used by or in connection with any instruction execution system that can fetch or obtain the logic from non-transitory CRM 1 12 and execute the instructions contained therein.
[0010] Non-transitory CRM 112 may comprise any one of many physical media such as, for example, electronic, magnetic, optical, electromagnetic, or semiconductor media. More specific examples of suitable non-transitory computer-readable media include, but are not limited to, a portable magnetic computer diskette such as floppy diskettes or hard drives, a read-only memory ("ROM"), an erasable programmable read-only memory, a portable compact disc or other storage devices that may be coupled to computer apparatus 100 directly or indirectly. Alternatively, non-transitory CRM 112 may be a random access memory ("RAM") device or may be divided into multiple memory segments organized as dual in-line memory modules ("DIMMs"). The non-transitory CRM 1 12 may also include any combination of one or more of the foregoing and/or other devices as well. While only one processor and one non-transitory CRM are shown in FIG. 1 , computer apparatus 100 may actually comprise additional processors and memories that may or may not be stored within the same physical housing or location.
[0011] The instructions residing in non-transitory CRM 1 12 (e.g., recording module 1 14 and replay module 116) may comprise any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by processor 1 10. In this regard, the terms "instructions," "scripts," and "applications" may be used interchangeably herein. The computer executable instructions may be stored in any computer language or format, such as in object code or modules of source code. Furthermore, it is understood that the instructions may be implemented in the form of hardware, software, or a combination of hardware and software and that the examples herein are merely illustrative.
[0013] Working examples of the system, method, and non- transitory computer-readable medium are shown in FIGS. 2-4. In particular, FIG. 2 illustrates a flow diagram of a process for recording and replaying user operations. FIGS. 3-4 show example screen shots in accordance with aspects of the techniques disclosed herein. The actions shown in FIGS. 3-4 will be discussed below with regard to the flow diagram of FIG. 2.
[0015] Referring now to FIG. 3, an example tree view 300 is shown. FIG. 3 shows an initial appearance of tree view 300 before user interactions with the tree view begin. Recording module 1 14 may record features of the initial appearance. As noted above, the features may include particular images, text, and pixel measurements. For example, in FIG. 3, recording module 1 14 may record pixel measurement 302 between image 304 and text 306. Furthermore, recording module 1 14 may record image 308, and image 304. Referring now to FIG. 4, a target appearance is shown after execution of at least one user operation. Here, image 308 has changed to image 402 and image 310 has been expanded to expanded tree view 404. The particular images, text, and pixel measurements of this target appearance may also be recorded.
[0016] The images, pixel measurements, and text may be recorded in a variety of ways. For example, coordinates (e.g., x and y coordinates or height and width) may be recorded via an application programming interface. For example, the pixel measurement 302 between image 304 and text 306 may be recorded by determining the x coordinates therebetween. By way of further example, the y coordinates of the expanded tree view 404 may be recorded. In another example, optical character recognition ("OCR") may be used to record text, such as text 306 in FIG. 3 or the text in expanded tree view 404. In yet a further example, images, such as image 308 and image 402, may be parsed and recorded. The parsing may comprise parsing the pixels within the images.
[0017] Referring back to FIG. 2, it is determined whether the current appearance of the graphical object is different than the target appearance, as shown in block 204. In one example, replay module 1 16 may determine the current appearance of the graphical object by comparing particular images, text, and pixel measurements of the current appearance to those of the recorded initial appearance. In another example, the replay module 1 16 may detect features of the current appearance and compare the detected features to the recorded features of the target appearance. Referring back to FIG. 2, if the current appearance is different than the target appearance, the current appearance may be adjusted or changed to look like the target appearance so as to repeat or replay the at least one user operation, as shown in block 206. Replay module 1 16 may transmit commands to the AUT to change the graphical object to the target appearance.
[0018] Advantageously, the above-described system, non- transitory computer readable medium, and method record details of a graphical object as a user interacts therewith and replays the user interactions by adjusting the image as did the user interactions. In this regard, fine details of the image may be recorded so that complex graphical objects may be adjusted in accordance with the recorded user operations. In turn, testing engineers can be rest assured that a high quality automated test can be carried out, even if an agent based approach is not possible.
[0019] Although the disclosure herein has been described with reference to particular examples, it is to be understood that these examples are merely illustrative of the principles of the disclosure. It is therefore to be understood that numerous modifications may be made to the examples and that other arrangements may be devised without departing from the spirit and scope of the application as defined by the appended claims. Furthermore, while particular processes are shown in a specific order in the appended drawings, such processes are not limited to any particular order unless such order is expressly set forth herein. Rather, processes may be performed in a different order or concurrently.

中文:
[0001]测试自动化软件可以记录并自动地重放用户与图形用户界面(“GUI”)的交互。GUI开发者可以使用自动化软件以记录并重放鼠标输入(例如指向和点击)并且网络开发者可以使用自动化软件以记录并重放网页导航。这些工具减轻了通常是费力和耗时的手动测试。
[0002]图1是根据本公开特征方面的示例性系统。
[0003]图2是根据本公开特征方面的示例性流程图。
[0004]图3是根据本公开特征方面的工作示例。
[0005]图4是根据本公开特征方面的进一步工作示例。
[0006]如上所述,GUI开发者可以使用测试自动化以记录用户操作并且自动地重放那些操作如所需之多的次数。这些工具在质量保证和回归测试中节省了大量时间。一些测试自动化工具使用基于代理的方案。在该方案中,代理模块可以被插入至测试下的应用程序(“AUT”)中。代理在用户与图形对象交互时可以记录与图形对象相关联的信息并且在需要时也可以重放那些用户交互。基于代理的方案的一个优点是在记录阶段期间可以捕捉图形数据的精确度。代理可以捕捉与复杂图形用户界面相关联的详细数据,诸如图形树图和日历。然而,基于代理的方案具有一些缺点。例如,如果AUT远程地执行,则安全约束可以防止代理插入至远程计算机中。此外,可能不得不对于每个平台开发定制的代理。例如,可能需要对于Internet的每个版本开发特定的代理。
[0007]考虑到前述,在此所公开的各个示例提供了用于记录并重放用户操作的一种系统、非暂时性计算机可读介质和方法。在一个方面中,可以记录图形对象的外观。在另一方面中,可以确定所记录的外观是否不同于图形对象的当前外观。在另一示例中,可以记录图形对象的特定图像、文本和像素测量结果(measurement)。因此,在此所公开的系统、非暂时性计算机可读介质和方法可以基于显示在屏幕上的用户界面捕捉复杂图形对象的细节而无需将代理插入至AUT中。在此所公开的技术捕捉与使用基于代理的方案所收集的细节相应的所显示图形对象的精确细节。当参考示例的以下描述和附图考虑时将意识到本申请的方面、特征和优点。以下描述并未限制本申请;相反,由所附权利要求及其等价形式限定本申请的范围。
[0008]图1呈现了包括示意性计算机设备100的示例性系统的示意图。图1描绘了根据本公开的方面的各个部件。计算机设备100可以包括通常与计算机结合使用的所有部件。例如,其可以具有键盘、鼠标和/或诸如输入笔、操纵杆、按钮、触摸屏等的各种其他类型的输入装置、以及可以包括例如CRT、LCD、等离子屏幕监视器、TV、投影仪等的显示器。计算机设备100还可以包括网络接口(未示出)以在使用传统协议(例如以太网、Wi-Fi、蓝牙等)的网络之上与其他装置通信。计算机设备100还可以包含处理器110,其可以是任意数目的已知处理器,诸如来自公司的处理器。在另一示例中,处理器110可以是专用集成电路(“ASIC”)。
[0009]非暂时性计算机可读介质(“CRM”)112可以存储可以由处理器110检索并执行的指令,如以下将更详细讨论,指令可以包括记录模块114和重放模块116。非暂时性CRM112可以由可以从非暂时性CRM112获取或获得逻辑并且执行其中所包含的指令的任何指令执行系统所使用或者与其结合使用。
[0010]非暂时性CRM112可以包括许多物理介质中的任意一种,诸如例如电子、磁性、光学、电磁、或半导体介质。合适的非暂时性计算机可读介质的更具体示例包括但不限于,诸如软盘或硬驱动的便携式磁性计算机磁盘、只读存储器(“ROM”)、可擦除可编程只读存储器、便携式小型盘碟、或者可以直接地或间接地联接至计算机设备100的其他存储装置。备选地,非暂时性CRM112可以是随机存取存储器(“RAM”)装置,或者可以划分为编组为双列直插式存储器模块(“DIMM”)的多个存储器区段。非暂时性CRM112也可以包括前述和/或其他装置中的一个或多个的任意组合。尽管图1中仅示出了一个处理器和一个非暂时性CRM,计算机设备100可以实际上包括可以存储或不存储在相同物理外壳或位置内的额外的处理器和存储器。
[0011]驻留在非暂时性CRM112(例如记录模块114和重放模块116)中的指令可以包括将要由处理器110直接地(诸如机器码)或间接地(诸如脚本)执行的任何指令集。在这点上,术语“指令”、“脚本”和“应用程序”可以在此可互换地使用。计算机可执行指令可以以任何计算机语言或格式存储,诸如以对象代码或者源代码的模块。此外,应该理解的是,可以以硬件、软件、或者硬件与软件的组合的形式而实施指令,并且在此的示例仅是示意性的。
[0013]在图2-图4中示出了系统、方法和非暂时性计算机可读介质的工作示例。特别地,图2示出了用于记录并重放用户操作的进程的流程图。图3-图4示出了根据在此所公开的技术的方面的示例性屏幕截图。以下将参照图2的流程图讨论图3-图4中所示的动作。
[0015]现在参照图3,示出了示例性树图300。图3示出了在用户与树图的交互开始之前的树图300的初始外观。记录模块114可以记录初始外观的特征。如上所述,特征可以包括特定的图像、文本和像素测量结果。例如,在图3中,记录模块114可以记录在图像304与文本306之间的像素测量结果302。此外,记录模块114可以记录图像308以及图像304。现在参照图4,在至少一个用户操作执行之后示出目标外观。在此,图像308已经改变为图像402,并且图像310已经被扩展为扩展树图404。还可以记录该目标外观的特定的图像、文本和像素测量结果。
[0016]可以以多种方式记录图像、像素测量结果以及文本。例如,可以经由应用程序编程接口记录坐标(例如x和y坐标、或高度和宽度)。例如,可以通过确定图像304与文本306之间的x坐标而记录它们之间的像素测量结果302。通过其他示例的方式,可以记录扩展树图404的y坐标。在另一示例中,光学字符识别(“OCR”)可以用于记录文本,诸如图3中的文本306或者扩展树图404中的文本。在另外其他示例中,可以解析并记录诸如图像308和图像402的图像。解析可以包括解析图像内的像素。
[0017]返回参照图2,如方框204中所示,确定图形对象的当前外观是否不同于目标外观。在一个示例中,重放模块116可以通过将当前外观的特定图像、文本和像素测量结果与所记录初始外观的特定图像、文本和像素测量结果比较而确定图形对象的当前外观。在另一示例中,重放模块116可以检测当前外观的特征,并且将所检测的特征与目标外观的所记录特征比较。返回参照图2,如果当前外观不同于目标外观,则可以将当前外观调整或改变为看起来像目标外观以便于重复或重放至少一个用户操作,如方框206中所示。重放模块116可以将命令发送至AUT以将图形对象改变为目标外观。
[0018]有利地,上述系统、非暂时性计算机可读介质和方法记录图形对象的细节作为用户与其的交互,并且通过调整用户与其交互的图像而重放用户交互。在这点上,可以记录图像的精细细节以使得复杂的图形对象可以根据所记录用户操作而被调整。接着,测试工程师可以确信无疑,即使基于代理的方案是不可行的,也可以实施高质量自动化测试。
[0019]尽管已经在此参照特定示例描述了本公开,应该理解的是,这些示例仅是本公开的原理的示意说明。因此应该理解的是,在不脱离由所附权利要求所限定的本申请的精神和范围的情况下,可以对示例做出许多修改并且可以得到其他布置。此外,尽管在附图中以特定顺序示出了特定的进程,但这些进程不限于任何特定顺序,除非在此明确阐述了该顺序。相反,可以以不同的顺序或同时地执行进程。