Abstract:
Described is a technology in which test case content in the form of a web application is provided to a client browser from a test case management system over a web server. Results of running the test case are similarly communicated back. This allows different web application test harnesses to be run on whatever Internet browser the client computing device is running, and is independent of any operating system. The client registers with the test case management system through the website, and receives a browser identifier for use in future communications. In one protocol, the client uses the identifier in heartbeats sent to the test case management system, including a heartbeat indicating when the client is available to run a test case, when the test case is complete, and the results of performing the test case. Also described are various interfaces that facilitate component communication.

Description:
BACKGROUND 
       [0001]    A software system called TCM (Test Case Management) automatically executes test cases and manages test machines. In current implementations, TCM can manipulate test machines only if a TCM client is installed and run as a client application on each test machine. 
         [0002]    Web applications also may be tested. However, customers using a web browser other than Internet Explorer running on a Windows® operating system are not supported. Existing TCM systems fail to support this case because no client applications are available, and/or because of difficulties in developing client application for non-Windows® operating systems. As a result test cases for non-Windows® operating systems have to be executed or launched manually, which increases the cost of software development. 
       SUMMARY 
       [0003]    This Summary is provided to introduce a selection of representative concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in any way that would limit the scope of the claimed subject matter. 
         [0004]    Briefly, various aspects of the subject matter described herein are directed towards a technology by which test case content in the form of a web application and its post-execution results are communicated between a client and a test case management system over a web server. This allows different web application test harnesses to be run on whatever Internet browser the client computing device is running. As one result, the test case management system and client computing device that runs the browser are independent of any particular operating system. 
         [0005]    In one aspect, the client registers with the test case management system through a website. The test case management system returns an identifier for the browser for use in future communications between the browser and the test case management system. The client uses the identifier in heartbeats sent to the test case management system, including a heartbeat indicating when the client is available to run a test case. 
         [0006]    In one aspect, when the client is available and the test case management system decides when to provide the client with a test case, a test case configured as a web application is provided to the client. The client runs the test case, and when complete, returns results to the test case management system, e.g., in a later heartbeat. In this manner, any client running any contemporary web browser and/or operating system may be supported by the test case management system. 
         [0007]    Other advantages may become apparent from the following detailed description when taken in conjunction with the drawings. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0008]    The present invention is illustrated by way of example and not limited in the accompanying figures in which like reference numerals indicate similar elements and in which: 
           [0009]      FIG. 1  is a block diagram showing an example test case management controller web access system. 
           [0010]      FIGS. 2-4  are representations of communications and control flow between various components of the test case management controller web access system. 
           [0011]      FIG. 5  shows an illustrative example of a computing environment into which various aspects of the present invention may be incorporated. 
       
    
    
     DETAILED DESCRIPTION 
       [0012]    Various aspects of the technology described herein are generally directed towards delivering test case content and execution results over a web server, e.g., instead of via a test case management client application. While some of the examples described herein are directed towards protocols and interfaces to facilitate the delivery, along with interfaces to accommodate different test case management systems for test case management controller web access and interface to accommodate different web application test harnesses running on internet browsers, it is understood that these are only examples. Indeed, numerous other protocols and/or interfaces may be used to provide a similar benefit. 
         [0013]    As such, the present invention is not limited to any particular embodiments, aspects, concepts, structures, functionalities or examples described herein. Rather, any of the embodiments, aspects, concepts, structures, functionalities or examples described herein are non-limiting, and the present invention may be used various ways that provide benefits and advantages in computing and test case management in general. 
         [0014]    Turning to  FIG. 1 , there is shown an example TCM controller web access implementation in which computing devices corresponding to test browser A and test browser B are being remotely test case managed via a web server  102 . Note that while test browser A and test browser B may be running on separate machines, it is understood that they may possibly share one test machine. 
         [0015]    A virtual client manager (VCM)  104  acts as a bridge between a test controller web access (TCWA) subsystem  106  and the web server  102 , generally for handling test cases, execution results and task distribution. A job session service may be used to separate test case and web browser registration, e.g., to enable test case execution that may use multiple browsers and/or other execution units. 
         [0016]    Note that as also shown in  FIG. 1 , a virtual TCM (VTCM)  110  is an abstract of TCM with which the test controller web access (TCWA) subsystem communicates. The job Codec  112  provides a translation mechanism between a TCM system-specific job/result description and a test case running inside the virtual client manager  102 . 
         [0017]    A virtual client  114  comprises a stateless website, which acts as HTTP channel with AJAX (asynchronous JavaScript and XML), and a web application running on a web browser, which manages the state of a web browser as an execution unit. Other components  131 - 134  of the product web server are shown for completeness. 
         [0018]    Each text browser includes a JavaScript Test Arena (JTA)  120 A or  120 B): interface to test case running on the web browser. In each tested browser, a client Monitor (CM,  122 A or  122 B) comprises an application program running along with the web browser. One task of the client monitor is to restart the web browser for each test case, such as to compensate for any resource leaks and/or exception halts. 
         [0019]    Turning to various communication scenarios,  FIG. 2  represents a virtual client  120  registering with a test case management system. To this end, the virtual client manager  104  provides necessary information to the test case management system (e.g., adds the machine and gets back a browser identifier via the virtual test case manage  110 ), adds the browser identifier as a new execution unit and returns it to the client  120 . 
         [0020]      FIG. 3  exemplifies a heartbeat sent from the virtual client  120  to the test case management system. More particularly, the virtual client  120  periodically sends a heartbeat request to announce its aliveness/availability, and to poll for whether there is a test case assigned to it. Note that the browser identifier received during the registration process of  FIG. 2  is carried to the test case management system in the heartbeat request so that the system knows which client is reporting. Moreover, because the identifier is associated with the browser, multiple browsers running on the same machine may be differentiated via each one&#39;s identifier. 
         [0021]    If a job is assigned, it is returned by the system  1   10  (via the manager  104 ) to the virtual client  120 ; the system waits for the job to finish with respect to that client. More particularly, when the test case management system decides to assign a test case as a job to one or more web browsers, it waits for the virtual client  120  to fetch the job; that is, the virtual client  120  will pass over a web browser&#39;s request. Given the fact that there may be many browsers being assigned a job at the same time, each job is buffered in the test case management system until it is fetched to the virtual client  120 . 
         [0022]      FIG. 4  exemplifies result reporting from the virtual client  120  to the test case management system. As shown in  FIG. 4 , when the test case as a job is finished in the virtual client  120 , the virtual client  120  sends a heartbeat with an attached execution result. The test case management system stops waiting (by the signal passed over from the virtual client  120 ) and reports the results. 
         [0023]    In this manner, a web browser may be launched to navigate to a URL, whereby a test case from an existing text case management system is delivered to the browser, and executed, with the execution result sent back to the test case management system independent of the client application. This framework thus enables this process to work on any operating system. Note that the tested machine only needs a very simple client manager application ( 122 A or  122 b in  FIG. 1 ) to handle exception and cleanup issues. 
       EXAMPLE INTERFACES 
       [0000]    
       
         Interface IWebTCM—implemented by VTCM as the interface between a VTCM and a VCM: 
       
     
         [0000]    
       
         
               
               
               
             
               
               
               
               
             
           
               
                   
               
               
                 Method 
                 Parameter/Result 
                 Description 
               
               
                   
               
             
             
               
                   
               
             
          
           
               
                 Response Register( 
                 assetName 
                 VA name to help build 
                 VCM wants to register a 
               
               
                  string assetName, 
                   
                 unique ID 
                 VA to VTCM 
               
               
                  string assetInfo) 
                 browserInfo 
                 VA information 
               
               
                   
                 Return value 
                 Asset ID returned if 
               
               
                   
                   
                 succeeds 
               
               
                 Response Unregister( 
                 assetID 
                 Specifying VA to 
                 VCM wants to unregister 
               
               
                  string assetID) 
                   
                 unregister 
                 a VA from VTCM 
               
               
                   
                 Return value 
                 Asset ID returned if 
               
               
                   
                   
                 succeeds 
               
               
                 Response Poll( 
                 assetID 
                 Specifying VA to query 
                 VCM wants to query 
               
               
                  string assetID, 
                 assetStatus 
                 Specifying current VA 
                 whether a job is assigned 
               
               
                  TcwaAssetStatus 
                   
                 status 
                 to specific VA 
               
               
                  assetStatus) 
                 Return value 
                 Job description text if 
               
               
                   
                   
                 any, otherwise empty 
               
               
                   
                   
                 string is returned 
               
               
                 Response 
                 assetID 
                 Specifying VA to send 
                 VCM wants to send back 
               
               
                 SendResult( 
                   
                 result 
                 the job result for specific 
               
               
                  string assetID, 
                 result 
                 Job result 
                 VA 
               
               
                  string result) 
                 Return value 
                 ACK specifies feedback 
               
               
                   
               
             
          
         
       
       
         Interface IJobCodec—implemented by Job codec: 
       
     
         [0000]    
       
         
               
               
               
             
               
               
               
               
             
           
               
                   
               
               
                 Method 
                 Parameter/Result 
                 Description 
               
               
                   
               
             
             
               
                   
               
             
          
           
               
                 string EncodeJob( 
                 job 
                 Job description from 
                 When IWebTCM.Poll is 
               
               
                  string job) 
                   
                 VTCM 
                 about to return the job 
               
               
                   
                 Return value 
                 Encoded job 
                 description he should call 
               
               
                   
                   
                 description for test 
                 this method to encode so 
               
               
                   
                   
                 harness 
                 that test harness gets a 
               
               
                   
                   
                   
                 TCM-independent format 
               
               
                 string DecodeResult( 
                 result 
                 Result expression from 
                 Before 
               
               
                  string result) 
                   
                 test harness 
                 IWebTCM.SendResult 
               
               
                   
                 Return value 
                 Decoded result 
                 really sends the result he 
               
               
                   
                   
                 expression for VTCM 
                 should call this method to 
               
               
                   
                   
                   
                 decode so that TCM gets 
               
               
                   
                   
                   
                 a test harness 
               
               
                   
                   
                   
                 independent format 
               
               
                 UInt32 JobTimeout( 
                 job 
                 Job description from 
                 Job timeout is required 
               
               
                  string job) 
                   
                 VTCM 
                 for plug-in and JTA to 
               
               
                   
                 Return value 
                 Timeout value in 
                 maintain reliability 
               
               
                   
                   
                 milliseconds, zero if not 
               
               
                   
                   
                 found 
               
               
                 bool SendResult( 
                 TCMReporter 
                 Result sender 
                 Send result in VTCM way 
               
               
                  object TCMReporter, 
                 job 
                 Job description from 
               
               
                  string job, 
                   
                 VTCM 
               
               
                  string result) 
                 result 
                 Result to send for 
               
               
                   
                   
                 VTCM 
               
               
                   
                 Return value 
                 Succeeds or not 
               
               
                 bool SendTimeout( 
                 TCMReporter 
                 Result sender 
                 Send timeout result in 
               
               
                  object TCMReporter, 
                 job 
                 Job description from 
                 VTCM way 
               
               
                  string job) 
                   
                 VTCM 
               
               
                   
                 Return value 
                 Succeeds or not 
               
               
                   
               
             
          
         
       
       
         Interface ITcwaService—implemented by VCM as the interface between a VC and a VCM: 
       
     
         [0000]    
       
         
               
               
               
             
               
               
               
               
             
           
               
                   
               
               
                 Method 
                 Parameter/Result 
                 Description 
               
               
                   
               
             
             
               
                   
               
             
          
           
               
                 Response 
                 browserName 
                 Browser name to 
                 VC wants to register a 
               
               
                 RegisterBrowserInstance( 
                   
                 help build unique ID 
                 browser to VCM 
               
               
                  string browserName, 
                 browserInfo 
                 Browser information 
               
               
                  string browserInfo) 
                 Return value 
                 Browser ID returned 
               
               
                   
                   
                 if succeeds 
               
               
                 Response 
                 browserID 
                 Specifying browser 
                 VC wants to 
               
               
                 UnregisterBrowserInstance( 
                   
                 to unregister 
                 unregister a browser 
               
               
                  string browserID) 
                 Return value 
                 Browser ID returned 
                 from VCM 
               
               
                   
                   
                 if succeeds 
               
               
                 Response Heartbeat( 
                 browserID 
                 Specifying browser 
                 VC wants to query 
               
               
                  string browserID, 
                   
                 to query 
                 whether a job is 
               
               
                  string browserStatus, 
                 browserStatus 
                 Specifying current 
                 assigned to specific 
               
               
                  string browserData) 
                   
                 browser status 
                 browser 
               
               
                   
                 browserData 
                 Data to sent by VC 
               
               
                   
                 Return value 
                 Job description text if 
               
               
                   
                   
                 any, otherwise empty 
               
               
                   
                   
                 string is returned 
               
               
                   
               
             
          
         
       
       
         Interface JTAConfig is implemented by JTA. A JTAConfig.js file is used to provide information to VC: 
         JTAConfig.EntryPage—specify entry HTM file that hosts for test code. 
         JTAConfig.ReloadEntryPageBeforeEachTest—specify whether reloading entry page above, used in JSS scenario. 
         JTACOnfig.ResetDevPageBeforeEachTest—specify whether reset product code page to about:blank, used in JSS scenario. 
         JTAConfig.fn_IsLoaded—specify function to detect whether entry page above is loaded. 
         JTAConfig.fn_Initialize—specify function to initialize for given task, with task description and reference to the frame hosting product code. 
         JTAConfig.fn_Execute—specify function to start executing the task. 
         JTAConfig.fn_IsFinished—specify function to detect whether task is done, with null for not done and not null for task result. 
         JTAConfig.fn_IsLastTest—specify function to check whether this is the end of the whole test case, used in JSS scenario. 
       
     
       Exemplary Operating Environment 
       [0036]      FIG. 5  illustrates an example of a suitable computing and networking environment  500  on which the examples of  FIGS. 1-4  may be implemented. The computing system environment  500  is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment  500  be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment  500 . 
         [0037]    The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to: personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. 
         [0038]    The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in local and/or remote computer storage media including memory storage devices. 
         [0039]    With reference to  FIG. 5 , an exemplary system for implementing various aspects of the invention may include a general purpose computing device in the form of a computer  510 . Components of the computer  510  may include, but are not limited to, a processing unit  520 , a system memory  530 , and a system bus  521  that couples various system components including the system memory to the processing unit  520 . The system bus  521  may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus. 
         [0040]    The computer  510  typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computer  510  and includes both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the computer  510 . Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above may also be included within the scope of computer-readable media. 
         [0041]    The system memory  530  includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)  531  and random access memory (RAM)  532 . A basic input/output system  533  (BIOS), containing the basic routines that help to transfer information between elements within computer  510 , such as during start-up, is typically stored in ROM  531 . RAM  532  typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit  520 . By way of example, and not limitation,  FIG. 5  illustrates operating system  534 , application programs  535 , other program modules  536  and program data  537 . 
         [0042]    The computer  510  may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,  FIG. 5  illustrates a hard disk drive  541  that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive  551  that reads from or writes to a removable, nonvolatile magnetic disk  552 , and an optical disk drive  555  that reads from or writes to a removable, nonvolatile optical disk  556  such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive  541  is typically connected to the system bus  521  through a non-removable memory interface such as interface  540 , and magnetic disk drive  551  and optical disk drive  555  are typically connected to the system bus  521  by a removable memory interface, such as interface  550 . 
         [0043]    The drives and their associated computer storage media, described above and illustrated in  FIG. 5 , provide storage of computer-readable instructions, data structures, program modules and other data for the computer  510 . In  FIG. 5 , for example, hard disk drive  541  is illustrated as storing operating system  544 , application programs  545 , other program modules  546  and program data  547 . Note that these components can either be the same as or different from operating system  534 , application programs  535 , other program modules  536 , and program data  537 . Operating system  544 , application programs  545 , other program modules  546 , and program data  547  are given different numbers herein to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer  510  through input devices such as a tablet, or electronic digitizer,  564 , a microphone  563 , a keyboard  562  and pointing device  561 , commonly referred to as mouse, trackball or touch pad. Other input devices not shown in  FIG. 5  may include a joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit  520  through a user input interface  560  that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor  591  or other type of display device is also connected to the system bus  521  via an interface, such as a video interface  590 . The monitor  591  may also be integrated with a touch-screen panel or the like. Note that the monitor and/or touch screen panel can be physically coupled to a housing in which the computing device  510  is incorporated, such as in a tablet-type personal computer. In addition, computers such as the computing device  510  may also include other peripheral output devices such as speakers  595  and printer  596 , which may be connected through an output peripheral interface  594  or the like. 
         [0044]    The computer  510  may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer  580 . The remote computer  580  may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer  510 , although only a memory storage device  581  has been illustrated in  FIG. 5 . The logical connections depicted in  FIG. 5  include one or more local area networks (LAN)  571  and one or more wide area networks (WAN)  573 , but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet. 
         [0045]    When used in a LAN networking environment, the computer  510  is connected to the LAN  571  through a network interface or adapter  570 . When used in a WAN networking environment, the computer  510  typically includes a modem  572  or other means for establishing communications over the WAN  573 , such as the Internet. The modem  572 , which may be internal or external, may be connected to the system bus  521  via the user input interface  560  or other appropriate mechanism. A wireless networking component  574  such as comprising an interface and antenna may be coupled through a suitable device such as an access point or peer computer to a WAN or LAN. In a networked environment, program modules depicted relative to the computer  510 , or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,  FIG. 5  illustrates remote application programs  585  as residing on memory device  581 . It may be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used. 
         [0046]    An auxiliary subsystem  599  (e.g., for auxiliary display of content) may be connected via the user interface  560  to allow data such as program content, system status and event notifications to be provided to the user, even if the main portions of the computer system are in a low power state. The auxiliary subsystem  599  may be connected to the modem  572  and/or network interface  570  to allow communication between these systems while the main processing unit  520  is in a low power state. 
       CONCLUSION 
       [0047]    While the invention is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the invention to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention.