Mechanism for testing execution of applets with plug-ins and applications

A mechanism for testing execution of applets with plug-ins and applications is described. In one embodiment, a data structure that specifies at least two applets of a plurality of applets is accessed. Based on the data structure that specifies the two applets, a plug-in that executes with an application is caused to attempt to execute each of the two applets. After causing the plug-in to attempt to execute the two applets, results are generated based on the attempt to execute each of the two applets. The mechanism may be implemented to test multiple test suites that each comprise one or more test cases that each include an applet, and reports may be generated for individual test cases, test suites, or a set of test suites. This embodiment of the present invention minimizes the time required for testing and allows for testing the effects of applets that are not visible to a user.

FIELD OF THE INVENTION

The invention relates generally to computer systems, and more particularly to a mechanism for testing execution of applets with plug-ins and applications.

BACKGROUND OF THE INVENTION

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section

Applications that are used with computing systems can be enhanced or extended by using plug-in applications, also commonly referred to simply as “plug-ins.” As used herein, the term “plug-in” refers to an auxiliary application that works with another application to enhance one or more features or capabilities of the other application. For example, plug-ins are often used with Web browser applications to provide support for additional types of Web page content, such as audio, video, etc. A plug-in is typically referenced by a special hypertext markup language (HTML) tag on a Web page, and when the Web browser loads the Web page, the browser retrieves and executes the code for the plug-in. Plug-ins often refer to auxiliary applications in software, but may also refer to auxiliary applications implemented in hardware or implemented through a combination of hardware and software.

Applications can be enhanced or extended by using applet applications, also commonly referred to simply as “applets.” As used herein, the term “applet” refers to an application that has limited capabilities as compared to typical applications. For example, an applet may contain one or a few number of features, require limited memory, be limited in size, be portable among different operating systems, or may include one or more of such characteristics. Applets are commonly distributed with Web pages and are executed using Web browsers. Applets are typically executed by a plug-in that operates in conjunction with the Web browser.

Examples of applets include Java applets that are based on the Java programming language. An example of a plug-in is the Java Plug-in that is provided by Sun Microsystems. The Java Plug-in allows applets included in Web pages to be executed using Sun's Java 2 Runtime Environment, Standard Edition (J2RE) instead of the default Java virtual machine (JVM) that is included in a Web browser. Examples of Web browsers include Netscape Navigator and Microsoft Internet Explorer. The use of the Java Plug-in allows users of Web browsers to use a more current version of the JVM than the default JVM included in the user's particular version of their Web browser because a newer version of the Java Plug-in can be obtained and installed more easily than obtaining and installing a new Web browser version.

The applets and plug-ins that are executed with an application are generally independent of the “operating environment” used to execute the applets and plug-ins. As used herein, the term “operating environment” refers to one or more parameters that can affect the execution of the applets, plug-ins, and applications, including but not limited to, one or more of the following: the particular application, the particular version of the application, the operating system, and the hardware platform that executes the particular application. For example, the operating environment may include version 2.0 of the Netscape Navigator Web browser running on an Intel-based computer that uses the Microsoft Windows XP operating system.

Although applets and plug-ins are designed to be independent of the operating environment, the differences in applications, application versions, operating systems, and hardware platforms may lead to differences when executing applets and plug-ins in different operating environments. As a result, developers of applets and plug-ins test their products in many different operating environments. Similarly, developers of applications test their products to ensure that the applications correctly execute different applets and plug-ins in different operating environments.

One problem with such testing of applets, plug-ins, and applications in different operating environments is the large number of potential tests based on the many different combinations of operating environment parameters, such as the number of different applications, application versions, operating systems, hardware platforms, applets, applet versions, plug-ins, and plug-in versions. As a result of so many different parameters, the number of different operating environments are often at least in the dozens, if not hundreds. Although not all operating environments are always supported, the supported number can still be considerable, and therefore the testing effort is significant.

One approach for testing applications in different operating environments is to execute a number of test cases in each operating environment. For example, to test a set of five applets with a particular version of the Java Plug-in, a user can use a Web page that includes a link to each of the five applets. By clicking on each link to each applet, the user can execute the set of applets, thereby testing whether the applets are successfully executed by the Java Plug-in.

However, there are several disadvantages to this approach. For example, the user manually executes each applet by clicking on each link to each applet used in testing. For tests that involve a large number of applets, the time required for a user to click on each link to each applet can be considerable. Furthermore, the time required for testing rapidly increases as the number of operating environments increases. It is not uncommon for a user to need several days to execute a full testing sequence for a set of operating environments. Finally, the user may make mistakes in running a series of test cases, resulting in some cases not being executed, other cases being executed multiple times, or some cases being improperly executed. This can be a problem when the same test cases are used later to test another set of conditions (e.g., to test another plug-in or application) because the test cases may not be run the second time in the same way as the first time. This can lead to differences in the test results that are due to the testing process, not the things being tested.

Another problem with this approach is that applets often work “behind the scenes” with the application. For example, applets that run with a plug-in for a Web browser may add capabilities to the Web browser that are not visible to the user. As a result, the user may be unable to determine whether or not the applet was successfully executed.

Based on the foregoing, there exists a need for a mechanism for testing execution of applets with plug-ins and applications that minimizes the time required for the testing. There also exists a need for a mechanism for testing execution of applets with plug-ins and applications that allows for a determination to be made whether the applet was successfully executed with the plug-in and application when the effect of executing the applet is not visible to a user.

SUMMARY

To overcome the shortcomings of prior approaches, one embodiment of the present invention provides a mechanism for testing execution of applets with plug-ins and applications. In one embodiment, a data structure that specifies at least two applets of a plurality of applets is accessed. Based on the data structure that specifies the two applets, a plug-in that executes with an application is caused to attempt to execute each of the two applets. After causing the plug-in to attempt to execute the two applets, results are generated based on the attempt to execute each of the two applets. The mechanism may be implemented to test multiple test suites that each comprise one or more test cases that each include an applet, and reports may be generated for individual test cases, test suites, or a set of test suites.

One embodiment of the present invention minimizes the time required for testing by automatically testing a plurality of test cases without requiring manual execution of the test cases by a user and allows for testing the effects of applets that are not visible to a user that otherwise are not observable in the prior approaches. Thus, this embodiment of the present invention provides an improved mechanism for testing execution of applets with plug-ins and applications.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following description, the various functions shall be discussed under topic headings that appear in the following order:I. STRUCTURAL AND FUNCTIONAL OVERVIEWII. USING A TRIGGER TO EXECUTE A TEST SUITE BY INVOKING A RUNNER AND TO INVOKE A REPORTERA. Trigger Input, Variables, and ValuesB. Trigger Functions and Configuration FilesIII. USING A RUNNER TO EXECUTE A TEST SUITE BY INVOKING A SCHEDULER AND TO GENERATE INTERMEDIATE REPORTSIV. USING A SCHEDULER TO EXECUTE TEST CASES IN A TEST SUITE BY INVOKING CLIENTSV. USING A CLIENT TO EXECUTE A TEST CASE AND GENERATE A TRACE FILEVI. USING A REPORTER TO GENERATE A FINAL REPORT BASED ON INTERMEDIATE REPORTSVII. IMPLEMENTATION MECHANISMSVIII. EXTENSIONS AND ALTERNATIVES

I. Structural and Functional Overview

FIG. 1is a block diagram100that depicts a high level overview of one embodiment of a mechanism for testing execution of applets with plug-ins and applications.FIG. 1is described with reference to three test suites (not shown), and each test suite is comprised of three test cases (also not shown), although any number of test suites each having one or more test cases can be used. Specifically,FIG. 1is described in the context of three test suites X, Y, Z, and in which test suite X includes applets A, B, C, as the test cases, and each test case is executed by a Java Plug-in that executes with a Web browser, although other types of plug-ins and applications can be used.

Block diagram100includes a trigger110, a runner120, a scheduler130, clients140A,140B,140C, and a reporter160. Block diagram100also includes trace files142A,142B,142C, intermediate reports150X,150Y,150Z, a finish file170, and a final report180.

Trigger110may be implemented as a shell script that invokes runner120and reporter160based on an input text file (not shown) that lists test suites X, Y, Z. Runner120may be implemented as a Java based application that invokes scheduler130for each of test suites X, Y, Z, and generates intermediate reports150X,150Y,150Z, after executing test suites X, Y, Z, respectively.

Scheduler130may be implemented as an instance of the Web browser application that loads an HTML page that lists the test cases for a test suite in a JavaScript-controlled HTML list box for the test suite. Scheduler130invokes a client140A,140B,140C for each test case in the a suite. Thus, in the current example, scheduler130invokes client104A to execute applet A, invokes client140B to execute applet B, and invokes client140C to execute applet C. In one embodiment, scheduler130is run in one window, while the clients140A,140B,140C are run one after the other in a second window. Thus, client140A is run first in the second window; client140B is then run in the second window; and client140C is then run in the second window.

Trace files142A,142B,142C are generated by clients140A,140B,140C, respectively, as they execute each test case. Trace files142A,142B,142C store the results of executing the test cases. For example, applet A may include a set of one or more criteria to determine whether applet A has been successfully executed by the Java Plug-in with the Web browser application. The one or more criteria may be satisfied if applet A was successfully executed. Trace file142A is used to record the result of the test as a “PASSED” or a “FAILED” depending on whether or not the test criteria were satisfied. Similarly, trace files142B,142C are used to record the results of testing applets B and C, respectively.

Finish file170is generated after execution of the test cases in a test suite has completed. For example, when clients140A,140B,140C have been executed to test applets A, B, C, and trace files142A,142B,142C are generated, finish file170is generated and stored on a persistent storage device, such as a hard disk. In one embodiment, when scheduler130is done with all of the test cases in a test suite, it loads a last HTML page, which invokes a finish applet. It is the finish applet that generates the finish file170.

Intermediate reports150X,150Y,150Z are generated by runner120after each of test suites X, Y, Z are executed, respectively. For example, intermediate report150X can be generated by runner120based on trace files142A,142B,142C following completion of test suite X. Intermediate report150X may indicate the number of test cases with a “PASSED,” the number of test cases with a “FAILED,” and whether or not the test suite “PASSED” or “FAILED.” Intermediate reports150Y,150Z are similarly generated based on the results of test suites Y, Z, respectively.

Final report180is generated by reporter160based on intermediate reports150X,150Y,150Z. For example, final report180may indicate the number of test suites that “PASSED,” the number that “FAILED,” and whether the set of test suites X, Y, Z “PASSED” or “FAILED.”

Note that whileFIG. 1is described with reference to several enumerated mechanisms, such as trigger110, runner120, scheduler130, clients140A,140B,140C, and reporter160, fewer or more mechanisms may be included. For example, the functions performed by a particular mechanism inFIG. 1may be divided among one or more mechanisms, including among the other mechanisms depicted inFIG. 1and additional mechanisms not depicted. Furthermore, the functions of two or more mechanisms ofFIG. 1may be performed by the same mechanism. In general, any mechanism, including but not limited to those ofFIG. 1, may perform one or more of the functions described herein.

FIG. 2is a flow diagram200that depicts a high level overview of one embodiment of a mechanism for testing execution of applets with plug-ins and applications. WhileFIG. 2is described with reference toFIG. 1, the embodiment ofFIG. 2is not limited to the structural elements ofFIG. 1. Furthermore, whileFIG. 2depicts a particular set of blocks and functions in a particular order, fewer or more blocks and different orders may be used in a particular implementation.

In block210, trigger110retrieves an input file for a test suite. For example, trigger110may be executed in response to command line input from a user, and the input may specify the name and/or location of the input file for the test suite. The input file specifies one or more test suites, such as test suites X, Y, Z. The command line input may include one or more additional data items that specify the plug-in, such as the Java Plug-in, the application, such as the Netscape Navigator or Microsoft Internet Explorer Web browsers, and the versions of the plug-in and application.

In block214, trigger110retrieves a configuration file and identifies a template file for the first test suite specified in the input file for the test suite that is retrieved in block210. For example, configuration files may be provided for each test suite X, Y, Z. Each configuration file may include a list of template files, with each template file corresponding to a particular set of input values, such as the Web browser application type and output desired. Based on the input, trigger110identifies the proper template file from the set of template files that are specified in the configuration file.

In block218, trigger110invokes runner120. For example, for the first test suite X, trigger110may invoke runner120as an instance of the Web browser application specified by the user and then pass the location of the template file that is identified in block214to runner120.

In block220, runner120retrieves the template file identified by trigger110for the test suite. For example, for the first test suite X, runner120receives the proper template file for test suite X that was identified in block214. Each template file may be a JavaScript HTML Web page that lists the test cases in a JavaScript-controlled HTML list box, such as applets A, B, C, for test suite X.

In block224, runner120invokes scheduler130for the test suite. For example, runner120may invoke scheduler130as an instance of the Web browser application specified by the user via the command line input, and load the template file for test suite X into the instance of the Web browser.

In block230, scheduler130invokes a client for each test case in the test suite. For example, for test suite X, the instance of the Web browser that was invoked in block224invokes clients142A,142B,142C for applets A, B, C, in test suite X. In one embodiment, clients142A,142B,142C are invoked one after the other in one window of the Web browser application.

In block244, clients140A,140B,140C write trace files142A,142B,142C, respectively. For example, based on the set of testing criteria included in applet A, client140A writes trace file142A that indicates whether the test with applet A was a “PASSED” or a “FAILED.”

In block234, scheduler130causes finish file170to be generated when the test cases are complete. For example, after clients140A,140B,140C have completed running applets A, B, C, and written trace files142A,142B,142C, scheduler130causes finish file170to be created. In one embodiment, when scheduler130is done with all of the test cases in a test suite, it loads a last HTML page (Closewin.html), which invokes a finish applet. It is the finish applet that generates the finish file170.

In block250, trigger110periodically polls for finish file170. For example, after invoking runner120in block218, trigger110may begin to periodically check for the presence of finish file170.

In block258, trigger110kills scheduler130when finish file170is found. For example, trigger110may terminate the instance of the Web browser application that is executing the template file for test suite X, thereby killing any of the instances of the Web browser that are executing as clients140A,140B,140C that may be still executing.

In block254, runner120generates an intermediate report based on the trace files for the test suite. For example, for test suite X, runner120generates intermediate report150X based on trace files142A,142B,142C. After generating the intermediate report, runner120terminates execution.

In block260, a determination is made whether the last test suite has been executed. If so, the approach continues to block270that is described below. If not, the approach returns to block214, and the trigger110retrieves the configuration file for the next test suite and identifies the proper template file in block214before invoking runner120for the next test suite in block218. The determination of whether the last test suite has been executed may be made in one or more ways, such as by runner120before runner120terminates execution following the generation of intermediate report150X or by trigger110after runner120terminates execution.

In block290, reporter160generates a final report based on the intermediate reports. For example, reporter160may generate final report180based on intermediate reports150X,150Y,150Z.

Note that a particular test may fail due to a variety of reasons, including but not limited to, problems with the test case (e.g., the applet), the plug-in, or the application. As a result, the testing approach described herein can be used to test new versions of applets, plug-ins, or applications. Furthermore, a test case may fail due to a combination of parameters, such as the combination of a particular applet and application version, a plug-in and an application type, etc. As a result, the testing approach described herein can be used to test different combinations of applets, plug-ins, and applications. Furthermore, test results can be affected by other parameters, such as the operating system and hardware platform. As a result, the testing approach described herein can be used to test new operating systems, operating system versions, hardware platforms, or particular hardware changes or variations. Thus, in general the testing approach described herein can be used to test any parameter, application, or anything else that can effect the execution of a test.

II. Using a Trigger to Execute a Test Suite by Invoking a Runner and to Invoke a Reporter

According to one embodiment, an application is used to initiate testing of the execution of applets with plug-ins and applications. For example, the application may be referred to as a “trigger,” and the trigger is initiated by a user to begin the testing process. The user may initiate the trigger using input, such as that provided via a command line or a graphical user interface. The trigger is typically implemented as a stand alone application, such as a shell script or as a Java application. As a specific example, the trigger may be implemented as a Korn shell script on a computer running the Unix Operating System.

The trigger remains running until the end of the testing process, although other applications as described herein may be initiated directly or indirectly by the trigger. For example, the trigger may invoke a runner application as described herein for each test suite to be used and a reporter application to generate a final report.

The input from the user may initiate the trigger, with other required inputs retrieved from one or more other data structures, such as a file stored on a fixed disk or other type of persistent storage device or from a transient storage device, such as memory. The input information may identify a number of parameters, including but not limited to, one or more test cases, a plug-in, and an application. Optionally, the input information may identify other items and options, such as the versions of the plug-in and application, working directories to store trace files, report files, and other data structures generated or used during the testing process, and testing options, such as checking reliability (e.g., the ability of the tests to be repeatedly executed, which is discussed more fully below).

Below is an example of a text file named “test.txt” that contains input variables and input values for a trigger. The particular input variables, input values, and organization depicted in the “test.txt” file are for illustration only, and fewer or more input variables and different input variables and values may be used besides those described herein.

The variable “WORK_DIR” specifies a working directory for the testing process. The working directory may contain other files, such as configuration files and template files, which are discussed below. In the example above, the working directory is specified to be “Y:/TESTDIR”.

The variable “JAVA_HOME” specifies the location of a Java version (e.g., a particular Java Virtual Machine, or JVM) that is suitable for use in executing a runner application. For example, the example above has an input value of “C:/JDK1.3” that points to version 1.3 of the Java Development Kit (JDK). The Java version specified might be a version that is known to be stable based on past use and testing, as compared to another Java version or Java Plug-in that is to be tested. As a result, when the goal of the testing is to test a different Java Plug-in, the Java version specified by “JAVA_HOME” is typically different than that specified by “TEST_JRE_VERSION”, which is discussed below.

The variable “RES_DIR” specifies the location of a result directory for holding files such as trace files, intermediate reports, and final reports. In the example above, the result directory is specified to be “Y:/TESTDIR/REPORT/TEST”.

The variable “BROWSER_TYPE” specifies the browser type to be used for testing. In the example above, the input value is “NS” to denote the Netscape Navigator browser application. Other input values may be used, such as “IE” to denote the Microsoft Internet Explorer application.

The variable “BROWSER_VERSION” specifies the version of the specified browser to be used for testing. In the example above, the input value is “4.78” to denote version 4.78 of Netscape Navigator.

The variable “TEST_JRE_VERSION” specifies the version of the plug-in to be used for testing. In the example above, the input value is “1.4” which specifies version 1.4. A particular computer may have more than one version of a plug-in installed, and this input variable allows the user to specify a particular version from the versions that are available. In addition, even if the computer has not had the specified plug-in version installed, the trigger can be configured to install the specified version before running any test cases or test suites. Note that in this example, the plug-in is not specified because the input assumes that the plug-in is the Java Plug-in. However, in other implementations, a plug-in type input variable may be used to specify the type of plug-in. Note also that a plug-in version need not be specified, nor even a plug-in version variable used, and as a result, the most recent installed plug-in version or a default version would be used.

The variable “RELIABILITY” specifies the reliability option. The testing of reliability is performed by repeatedly executing each test case until one or more criteria are satisfied. Reliability testing is different than functional testing, which determines whether a test can be executed and therefore usually executes each test case one time. Reliability testing is also different from performance testing, which determines performance measures such as speed, efficiency, resources used, etc. In the example above, the input value is “YES” to indicate that reliability testing is to be performed. The input value could also be “NO” to indicate that no reliability testing is to be performed. Note also that this input value, or any other, can be blank to specify that a default value is to be used, such as to not perform reliability testing.

The variable “RELIABLETIME=100” specifies the time for the reliability testing. In the example above, the input value is “100” that specifies that reliability testing is to be performed for each test case for one hundred seconds.

The variable “TEST_CYCLE” specifies the output option. In the example above, the input value is “QL” that specifies that a “quick look” is desired, in contrast to a “full look (FL)”. In one embodiment, a quick look set of test results is a subset of the full look set of test results. Quick look tests test the breadth-wise functionality of a plug-in, whereas full look tests test the depth-wise functionality of a plug-in.

Note that variations on the input discussed above may be implemented, including but not limited to, testing multiple browser types, multiple browser versions, multiple plug-ins and plug-in versions. For example, the trigger can accept command line input options that modify the input subsequently retrieved from a stored file, such as by specifying that particular test cases be executed instead of all test cases in a test suite as would normally be done.

According to one embodiment, the trigger obtains input that identifies one or more test suites to be used for testing. For example, the working directory may include a test suite input file, called “properties.txt”, that lists a set of test suites to be used for testing. As other examples, another input variable may be used to specify the test suite input file, or the test suite input file may be specified as a command line argument when the trigger is executed. Each test suite may be identified using a feature name that is representative of the feature exercised by the test cases in the test suite.

According to yet another embodiment, a set of inputs can be provided and one or more test cases can be executed based on one or more combinations of the inputs. For example, the input can include a list of browser types, browser versions for one or more of the browser types, one or more operating systems, one or more hardware platforms, one or more plug-in types, and one or more versions of each plug-in type. The trigger can then determine how many different combinations of the inputs are possible, such as how many different combination of inputs can be created based on the browser types, browser versions, application types, application versions, operating systems, hardware platforms, plug-in types, and plug-in versions. The test case or cases, or test suite or test suites, can then be executed for one or more of the different combinations of such inputs, and the reporter can summarize the results based on the different combinations of inputs. Each combination of such inputs may be described as defining an “operating environment” or “testing environment.”

B. Trigger Functions and Configuration Files

The trigger performs a number of functions, including but not limited to those described herein. For example, the trigger can be configured to make a specified number of attempts, such as three, to load and launch the plug-in version and application version specified in the input. If the plug-in and application are successfully loaded and launched, the testing process continues. Otherwise, the testing process ends, and a suitable error message or report can be generated.

The main function of the trigger is to invoke the runner and monitor the execution of the test cases. For example, the trigger can invoke the runner at the start of testing each test suite. For a particular test suite, the trigger retrieves a corresponding configuration file. Based on the configuration file, the trigger identifies the appropriate template file to be used for the test suite and passes the template file to the runner. Alternatively, the trigger passes the location or identify of the template file to the runner, and the runner retrieves the template file.

Below is an example of a configuration file called “config.txt” for a feature called “FEATURE X”. The particular input variables, input values, and organization depicted in the “config.txt” file are for illustration only, and fewer or more items and different values may be used besides those described herein.

The first item in each line of the configuration file “config.txt” above is a numerical value that specifies the number of test cases in the test suite. The second item is a variable that specifies the output option. In this example, both “quick look (QL)” and “full look (FL)” output options are supported. The third item is a variable that specifies the browser type. In this example, the supported browser types are Netscape Navigator (NS) and Microsoft Internet Explorer (IE), although fewer or more browser types can be used in other implementations. The fourth item is an arbitrary text description of the feature, which in this example is “FEATURE X”. Finally, the fifth and last item is the path to an HTML file that contains the test cases that correspond to the particular combination of quick look/full look and browser type values. For example, the path for “http://sun.com/x2.html” points to an HTML template file with three test cases for “FEATURE X” for use with the Internet Explorer Web browser application and for which a “quick look” report is to be generated.

Once the trigger has retrieved the configuration file, the path to the HTML file for the test cases to be used is identified based on the browser type specified in the “test.txt” input file and the output option. For example, in the previous input example, the input value for the browser type variable is “NS” and the input value for the output variable is “QL”, therefore the “x1.html” template file will be used for the test suite for testing “FEATURE X”. The trigger may either retrieve the appropriate template file and pass the template file to the runner, or simply pass the path for the template file to the runner so that the runner can retrieve the template file.

After the trigger invokes the runner, the trigger begins polling for the presence of a “finish file” or some other indication that the testing for the test suite is complete. For example, the trigger can periodically check for the presence of a “finish.txt” file that is written by the scheduler upon completion of all the test cases in the test suite. In addition, at the start of the polling for the finish file, the trigger may begin a clock or timer. If the finish file is not found by the expiration of the clock or timer, the trigger proceeds on as if the finish file had been found. The clock or time therefore handles the situation in which for some reason the finish file is not created, such as when an error occurs with the runner or scheduler or one of the clients that prevents the creation of one or more trace files or the finish file.

When the finish file is found, the clock or time expires, or some other criterion is satisfied, the trigger terminates, or “kills,” the scheduler, thereby allowing control to return to the runner to allow the runner to generate an intermediate report for the test suite based on the trace files written by the clients. If any clients are still executing or have failed to terminate normally, such clients will be killed when the scheduler is killed since the clients are additional instances of the same application used to execute the scheduler.

After the runner completes execution and terminates, the trigger invokes the reporter application that generates the final report. Upon killing the scheduler, the trigger can begin a new clock or timer in the event an error occurs that prevents the runner from generating the intermediate report and terminating normally.

III. Using a Runner to Execute a Test Suite by Invoking a  Scheduler and to Generate Intermediate Reports

According to one embodiment, an application is used to initiate testing of one or more applets with plug-ins and applications. For example, the application may be referred to as a “runner,” and the runner is invoked by the application that initiates the testing process, such as a trigger. The runner is typically implemented as a stand along application, such as a shell script or as a Java application. As a specific example, the runner may be implemented as a Java application executed using a JVM that is known from previous use and testing to be stable. By using such a known stable version instead of an untested JVM for the runner, errors that might occur due to the runner may be minimized.

The trigger can pass the identity or location of the template file for the particular test suite to be executed to the runner. Alternatively, the trigger can retrieve the template file and pass the template file to the runner. As yet another alternative, the trigger can pass the identity of the test suite to the runner, and then the runner can retrieve the appropriate configuration file, identify the proper template file from the configuration file, and then retrieve the template file.

After the runner has retrieved the proper template file for the test suite, the runner invokes the scheduler based on the template file.

After the scheduler has executed each test case by invoking the clients, or at least attempted to execute each test case, and written the finish file, been timed out by the trigger, or satisfied some other criterion, the trigger terminates the scheduler and control returns to the runner.

The runner then generates results for the test suite, such as in an intermediate report based on the trace files that are generated by the clients. For example, the intermediate report may list the result of each test case in the test suite, indicating whether each applet was successfully loaded or not, such as by listing each test case by name or number with either a “PASSED” or a “FAILED” indication. The result for each test case is determined by the runner based on the trace file for each test case, such as by retrieving each trace file and identifying the status line that indicates the result of the test case. If a test case failed such that a trace file was not generated, the runner may indicate the result for that test case as “FAILED.”

According to one embodiment, the results for the test suite are modified according to an exclude list. For example, a file called “exclude.txt” may be used to list known failures, such as that test case five of test suite X is known to fail for Netscape Navigator version 4.78 with Java Plug-in version 1.4. The runner can check the failed test cases against the exclude list to identify whether any failed test cases are a known failure. For any test cases on the exclude list, the runner can change the result for the test case from “FAILED” to “PASSED” or some other indication, such as “EXCLUDED.” As a result, known failures are accounted for in the results for the test suite, and the intermediate report only reflects test cases that “FAILED” due to unknown failures.

IV. Using a Scheduler to Execute Test Cases in a Test Suite by Invoking Clients

According to one embodiment, an application is used to schedule the testing of one or more applets. For example, the application may be referred to as a “scheduler,” and the scheduler is invoked by the application that initiates testing of one or more applets, such as the runner. The scheduler is typically implemented as an instance of the application that is part of the testing, such as an instance of a Web browser application. The one or more test cases are identified based on a template file, such as an HTML file that lists the applets for a particular test suite. As a specific example, the scheduler may be implemented using an instance of the application type and version specified in the input, such as a particular version of a specified Web browser application, and the template file may be a JavaScript-based HTML Web page that refers to one or more applets to be tested with the particular browser.

Below is an example of a portion of a template file called “x1.html” that was previously identified as discussed above. The “x1.html” file lists each of the five applets, or test cases, for the test suite for “FEATURE X” in a JavaScript list box.

Each of the “Testn.html” references identifies a particular applet for that test case. The scheduler executes the test cases by instantiating a browser window, and then executing each test case one after the other in that window. For the example shown above, the scheduler instantiates a window, and then executes each of the applets identified in the JavaScript list box in that window.

In addition to listing the test cases, the template file includes a reference to a “Closewin.html” file that is used to generate the indication that the test suite has been executed, such as by creating a “finish file.”

According to one embodiment, the scheduler includes a timer that spaces out the execution of the test cases. For example, the test cases may be invoked in clients every ten, twenty, or thirty seconds. Spacing out the execution of the test cases may reduce the contention for shared resources that can occur if the test cases were not spaced out over time. If necessary, the scheduler can use an interval that is long enough to allow each test case to complete executing before invoking the next test case.

V. Using a Client to Execute a Test Case and Generate a Trace File

According to one embodiment, an instance of an application is used to test execution of applets with plug-ins and applications. For example, the instance of the application may be referred to as a “client,” and the client is invoked by the application, such as the scheduler, to execute the test case. The client is typically implemented as a window of another application. As a specific example, the client may be a browser window that executes an applet, and the browser window is invoked by a browser, such as when the scheduler is implemented as an instance of a Web browser application that is specified by the input, and the scheduler uses the appropriate template file to execute the test cases for a test suite.

For example, for the test suite for “FEATURE X” discussed above, in which the first test case is included with “Test1.html”, the second test case with “Test2.html”, etc., the scheduler invokes a browser window, and then tests each test case in that browser window. For each test case, the scheduler uses the appropriate HTML file. Each HTML file includes the applet for the particular test case, which is loaded by the plug-in specified by the input.

According to one embodiment, an applet includes one or more criteria for determining whether the test is successful or not. For example, the applet can include one or more instructions that are only executed if the applet is successfully loaded by the plug-in with the application and is functioning properly. When executed, the one or more instructions cause a trace file to be written, which includes a status line indicated the result of the test case (e.g., whether the test case “PASSED” or “FAILED”).

According to another embodiment, a portion of the plug-in monitors the performance of the applet for the test case and generates a result. For example, the Java Plug-in includes a Java console that can monitor the execution of the applet by the plug-in. The Java console can generate a result, such as a trace file discussed above, based on the results of the test case. The result may be based on one or more criteria that are included in the test case. An applet that includes the one or more criteria can generate an indication based on the execution of the test, and then pass the indication to the Java console that then generates the trace file.

VI. Using a Reporter to Generate a Final Report Based on Intermediate Reports

According to one embodiment, an application is used to generate a report of the results from execution of a group of applets with plug-ins and applications. For example, the application may be referred to as a “reporter,” and the reporter is initiated by another application, such as the trigger. The reporter is typically implemented as a stand-alone application, such as a shell script or as a Java application. The report generated by the reporter may be referred to as a “final report” to distinguish from the “intermediate reports” that may be generated by another application, such as the runner that generates intermediate reports for each test suite, as described above.

For example, the reporter can generate a final report based on the intermediate report for each test suite. While the intermediate reports include the results for the test cases within the test suite and the overall result for the test suite, the reporter can be configured to include results for the test suites without detailed results for the test cases. The final report can therefore be a summary by test suite of the results, and an overall result for the testing performed, with summary information indicating the number of test suites that passed or failed, the result for each test suite, and perhaps summary information for the test cases such as the overall percentage of test cases that passed or failed. The final report can also indicate how many or which test cases were excluded based on the exclude list.

In general, the reporter can be configured to present any or all of the information from the intermediate reports for the test suites and trace files for the test cases or any information that can be generated based on such information. In addition, the final report can include information about the applets used in testing, such as a list of the features tested, and details about the plug-in and/or application used for testing, such as the plug-in type and version and the application type and version. The final report can also include file information, such as the location of the trace files, test cases, and intermediate reports.

VII. Implementation Mechanisms

The approach for testing execution of applets with plug-ins and applications described herein may be implemented in a variety of ways and the invention is not limited to any particular implementation. The approach may be integrated into a computer system or a computing device, or may be implemented as a stand-alone mechanism. Furthermore, the approach may be implemented in computer software, hardware, or a combination thereof.

The term “computer-readable medium” as used herein refers to any medium that participates in providing instructions to processor304for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device310. Volatile media includes dynamic memory, such as main memory306. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus302. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

The received code may be executed by processor304as it is received, and/or stored in storage device310, or other non-volatile storage for later execution. In this manner, computer system300may obtain application code in the form of a carrier wave.

VIII. Extensions and Alternatives

In the foregoing specification, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention. For example, although examples have illustrated the use of Java Plug-ins with Web browser applications, the Java Plug-ins and Web browser applications are used for explanation purposes only as embodiments of the invention are not limited to any particular type of plug-in or application. Thus, the specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The invention includes other contexts and applications in which the mechanisms and processes described herein are available to other mechanisms, methods, programs, and processes.

In addition, in this disclosure, certain process steps are set forth in a particular order, and alphabetic and alphanumeric labels are used to identify certain steps. Unless specifically stated in the disclosure, embodiments of the invention are not limited to any particular order of carrying out such steps. In particular, the labels are used merely for convenient identification of steps, and are not intended to imply, specify or require a particular order of carrying out such steps.