Abstract:
On a test apparatus for a telecommunication network a plurality of quality rules are defined as a function of the results of counts of events for a plurality of test parameters. The events relating to the plurality of test parameters during the performance of a test are counted. A plurality of statistics are created from the results of the counts. The statistics and quality rules which belong together are compared. A display is generated on a display unit in accordance with the results of the comparisons showing success or failure for each test run. Details of errors that caused any failure may be displayed using a drill-down technique so that test runs may be run rapidly without requiring expert knowledge of the test apparatus.

Description:
BACKGROUND OF THE INVENTION 
   The present invention relates to telecommunications network testing, and more particularly to displaying test results (success or failure) upon completion of load and systems tests. 
   Previously all relevant information generated during each telecommunications network test run is stored in a log file in the form of a protocol. From this information the success or failure of the test is extracted. The information has to be sifted through manually, which is very time-consuming—often lasting several hours—particularly when problems occur during the test run. It is presently not possible for a user to determine a cause for the problems or errors without having a very accurate knowledge of the test apparatus being used. As a result determining success or failure of each test run is both time-consuming and knowledge-intensive. 
   What is desired is a means and method of providing results about each test run on a telecommunications network that is faster and less knowledge-intensive than previously. 
   BRIEF SUMMARY OF THE INVENTION 
   Accordingly the present invention provides a means and method for displaying test results for each test run of a telecommunication network that is fast and does not require expert knowledge about any test apparatus. Protocols are no longer consulted for arriving at a determination on the success or failure of a test run, but instead there is a pre-analysis involving the use of statistics where the statistics relate to results of test-relevant events. These may then be related to freely definable quality rules in order to determine the success or failure of a test run. It is then possible to base thereon extended analyses, such as determining causes for errors. By generating appropriate statistics and defining appropriate quality rules, a considerable acceleration of the analysis of the test results is achieved so that there may be performed in a given time a multiple of the number of tests performed in the prior art, resulting in an increase in the quality of the telecommunication network. Further specified tests may be performed in a fraction of the time as required previously, so that considerable cost reductions are realized. 
   The objects, advantages and other novel features are apparent from the following detailed description when read in conjunction with the appended claims and attached drawing. 

   
     BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING 
       FIG. 1  is a plan view of a display for a test results analysis summary according to one embodiment of the present invention. 
       FIG. 2  is a plan view of a display for a test results analysis problems according to the one embodiment of the present invention. 
       FIG. 3  is a plan view of a display for a test results analysis details according to the one embodiment of the present invention. 
       FIG. 4  is a plan view of a display for a test analysis summary according to a second embodiment of the present invention. 
       FIG. 5  is a plan view of a display for a test analysis details according to the second embodiment of the present invention. 
       FIG. 6  is a plan view of a display for an expanded test analysis details according to the second embodiment of the present invention. 
       FIG. 7  is a plan view of a display for a test analysis failures according to the second embodiment of the present invention. 
       FIG. 8  is a plan view of a display define statistic limits according to the second embodiment of the present invention. 
   

   DETAILED DESCRIPTION OF THE INVENTION 
   Referring now to  FIG. 1  a user interface display  10 , such as a liquid crystal display (LCD) display, for a telecommunications network test apparatus is shown. A first window  12  provides information to a user about aspects of a test run, which aspects are displayed in more detail in a second window  14 . As shown the user may select from Overview, Properties and Test Results by “clicking” on the appropriate term. For the present illustration Test Results is selected, and further aspects are shown in an area  16  of the first window  12  that relate to Test Results. The user may select from Summary, Problems and Details to determine the details that are displayed in the second window  14 . The details corresponding to the term Summary are shown in the second window  14  in this illustration. At a top area  18  of the second window  14  are shown the various test cases that have been run by Name, Result, time details (Started and Finished) and number of Errors. According to quality rules that are defined by the user, the relevant test case is marked under Result as having Succeeded or Failed. By selecting, such as by “double-clicking”, one of the test cases further details may be displayed, as described below. 
     FIG. 2  shows a user interface  25  for the errors (Problems) aspect of Test Results as selected by the user in area  16  of the first window  12 . The rest of the information in the first window  12  corresponds to that of  FIG. 1 , and has been omitted for clarity&#39;s sake. In a Problems window  26  the errors are displayed in a Problems area  28  by name, type and frequency. Via a scroll bar  30 , similar to the scroll bar  18  of  FIG. 1 , the user may scroll through the total number of errors. Via a Filter area  32  the user may define an error filter, so that only errors that meet the filter conditions are shown. For this illustration it is possible to include in the filter information the name, type, frequency and link (operation) of the information. If in the Problems area  28  a specific error is selected by clicking on it, details of the error are shown in a Problem Details area  34 , such as when the error occurred, within the framework of which statistic and in which context. The term “statistic” is a plurality of counts of events of certain test parameters performed during the execution of the test. From the results of the counts there is set up a plurality of statistics. The user may define for a plurality of test parameters quality rules as a function of results of counts of events. By comparing statistics and related quality rules, it is inferred whether a corresponding test parameter has completed the test successfully or unsuccessfully. What is of importance is that an error is not only defined by the non-fulfillment of a quality rule relating to a plurality of events, but possibly by the non-fulfillment of one single event. “Context” means that certain statistics and/or events may be related to each other. The relatedness may be that they were recorded at the same time as the test parameter event triggering the error; that they were recorded within a specifiable time interval before and/or after the time of the recording of the parameter event triggering the error and relate to the same test parameter or a specifiable selection of test parameters, particularly all test parameters; and/or that they are a specifiable selection of test parameters, particularly all test parameters, that belong to the same call or the same network element and/or to the same subscriber as the event of the test parameter triggering the error. 
   By double-clicking on an event or a statistic in a statistic area  36  a user interface is displayed on which there may be shown a graphical view or a tabular view of the corresponding event or of the corresponding statistic. By double-clicking on an entry in a context area  38  there are displayed related events and/or statistics in another user interface. 
     FIG. 3  shows a user interface  40  that occurs if the Details entry in the first window  12  of  FIG. 1  in the Test Result area  16  is activated by clicking on it. Details on the test results are displayed in a Details window  42  of the user interface  40 . In an area  44  the details of the present test are grouped and sorted according to a tree structure so that the user, by clicking on the corresponding entries, may branch to more detailed information or, by clicking on the minus sign, may go to higher levels. Using a scroll bar  46  enables display of another section in area  44 . By clicking on an entry in area  44 , corresponding details are shown in a details area  48 , such as the name of the test case, the result, the number of the errors that have occurred and a table containing information on the events and/or statistics that belong to the test case. In a graphic area  50  the course of the events of a statistic over time is shown. 
   While  FIGS. 1 to 3  relate to a first embodiment of the test apparatus,  FIGS. 4 to 8  relate to a second embodiment with  FIG. 4  corresponding to  FIG. 1 ,  FIGS. 5 and 6  corresponding to  FIG. 2 , and  FIG. 7  corresponding to  FIG. 3 . 
     FIG. 4  shows a user interface  52 , in which Analysis has been selected in an area  54 . In an area  56  there is indicated in an overview that the present test has only one test case # 1 . In an area  58  there are provided capabilities for exporting or deleting analysis details. Finally in an area  60  there are shown three entries for selection by the user. In the present case the Summary entry has been selected, as indicated by the color (shaded) background. The other two entries lead to the user interfaces of  FIGS. 5 and 6  (Details) and to the user interface of  FIG. 7  (Failures=errors). In a window  61  of user interface  52  there is shown a summary of the analysis results. In a top area  62 , there are shown the three elements of the highest level of a tree structure relating to this test case, with two elements (Network Interface and Node-B) shown as OK (preferably using a green color), i.e., these elements have passed the test, while the System Component element is shown as Failed (preferably using a red color) to draw the attention of the user to the fact that an error has occurred for this element. The second column of the table in area  62  gives the number of errors, and the third column indicates whether the test of the corresponding element is considered to have passed or failed. In an area  64  there is shown the main errors and their frequencies. It is possible for the user to enter comments. 
     FIG. 5  shows a user interface  66  resulting when the term “Details” is selected in area  60  of user interface  52  in  FIG. 4 . As shown by the entry in line  68 , the elements of the tree structure that lie below the “System Component” element are displayed in area  70 . By clicking on arrow  72  in line  68 , the other elements from area  62  of  FIG. 4  may be selected. At this level, too, there are marked, preferably in color, which element(s) errors have resulted. In the present case, the “ALCAP” entry is marked by a symbol as being afflicted with an error, while the other four entries indicate that they have been completed without an error. The table in area  70  gives the number of errors and provides, where quality rules on a specific test parameter are definable and have been defined, a display of the associated lower limit and/or the associated upper limit. Selecting icon  76  allows quality rules to be defined, as described below with respect to  FIG. 8 . 
     FIG. 6  shows further details of the tree structure in area  78  which belongs to the “ALCAP” entry that is marked as being afflicted with an error. In the present case the entry “Number of received ERQ messages” is marked by a symbol as being afflicted with an error. As is seen from the entry in the Upper Limit column, the upper limit is  40  events, while in this case  42  events have occurred. In area  80  details on the associated error from area  78  are shown and may be exported by clicking on icon  82 . 
     FIG. 7  shows a user interface  84  that results when in area  60  of the user interface  52  of  FIG. 4  the Failures (error) area is selected. In area  86  of the user interface  84  there is given a table in which the occurring errors are given by their name and the associated count. In area  88  there are shown details on the errors selected in area  86 , besides the name and the count in area  90 , some information on the occurrence, particularly the time and value, and in area  92  information on related events and/or statistics, again with information on the time of the occurrence and the associated value. By clicking on entries in area  92  there may be displayed on the display further details on the entries that are seen there. 
   Finally,  FIG. 8  shows a user interface  94  that occurs after clicking on the icon  76  of  FIG. 5 . A field  96  shows the selected statistic or the selected event, such as the failed attempt on a Gateway GPRS (General Packet Radio Service) Support Node (GGSN) to establish a link (Activation rejected by GGSN). The lower limit may be indicated in an area  98  upon the activation of an icon  100 , and the upper limit in an area  102  after clicking on an icon  104 . Following completion of the entry, it may then be recorded in the system by clicking on icon  106 . 
   Thus the test apparatus provides during a test statistics relating to events for a plurality of test parameters. The user may define quality rules, particularly thresholds, by which the associated statistics are defined as being afflicted with an error when the threshold is exceeded. Just like the statistics, individual events, too, may be defined as errors if they are not executed correctly. In a preferred embodiment more than 200 statistics may be recorded, which corresponds to a data volume of 80 gigabytes per day. Statistics may be grouped in a tree structure that may be predefined or user-defined. One element of the tree structure contains its result, i.e., its assessment as to whether it was completed successfully or unsuccessfully, on the basis of the statistics and events contained in it. 
   On the test apparatus a so-called “drill-down approach” is implemented by which the user preferably starts with the Summary user interface and tracks the errors within the tree structure downwards until the statistic or the event that has caused the error is found. There is provided the ability to display details on the statistic or the event that caused the error, particularly statistics and events which are related, i.e., stand in context with the statistic/event causing the error. 
   In the Summary user interface there is shown particular test procedure information, such as the time of the start and the time of the end of the test, the user who started the test, the test environment, a summary of the results with an assessment as to whether the relevant test case was successful or unsuccessful, an indication of the errors of the relevant test case, and finally an overview of the main errors given by their name, type, threshold exceeded or one single event executed incorrectly, and the frequency. 
   On the error user interface there is displayed a list of the errors, and there is provided the ability to filter for specific errors as well as to list further information on the errors, such as name, type, frequency. In an area of the error user interface there are displayed details on the selected error apart from the name, the type and the frequency, such as information on its occurrence (when, where—in which statistic) and information on the context (giving of related values) which might have something to do with the error. 
   In another user interface with details of the analysis, there may be selected via the tree structure the test elements on which further details are to be displayed. Moreover, for each element of the tree structure there is given the associated result—whether it was completed successfully or unsuccessfully. In another area of the user interface there are given details on the selected elements of the tree structure, such as the name, the result, the number of errors, possibly a list of the errors, the statistics and results, which were completed unsuccessfully, and which are subordinate to the selected element of the tree structure. Moreover, selected statistics may be shown graphically or in the form of a table.