Abstract:
A system searches for, and verifies, according to certain criteria, a database of records, typically call records generated during the testing of a telecommunications network after software or hardware updates have been applied to the system. Multiple instances of collecting and decoding processes embodied in stored programs running in a computer system act upon blocks of incoming data records to store both a raw image of the received data and a pre-parsed version of the data suitable for database searching and retrieval. Three-step partitioned processing comprises a set of collector processes for collecting data records, a set of decoder processes for decoding and parsing such records, and a set of loader processes for loading records into a database. A client can request certain call records or request verification of certain records. A rules mechanism embodied in stored templates operates to tie client requests to asynchronously received data.

Description:
BACKGROUND 
     This invention relates to telecommunications systems, and, in particular, a system and method for searching and verifying a database of records, typically call records, generated during the testing of a telecommunications network after software or hardware updates, or both, have been applied to the telecommunications system. 
     Typically, new services to be implemented in a telecommunications network are tested in a mock network testbed before implementation in a production network. Untested software or hardware updates to a functioning production telecommunications system could cause disastrous results if those updates contain software or hardware bugs. The network testbed is designed to emulate the production telecommunications network as closely as possible. During testing, the many heterogeneous devices in the network create call records which simulate the type and volume of call records which would be generated by the actual network. This stream of call records from the test network offers a valuable audit of network operation. It is necessary to collect these records, store them, and allow easy access to them so they may be analyzed for information about the state of the system. Realistic testing will generate a high volume and high-speed flow of call records. It is important that the verification system catch all incoming records. 
     Thus a system is required for receiving a high-speed stream of call records in a test network and efficiently organizing and storing the records for verification access. The present invention is designed and optimized for receiving and analyzing multiple data streams, such as call records, from a testbed telecommunications network. Although the preferred embodiment of the invention described below discloses a use of the invention for the processing of call records in a telecommunications system, it should be realized that the invention may be used to process incoming data streams other than call records. 
     SUMMARY 
     These and other features and advantages are accomplished in the system and method for call record search and verification disclosed in this application. In general, multiple instances of collecting and decoding processes embodied in stored programs running in a computer system act upon blocks of incoming data to store both a raw image of the received data and a pre-parsed version of the data suitable for database searching and retrieval. Three-step partitioned processing is disclosed comprising a set of collector processes for collecting data records, a set of decoder processes for decoding and parsing such records, and a set of loader processes for loading records into a database. A client can request certain call records or request verification of certain records. A rules mechanism embodied in stored templates operates to link client requests to asynchronously received data. The system provides data to a client in minimal time, regardless of when data becomes available. 
     In general, a computer software system for receiving, storing, analyzing and optionally filtering multiple data streams, and for retrieving and verifying data records from the data streams, comprises at least one processor executing a sequence of instructions embodied in a computer-readable medium. The system further comprises: 
     A service manager process executing asynchronously for starting and stopping all system processes; at least one collector process executing asynchronously for collecting data records from the data streams and placing the data records in a record queue; and, a store of one or more first pre-determined templates. The first pre-determined templates contain rules for filtering and parsing the data records. At least one decoder process asychronously parses data records in the record queue according to the first predetermined templates and stores such parsed records. At least one loader process asychronously loads the stored parsed data records into a database. The system has at least one asynchronous client manager process for accepting verification requests for data records from a client, acknowledging such requests, and placing such requests in a request queue. A store of one or more second pre-determined templates is provided; the second templates contain rules for finding and verifying data records. 
     At least one verification request processing process asynchronously reads requests from the request queue, reads requested data records from the database according to the second pre-determined templates, stores the requested data records, and stores requests for which no data records are yet available. The system also has an asynchronous query refresh futures process which reads the stored requests for which no data records are yet available and places on the request queue those requests for data records which require a retry. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIG. 1 is a flowchart depicting the service manager process in the preferred embodiment. 
     FIGS. 2 and 3 are schematic overviews of the preferred embodiment of the invention, depicting the activity of the processes which collect call records from a network information center and eventually place formatted records in a database. 
     FIG. 4 is a flowchart depicting the call-record collector process in the call-record component of the preferred embodiment. 
     FIGS. 5 and 6 are flowcharts depicting the decoder process in the call-record component of the preferred embodiment. 
     FIG. 7 is a flowchart depicting the processes used to accomplish a database load in the preferred embodiment. 
     FIG. 8 is a flowchart depicting the time server process in the preferred embodiment. 
     FIG. 9 is a flowchart depicting the queue refresh futures process in the preferred embodiment. 
     FIGS. 10 through 13 are flowcharts depicting the client manager process in the preferred embodiment. 
     FIGS. 14 through 16 are flowcharts depicting the request-processing and request-verification process in the preferred embodiment. 
     FIG. 17 illustrates a block diagram of a preferred embodiment of the present invention. 
     FIG. 18 illustrates an exemplary design for multiple instances of the VER,LOG and COL functions of FIG.  17 . 
     FIG. 19 is an exemplary high-level view of the Collector (COL) function, consistent with an embodiment of the present invention. 
     FIG. 20 illustrates an exemplary block diagram of message queuing consistent with an embodiment of the present invention. 
     FIG. 21 is an exemplary high-level view of the Verification (VER) function, consistent with an embodiment of the present invention. 
     FIG. 22 is an exemplary block diagram of the Time Server (TS) function, consistent with an embodiment of the present invention. 
     FIG. 23 is an exemplary block diagram of the Queue Refresh Futures (QRF) function, consistent with an embodiment of the present invention. 
     FIG. 24 is an exemplary block diagram of the Shared Memory Refresh (SMR) function, consistent with an embodiment of the present invention. 
     FIG. 25 is an exemplary block diagram of the interaction between the shared memory and other functions, consistent with an embodiment of the present invention. 
     FIG. 26 is an exemplary block diagram of the Client Manager (CM) function, consistent with an embodiment of the present invention. 
     FIG. 27 is an exemplary block diagram of shared memory and logger (LOG) function, consistent with an embodiment of the present invention. 
     FIG. 28 is an exemplary block diagram of the Verification function, consistent with an embodiment of the present invention. 
    
    
     DETAILED DESCRIPTION 
     In this disclosure, we assume the preferred embodiment is implemented on a programmable computer system running some version of the UNIX operating system, although implementation on most other operating systems could be accomplished by persons skilled in the art, given the disclosure of the preferred embodiment in this application. Accordingly, the terms in this disclosure which describe functions of the preferred embodiment are terms commonly understood by users of the UNIX operating system, but their use should not be construed to limit the application of the invention to UNIX operating systems. 
     In the preferred embodiment, the invention is implemented on a programmable computer, or a network of such computers, as a set of asynchronous processes. FIG. 17 depicts the high-level design of the preferred embodiment. FIGS. 2 and 3 show a simplified block diagram of the preferred embodiment and its participating asynchronous processes. One or more network information concentrators (NIC&#39;s)  100 ,  105  provide call records  102  from telecommunications switches in a telecommunications network. In this disclosure the data records of interest are call records from telecommunications switches; however, other embodiments of the invention could generally process a stream of data records from other devices, using the claimed improvements. The service manager process  125  (discussed below and depicted in FIG. 1) spawns one or more collector processes  10 ,  115 ,  120 , etc. (discussed below and depicted in FIG.  4 ), as well as all other processes of the preferred embodiment. Each collector process  110  writes blocks of call records to a common memory record queue  130 . This, and other queues described in this disclosure, may also be written to disk storage, with a considerable loss of processing speed. In the preferred embodiment, the NIC  100  collects and feeds call records  102  in blocks of some convenient predetermined size, such as  32  call records. Decoder processes  135 ,  140 ,  145 , etc. write decoded records to buffers in a memory file system  150 . One advantage of having multiple decoder processes  135  is having more processes to handle the work load from the record queue  130 . A disk file system could be used in place of the memory file system  150 . The decoder process  135  is discussed below. Loader processes  155 ,  160 , etc. take decoded records from the memory file system and mass load the records into a database  165 . Preferably, the database  165  is a Structured-Query Language (SQL) database which accepts mass insertion of records and high performance query processing by other computer programs. The preferred embodiment of the invention comprises a shared memory scheme. Multiple processes can access data in shared memory, thus conserving system memory and also enhancing performance by not maintaining all data in the database  165 . The loaders  155 ,  160  also write records and other information to log files and archive files  170 . The archive files  170  contain an image of records in mass load form. As shown on FIG. 3, one or more clients  196 ,  198  connect to the system, preferably by a telnet connection to a known port, which in turn spawns a client manager process (CLM)  192 ,  194 , etc. Each CLM  192 ,  194  communicates with the respective clients  196 ,  198  over TCP/IP, accepts requests from the respective clients  196 ,  198 , and sends back responses. Each CLM  192 ,  194  writes requests to the database  165  and to a request memory queue  175 . A configurable pool of verification request processing processes (VER&#39;s)  180 ,  185 , etc. feed from the request memory queue  175 . A given VER  180  stays blocked until a request is available form the queue. The goal is to give every request its own thread of processing as seen by the client  196 . An instance of a VER  180  processes every type of request and returns the result to the client as quickly as possible. If the billing record of interest is not yet loaded from the switch the billing is sought from, then the request in the database  165  is updated for a future retry, and the VER  180  continues processing the next request, as explained below and in FIGS. 13-15. The requests of the request queue  175  are disposed of after they are processed. The image of the requests lives in the SQL database  165 . 
     The preferred embodiment provides an ASCII text string interface for its clients. An ASCII interface is not necessary for practice of the invention, but it makes debugging easier. In the UNIX operating system configuration files /etc/inetd.conf and /etc/services (or corresponding files with different names on different versions of UNIX) are modified to provide automatic spawning of a client-manager process (CLM)  192  when a client telnets to a predetermined port on the system. When a CLM  192  is launched, it reads a configuration file which lists the supported commands and corresponding parameters. This permits convenient administration of CLM  192 . The CLM  192  configuration file contains syntactical requirements of supported commands and corresponding parameters, and the semantical requirements of how to deal with parameters. For example, the type of the parameter can be defined for how to parse it, the type of SQL database column defined for how to convert and store it, and other special handling. The CLM  192  configuration file provides all intelligence in request processing. The CLM  192  processing is generic and behaves according to the configuration in the configuration file. As described in more detail below, a client could submit a string request containing a command prefix with individual parameters such as: VREQTC TCSECTION=800654 TCNUMBER=33 START_DT=19980512125603 
     END_DT=19980512125603 OSW_NAME=RES 1  TSW_NAME=RES 1   
     RES_NAME=RES 1  NRETRY=0 MAXRETURN=5 TMPOVR=@=5;[=3;]=1;{=3}=1 
     PRI=HIGH 
     The parameter names indicate the type of parameter; for example, START_DT refers to a start date and time; TCNUMBER refers to a test case number; etc. 
     In the preferred embodiment, the system responds to a command string with a string that may consist of comments, errors and results. The comments and error codes returned can be flexibly written to test any of the many possible pathways and failure conditions as indicated by call records generated in a telecommunications system. For example, using the arbitrary convention that responses begin with a “+” and error codes with a “−”, we could have as possible responses: “−10000 error: invalid tc section/number section: 4  number: 4 ” or “+1003 CompareResult: filedName=PD;expected value= 2 ;operator===; reportedValue= 5 ;failDescript=Passed with Problem Code; problem Code= 276 ;When MCI 05  is loaded with 3 digit CIC (as opposed to 4 dig CIC), the leading bit in the CN is now 0, not TBCD NULL” 
     The reader should understand that the particular text strings used for commands, parameters, responses, comments, and error codes does not define the invention. These text strings may be crafted by designers of the system to display the system functions to operators in the most convenient way. Many other conventional command string and response formats could be used. The client manager process is described below and in FIGS. 10-12. 
     The service manager process  125  is depicted in FIG.  1 . In step  900  the service manager  125  creates shared memory, creates memory queues, and starts all other processes. Steps  910  and  920  form the main processing loop for the system, as the service manager  125  processes all inbound messages or signals from processes. If a request is received to terminate all processes, the service manager, as shown in steps  930  and  940  stops all processes and exits. 
     Each collector process  110  is an asynchronous process started by the service manager  125 . The flow chart of FIG. 4 describes the main steps in each instance of the collector process  110 . In the first step  200  the collector  110  initializes its variables by reading from a registry file (not shown) for appropriate initialization. (The reader will understand that when this disclosure speaks of “the collector” or of an instance of any of the other asynchronous processes described in this disclosure, it is intended to refer to any number of similar processes which may be running.) In the next step  210  the collector  110  initializes its session to an NIC  100  by making a connection to the NIC  100  through the network and receiving confirmation from the NIC  100 . In step  220 , the collector  110  gets the next block of records from the NIC  100  and verifies that the block of records is good in step  230 . If the block of records is not good, it is logged in step  240 , and execution returns to step  220 . If the block of records is good, the collector  110  checks for the end of available blocks of records in step  250 . If blocks of records are available, the block of records is written to the queue  130 , and execution returns to step  220  to get another block of records. If no more blocks of records are available from the NIC  100 , the collector  110  stays blocked on I/O from the NIC. The end result of the collector  110  processing is the placement of valid blocks of call records onto the record queue  130 . 
     The service manager  125  starts a number of decoders  135 , as shown in FIGS. 5 and 6. Each instance of a decoder  135  gets blocks of call records from the record queue  130  and processes each call record in the each block according to certain rules embodied in first predefined templates  342 . These templates  342  generally comprise the rules for formatting of call records for insertion into the database  165 , and the decision to load or not to load certain call records  102  according to filter rules. For example, blocks may be filtered by device, by record type, by particular fields within a record, or by characteristic of the data. In the preferred embodiment, the first templates support equality or inequality tests on values in the call records. In the preferred embodiment, a decoder retrieves its filter specifications by reading a table specified by a registry variable. 
     Referring to FIG. 5, the decoder  135  begins by initializing its parsing and filter rules in step  300 , reading from templates  342  for the decoding process. It then gets a block from the memory queue  130 . Step  310  tests if all records in the block are processed; if so, execution returns to step  305  to retrieve another block. If not, the next record is retrieved in step  315 . The retrieved record is parsed to determine its record type and filter criteria in step  320 ; then a check is made to determine if a filter is set for the record in step  330 . If, so, the record is skipped, and execution returns to step  310 . If no filter is set, execution continues to step  335 , continuation block B shown on FIG.  6 . The date and time is set for the first record in the buffer in step  335 . Then step  340  prepares the record for insertion into the database according to predetermined rules. Step  350  checks to see if the predetermined maximum buffer count is reached. If not, execution continues at step  305 ; if so, then the buffer must be written to a queue for a database loader process  155 . Step  360  makes a queue entry in a loader queue. At step  370 , the database loader process  155  is signaled to load the filtered and parsed records to the database, and execution continues at step  310 . 
     The next component of interest in the preferred embodiment is the loader process  155 , described in the flowchart of FIG.  7 . When a decoder  135  signals the database loader to load in step  370 , the signal is caught by the loader process  155 . In steps  400  and  405  of the loader, the process checks the buffer of interest to see if it has been processed; that is if there are any old (unprocessed) records in the buffer, which will be evidenced by a file name in the queue. If not, the process exits. If old records exist, these are written to the database  165  in the following steps. At step  410 , the set of records of interest is retrieved from the loader queue. At step  415 , the actual set of records is retrieved from the memory file system (MFS), and at step  435 , the set of records is loaded into the database  165 . The buffer is reinitialized from its start at step  440 , and the process stays blocked on I/O, awaiting its signal. 
     A time server process, depicted in FIG. 8, deals with the problem of time correlation on the network. On the network, a conventional network time protocol (NTP) keeps devices on the network synchronized. However, devices sending data records may not use a network time protocol. It is thus necessary to correlate the time of such devices to the system time, so that the system processes see time that correlates to the call record switch times. The time server process accomplishes this by using the offset between the switch time and the system time to properly offset the request time parameters in the system to times in the call records from that particular switch. The time server process first initializes itself at step  510  for the devices of interest by reading, in step  500 , from a table containing device data. This device data will include information about where the device address is located, how to log in (for example, a user name and password), any reply which is to be ignored, and the appropriate command to issue. The time server process polls, at step  520 , each device for time. In the disclosed embodiment, these devices are telecommunications switches, but the scope of the invention is not limited to such switches. At step  530 , the process updates the device time-change history table for each device. Finally, it sleeps for a predetermines time in step  540 , and execution returns to step  520 . 
     In general, telecommunications switches will have an administrative interface, which allows the switch time to be changed. The time server process of FIG. 8 handles time drift to account for varying clock speeds. The administration interface (not shown) ensures that a time change to a device is reflected with an appropriate update to the device time-change history table in a manner where polling depicted in FIG. 8 is properly synchronized to direct changes to the device time change history table. In the preferred embodiment, time changes are sent to the database  165 , where a record is kept of device time changes. Data in the device time-change history table is guaranteed to reflect correct correlation of the preferred embodiment&#39;s system time with the device times of interest. 
     The verification processes  180  may request call records  102  which are not yet available. In this case, it is necessary to store such requests and periodically attempt to retrieve the records. In the preferred embodiment, this is handled by a query-refresh futures process (QRF), described in FIG.  9 . The process begins at step  600  by selecting all rows of the database  165  with a status of initial or pending requests. At step  605 , the process fetches a row and checks for and end-of-file condition (EOF) at step  615 . If the end of file is not reached, the row is formatted into a structure in step  620  and this request structure is deposited into the VER queue in step  610 . Execution then returns to step  605  to fetch another row from the database  165 . Steps  600 ,  605 ,  615 ,  620 , and  610  process requests that have not yet been seen by a verification process; for example, if the system was powered off or terminated prematurely. 
     If EOF was detected in step  615 , the QRF process then selects all rows with a requeued status in step  625 , and fetches such a row from the database  165  in step  650 . If EOF is detected in step  635 , the QRF process sleeps for a predetermined time in step  640 , then returns to step  625 . If EOF was not detected in step  635 , the row is formatted into a request structure in step  630 ; the structure is deposited onto the VER queue in step  640 , the corresponding row in the database  165  is updated to pending status in step  645 , and execution returns to step  650  where another attempt is made to fetch a row having requeued status from the database  165 . Steps  625 ,  635 ,  640 ,  645 , and  650  process requests seen by a verification request that require a future retry. 
     It is convenient to next describe the client manager process (CLM)  192  of the preferred embodiment, before explaining the steps of the verification process (VER)  180 . Each client  196  which connects to a well-known socket, causing a spawn of a corresponding CLM  192 . The CLM  192  accepts requests from the client  196  and sends back responses. In the preferred embodiment, a client  196  receives a request identification (request ID) from the CLM  192  as an acknowledgment to its request. That request ID is the handle for receiving a later response. Typically, automated clients will bombard the system with many requests, often before billing arrives from the NIC&#39;s  100 ,  105 , etc. Thus, we have a requirement to store requests for call records in the sought call record has not yet arrived. The CLM writes requests to the database  165  and to a request memory queue  175 , as shown in FIG.  1 B. The database  165  provides persistent storage for the requests. 
     As shown in FIG. 10, the CLM  192  begins at step  700  by reading its environment variables, and the its behavior configuration from a CLM  192  configuration file (step  701 ). The CLM  192  is spawned with appropriate login parameters. At step  702  it gets the IP address and port of a process providing communications between the verification process and clients  196 . In this disclosure this process is called CSIS. The CSIS process implements a conventional means to match the socket ID of a CLM  192  with the session ID and the STDOUT ID (standard output on UNIX systems), so that verification processes know where to send responses for routing back to a client. Step  704  verifies login parameters, and if these are invalid, the CLM  192  exits. Additionally, the communications link access in step  702  is checked in step  704 , and if it was not successful, the CLM  192  exits. If step  702  was successful, then in step  706  the process loads its request types and parameters from the CLM  192  configuration file and attempts to connect to CSIS. Connection to CSIS is checked in step  708 . If not successful, the CLM  192  exits. If access was successful the CLM creates and initializes its various timers and signals in step  710 , and gets socket descriptors for CSIS and the client  196 . Execution continues as depicted on FIG.  10  through continuation block B. We check in step  714  if the CSIS socket is ready, with response data from a verification process to return to the client. If it is, step  716  reads data from the socket and sends it to the client that is connected to this CLM  192 . Execution then continues at step  712 , as shown by continuation block A. If the CSIS socket is not ready with response data, we check for any STDFN (standard input device on UNIX systems) data ready in step  718 . When ready, the CLM  192  reads from STDIN in step  720  until a terminator is detected, and checks for a valid login by the client in step  722 . The read from STDIN implies a wait for input. If has been no valid login, the CLM  192  exits. If the client has made a previous valid login, the CLM  192  begins to handle commands from the client  196  in step  724  and following steps. It should be noted that step  712  implicitly waits for the system to indicate availability of STDIN (request data from a client), STDOUT (response data to send back to the client, or a signal from the service manager. Thus step  712  waits for one of these events. 
     First, the CLM  192  checks in step  728  for a request for verification or a verification ID. If either request is received, the request is processed and inserted into the request queue in step  729 . Successful processing of the request is checked in step  730 . If the request could not be processed, the transaction is rolled back in step  732  and execution continues at step  712 , depicted in FIG.  10 . If request processing for the verification request is successful, execution continues at step  734  depicted on FIG. 12, through continuation block D. Step  734  checks to see if a certification ID is present. A “certification ID” in the preferred embodiment is an identifier of a list of test cases to be verified. This allows tagging of a group of test cases to a batch ID. A test case may belong to a plurality of certification batches. Such certification ID&#39;s are stored in step  736  with the request. If no certification is present, execution flows to step  742 , where the record is committed to the database  165 . Step  740  checks for a successful add to the request&#39;s certification table; if the add was not successful, the transaction is rolled back in step  738 , and execution returns to step  712 . If step  744  determines the transaction was not successfully committed to the database  165  in step  742 , then the transaction is rolled back in step  738  and execution proceeds to step  712 . If the transaction was successfully committed, then the verification request is inserted in the memory request queue in step  746 . If step  748  determines this insertion was successful, then execution returns to step  712 ; if not, execution returns to step  712 . Steps  746  and  748  wait until the memory queue is available for insertion (queue full condition). This situation rarely occurs because verification processes handle requests quickly. 
     If, at step  728 , the system determines that the request is not for verification, execution proceeds to step  752 , depicted on FIG. 13, through continuation block C. A test is made at step  752  to see if the request is a request to get a record from the database  165 . If not, a test is made at step  776  to see if the request is for help. If so, the corresponding help file is read and displayed at step  778 , and execution returns to step  712 . If the request was not for help, a test is made at step  774  to see if the request is for debugging. If so, at step  772 , the line containing the command is read from the debug file maintained for records generating error messages, and execution proceeds to the command handler at step  724  on FIG. 10, through continuation block E. The reader should understand that procedures for implementing help and debugging features are well-known in the art and do not define the invention. 
     If the test at step  752  found a request to get a record (i.e., perform a search), execution proceeds to step  754 , where the request is inserted into the request queue. A test is made for success at step  756 . If the insertion was not successful, the transaction is rolled back at step  764  and execution proceeds to step  712 . If the insertion was successful, the request is tested at step  758  to determine if a request for particular fields of the call record requested is present. If so, the fields are inserted into the a database table at step  760 . If the insertion was not a success, the transaction is rolled back at step  764 , and execution returns to step  712 . Otherwise, the request is committed to the database in step  763  and inserted into the database request queue at step  766 . If the insert request for the queue tests successfully at step  768 , execution proceeds to step  712 ; otherwise execution continues step to  766 . 
     We now turn to a description of call record verification request processing in the preferred embodiment. FIG. 14 depicts the beginning of a verification (VER) process  180 . A pseudocode listing of the VER process  180  of the preferred embodiment may be found at pages 35 through 37 of the Appendix. The VER  180  gets the next request for record verification from the appropriate memory queue in step  800 , with an implicit wait. Next, the process checks at step  805  if a signal has been received from the service manager  125  to terminate, that being the reason for the exit from step  800 . If such a signal is received, the VER process  180  terminates, otherwise, execution flows to step  810  to get all templates for the request from a store of second predefined templates  812 . The templates retrieved from second predefined templates  812  contain information for how to seek for a call record associated with a testcase (i.e. a call made into the test network) from the SQL database  165 , including the device that should generate the record, and how to verify the record after it is found in the SQL database  165 . Then, step  814  correlates system time of the preferred embodiment (i.e., client request time parameters) with device times in templates, so a proper search query is built to find the call record in the SQL database  165 . In step  815  the VER  180  gets the most recent billing date-time for devices associated with the second templates. Step  820  checks to see if all billing records should have been loaded yet for the second templates; if not, such requests are marked in the database for retry in step  825 , and execution returns to step  800 . There are provided database triggers in the SQL database  165  that update the most recently received date/time stamp for call records received by devices. Step  820  accesses the values for devices of the second predefined templates  812  to see if all records of the test case are indeed loaded yet. Time correlations from step  814  are used to compute the date/time stamp of the most recent device call record date/time stamps. Not until all call records should be present as determined by step  820 , will step  830  continue processing. If all billing records should be loaded, the VER  180  outputs a partially constructed overall response testcase line into the response buffer at step  830 . In the preferred embodiment, the output of the VER  180  is built on the fly and response lines are collected in a buffer (not shown). Execution continues through continuation block B to FIG.  15 . Step  835  gets the next template  812  for the request. If all testcase templates  812  are processed, step  840  sends execution to step  845 , where the return code in the test-case line is set to the worst-case template result. Then step  847  puts the testcase results to a certifications data if one or more certification ID&#39;s were associated with the testcase. The results posted allow a certification interface to access results from the SQL database  165 . Thereafter, the built response is then sent to the client through the CSIS process in step  850 . If all templates  812  are not processed, execution proceeds to step  835  where a partially-constructed template result line is appended to response output. Then, step  855  reads billing record search criteria from the template  812 . Step  860  performs billing record search using the search criteria just obtained. Step  865  initializes the greatest success to “record not found,” and passes execution to step  870  to get the next billing record found. Step  875  checks whether all billing records are processed. If so, step  880  sets a return code in the response template line to the best case of verification results of the billing record found, and passes execution to step  835  to get the next template  812  for the request. 
     If all billing records are not processed, execution proceeds through continuation block C to step  885  on FIG.  16 . At step  885 , a partially constructed billing record response line is appended to the response buffer. Execution then passes to step  890  to verify billing record fields to template expected values of second predefined templates  812 . Step  890  appends billing record field results to the partially constructed response output buffer and updates the status of the line output at step  885 . The worst case result of a field comparison during verification is set in the billing record line of step  885 . After step  890 , execution returns to step  870  through continuation block D to get the next billing record. 
     The reader will understand that many different call record verification requests may be conceived. The preferred embodiment provides a flexible and scaleable system for generating different test cases to fully test a telecommunications network, and for storing the results of applying such test cases to call records. 
     As described in more detail below, a preferred embodiment may include conventional logger and janitor processes. FIGS. 17-28 illustrate additional exemplary embodiments of the present invention described above.