Abstract:
A routing chip test case generator operable to generate test cases for testing microcode allows test cases to be directed to specific operations, such as those executed by an Application Specific Integrated Circuit (ASIC), without burdening the test cases with other operations performed during packet traffic routing in a routing engine. Such test cases are directed to specific microcode operations to be tested by simulating the other traffic processing operations performed during packet routing. A router packet is generated to simulate message traffic in a routing engine. The router packet is modified to produce a router test packet that emulates the packet at a point prior to the specific microcode instructions to be tested. A test verification packet is generated to simulate the router test packet after the specific microcode instructions to be tested are applied to the router test packet. The specific microcode instructions are applied to the router test packet to produce a test results packet, and compared to the test verification packet. In this manner, ASICs directed to specific microcode operations can be discretely tested without encompassing or anticipating other packet routing operations which typically occur as a packet is processed by the routing engine.

Description:
BACKGROUND OF THE INVENTION 
     Systems and methods are known for testing routing engines which perform message traffic routing in a computer network. Typically, a router packet is generated to target a specific traffic processing operation performed by a hardware, software, or firmware unit under test. An expected output packet is also generated to correspond to the state of the router packet after the specific traffic processing operation has been performed. The expected output packet is then compared to the actual output packet received from the routing engine to determine the success or failure of the test. By generating a series of router packets and corresponding expected output packets, a range of test cases covering the desired traffic processing operations to be tested is produced to fully test the routing engine. 
     In a routing engine, input packets are received, processed, and then sent on as output packets. Various traffic processing operations may add to, delete from, or otherwise modify the router packet, typically according to a protocol such as the Transmission Control Protocol/Internet Protocol (TCP/IP). In the routing engine, one or more routing chips apply traffic routing instructions to the packet in series according to the traffic processing operations to be performed. Each traffic processing operation includes one or more traffic routing instructions. During a traffic processing operation, many traffic routing instructions may be applied to and may modify the router packet. 
     When a test case is generated, the expected output packet is generated to correspond to the modifications the router engine will apply to the router packet. The expected output packet reflects the modifications that will be made by the traffic processing operation that the test case is directed to. Each router packet, however, is processed according to a full series of traffic routing instructions occurring as the input packet is processed into an output packet. Therefore, each test case must anticipate not only the modification made by the traffic routing instructions that the test case is directed to, but to all the traffic routing instructions which will modify the packet from the packet through processing as an input packet to the output packet. 
     Some traffic routing instructions, such as those driven by microcode residing in Application Specific Integrated Circuits (ASICs), result in an unwieldy number of test cases due to the a complex arrangement of control paths exhibited by the traffic routing instructions to be tested. The router packet and the expected output packet must nonetheless be generated to anticipate not only the traffic routing instructions applied by the microcode, but also other traffic routing instructions such as those driven by TCP/IP and other protocols employed by the router engine. The effort required to generate and analyze all the packets of the test cases which result from microcode is substantial. Accordingly, it would be beneficial to provide a system and method to generate router packets directed to test only a set of instructions corresponding to microcode residing in ASICs, without the need to generate router packets and expected output packets which anticipate other traffic routing instructions occurring in the router engine. 
     SUMMARY OF THE INVENTION 
     A routing chip test case generation system and analyzer operable to generate and analyze test cases for testing microcode allows test cases to be directed to specific microcode operations, such as those executed by an Application Specific Integrated Circuit (ASIC), without burdening the test cases with other operations performed during packet traffic routing in a routing engine. Such ASIC-based test cases are directed to specific microcode operations to be tested by simulating the other non-microcode traffic processing operations performed during packet routing. A router packet is generated to simulate message traffic in a routing engine. The router packet is modified to produce a router test packet that emulates the state of the packet at a point prior to the specific microcode instructions to be tested. A test verification packet is generated to simulate the router test packet after the specific microcode instructions to be tested are applied to the router test packet. The specific microcode instructions are applied to the router test packet to produce a test results packet, and compared to the test verification packet. In this manner, ASICs directed to specific microcode operations can be discretely tested without encompassing or anticipating other traffic routing instructions which typically occur as a packet is processed by the routing engine. 
     In this manner, the test case generation system generates a router packet directed to testing a specific traffic processing operation in the routing engine. The test case generator determines the processing operations that modify the packet in the routing engine just prior to the execution of the set of test instructions, generates a router test packet indicative of the processing operations prior to the set of test instructions being applied to the router packet, and generates a test verification packet indicative of the set of test instructions being applied to the router test packet. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular description of preferred embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. 
     FIG. 1 is a block diagram of the automated test case generation system as defined herein; 
     FIG. 2 shows a prior art packet-based test system; 
     FIG. 3 shows the test case generation system being used for microcode testing as defined herein; 
     FIG. 4 shows a block diagram of the automated test case generation system of FIG. 3 in greater detail; 
     FIG. 5 show a flowchart of packet-based testing using the automated test case generation system; and 
     FIG. 6 shows an example of a packet-based test scenario using the system of FIG.  4 . 
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     A description of particular embodiments follows. 
     FIG. 1 shows a block diagram of the automated test case generation system  10  operable for performing packet-based testing as defined herein. Referring to FIG. 1, a test case generator  12  generates test cases indicative of a router test packet and a test verification packet. The packet generator  14  generates the router test packet and sends the router test packet to the unit under test  16 . The packet analyzer  18  generates the test verification packet corresponding to the test case. The unit under test  16  applies a set of test instructions corresponding to the traffic processing operation to which the test case is directed to generate a test results packet. The test results packet is sent to the packet analyzer  18 , where it is compared with the test verification packet. In this manner, the unit under test, such as an ASIC or other router chip, can be tested by generating a router test packet and test verification packet corresponding only to the traffic processing operations to be tested. Further, such testing may be performed on a simulator designed to simulate the actual operations to be performed by the unit under test, rather than the actual hardware, software, or firmware. 
     FIG. 2 shows a prior art packet-based test apparatus. A line card  20  or simulator provides a router packet  21  to a routing engine  22 . The routing engine  22  processes the router packet according to processing operations from a router processor  24 . The message is then sent to two groups of ASIC processor banks  26   a  and  26   b , respectively. The router packet  21  directed to testing a traffic processing operation in the processor banks  26   a  and  26   b , therefore, must anticipate the processing operations to be performed by the router processor  24 . Accordingly, the router packet  21  and the corresponding expected result packet must be generated with information corresponding to the processing operations to be performed by the router processor  24 , in addition to the processing operations to be performed by the processor banks  26   a  and  26   b  under test. 
     FIG. 3 shows a particular embodiment of the system operable to use the test case generator for testing microcode as defined herein. A test bed  28  includes the routing engine  22  and the processor banks  26   a  and  26   b . In a particular embodiment, each processor bank includes a 4*4 array of processors. These processors are arranged in eight columns, comprising eight processor groups defined by processor columns  32   a - 32   h . Each of the processor columns  32   a - 32   h  is directed to a particular type of traffic processing operations, such as routing table lookup, quality of service (QOS), IP sanity check, access lists, multicasting, and others. A router test packet  30  is generated to test a set of instructions corresponding to a traffic processing operation to be tested, as will be described further below. The router test packet  30  is sent directly to the processor banks  26   a  and  26   b , and is processed in sequence by each of the processor columns  32   a - 32   h . The router test packet  30  is not processed by the router processor  24  (FIG.  2 ), so the router test packet  30  does not need to contain fields corresponding to operations performed by the router processor  24  (FIG.  2 ). A particular router test packet  30  is typically directed to a traffic processing operation to be performed by one of the processor columns  32   a - 32   h . A router test packet, however, can be directed to multiple processor columns  32   a - 32   h . Following processing by each of the processor columns,  32   a - 32   h , the output packet  34  is generated. 
     Prior to discussing the test bed  28  in conjunction with the test case generator  12 , a further discussion of prior art microcode testing may be beneficial. As indicated above with respect to FIG. 2, the routing engine  22  receives a router packet  21  from the line card  20 . In a testing environment, a traffic generation module (TGM), described further below, is used to generate packets to emulate router packets from a line card  20 . Since the TGM generates complete router packets, router processor  24  operations, as well as processor bank  26   a  and  26   b  operations, are anticipated. Since the TGM generates all fields in a router packet  21 , it can be beneficial to modify the TGM generated router packet  21  to generate a router test packet  30  which does not include information directed to the router processor  24 . 
     FIG. 4 shows the test bed of FIG. 3 with the test case generator  12 . Referring to FIGS. 3 and 4, the test case generator  12  generates an input case list  36  and an output case list  38 . The packet generator  14 , by modifying a router packet from the TGM  40 , generates a router test packet  30  (FIG. 3) and stores the packet  30  in the input packet file  42 . For each router test packet  30  in the input packet file  42 , a corresponding entry in the output case list  38  is generated by the test case generator  12 . Following testing, an output packet file  44  contains a test results packet corresponding to each router test packet  30  from the input packet file  42 . The packet analyzer  18  receives and compares the test verification packet generated from the output case list  38  with the test results packet from the output packet file  44  to determine the test outcome. 
     FIG. 5 depicts a flowchart of packet-based testing as described herein. Referring to FIGS. 3,  4 , and  5 , an input case list is generated through the test case generator, as shown at step  100 . A corresponding output case list  38  is generated, as depicted at step  102 . For each entry in the input case list  36 , the packet generator  14  receives a TGM generated router packet  21  as disclosed at step  104 , and generates a router test packet  30  based on the input case list entry  36 , as shown at step  106 . The router test packet  30  is written to the input packet file  42 , as depicted at step  108 , and a check is made for more entries in the input case list  36 , as shown at step  110 . If there are more entries in the input case list  36 , control reverts to step  104  to process the next entry, as shown at step  112 . If all entries in the input case list  36  have been processed, the input packet file  42  is complete and is sent, packet by packet, to the test bed  28 , as shown at step  114 . Each router test packet  30  in the input packet file is processed by each of the processor columns  32   a - 32   h  in the processor banks  26   a  and  26   b , as depicted at step  116 . Following processing by the processor columns  26   a  and  26   b , the output packet  34  is generated and is written to the output packet file  34 , as disclosed at step  118 . A check is made at step  120  to determine if there are more router test packets  30  in the input packet file  42 . If there are more router test packets  30 , control reverts to step  114 . If all of the router test packets  30  have been processed, the output packet file  44  is sent to the packet analyzer  18 , as depicted at step  122 . For each output packet  34  in the output packet file  44 , the packet analyzer generates a test verification packet from the corresponding entry in the output case list  38  and compares the test verification packet to the output packet  34 , as depicted at step  124 . In this manner, automatic generation of router test packets  30  for testing specific functions in the processor banks  26   a  and  26   b  avoids manual generation and validation for each test case. 
     
       
         
               
               
               
             
           
               
                   
                 TABLE I 
               
               
                   
                   
               
             
             
               
                   
                 LC1 
                 0030.965A.933A 
               
               
                   
                 LC2 
                 0030.965A.933B 
               
               
                   
                 LC3 
                 0030.965A.933C 
               
               
                   
                 LC4 
                 0030.965A.933D 
               
               
                   
                 LC5 
                 0030.965A.933E 
               
               
                   
                 LC6 
                 0030.965A.933F 
               
               
                   
                   
               
             
          
         
       
     
     FIG. 6 shows an example of a test case using the test case generator. Three test cases are developed to route a packet from line card one (LC 1 ) to line card two (LC 2 ), from line card three (LC 3 ) to line card four (LC 4 ), and from line card five (LC 5 ) to line card six (LC 6 ), respectively. Note that the substance of the raw test cases  46  developed may be from a variety of sources, such as from a unit test plan, engineering notes, iterations through a range of possible values, boundary values, quality control requirements, and others. The test case generator produces the input case list  36  and the corresponding output case list  38 . The test case illustrated uses the supporting test data from TABLE I associating MAC address data with line cards. 
     The supporting test data is used by the packet generator  14  and by the packet analyzer  18  to derive the raw data to be written to each router test packet  30  generally and expected by each test results packet  34  generally. The input case list entries  36   a - 36   c  are supplied to the packet generator  14 . The packet generator  14  receives the router packets  21   a - 21   c  from the TGM  40 , and strips off the router fields  46 . The router fields  46  correspond to processing performed by the router processor  24  (FIG.  2 ). MAC address fields and payload fields,  48  and  50  respectively, are left intact by the packet generator to produce the input packet file  42 , including the router test packets  30   a - 30   c . The router test packets  30   a - 30   c  are now indicative of a router packet following processing by the router processor  24  (FIG.  2 ), but prior to processing by the processor banks  26   a  and  26   b.    
     The router test packets  30   a - 30   b  are supplied, in order, from the input packet file  42  to the processor banks  26   a  and  26   b . Following processing by the processor banks  26   a  and  26   b , the test results packets  34   a - 34   c , corresponding respectively to the router test packets  30   a - 30   c , are written to the output packet file  44 . 
     The packet analyzer receives the output case list  38  from the test case generator, and, also using the supporting MAC address test data from TABLE I, generates a test verification packet  52   a - 52   c  for each of the test cases to compare to each of the test results packets  34   a - 34   c . Test success is indicated by a match between the test results packet  34  and the test verification packet  52 . 
     The examples of FIGS. 3-6 employ a plurality of processor banks corresponding to ASIC processors having microcode, and is intended as illustrative of the automated test packet generation engine as described and claimed herein. The unit under test need not be an ASIC processor bank, but could be a variety of hardware, software, or firmware units under test. The system and methods described in the particular embodiments disclosed are directed to automated generic packet-based testing by generating a packet directed to a specific traffic processing operation performed by a unit under test, without encumbering the test with extraneous operations which may otherwise need to be anticipated in a black-box testing environment. The testing system and methods described herein, therefore, target a discrete operation which may be performed in a series of operations in an actual system, and is directed to generating router test packets and test results packets adapted to be readily compared to ascertain test success or failure. Other packet-based systems can be tested using the methods described herein, such as other routing chips, processors, and protocols, and various hardware, software, and firmware included therein. 
     Those skilled in the art should readily appreciate that the programs defining the operations and methods defined herein are both deliverable to the test case generation system and directed to a unit under test encompassing many forms, including but not limited to a) information permanently stored on non-writeable storage media such as ROM devices, b) information alterable stored on writeable storage media such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media, or c) information conveyed to a computer through communication media, for example using baseband signaling or broadband signaling techniques, as in an electronic network such as the Internet or telephone modem lines. The operations and methods may be implemented in a software executable out of a memory by a processor or as a set of instructions embedded in a carrier wave. Alternatively, the operations and methods may be embodied in whole or in part using hardware components, such as Application Specific Integrated Circuits (ASICs), state machines, controllers or other hardware components or devices, or a combination of hardware and software components, or hardware, software, or firmware simulators. 
     While the system and method for automated, generic packet-based test case generation and verification have been particularly shown and described with references to embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims. Accordingly, the present invention is not intended to be limited except by the following claims.