Abstract:
A modular data network protocol conformance test system is designed to make distinct the key functionality that is required in protocol conformance testing such that the system can be reused across different platform testing, protocol testing, and network topology. As a result, in its current design, it can be used to test devices as distinct as cable modems, routers, and tag-switching routers. Consequently, the system has a packet generation module that generates send packets for transmission to the UUT (unit under test) and that analyzes the receive packets from the UUT. A separate network handling module transfers the send packets to and acquires the receive packets from the UUT. 13

Description:
BACKGROUND OF THE INVENTION 
     Protocol conformance testing for network communication devices is relevant in a number of different environments. Manufactures must implement rigorous device verification before release for beta-testing and then finally before shipments for revenue. Moreover, especially in mission-critical applications, the end-user will typically implement its own protocol testing to ensure that the communications device is operating according to specifications and is compatible with the other network communication devices in existing networks. 
     The spectrum of network communication devices to which protocol testing is applied is similarly varied, extending from devices which have relatively simple behavior such as asynchronous transfer mode (ATM) switches to switches/bridges, with or without learning capabilities, and cable modems. It is most critical, however, for devices with the most complex behavior such as routers, multiprotocol routers, and tag-switching routers. 
     Within these network communication devices, the protocols to be tested are extensive. Hand-shaking to establish a level of service across a link between two devices represents some of the more complex types of protocols, due to the state-driven nature. For example, communicating tag-switching-enabled routers establish level of support using a tag distribution protocol. Whereas, lower-level protocols, operating nearer to the physical layer, must also be verified to determine, for example, whether the frame checksums/cyclical redundancy check (CRC) fields are properly set and whether other fields are within ranges dictated by the relevant protocol. 
     In the past, protocol testing systems have been designed to test individual protocols in the context of specific platforms, specific topologies, and specific states. Commonly, a workstation would be connected to the network in which the unit under test (UUT) is deployed. At the workstation, a test engineer typically hard-codes a test script that generates communications on the network to essentially interrogate the UUT&#39;s regarding their behavior under the protocol of interest. 
     SUMMARY OF THE INVENTION 
     The problem with this hard-coding technique for generating the scripts used to test network communications device protocol conformance surrounds the fact that the test engineer must typically write a separate script for each device, each implementation of that device, each protocol that must be tested, and each topology on which the device operates. In the context of the situation where many network communications devices must be tested over periods of time, commonality exists between protocols executed by the devices, network topologies, and the numerous test scripts that are developed, but due to the situation-specific nature of the scripts and limitations on the documentation of these scripts, there is little opportunity to reuse the core intelligence from one device-test to another. 
     The present invention is directed to a modular data network protocol conformance test system. The modularity at the software module, procedure, and/or object level is designed to make distinct the key functionality that is required in protocol conformance testing such that the system can be reused across different platform testing, protocol testing, and network topologies. As a result, in the current design, it can be used to test devices as distinct as cable modems, multiprotocol routers, and tag-switching routers. 
     In general, according to one aspect, the invention concerns a data network protocol conformance testing system. It comprises a packet generation module that generates send packets for transmission to the UUT (unit under test) and that analyzes the receive packets from the UUT. A separate network handling module transfers the send packets to and acquires the receive packets from the UUT. 
     The system can be deployed in a number of implementations. The modules can run on a workstation computer that directly interfaces with a network on which the UUT operates. Alternatively, interface can be achieved between the workstation executing the modules and the UUT via a packet generating router. The advantage of this latter system is that the intelligence required to transfer the packet or frame, for example, onto the network can essentially be out-sourced to this separate network communications device, i.e., router. 
     In a preferred embodiment, an input definition file is used to hold a testing value section that contains fields of the send packets. This input definition file preferably also has protocol header information that determines the encapsulation for the send packets. Further, the testing value section can comprise minimum and maximum values to be used in specified fields of the send packets in addition to incrementing values used so that the value of the field is changed, incrementally, in successive packet transmissions. 
     In order to test more complex protocol operation, the input definition file also preferably has state definition information. The packet generation module extracts state packets for transmission to the UUT so that protocol conformance in a specific state, or proper state-transition behavior of the UUT can be tested. 
     Also according to the preferred embodiment, an output-definition file is further included containing expected values in the receive packets. The packet generation module accesses this output definition to analyze the receive packets from the UUT for protocol conformance. 
     In general, according to another aspect, the invention also features a data network protocol conformance test method. This method comprises forming the send packets for transmission to a UUT and then transferring the send packets to a network handling module, which transfers the send packets onto the network. Receive packets are then acquired and transferred to the packet generation module where they are again analyzed for protocol conformance. 
     Finally, the invention also concerns an article of manufacturer comprising a machine readable medium holding a computer program. This computer program comprises the packet generation module which enables a computer workstation to generate send packets for transmission to the UUT and that then analyzes the receive packets from that unit. The program further comprises a network handling module that enables the workstation to transfer and receive the send and receive packets to and from the UUT. 
     The above and other features of the invention including various novel details of construction and combinations of parts, and other advantages, will now be more particularly described with reference to the accompanying drawings and pointed out in the claims. It will be understood that the particular method and device embodying the invention are shown by way of illustration and not as a limitation of the invention. The principles and features of this invention may be employed in various and numerous embodiments without departing from the scope of the invention. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     In the drawings, like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention. 
     FIG. 1 is a block diagram illustrating a test bed configuration in which the data network protocol conformance testing system of the present invention is implemented; 
     FIG. 2 is a software architecture diagram illustrating the organization of the definition files, packet handling module, and network handling module according to the present invention; 
     FIG. 3 shows exemplary packet field definitions in the packet definition section of the input definition file; 
     FIG. 4 is an exemplary state diagram of a UUT; 
     FIG. 5 is a flow diagram illustrating the operation of the inventive system to control UUT state; 
     FIG. 6 is a flow diagram illustrating the steps performed by the protocol performance testing system of the present invention; 
     FIGS. 7-11 show other test bed configurations in which the system of the present invention is implemented; and 
     FIG. 12 shows an implementation in which the inventive system is used to test the tag distribution protocol of a tag switching router for conformance. 
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     FIG. 1 is a block diagram illustrating a typical environment in which the present invention is deployed. A computer workstation  110  and packet generating router  112  function together to act as the data network protocol conformance testing system  100 . The router  112  interfaces with the network in which the UUT (unit under test)  200  operates. 
     Depending on the implementation, the packet generating router  112  has a common link with UUT  200  or is connected indirectly through other data network communication devices. Further, the UUT  200  may be additionally connected to other data communication devices in network cloud  114 . 
     The testing performed by system  100  is described as protocol conformance testing. This classification includes, for example, protocol finite state machine testing in which the protocol state machine of the UUT  200  is tested for proper transition from one state to another in response to external stimuli, such as state packets sent from system  100 . Additionally, negative testing of the protocol state machine UUT  200  is provided in the preferred embodiment. Conditions for negative protocol testing are created with abnormal state packet sequences or packets with erroneous or invalid fields that relate to state transition. Protocol data structure testing is also contemplated for testing by system  100 . Here, fields of packets containing data values at upper and lower limits as dictated by specifications are sent. Further, full structure or linked list implementation of data structures is tested by sending packets with single or multiple linked value formats. Extreme and out-of-bound values are also sent in the packets and the responses of the UUT  200  are verified. Abnormal or illegal values can be sent by system  100  to the UUT  200  to verify the operation of the database. Further, the database handling is tested by populating it to a large size and verifying normal operations. Finally, inter-operability is tested along with compatibility with different versions of protocols and protocols of different devices. 
     FIG. 2 is a software architecture diagram illustrating the modularity of the software running on the workstation  110 . 
     The software comprises an input definition file  120 , which dictates both the construction of the packets that are sent to the UUT  200  and the information necessary to get those packets to the UUT, i.e., encapsulation. As such, the input definition file  120  comprises a protocol header section  122 . This contains, in the preferred embodiment, hexadecimal (HEX) formatted testing protocol header data. In a specific example of the system  100  being connected to the UUT  200  via an CSMA/CD link, the protocol header section contains the header information, source and destination information, for the CSMA/CD frame that conveys the packets to the UUT  200 . 
     The packet definition section  124  has both a test values section, which contains the fields of the send packet, and state definition section, which is the state packets that are required to be sent to the UUT  200  to get it to the particular state, in which the protocol testing conformance is to take place. 
     The following sets forth organization of the testing values section of the packet definition section  124  of the input definition file  120 . Each line defines a field of the packet. As a result, if a packet has 50 fields, then there will be 50 lines in the testing values section of the packet definition section  124 . In the preferred embodiment, the format of each line is: 
     
       
         name length type delta MIN MAX SET{ . . .} INCR 
       
     
     In this illustrated implementation, the following rules are used when constructing each line. 
     the first four fields are mandatory, the fields order must be followed. 
     name is the field&#39;s name. e.g. Version_number. 
     length is this field&#39;s length, number of bytes. 
     type specifies this field&#39;s data type. NUM (number), CHAR (character), BIT are the data types. 
     delta is used for the negative test that are below and above the data range boundary. It is based on MIN/MAX or SET, depending on which one is used. The first/last value in the MIN/MAX or SET data range minus/plus the delta value will be the first/last testing data. To create one testing value of the field, use 0 as delta with one testing value defined in either SET or MIN/MAX value (MIN=MAX=testing value). 
     MIN, MAX, SET, INCR are keywords; the data must follow them and have a space separator. The MIN/MAX pair or SET must exist. 
     MIN/MAX specifies the minimum/maximum value of this field. 
     SET specifies a set of test values, separated by space within “{ }”. 
     If SET is present, its value(s) override MIN/MAX for use. ie. SET {2 7 20} 
     INCR specifies the increment of MIN/MAX. The default value is 1. 
     FIG. 3 illustrates a packet constructed from the following definition file to illustrate a specific example: 
     version_id  1  NUM 0 MIN 0 MAX 5 SET {1} 
     fun_code  2  NUM 0 SET {1} 
     pg_len  1  NUM 0 MIN 8 MAX 8 
     Returning to FIG. 2, the state definition section of the packet definition section  124  contains the state packets that are required to initiate the change-overs required to get the UUT  200  from its initial state to the particular state in which protocol testing is desired, for example, or to test state transitions themselves. Such operation is relevant to test, for example, the hand-shaking involved between routers to establish some level of service across the link. For example, these packets are sent between the packet generating router  112  to the UUT  200  to first open a link between the devices and then establish some level of data transfer such as tag switching between the router  112  and UUT  200 . 
     In one implementation, each line of the state definition section of the input definition file is formatted as follows: 
     STA_# state_name&lt;state definition packet files&gt; 
     ▪STA_#, STA_ is fixed format, # are sequential line by line. 
     FIG. 4 illustrates one example. To get UUT  200  into State3 requires it to have received Pk 1  and Pk 3 , in order. The state definition section thus has at least two state definition packet files. One for PkI and one for Pk 3 . Or, the UUT can also be transitioned with packets Pk 2 , Pk 1  and Pk 3 . In this case, the files needs at least three state definition packet: Pk 1 , Pk 2 , Pk 3 . 
     A complete state definition section is as follows: 
     STA_ 0  Initial 
     STA_ 1  State1 Pk 1   
     STA_ 2  State1 Pk 2  Pk 1   
     STA_ 3  State2 Pk 2   
     STA_ 4  State3 Pk 1  Pk 3   
     STA_ 5  State3 Pk 2  Pk 1  Pk 3   
     Pk 1 , Pk 2 , Pk 3  are the state packet file&#39;s file name. 
     For example, the line number  5  (STA_ 5 ) is interpreted as: the UUT to get to the State3, it must receive packets Pk 2 , Pk 1  and Pk 3 . 
     Returning to FIG. 2, the input definition file  120  is used by the packet handling module  126 . The packet handling module has a definition process  128 , which is used to access the input definition file  120 . This process passes the information from the file to generation process  130 . This creates the state packets and packets with the testing values, which are then placed in a send buffer  132 . 
     A network handling module  134  successively picks the packets out of the send buffer  132 , preferably in a first-in-first-out basis, for transmission to the UUT  200 . 
     In the context of the example illustrated in FIG. 2, the network handling module also has a configure process  136  that configures the packet generating router  112  to receive the send packets from the workstation  110  and then send them to the UUT  200 . This embodiment has certain advantages in complexity of implementation since details concerning formation of the frames, for example, for transmission to the UUT, is implicit in the capabilities of the router  112 . 
     In an alternative embodiment, where the workstation  110  is directly interfaced to the network on which the UUT  200  is operating, via a network interface card, for example, the sending sent packet process  138  must have the additional intelligence necessary to form the data transmission structure to get the packet to the UUT  200 . 
     Also in the network handling module  134  is a capture response process  140 . This, in the illustrated example of FIG. 1, interfaces with the packet generating router  112  to record and capture the response packets received by the router  112  from the UUT  200 . These response packets are placed by the capture response process in received buffer  142 . 
     The packet handling module  126  has a compare received packet process  144 . This process  144  accesses a validation file  146  and output definition file  148 . 
     The validation file  146  identifies the output definition file  148 . It also specifies the fields of the receive packet(s) that are to be verified by the packet handling module  126 . If the field is not identified by the validation file  146 , then it is ignored by the packet handling module  126 . 
     The output definition file is formatted, in one implementation, identically to the input definition file  120 . Each line of the testing values section identifies a field to be tested in the receive packets. The line format is as follows: 
     
       
         name length type delta MIN MAX SET{ . . . } INCR 
       
     
     The output definition file  148  also has a state packet section in the packet definition section. Each line contains one of the state packets. The difference between the two files, however, is that the input definition file identifies multiple test packets whereas the state packet section of the output definition file  148  defines only one valid packet. Thus, each field in the state packet section only contains one value. 
     Once the compare received packet process  144  has analyzed the receive packets, the results are passed to a generate result, summary file process  150 . This process creates the result and summary file  152  that the tester reviews to assess the performance of the UUT  200 . 
     FIG. 5 is a flow diagram illustrating the UUT state control/validation process performed by the packet handling module  126 . Initially, the response or receive packet from the UUT  200  is analyzed by the packet handling module  126  in step  310 . It is then queried whether or not this response packet is from the expected initial state of the UUT  200 . If the UUT  200  is determined to be at its expected, initial state, then the state packets are sent from the state packet section of the packet definition section  124  contained in the input definition file  120 . This creates the required state for sending further test packets in step  314 . Alternatively, if in step  312 , the response packet is determined not to be from the initial, expected state, it is then determined whether or not the UUT  200  is in the state required for the test packet in step  316 . That is, it is queried whether the UUT  200  is already in the state for the test packets defined in the testing values section of the input definition file  120 . If so, then the test packet is sent in step  318 . Alternatively, an invalid state is indicated in step  320 . 
     FIG. 6 is a flow chart illustrating the overall operation of the protocol conformance testing system. First, the link to router  112  is configured in step  410 , Next, the packet handling module  126  accesses the input definition file  210 . The state of the UUT  200  is interrogated in step  414 , and the packet handling module  126  successively picks the state packets out of the state packet definition section of the packet definition section  124  of the input definition file  120 , passes the state packets to the send buffer  132  where they are transmitted to the UUT  200 . As described in FIG. 5, this process get the UUT to the required state. State transitions of the UUT are tested by analysis of the response packets. The response packets are stored to the result and summary file  152  by the compare receive process  144  and file generation process  150 . In parallel, the generate send packet process  130  creates the testing values packets from the testing values section of the packet definition section  124 . These are also placed in the buffer  132  for transmission to the UUT in step  418 . -The receive packets are then processed with the compare receive packet process  144 . Finally, in the preferred embodiment, after all the testing values packets have been sent, the UUT  200  is returned to its initial state by sending the required state packets in step  420 . 
     FIGS. 7-11 show a number of other bed configurations for the protocol conformance testing system. 
     In FIG. 7, the workstation  110  is configured to interface directly with the network on which the UUT  200  operates. Relative to FIG. 1, the intervening router  112  has been removed. This configuration requires that the combined workstation  110 /system  100  have the network interface card necessary to communicate on the network or via a console port of the UUT  200 , for example. Further, it complicates, somewhat, the construction of the network handling module since the particularities of packet construction and encapsulation must now be included in this module  134 . 
     FIG. 8 shows another test bed in which the workstation  100 , or system  110 , is connected directly to the UUT  200 . An additional router is also connected to the network. In this configuration, the UUT&#39;s behavior with the additional router  210  is confirmed. 
     FIG. 9 shows still another configuration in which two protocol conformance testing systems, workstations  110 A,  10 B, are used to test a single UUT  200 . In the specific illustrated embodiment, both systems  110 A,  110 B are connected directly to the UUT&#39;s network. 
     In the FIG. 10 test bed, two systems  110 A,  110 B are used to test two UUT&#39;s  200 A,  200 B. This test bed is useful to test interoperability, especially for new protocols. 
     FIG. 11 shows a test bed configuration in which the testing system  110  is connected to the UUT  200  via a network  114 . This system is useful for a manufacturer, for example, to test a UUT  200  at a customer facility. The intervening network  114  is conveniently the InterNet, for example. 
     FIG. 12 shows still another test bed configuration. In this example, the UUTs are two tag switching routers  200 A,  200 B with a mutual ATM link. This system is used to test ATM tag distribution protocols. For example, ATM TDP is using the path VPI=0, VCI=32 to communicate between two TC-ATM interfaces. This path (0, 32) and the TDP sessions get created automatically, when a TC-ATM interface is configured on both sides of the ATM link. In order to avoid the automatic creation of the TDP session, just one side of the ATM link is configured as a TC-ATM sub interface and other side is configured as a regular point-to-point or point-to-multipoint sub interface. The TC-ATM interface configuration already created an unidirectional path with VPI=0, VCI=32. So an ATM PVC is configured with the same VCD, VPI and VCI numbers at the point-to-point or point-to-multipoint subinterface. Now the workstation  100 ,  110 , pings the TC-ATM through the VPI=0 and VCI=32 path. The test bed is ready to open the ATM TDP. 
     In the above test bed, routing protocols such as OSPF will not work between TSR 1   200 A and TSR 2   200 B through the ATM link, because the routing protocols will not see them as neighbors. So if the routing protocol is OSPF, then configure ‘ip ospf network broadcast’ on both sides of the ATM link. The routing protocol discovers the TSR 1  and TSR 2  as neighbors from the broadcast messages. 
     While this invention has been particularly shown and described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.