Letters

Network Conformance Test System for IP
Networks
Nobuo Takagi, Yukio Nagafuchi†, Kouji Morishita,
Kiyoshi Yanagimoto, and Shin-ichi Kuribayashi
Abstract
The use of IP networks has recently been expanding from the simple transfer of data to the handlingof voice and video signals and to communication within and between business enterprises. Evaluating
the performance of such complex IP networks requires sophisticated expertise to comprehend protocol
mechanisms and test results. Moreover, as the scale of a network increases, the effort of analyzing theresults will increase exponentially. Here, we introduce a NCT (Network Conformance Test) system thatoffers test scenarios including sophisticated expertise, so operators do not require these skills. Moreover,this system can perform several kinds of tests automatically, changing network and measurement equipment configurations, generating and measuring traffic, and analyzing test results, which greatly reduces
the workload on the user.

1. Network conformance test
The network conformance test (NCT) system isused to verify proper operation of the IP network
when multiple networks are interconnected prior to
the beginning of service and to test the limits of network performance. The system automatically performs a series of processes, including configuring
network equipment (i.e., routers and switches) andmeasurement equipment and performing traffic generation, observation, and analysis, according to a scenario written as a time series. Multiple tests can be
run consecutively by preparing multiple scenarios. So
if the user has many choices of network settings, theNCT system lets him/her choose the best setting.
The system can generate traffic that ranges from
simple traffic for confirming the operation of protocols to the complex traffic that arises when multiple
networks are interconnected. Furthermore, theadvanced testing expertise that is incorporated into
the traffic analysis function makes advanced analysis
available even to users who do not possess such
† NTT Information Sharing Platform Laboratories
Musashino-shi, 180-8585 Japan
E-mail: Nagafuchi.Yukio@lab.ntt.co.jp

82

expertise.
Tests that conventionally require multiple hardware
systems and testing personnel in 24-hour continuous
testing can be performed with small equipment andpersonnel requirements with this system. This makes
it possible to shorten the time until the beginning ofservice, reduce the cost of facilities, and construct astable network environment backed by test data. One
technical point of this system is that the test flow isbased on scenarios that include sophisticated expertise, and it automatically performs tests, analyzes
them, and draws test result graphs. A scenario caneasily be reconstructed by the user.
2. Problems in IP network evaluation testing
The router is the key component of an IP network.
It does more than simply transfer packets; it also performs message processing for the exchange of therouting information contained in routing tables, computational processing for route optimization based on
routing data exchanged with other nodes, and encapsulization processing for encryption and destination
address concealment. In addition to these functions,the router collects various kinds of statistical data foruse in network management. In ordinary routers, theNTT Technical Review

Letters

w The test scenarios are input via a Web interface.
Here, events are described in terms of time
elapsed since the beginning of the test. The events
that can be described include the test start time,
the onset of load generation, the cessation of load
generation, the end of measurement, and changes
in settings.
e The load generation patterns are described. For
packet transfer load, large-volume loads are generated by controlling commercial measurement
devices such as the SmartBits 2000 (Spirent),
IXIA 1600 (IXIA), and AX/4000 (Spirent). The
addition of the general-purpose workstation
makes it possible to generate loads that havebiased distribution characteristics (the number ofsending and receiving IP addresses and the number of their combinations, the frequency distribution of sending and receiving IP addresses, thepacket length distribution, total bandwidth, etc.),
which is difficult to do with commercial measurement equipment. For the message processing
load, the protocol classification is selected. The
available protocols are I-BGP (Internet Border
Gateway Protocol), E-BGP (External BGP),SNMP (Signaling Network Management Protocol), and PPP (Point-to-Point Protocol). When arouting information load is to be applied, we set
the id numbers of the autonomous systems, theAS path, network address, and network mask foruse in exchanging information. When an SNMP
load is to be applied, the management information base tree address and continuous load speed
are set. For a PPP processing load, the authentication load for PPP over L2TP (layer 2 transfer
protocol) can be confirmed. The password error

processing for packet transfer, encapsulization, andmessage handling is all implemented on the sameprocessor, so the processing load has a large effect on
the packet transfer performance. Measurement
devices for evaluating these router functions areavailable, but simultaneous measurement of multiple
items and automatic testing over a long period of time
has not previously been possible.
Although the evolution of IP networks has centeredaround the Internet service provider (ISP) in the past,telecommunication companies and system integrators have appeared recently to provide means oftransmitting voice and video data on IP networks andusing IP networks for communication within business enterprises. Furthermore, IP networks haveattained a global scale through the interconnection of
network enterprises, and failures that occur in networks often propagate to other networks in the farcorners of the world. This situation has created theneed for a system that can support and automate theperformance evaluation of IP networks by modelingthe traffic of large-scale networks assuming network
interconnection and running multiple types of load
tests simultaneously and multiple tests consecutively.
3. Procedure employed in the NCT system
The NCT system (Fig. 1) was developed to solve
the problems described above.
q First, the test configuration, which includes thesimulated network that is being tested, is studied.
A LAN is set up to control the measurement
equipment. A general-purpose workstation and a
Windows terminal with a Web browser are connected to the control LAN.
Control workstation (WS)Test setting file(configuration, items,
procedure)

Resultanalysis module

Test condition
setting module

RecordingmoduleTest control
moduleTransfer load generation module
Load generator (WS)

Result analysis
moduleTest result display
moduleLong-term observation module

Access system load generation module
Load generator (WS)
Routing information load generation module
Load generator (WS, Windows-PC)

Web interface (Windows-PC)
GUI test
condition
input
moduleMeasurement equipment (WS)

Network

Large-volume load generation control module
Load generator (WS)Counting observation
control module
Measurement equipment (WS)Counting observation
moduleCommercial testing equipment

Large-volume load generation module
Commercial testing equipment

WS: workstation

Fig. 1. NCT system.

Vol. 1 No. 6 Sep. 2003

83

Letters

rate and continuous load speed for PPP authentication are set.
r The flow of attention to be used in the measurements is set. Specifying this attention flow makes
it possible to evaluate fluctuations in the transfer
speed, transient loss, and path convergence time
under simulated failure. It is also possible to measure the encapsulization processing performance
because the delay for various packet lengths canbe analyzed at the same time.
4. Main features of the NCT system
The NCT system automates and increases the efficiency of the entire conventional verification procedure from measurement equipment setup to result
analysis, as shown in Fig. 2. Conventionally, the testing work begins after test items have been identified,
but with this system, it begins after a test scenario has
been created. The testing work is executed automatically from the completion of the test system configuration up to the analysis. Thus, work that took 400
hours in the past can be accomplished in 14 hours, forexample. It is also possible to perform unmanned
testing that includes simulated failures, a task that
required manual intervention in the past. The ability
to change the settings of commercial measurement
equipment also makes automatic changing of load
patterns in long-term stability testing easy.
The features of the NCT system are summarized
below.
q Because the settings of the equipment under test
Study of evaluation items andmethods

and the measurement equipment are changed
automatically according to the scenario, it is possible to run consecutive tests with different parameters and tests of different items automatically
24 hours a day.
w The evaluation of message processing, computation, and encapsulization loads, which is difficult
to do with existing measuring equipment, can bedone by software load generation of the NCT
traffic generator.
e Specifying changes to be made in the testing
environment settings over time after the testinghas begun makes it possible to repeatedly perform tests that simulate transport path failuresand other such transmission path interruptions
during communication.
r Because there is a function for graphing theagreement conditions of test packets, the test
result data can be analyzed from various points ofview.
t Because tests can be done repeatedly with various
network configurations, it is easy to perform
comparative studies of network configurations
that shorten recovery time after failure and investigate aspects of performance degradation with alarge number of users to clarify the criteria for
deciding whether to expand facilities.
y Automatic control of commercial network measurement equipment and cooperative testing ispossible.
u The system can be implemented by software running on multi-purpose workstations or personal
computers rather than on special dedicated equipment.
5. Example of IP network evaluation using theNCT system

Study of test configuration
Study of test conditions
Study of measurement method(Work load studies)
Measurement equipment settingsand system configuration
Execution of testing
Result collection

Automated testing work that
previously done manually

Result analysis
Evaluation of analysis result

Fig. 2. Automation with the NCT system.

84(1) Evaluation of the number of paths and routertransfer performance
The router has a cache for storing packet destinations, but if the volume of destination data (i.e., thenumber of paths or the number of BGP lines) is large,
the time required for looking up the destination
decreases the transfer performance (forwarding ratio= number of outgoing packets (number of incoming
packets). That tendency increases as the incoming
packet rate increases. An evaluation example isshown in Fig. 3.
(2) Evaluation of IP network topology
For networks other than IP networks, a mesh topology generally has the highest performance against
NTT Technical Review

Letters

Forwarding ratio (%)

failures. For IP networks, the routing information is
updated if a router fails. The path information propagates on the network, so the convergence time of thepath information varies with the network topology.
The NCT system can evaluate the routing information convergence performance by simulating failures
within the connected networks and the traffic transactions at the time of failure. The evaluation results
show that a parallel topology is desirable from theviewpoint of robustness of performance against failures in an IP network (Fig. 4).
(3) Evaluation of router packet loss rate based onpriority and packet length
The results of an investigation of how the numberof discarded packets is affected by priority settingsand differences in packet length when the router isunder a high load are presented in Fig. 5. The router
behavior with respect to priority control and othersuch processing varies with the type of machine, so it
100
98
96
94
92
90
88
86
84
82
80
3
,63 32 214 22,6 0,63 ,633 34 4
3 36 2,6 ,63 33
4 50 9,6
5
Number of paths(number of BGP lines)

is necessary to verify operation in advance. The
groups in the figure indicate priority (the larger thenumber, the higher the priority). For the example
shown in Fig. 5, we can see that no 1518-byte packets belonging to the highest priority group were discarded by the router at the time of congestion in theinterval from 20 to 30 ms.
(4) Evaluation of priority control
An evaluation of the priority control effectivenessof the router and the network is shown in Fig. 6,
where the groups indicate priority classes. From thisexample, we can understand the delay distribution forincoming packets that have different priorities as they
pass through the network and the router. The traffic
distributions are shown by packet size and delay time
for a test traffic flow that involves mixed protocols
with priority control set at the router. These results
reveal that router packet processing is not constantwith respect to traffic pattern and packet size, even(a) Parallel(b) Mesh
9,000
18,00025,500 Transfer performance(packets per second)Fig. 3. Example of evaluating the number of network
paths and packet transfer performance.

Fig. 4. Example of network configuration evaluation.

Number of packets discarded

700,000
600,000Size(byte)

500,000

1,518
1,024
512
256
128
64

400,000
300,000
200,000
100,000
0

0
1
2
0
1
2
0
10
20
Upper: group (priority)
Lower: time elapsed from beginning of test (ms)

1
30

2

Fig. 5. Example of evaluating packet discarding rate using the NCT system.

Vol. 1 No. 6 Sep. 2003

85

Letters

Total: proportion
100%

Ratio for each group

90%
80%
70%
60%

Group

50%

2
1
0

40%
30%
20%
10%
0%

10 20 30 10 20 30 10 20 30 10 20 30 10 20 30 10 20 30
64

128

256
512
1,024
Upper: delay time (ms)Lower: packet size (byte)

1,518

Fig. 6. Example of evaluating the effectiveness of network and routerpriority control.

6. Future development
The functions shown in Fig. 1 havealready been developed. Our futuresystem development plans include
evaluating online performance after
service has begun, automaticallydetecting attacks on the network(expanding into the field of security),
detecting predictors of service quality degradation or failure, and identifying bottlenecks.

Nobuo Takagi
Senior Research Engineer, NTT Information
Sharing Platform Laboratories.
He received the B.E. and M.E. degrees in engineering from Nagoya Institute of Technology,
Nagoya in 1987 and 1989, respectively. He
joined NTT Switching Systems Laboratories in1989 and has been engaged in the design anddevelopment of voice encoding for ATM and IP
networking structures. He is a member of theInstitute of Electronics, Information and Communication Engineers (IEICE).

Kiyoshi Yanagimoto
Research Engineer, NTT Information Sharing
Platform Laboratories.
Since joining NTT in 1983, he has been
engaged in research on IP networking structureand service quality measurement. He is a member of IEICE.

Yukio Nagafuchi
NTT Information Sharing Platform Laboratories.
He received the B.S. and M.S. degrees in science and engineering from Saga University, Sagain 1996 and 1998, respectively. He joined NTT
Network System Service Laboratories in 1998
and has been engaged in the design and development of the IMT2000 system, VoIP network systems, and traffic engineering systems. He is amember of IEICE.

Shin-ichi Kuribayashi
Senior Research Engineer, Supervisor, NTT
Information Sharing Platform Laboratories.
He received the B.S., M.S., and Ph.D. degrees
in engineering from Tohoku University, Sendai,
Miyagi in 1978, 1980, and 1988 respectively. He
joined NTT Electrical Communications Laboratories in 1980. He has been engaged in the designand development of DDX and ISDN packet
switching, ATM, PHS, IMT2000 and IP-VPN
systems. He researched distributed communication systems at Stanford University from December 1988 through December 1989. He participated in international standardization activities on
ATM signaling and IMT2000 signaling protocols at ITU-T SG11 from 1990 through 2000. He
is now responsible for the next-generation IP system in NTT Information Sharing Platform Laboratories. He is a member of IEICE.

Kouji Morishita
NTT Information Sharing Platform Laboratories.
Since joining NTT in 1985, he has been
engaged in research on IP networking structureand service quality measurement.

86

with priority control. This shows theneed to conduct tests with various
traffic patterns before the beginningof actual network service to understand the characteristics of the network and its devices.

NTT Technical Review