WIDE Technical-Report in 2008

Gulliver Project : Building
Distributed Active Measurement
Appliances
wide-tr-mawi-gulliver-00.pdf

WIDE Project : http://www.wide.ad.jp/

If you have any comments on this document, please contact to ad@wide.ad.jp

Gulliver Project : Building Distributed Active
Measurement Appliances
Yuji Sekiya, Yohei Kuga and Kenjiro Cho
January 4, 2008

1

Overview

This paper describes the design and implementation of a plug-and-measureappliance and its remote management framework. We use a small diskless box as an active measurement appliance, and the remote management
framework is built into the ﬁrmware of the appliance. A management server,
by both a GUI or scripts, remote appliances can be instructed to preform
measurements, current appliance status, and apply ﬁrmware updates. As acase study of this system , we show long-term DNS measurements obtained
from the appliances deployed around the world.
The purpose of this project is to build a measurement appliance andframework which can easly be deployed in a developing world instructor.
The core goals of the framework are (1) distributing probe boxes all over
the world, (2) managing the boxes easily, and (3) getting measured results
easily. We call out framework the “Gulliver Framework”. In this section theoverview of the appliance and the framework is described.
In order to build a distributed active measurement framework, there areseveral requirements to achieve. The requirements for the framework are (1)easy installation and robustness of probe boxes, (2) security of the probebox and communication, (3) low costs for management, and (4) applicabilityof various active measurements.
In consideration of the above requirements, we have selected to use theSEIL appliance, rather then a PC as a probe box. SEIL is a commercial
product produced by the Internet Initiative Japan (IIJ) Inc, developed foruse as a SOHO router. The advantages of using this platform for us are: (1)the hardware has been proven to work for a 24x7 operation, (2) a remotemanagement system for a SOHO router use is already developed, and (3) we
can easily import future bug or security ﬁxes from the commercial version of
the product. We have customized the SEIL’s ﬁrmware in order to support
our measurement activies. Should the need arise we could easily port theframework to some other hardware.

1

Most of the existing measurement eﬀorts use PCs as measurement boxes.
In constast to a PC an applications: is cheaper to purchase, cheaper tosend, take up less space at the hosting institution, and is has a lower power
consumption. In addition SEIL’s disk-less storage means improved reliablity
in the face of unreliable power supplies, common some developing regions.
makes it a more superior choice given our target hosting sights.
As for security, the communication between the probe boxes and theservers should be encrypted. The probe box should accept only connections
from our speciﬁed servers. Even in the unlikely case that one one of thebox is physically stolen, the over all security of the framework should not
be threatened.
The probe boxes should be managed and operated in an integrated fashion by the management server. One of the goals of the project is to deploy
more than 100 probe boxes world wide. If a vulnerability is found in theﬁrmware or if we want to build new measurement tools into the ﬁrmware,
it should be upgraded in all the probe boxes without pains and troubles. Inaddition to the ﬁrmware upgrading, daily monitoring of operations should
be provided.
The ﬁrmware of our appliances is a derivative of NetBSD, so measurement tools which run on NetBSD can be easily be deployed in futureﬁrmware updates. This provides an easy mechanism for development anddebugging for measurement tools. The ﬁrmware also includes a remotemanagement system. All the appliances can be controlled and monitoredfrom the management server through the remote management system, called
SEIL Management Framework(SMF).

2Framework Design

The framework consists of probe appliances, management servers and measurement collection servers. We have designed the framework based on theclient-server model. The set of servers manage, control and monitor all probe
appliances. The appliance does not start any measurement action until it isinstructed to do so by the management server.

2.1

Architecture Goals

As described before, the appliance has the remote management system built
into the ﬁrmware. The Gulliver Framework has the following architectural
goals.
(a) plug-and-measure of appliances; even if the network environment
which the appliance is located in is changed and an administrator can not login the appliance remotely, the conﬁguration of the appliance can be changed
at the management server and just cycling power of the appliance, the ap-2

pliance will boot with new applicable conﬁguration. We want to distributeappliances without circumstance in developing countries.
(b) low operational costs and eﬀective measurements; when the numberof activated appliances is increased, the operational costs is not increased
proportionately. An administrator can control a number of appliances ascontrolling just an appliance. However, the measured results are increased
proportionately and an administrator can collect the results easily.
(c) secure and safe against snooping and cracking; the appliance talks tothe servers with authenticated and encrypted datagrams, and the measured
results also should be retrieved securely.

2.2

Appliance bootstrap

When the appliance is connected to a UTP cable and power source, it obtains
an IP address using DHCP, and then the appliance bootstraps as shown inﬁgure 1. The appliance computes its own unique ID from its MAC address.
The appliance knows only the IP address of a master server and the server’s
SSL certiﬁcate.
The master server’s only job is to authenticate the appliance from its
ID and tell the appliance about a second-level server. At bootstrap, theappliance contacts the master server using XML-RPC over SSL, veriﬁes theserver using the built-in certiﬁcate, and passes its unique ID to the server.
The master server checks the appliance’s ID, and returns the IP address andthe SSL certiﬁcate of a second-level server.
The appliance contacts the second-level server using XML-RPC over
SSL. The reasoning behind this server hierarchy is to allow a set of secondlevel servers for future scalability as well as to have completely diﬀerent
services for the appliances. The second-level server returns the appropriate
conﬁguration for the appliance, if the authentication is succeeded. The conﬁguration includes the SSL certiﬁcates of the servers, and the SSH public key
of a measurement collection server. The appliance keeps this conﬁguration
on memory, and reboots itself using this conﬁguration.

2.3

Appliance management

When controlling the appliances after a reboot, ﬁrmware upgrading or requesting status reports, an administrator does not need to send the requests
directly to every appliance. Rather, all he or she needs to do is send therequests to the second-level server as shown in ﬁgure 2.
For example, if an administrator wants to reboot Box B and C, she cansend a reboot request to the second-level server with the IDs of Box B andC, then the second-level server sends reboot requests to Box B and C.
When an appliance talks to a server, it uses XML-RPC over SSL. On
the other hand, when a server talks to an appliance, it sends a command3

Figure 1: The ﬂow diagram of boot- Figure 2: The ﬂow diagram of manstrap
agement
over SSH. In addition to these communications, appliances can periodicallysend heartbeats to a server; a heartbeat message includes the status of theinterfaces, CPU and memory so that administrators can monitor the statusof the deployed appliances.

2.4

Appliance measurement

Administrators can send start and stop requests for measurement programs
to the appliances in a similar manner as the appliance management shownin ﬁgure 3. When an appliance receives a start request from second-levelserver, it starts the measurement. When the measurement is completed, theappliance sends the results to the measurement collection server speciﬁedby the administrator.

Figure 3: The ﬂow diagram of measurement

43

Framework implementation

There are a number of challenges in building the appliances and framework.
In this section, we have described details of our implementation, especiallyabout our modiﬁcations to the original SMF and challenges encountered.
The ﬁrst challenege we faced was to reduce ﬁrmware memory usage.
The appliance has only 64MB memory. In order to save memory we deleted
unused functions for active measurements from the original ﬁrmware.
The second challenge was building a wrapper software to run the existingmeasurement tools without modiﬁcations. The wrapper was integrated into
the management system ﬁrmware.
The third challenge was handling static conﬁguration of IPv4 addresses.
The original remote management system, SMF only supports DHCP environments. This means that if a DHCP server is not available, the appliancecan not get IPv4 address or communicate with the master server. Depending on the policy of the instillation site this can be problematic, because
there exist situations were only a static IPv4 address is available. In orderto adapt to a static IPv4 address, we modiﬁed the ﬁrmware so that the IPv4address and default gateways could be pre-conﬁgured. If the IPv4 address
is pre-conﬁgured, the appliance communicate to the master server using theaddress, then it boots in a manner similar to a DHCP environment.
The fourth challenge was the to build a more user-friendly administrativeinterface. Thus reducing the cost to manage and monitor the appliances. Itis not user-friendly to send messages using XML-RPC with SSL for users
who would like to just run active measurements on this framework, so it isvery important to deploy and generalize the framework for various activemeasurements. The interface is shown in ﬁgure 4.

Figure 4: Web GUI for the managements

5

Figure 5: Details of the applianceIn the ﬁgure there is a list of the registered appliances. By clicking thename of appliance on the list, the detail of the appliance is shown as in ﬁgure
5. By clicking “reboot” button in the ﬁgure, the reboot request will be sent
to the appliance through the second-level server and the appliance will berebooted.

Figure 6: Monitoring memory usages Figure 7: Monitoring measurement
statusFigure 6 shows the memory usages of each appliance. The status ismonitored by heartbeats which every appliance send to the second-levelserver. Figure 7 also shows the status of the measured results collected bymeasurement collection server. As shown in the ﬁgures, administrators canmanage and monitor the appliances using web GUI.

4Measurements

As of October 2007, 18 appliances are distributed and activated. Table 1
summarizes the locations. Only one box is located in United States, while
the majority are located in Asian countries reﬂecting the project aims todeploy the appliances in developing regions.
Two measurement tools are implemented in the appliance at this time,dnsprobe and scamper. As a case study of the Gulliver framework, we
run two measurements and validate and prove advantages of the GulliverFramework. The results are described in this section.

5Project Status

A further directions of this study are (1) distributing more appliances all over
the world at least 100 boxes, especially developing regions, (2) implementing
6

Table 1: Location of measurement boxes
Box ID
12345678910111213141516171819Organization
WIDE
WIDERegistro.br
WIDE
WIDE
WIDE
WIDE
APNIC
NECTECAITJGN2The University of Tokyo
IIJ Inc.
Korea Telecom
MIMOS
CAIDA
University of Colombo School of Computing
University of Napoli Federico II
The University of Auckland

CityLos Angenes
Tokyo
Sao PauloTokyo
Osaka
Osaka
ParisBrisbane
Bangkok
PathumthaniTokyoTokyoTokyo
SeoulKuala Lumpur
San Diego
Colombo
NapoliAucklandCountry
U.S.A.
JapanBrazilJapan
JapanJapan
FranceAustralia
ThailandThailandJapanJapanJapan
Korea
Malaysia
U.S.A.
Sri Lanka
Italy
New Zealandmodule mechanism in the ﬁrmware in order to load and unload measurement
tools dynamically, and (3) enabling managements even if the appliance islocated behind NAT.
The status of gulliver project is available on http://gulliver.wide.ad.jp/

Copyright Notice
Copyright by WIDE Project 2007, 2008. All Rights Reserved.

7