Letters

Development of a Commercially Viable Besteffort Layer-2 Virtual Private Network System
Akira Saito, Masakazu Miyamoto†, and Kouichi Suto
Abstract
This article describes the development of edge routers and settings servers for adapting the existing
FLET’S network to provide a Layer-2 virtual private network (L2-VPN) service to small and mediumsized businesses and SOHO (small office, home office) users. This service can be easily provided byinstalling compact edge routers in users’ premises and settings servers within the service provider network. In contrast to earlier Wide-Area Ethernet services, which were designed mainly for use by large
corporations, this system can provide smaller users with an L2-VPN service at a low cost.

1. Virtual private networks

2. L3-VPN

Virtual private network (VPN) technology isdesigned to provide a virtual environment thatappears to the user as a dedicated line. Until now, themain design consisted of overlay model networks inwhich user nodes were connected by asynchronoustransfer mode (ATM) or frame relay (FR) technologies. Networks of this type have a number of advantages in that they support redundancy, a high degreeof reliability, and detailed quality of service (QoS)
control. However, they also have a number of problems. Routers are needed between each pair of nodes,
resulting in higher device costs and the need foradvanced installation skills, and it becomes extremely difficult to manage and maintain paths when thenumber of nodes increases and the network becomeslarger. This is because nodes must be linked by oneto-one connections. Moreover, the processing load isplaced mainly on the core of the communications carrier network. To address such problems, new VPN
technologies were developed using IP or virtual LAN(VLAN) communications. These are classified as
Layer-3 (L3) or Layer-2 (L2) VPNs according to thepacket transmission methods used within the network.

In an L3-VPN, the VPN is controlled on the basisof IP packet header information. A common example
of an L3-VPN service is an IP-VPN service using virtual router technology, or a combination of bordergateway protocol (BGP) and multiprotocol label
switching (MPLS) technology. A network of this typecan reduce the need to specify settings for individual
node routers and reduce the processing load placedon such routers because the path control from each
VPN node is terminated by the edge routers (ERs)located within the network. The processing load
placed on internal network routers is also reduced
because no path control information is released to the
network from individual user nodes, and dynamic
path control protocols can be used to provide flexible
redundancy for transmission paths. Such a network
also presents problems, however, in that ERs must
assume the majority of the processing load and only
the IP protocol may be used.

† NTT Access Network Service Systems Laboratories
Chiba-shi, 261-0023 Japan
E-mail: miyamoto@ansl.ntt.co.jp

Vol. 2 No. 6 June 2004

3. L2-VPN
In an L2-VPN, the VPN is controlled on the basisof Ethernet frame packet header information. A common example of an L2-VPN service is a wide-area
Ethernet service using VLAN-VPN technology. In a
network of this type, the entire communications carrier network is composed of Layer-2 switches, witheach switch and its associated VPN being identified
61

Letters

by a VLAN tag. It can use protocols other than the IP
protocol (e.g., the Microsoft Windows NetBIOS or
Macintosh AppleTalk protocol) because the network
as a whole appears as a single switching hub to eachuser node. Furthermore, there is no need for any
advanced skills in specifying operating settings
because there is no need to install a router on the userside. However, it provides very limited QoS control
capabilities and one problem is that the transfer tables
for each switch become extremely large due to nonhierarchical media access control (MAC) addresses
being used to control transmission paths. Care must
be taken in specifying switch path control settings
because there is no limit on the number of transfer
hops that can occur in the MAC layer, and it is difficult to build large-scale networks even when VLAN
tags are stored in stacks because there are restrictions
on the number of VPNs that can be identified.

constructed as an Ethernet network. There are otheradvantages too. The MAC address, which must be
assigned to each user device to perform L2-VPN control, can be automatically learned by the L2-VPNedge routers installed at each user node, and thelearned addresses are handled separately by each ER.
This makes ERs easy to operate and maintain andreduces the load placed on internal routers. This inturn makes this type of system superior for constructing large-scale networks.
5. Packet transfer method
The way in which packets are transferred in an
Ether/IP-VPN system and the format of ER transfer
packets used in such a system are shown in Fig. 1. In
this system, an ordinary IP network is used for transferring data between ERs, and Ethernet traffic istransmitted along the IP network via tunneling connections*1. When an Ethernet frame is delivered to an
ingress ER from its associated device, the ER gives
the Ethernet packet a VPN header to identify the
associated VPN, together with an IP header
addressed to the egress ER. It then transfers the packet to the network. Network-internal IP routers thentransfer the packet to the egress ER, handling IP pathcontrol independently of any user nodes. The egress
ER then removes the attached VPN and IP headers

4. Best-effort L2-VPN system
This system uses an Ethernet over IP-VPN(Ether/IP-VPN) protocol, which makes it possible toprovide Ethernet connectivity over a best-effort IPVPN network. An Ether/IP-VPN network offers all of
the advantages of both IP-VPN and VLAN-VPN.
More specifically, while the communications carrier
network appears as a single switching hub from theviewpoint of a user node, IP path control is actually
performed within the network. This design has several advantages that result from eliminating the need to
install routers at individual user nodes: it is easier tospecify operating settings, protocols other than the IP
protocol can be used, and it is possible to use existing
IP networks because the network itself need not be
Residence*1 Tunneling connection: A technique whereby path control headers
are added and deleted when packets are transferred across network
borders so as to make it possible to perform end-to-end packet
transfers over two networks with differing transfer protocols. For
instance, if two IPv6 sites are connected via an IPv4 network, then
IPv6 packets are encapsulated in IPv4 packets for transfer within
the IPv4 network.

Residence

Carrier network

IP address of ingress ER
IP address of egress ER…………………………
Payload C-L3 C-Ether

Payload C-L3 C-Ether VPN-ID P-IP
IngressER

Terminal

RouterRouter

Payload C-L3 C-Ether
EgressER

Application
C-L3 (e.g., IP)
C-Ether

Terminal
Application
C-L3 (e.g., IP)
C-Ether

VPN-ID

VPN-ID

P-IP
P-L2 (e.g., Ethernet, ATM, PPP)ER

VPN-ID
Reserved (not used)
Destination terminal MAC address
Source terminal MAC address
Ethernet type……………………………………
IP, NetBIOS,
AppleTalk, etc.

Provider IP(P-IP) header

VPN-ID
Reserved
Customer Ethernet(C-Ether) header

Customer L3(C-L3) header
Payload (user data)
FCS

Product development: best-effort L2-VPN edge router(a) Packet transfer(b) Packet format

Fig. 1. Ether/IP-VPN packet transfer and packet format.

62

NTT Technical Review

Letters

and transfers the Ethernet frame to its own associateddevice.

remote node terminal B within a wide area network(WAN), an entry for the MAC address of the senderof the packet (terminal A) is added to the incoming
packet transfer table. By storing table entries in thisway, the ER can recognize that terminal A exists at its
own home node when a packet from a remote node
within the WAN addressed to terminal A arrives at the
ingress ER. Once the VPN-ID has been checked andthe IP header that allowed the packet to be transferredalong the network has been removed, the packet canbe transferred within the LAN.
Conversely, at the egress ER, upon the arrival of an
incoming packet from remote WAN terminal A
addressed to LAN terminal B, the VPN-ID from within the packet header is checked and an entry is added
to the outgoing packet transfer table listing the MAC
address of the sender (terminal A) and the IP addressof the sending (ingress) ER. By storing table entries
in this way, the egress ER can recognize that terminal
A exists at the ingress ER node when a packet from ahome node of the egress ER addressed to terminal A
arrives. This information is used to attach an IP header for transmitting the packet along the network, and6. Automatic learning of ER MAC addresses
Each ER can automatically learn information about
which devices exist at any given ER node. This meansthat traffic is never directed anywhere except to theappropriate node within the same VPN, making it
possible to reduce the usage of network bandwidthand reduce the processing load placed on individual
ERs. There is also the advantage that the network as
a whole may be treated as a single switching hub
from the viewpoint of the user.
The automatic MAC address learning method isshown in Fig. 2. Figure 2(a) shows the processes that
automatically build a transfer table for incoming
packets from outgoing packets transmitted from ahome node, and Fig. 2(b) shows the processes thatbuild a transfer table for outgoing packets fromincoming packets from the network.
First, when an outgoing packet from home node terminal A within a local area network (LAN) arrives atUser
terminal A Source MAC address○○-○○-○○-98-3A-CF

Ingress ER
Interface 1

Network router
IP address 10.3.1.1

Input frame
Source MAC address
Destination MAC address
Payload○○-○○-○○-98-3A-CF○○-○○-○○-33-DF-BD
2. Use VPN-ID to select MAC (FIB) transfer table
MAC (FIB) transfer table
IP address of
Destination
VPN-ID destination
ER
MAC address○○-○○-○○-00-00-05
5
10.2.1.1○○-○○-○○-98-3A-CF
5
10.3.1.1
3. Add source MAC address to table(a) Automatic learning by ingress ER
Network router

Input
interface
1

VPN-ID5
1. Use input interface
to identify VPN-ID

Outputinterface2
1

Egress ER

User terminal B

Interface 101
IP address 10.1.1.1

PayloadDestination MAC address
Input frame○○-○○-○○-33-DF-BD
Source MAC address Destination MAC address
IP address of egress ER IP address of ingress ER○○-○○-○○-98-3A-CF ○○-○○-○○-33-DF-BD VPN-ID 5
10.1.1.1
10.3.1.1
1. Use VPN-ID to select MAC (FIB) transfer table Determine output interface tobe used depending on routing protocol
MAC (FIB) transfer table
Destination
MAC address○○-○○-○○-00-00-05○○-○○-○○-98-3A-CF

Outputinterface
102
101

IP address of
VPN-ID destination ER
10.2.1.1
5
10.3.1.15

2. Add source MAC address and IP address of ingress ER to table(b) Automatic learning by egress ER

FIB: forwarding information baseFig. 2. Procedure for automatically learning MAC addresses.

Vol. 2 No. 6 June 2004

63

Letters

the packet is then transferred along the WAN.
Once these transfer tables have been automaticallybuilt, data transmissions between ERs are performed
via unicast transfers*2. However, if the IP address ofthe destination terminal is known but no information
is available to identify the ER to which it belongs,
then an ARP (address resolution protocol) packet
used to resolve the MAC address from the IP address
is delivered to the ER from the terminal. In suchcases, the ER creates a number of copies of the packet equivalent to the number of remote nodes existing
within the same VPN, and broadcasts the copies to alluser nodes within the VPN.
To allow for the movement of terminals between
ERs, existing entries within the transfer tables may beoverwritten and updated using the header information
contained in newly arrived packets. In addition, toallow for the fact that there are restrictions on thenumber of entries that can be maintained in an actual
implementation, an aging time*3 parameter has been
provided to allow for the deletion of entries for which
there have been no communications over an extended period.

ing FLET’S network is shown in Fig. 3. Figure 3(a)shows the type of network that will be used when service is to be provided by an NTT service company inconjunction with a VPN service provider, and Fig.
3(b) shows the type of network that will be used whenservice is to be provided by an NTT service company
on its own.
In cases where service is to be provided by a VPN
service provider in conjunction with an NTT service
company, the VPN service provider will supply settings servers used to manage ERs, and all the otherequipment and facilities supplied by any ordinary ISP
to provide connectivity to the FLET’S network, andwill earn revenue by providing VPN connections
between users. The NTT service company will obtainits revenues from fees for using the FLET’S network.
In cases where service is to be provided by an NTT
service company on its own, the company will provide inter-LAN connectivity services in an Ethernet
layer while ensuring security between different VPNs
using FLET’S group access. The NTT service com*2 Unicast transfer: A type of communication targeted at a single
specified node. Opposite of multicasting.
*3 Aging time: Rather than being designed simply to store the learned
MAC addresses permanently, the system is designed to return tothe pre-learning state (i.e., to forget) after a specified amount oftime has passed since learning. Aging time is used to refer to how
long MAC addresses are stored in this way.

7. Types of services
The structure of the networks needed when an L2VPN system is used to provide services over an existSetting server

VPN provider
revenue
NTT service
company
revenueER
Premises

VPN provider(e.g., ISP)

NTE
NTE
FLET’S network

PC

PC

VPN#1 site1

ER
PC

PC

VPN#2 site1

NTE
NTE
FLET’S network
ER
PC

ER
PC

ER

PC

PC

VPN#2 site2

VPN#1 site2
VPN#2 site3
NTE: network terminating equipment(a) Structure of service network when service is provided by NTT conjunction with a VPN service provider
FLET’S network
NTT service
company
revenue

FLET’S group accessER
Premises

PC

PC

ER
PC

PC

ER
PC

ER
PC

PC

ER
PC

VPN#1 site1
VPN#2 site1
VPN#2 site2
VPN#1 site2
VPN#2 site3(b) Structure of service network when service is provided by NTT alone

Fig. 3. Structure of service network when using an L2-VPN system over a FLET’S network.

64

NTT Technical Review

Letters

L2-VPN
Development ER
Bridgecapability

NTE

PPP

NTE

VPN tunnel

PPP

Development
ER

FLET’S
network

VPN provider

FLET’S network
Routercapability
FirewallUser B (office B)

PPP

Physicalinterface

NTE

Internet

ISP

Content server

User A (office A)

Fig. 4. Example of a system providing simultaneous access to the Internet and to an L2-VPN network.

pany earns revenues from fees for using the FLET’S
network, FLET’S group access, and L2-VPN service.
8. Use for simultaneous Internet access
If two separate PPPoE (point-to-point protocol over
Ethernet) sessions are used, then the ERs may be used
to provide simultaneous access to a VPN and theInternet. An example of such a system is shown in
Fig. 4. In a system such as this, it is possible to provide Internet access for multiple users over a single
VPN by obtaining an Internet account for only a single node within the VPN and allowing users at othernodes to access the Internet through the VPN. Notethat to allow for Internet connectivity the system has
been designed with a router, and to provide security
against illegal access to VPNs from the Internet it has
been given firewall capabilities.

suitable for use in the home. Settings servers may becreated by installing the applications software wehave developed on a commercially available PC. ER
settings may be specified either directly or from aserver. In the direct method, ER settings are specified
over a telnet connection from a LAN terminal. Whenspecifying settings from a server, ER settings are firstrecorded in the settings server and the settings arethen downloaded from the server when the ER is firststarted up. It is also possible to force the settings to be
downloaded whenever it is necessary to do so. An
example of a system using server-specified settings isshown in Fig. 6. Note that when the server settings

30 mm

12

6

9. Edge routers and settings serversmm

An edge router is shown in Fig. 5. It is more compact than an ordinary broadband router, making it

110

mm

Fig. 5. Appearance of L2-VPN edge router.

VPN provider(operated by any
available ISP)

Internet

Setting server• L2 VPN setting• Internet setting• Setting modification notification

L2-VPN ER• PPPoE client capability• L2VPN capability• IP router capability(Internet connectivity)• Remote management and maintenance(works in conjunction with setting server)

NTE
FLET’S
network BRAS
M/C

NTE
FLET’S
BRAS network
M/C
Setting

M/C

M/C
FLET’S contract(static IP)

OfficeFactory

BRAS: broadband remote
access server
M/C: media converter

Fig. 6. Handling of edge routers by settings server.

Vol. 2 No. 6 June 2004

65

Letters

method is used it also becomes possible to specify
user settings or perform isolation testing centrally
from a single remote location.
10. Future issues
In this article, we described how we developed i)
settings servers that can be used to manage edge
router settings remotely from a single location and ii)
compact edge routers designed for use in the home,which can be used to provide Ethernet access over an
IP-VPN network. They make it possible to provide a
low-cost best-effort wide-area Ethernet service tosmall and medium-sized businesses and SOHO users.
While this completes the work necessary to demonstrate that this is a commercially viable system, several issues remain. These include improving the easeof operability of the system and improving the userinterface.

Akira Saito
Chief, Research and Development Center,
NTT East Corporation.
In 1997, he joined NTT Access Network Service Systems Laboratories, Makuhari, Japan.
From 1991 to 2003, he was engaged in R&D ofhigh-speed downloading using multiple TCP
connections and system engineering of carrier
access networks. He moved to his present position in November 2003.

Masakazu Miyamoto
Access Service and Network Architecture Project, NTT Access Network Service Systems Laboratories.
He received the B.E. degree in electrical engineering and the M.S. degree in computer sciencefrom the University of Yamanashi, Kofu,
Yamanashi in 1997 and 1999, respectively. In
1999, he joined the NTT Access Network Service Systems Laboratories, Makuhari, Japan.
From 1999 to 2003, he was engaged in development of ATM access network systems andresearching dynamic traffic engineering on
IP/MPLS networks. Since 2004, he has been
improving maintenance and operation after
development of L2-VPN system.

Kouichi Suto
Senior Research Engineer, NTT Access Network Service Systems Laboratories.
He received the B.E. degree in electrical engineering from Iwate University, Morioka, Iwate
in 1977. Since joining NTT Laboratories in 1977,
he has been active in developmental research on
optical fiber trunk transmission systems and optical fiber subscriber transmission systems. From1999 to 2002, he was engaged in development ofan access control server system for IP-VPN service. Since 2002, he has been engaged in developmental research on access network services
using ADSL and/or FTTH systems. He received
the Electronics Letters Premium from IEE in1978. He is a member of IEEE.

66

NTT Technical Review