2. Focused Research (1)Running an Overlay Network in a Multi-tenant Setup—Challenges of IIJ GIO Infrastructure P2 Gen.2
2.1 P2 Gen.2—IIJ’s new-generation IaaS

long as customers want to use them. The development ofOn October 1, 2021, IIJ launched IIJ GIO Infrastructure P2

new systems based on cloud technology under the banner

Gen.2 (P2 Gen.2 for short), a new IaaS that fully combinesof DX (digital transformation) is on the rise. But maintaining

and takes advantage of the characteristics of the public

and modifying existing systems to support DX is also aIaaS and private IaaS developed and provided under the IIJ

key mission for enterprise IT teams. We developed P2

GIO brand. P2 Gen.2 is a new-generation IaaS designed toGen.2 to address this challenge by providing a third op-

confer the full benefits of public clouds while also taking

tion for cloud computing that can not only accommodate

advantage of VMware vSphere-based private clouds.

on-premise systems and private clouds that remain in
operation but also make it easy to migrate from public

To ensure customers can continue using P2 Gen.2 into theclouds as well as.

future, we adopted an architecture geared toward the next
generation of services. This report looks at the development

P2 Gen.2 is an IaaS offering that allows customers to useof the P2 Gen.2 service and the ideals and reality of its

resources in the form of flexible server resources. The

operation in the field with a focus on the network.

resource pool is virtualized, customers focus on system
operations within the resource pool, and IIJ runs all of the16

Our aim in developing and running P2 Gen.2 is to provide afunctions in the layers below it. With IaaS run in individual

service platform that will continue to be useful a decade or

VM instances, the size of the VM is made to conform tomore from now. Today’s rapidly changing world makes it

the instance model specified by the cloud service provider,

impossible to predict what will happen 10 years ahead. Yet

but with P2 Gen.2, customers are free to create VMs in

we believe that services should remain available for as

the resource pool. This means that customers can transfer© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

2. Focused Research (1)the specification of the machines running in their own

layer from users. This made it possible to offer resource

environments as is. Naturally, this design makes it possible

control permissions that are as flexible as with vSphereto bring in images or migrate VMs from the existing

while allowing IIJ, as service provider, to take on the hy-environment as is via V2V and P2V migrations.

pervisor and hardware lifecycle management role. Defining
a new joint-responsibility model like this made it possible

P2 Gen.2 provides an image provision mechanism forfor IIJ to manage and operate the hypervisor network.

deploying VMs; backup, network, and migration functions; andWe adopted VMware NSX-T Data Center (NSX-T), which

file servers for storing files. The hypervisor and hardware

enables integration with vSphere, to run the hypervisor

such as servers, storage, and networks are abstracted inlayer network efficiently, and this significantly improved

a form that customers are able to visualize, so they do not

the IaaS network. With P2 Gen.2, in Layer 3 an NSX-T

need to worry about these elements. This eliminates theoverlay network sits on top of an IP fabric underlay net-

workload encountered when renewing the hypervisor andwork, and each tenant’s network is completely separated

hardware that results from handing ESXi directly over tofrom the others and provided as a VPC (Virtual Private

the customer, something that had been a major issue withCloud). Combining this with the operational knowledge inthe Virtualization Platform VW Series, a vSphere-basedlarge-scale server pools we amassed through IIJ GIO, weservice provided on IIJ GIO.

have made it possible to allocate resources to users in amanner independent of the allocation of physical computing

We adopted VMware Cloud Director (vCD), a VMware prod-resources (CPU, memory, storage) (Figure 1).

uct for service providers, and hid the hypervisor (vSphere)

Customersite
IIJ Private
Backbone ServiceInternet

Gateway
Customer

VPC

LB
VM
Virtual server

VM
VM

VM
VM

Virtual server resource pool
Virtual CPU / memory
Virtual storage

IIJ GIO

Hypervisor VMware
Physical server resource pool
Physical storage resource pool
Physical network resource pool

Figure 1: Flexible Server Resources Overview© Internet Initiative Japan Inc.

17

2.2 Overlay Networks Using SDN Technology

it possible to manage the configuration centrally. This

Overlay networks are nothing new. Along with VLAN

creates an environment in which administrators can auto-(Virtual LAN), they are a long-standing network virtualization

matically configure devices according to how the network

method. In the past, they were often used for point-to-point

is defined, instead of having to change the configuration ofconnections due to protocol specifications/constraints andeach individual device, so configurations can be changed

for operational considerations, but with the evolution ofquickly to meet the requirements of large-scale networks.

SDN (software-defined network) technology, they can now

This makes it possible to have more than 4,096 network

be used in large-scale networks. Encapsulation protocols

segments, the limit imposed by IEEE 802.1Q VLAN,

like L2TPv3, VXLAN, and Geneve are used to build virtual

something that has posed issues in large-scale networks

L2 networks. SDN technology uses software to control thethat use many segments. It also enables the efficient

networks, and is made up of the control plane, which

control of north-south traffic (traffic between devices andcontrols what happens between devices, and the datathe network center) and east-west traffic (traffic amongplane, which forwards the actual frames and packets. Usingdevices) within virtualized networks (Figure 2).

software to control what happens between devices makes

External

Overlay Network

Edge Leaf

North-South

Spine

Leaf

East-West

Figure 2: Overlay Network in a Large-scale Environment

18© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

2. Focused Research (1)2.3 Benefits of Overlay Networks with VMware
NSX-T

IaaS solutions need to be multi-tenant (accommodate

Along with server virtualization technology, network

via software is also needed in order to provide functions

virtualization with overlay networks is one of the core

separately to the tenants and to maintain security andtechnologies underpinning the P2 Gen.2 infrastructure.

the specified service level. Network virtualization makes

Service networks can be configured separately from theit easier to link together the other components of theunderlay network that lives on the hardware layer. Using

IaaS solution as well as monitoring systems, facilitating

NSX-T for network virtualization on the IaaS layer of P2

proper control of a multi-tenant environment in which

Gen.2 provides a flexible network configuration and loose

each tenant operates their system in accord with theircoupling with the physical layer. When setting up networks

system requirements. NSX-T also implements VRF (virtualwith a view to expansion from minimum scale up torouting and forwarding), and VRF can be used to enhance

hyperscale, they tend to become complicated in terms

equipment efficiency. In multi-tenant environments, theof physical device specs because design and sizing need

need to divide up routing domains among the tenants posed

to be chosen in anticipation of that expansion. But beinga resource constraint. VRF makes it possible to increase

able to control expansion via overlays confers considerable

infrastructure consolidation.

multiple customers). Centralized control and automation

benefits on the operational front.
Interface control is centralized into NSX Manager, NSX-T’s
An IaaS also needs to handle multi-site configurations

management component, making API-based operations andacross multiple data centers. Using infrastructure acrossautomation easy. This also helps to reduce development

multiple, physically separated sites facilitates disasterhours spent on making the service’s backend applications

preparedness and makes it possible to obtain the necessary

work together.

computing resources without needing a large-scale
facility. Virtualizing the network between sites and making

When an overlay network is set up as an IaaS solution,it an overlay network enables you to configure networks

network exits are needed. NSX-T also cannot communicate

independent of their location. Physically separated

outside the NSX-T domain (under its control) with theenvironments involve more considerations to take into

NSX-T encapsulation protocol Geneve, so to communicate

account than with single-site configurations, such aswith the external world, you need to convert it to a VLAN.

network latency and bandwidth between sites, and thisWe had not fully automated external connections withincreases design and operational difficulty, but ensuring

IIJ GIO, and a major issue was that users were unable toproper resource monitoring and control of systems basedchange network configuration in a complete form whenon the monitoring data makes it possible to set up flexible

migrating systems. P2 Gen.2 improves this situation bynetworks.

automating route configuration for connections with theIIJ Private Backbone Service. NSX-T can also be run in a

Network virtualization remains uncommon despite thesebare metal configuration with NSX Edge installed directly

benefits. With traditional networks where the configuration

on a server with no virtualization, and while this is finewas expected to remain static, administrators were ablein performance terms, it is not really up to the task ofto continue maintaining and running systems without themulti-tenant configurations. Many challenges also present

benefits of network virtualization. In recent years, however,themselves particularly with external connections, suchthere is a need for IT infrastructure as a whole to adapt

as downtime during software upgrades. So many necessary

flexibly and swiftly to rapidly changing and unpredictableareas of improvement remain, and we continue to workrequirements. Network virtualization will likely become more

on resolving these issues.

and more important in this landscape.

© Internet Initiative Japan Inc.

19

2.4 Operational Issues and Solutions

conventional network engineers and server engineers—

Ahead of the October 1, 2021 release, in 2020 we conducted

and instead, the engineers need to be familiar with thea small-scale rollout of the NSX-T SDN-based overlay

entire technology stack used in building the infrastructure.

network, along with NSX-T’s NFV (network function

When we started out, our team was small but included

virtualization) capabilities, for some data center functions.

people with a strong knowledge of VMware technologies

This revealed a whole host of issues, which we continueand people with experience running large-scale service

working to address as part of our operations. Below, weinfrastructure, so as a team, we had the capabilities togo over the issues we discovered and their solutions.

handle ongoing development and operations. The mostimportant part of getting engineers with differing areas of20

Creating an overlay network using SDN requires a moreexpertise to work both autonomously and in collaboration

complicated configuration than with conventional networks.

with each other as a team is to have a vision for yourThe configuration has two layers: the physical layer (under-products and services and strive continuously to makelay) and the IaaS layer (overlay, logical layer).

them better based on the DevOps philosophy.

Another point is that to get the most out of the NSX-T

With P2 Gen.2, we revisited infrastructure control fromarchitecture, you need to be well-versed in vSpherescratch to ensure it would work the way we needed it

operations, so in addition to networking, the engineers

to. We put as much effort as possible into streamlining

also need to be able to work with server, hypervisor,operations and automation. Under DevOps principles, weand virtualization technologies. This is a departure frombuilt a Git-based CI/CD pipeline, and the engineers

the past practice of specializing in specific areas—à lacontrol the majority of elements that make up the IaaS,© Internet Initiative Japan Inc.

Feb.2022

Vol.

53

2. Focused Research (1)not just the VMs, under a policy of making changes to thephysical equipment involves delivery lead times, andoperations and configuration of the infrastructure without

setting up physical equipment in the data center is notdirectly manipulating the environment (Figure 3).

something that you can do via software control alone.
Configuring facilities involves some level of physicalWhile there is a lot of knowledge and software out there

human interaction, so making sure the associated leadfor controlling VMware products, the de facto standard

times do not become a bottleneck is key. Having anin server virtualization, it is not quite adequate for totalabundant inventory of resources can solve this issue, but

control of multi-tenant infrastructure in the form of athis then leads to inefficiencies because the overall

SDDC (Software-Defined Datacenter), so some of theresource utilization rate falls. Also, out-of-resources

tools and libraries we use were developed from scratch.

conditions are unacceptable in cloud services. So resource
management, monitoring, and demand forecasting are

We are not only simply carrying out operations tasks

crucial. Monitoring and demand forecasting should bethrough these efforts. We have also laid the groundwork

done not just on the overlay, it should also aggregate

for implementing SRE (site reliability engineering).

and facilitate visualization of the various underlay data
sources, including servers, network equipment, and storage.

Taking on board agile values and​​methods like this canSince the October 1, 2021 release, we have been

facilitate more flexible and agile infrastructure operations,collecting and analyzing data on actual usage conditions

which can otherwise tend to be fixed and inflexible, but

to enable just-in-time resource inventory deployments

you also need to be keenly aware of the opposing relationshipbased on demand forecasting.

between agile principles and facility operations. Procuring

Figure 3: The IaaS Components are Managed via Git© Internet Initiative Japan Inc.

21

2.5 Looking Ahead

will allow IIJ to deliver high-quality services on edge nodes

P2 Gen.2 still only has the minimum functionality, and wecloser to where the customer is actually located. In addition

plan to continue developing it to serve customer needs. The

to conventional networks, 5G mobile networks and the likeuse of computing resources in the form of cloud-native IT

can also be used to connect the resources. Linking the coresystems has become commonplace, and a more agile digitaland the edges requires not only VMware-based workflow

infrastructure is now being demanded. Our first priority ismanagement but also application container-based resource

to implement the features needed to ensure that customers

management, so also providing resources in the form of

will continue to use the service, but we also want to be able

Containers as a Service (CaaS), allowing applications to beto address future needs beyond that.

run inside containers, might be the optimal solution forcustomers (Figure 4).

In addition to migrations into the cloud, the need for edge
computing and distributed computing that deploys computing

Software control that brings the network closer to theresources close to where users are is also rising. IIJ’s role isstatus of an application is needed here. This goes beyond

to connect the cloud at the core, the distributed edge nodes,

just resource services that provide VMs, and I think it canand the network in between. These interconnected elements

be achieved by evolving the network from the current

Autonomous driving
and connected cars

5G
network
APP

loT gateway

APPAPP

APP

East Japan region
data center cluster

Shiroi DCC

APP

Factory loT

5Gnetwork

Industrial devices

Smart cityloT gateway

5G
network

APP

APP

Existing
IP network

Existing GIO sites
Figure 4: Looking Ahead

22

DR services

IIJ Backbone Network(Wide area / secure)© Internet Initiative Japan Inc.

West Japan region
data center cluster

Matsue DCP

Feb.2022

Vol.

53

2. Focused Research (1)architecture into something that is even less dependent

To make this sort of world a reality, we are also working

on physical resources. This will be difficult to accomplish

on a project to use the technology stack we have developed

simply by deploying resources in a compact form that can

not just for data center purposes but also as an edge

also be deployed on the edges. You also need to be able

computing platform. We have set up a micro data centerto configure the network flexibly so that it works together

outdoors at IIJ’s Shiroi Data Center Campus, and we arewith the other resources.

conducting tests in the area of cloud and edge computing
as well as multi-access edge computing (MEC), with our

Personally, I am interested in SmartNICs as a technology

objective being to develop the technology further along

suitable for this sort of edge computing and distributedwith local 5G, IoT, and other use cases. My hope is thatcomputing architecture. VMware is working on Projectevolving network services in this way will make it possible

Monterey, which runs ESXi on Arm-processor based

to provide customers with unique infrastructure of theSmartNICs. The hope is that managing SmartNICs as NSX-T

sort that only IIJ can deliver.

domains will make it possible to control and deploy networks
in a manner that makes them equivalent to applications.

Takehiro Yamamoto
Infrastructure Service Design Manager, Cloud Services Division 1, IIJ System Cloud Division.
Mr. Yamamoto is engaged in the planning, development, and operation of cloud-related services.

© Internet Initiative Japan Inc.

23