3. Focused Research (2)IIJ and the Evolution of Data Centers—Commemorating 30 Years
3.1 1990s—It Started with the Effective Use ofSpace

ISPs use a huge number of dedicated lines and telephone

Even before the privatization of the Nippon Telegraph andwithin communication stations and connect them toTelephone Public Corporation in 1985 (an event referred to astransmission equipment and exchanges on-site, but NTT’s

Japan’s telecommunications deregulation), system integrators

station facilities were not yet made broadly available,

were engaged in the business of housing other companies’

and because ISPs needed to connect both domesticallycomputers in their computing centers in space freed up

and internationally, they naturally decided to make use

by computer downsizing. International telecommunicationsof the communication stations of KDD and the NCCs.

companies were also in the business of housing dealer phone

NSPIXP-3 was opened in Osaka, and 1997 saw Japansystems and private telephone exchanges for foreign finan-Internet Exchange Co., Ltd. and Internet Multifeed Co.,cial institutions in space freed up in their telecommunications

Ltd. established, marking the beginning of commercial IX

spaces by the downsizing of exchange and transmission

services. This was the era in which Japan’s underlying

equipment. Both types of buildings were sturdier thanInternet framework was formed.

lines, so it makes sense to install network equipment

ordinary buildings, and people recall them as the predecessor
to today’s data center facilities.

The term data center was still not in use at this point. The
services were called “co-location,” because equipment was

Before Japan’s telecommunications deregulation, international

located alongside transmission equipment, and “housing,”

communications were handled exclusively by Kokusai

because they involved housing customer equipment. The

Denshin Denwa Corporation (KDD, now KDDI), but postrooms were heavily cooled and seemed cold and inorganic,

deregulation, several new communications carriers (NCCs)

and at the time, a supply of 1–2KVA per rack was quite

entered the market, and computer communications alsosufficient.

started to spread, driven by Type 2 Communications Carriers(General, Special), which do not have their own transmission

3.2 2000s—Internet Data Centers

equipment.

With the explosive growth of Internet connectivity services
in the late 1990s, all at once many ISPs began leasing

IIJ was founded in 1992 and, after much effort, was licensed

communications station buildings and providing services.

as a Special Type 2 Communications Carrier and thus per-Late in the era, in October 1998, IIJ established a joint

mitted to provide Internet services. IIJ’s first installation

venture with Sony and Toyota Motor Corporation calledof communications equipment was in the KDD Otemachi

Crosswave Communications Inc. (CWC), which set up

Building (now the KDDI Otemachi Building) in Otemachi,a nationwide network as an NCC. The following year, it

within Tokyo’s Chiyoda Ward. That was followed by theopened access points across Japan.

installation of NSPIXP-1 in the basement of Iwanami Shoten,
Publishers’ premises in Kanda Jimbocho, Chiyoda Ward, andAt the same time, we selected buildings with a fairly large

when Internet exchange (IX) interconnectivity tests began,

floor load capacity in major Japanese cities (Sapporo,

IIJ set up routers on dedicated lines. Once NSPIXP-2 was

Sendai, Tokyo, Nagoya, Osaka, and Fukuoka) and set up

subsequently installed at the KDD Otemachi Building,data centers equipped with uninterruptible power supplies

Internet service providers (ISPs) began to congregate atand emergency generators. Later, we built our own datathat location.

centers—encompassing land and structure—in Kawaguchi

Table 1: Differing Features of Computing Centers and Communications Stations
FeatureComputing center

Communications station (telephone & telegraph offices)

Air conditioning

Water cooled (large, general-purpose coolers)

Air cooled (transmission equipment, telephone exchanges)

Power supply

Main supply: AC (3-phase 200V)Generator backup

Main supply: DC (48V DC)

Building structure

Designed on par with office buildings

Designed as a telegraph and telephone office

Free-access flooring

Slab floor

Mainly space leasing

Mainly rack leasing

CVCF also allows AC supply

Provisioning

20© Internet Initiative Japan Inc.

Jun.2024

Vol.

62

3. Focused Research (2)City, Saitama Prefecture, and Yokohama City, Kanagawa

3.3 2010s—Accommodating Cloud Services

Prefecture. Our guiding concept at the time was to create

With cloud services, which provide computing resources

human-friendly data centers, with integrated people flow

over a network, service providers and the like own andlines, cafeterias for break times, conference spaces, kitting

operate the servers and other IT equipment essentially on

rooms, and more. This marked a shift away from inorganic

behalf of users. To achieve high equipment densities andcomputing centers and communications stations to intelli-efficiently run the huge amount of IT equipment involved,

gent buildings for the Internet era.

in the 2010s, operators began building hyperscale data
centers capable of adequately cooling the equipment andWhile it was opening data centers, CWC was also providing

supplying 10KVA or more per rack.

wide area LAN services to which customers could connect
via Ethernet, which was the standard LAN interface. This

Power receiving capacities were now on the order of 50MW,

communications service (called a “virtual building”) allowed

equivalent to some 10,000 ordinary households (on 100V

users to connect as if on a LAN even when physically distant

50A contracts). The facilities were not only large in scale but

and use the network as if they were all in the one building.

also designed to be energy efficient. The rise of hyperscale

Combined with a fee structure that was independent ofdata centers also saw improvements in an indicator of energy

physical distance, this service took the world by storm at

efficiency called PUE (power usage effectiveness; found by

the time, and it is fair to say that this setup has now become

dividing the power consumption of the entire data center

the de facto standard for L2 data services.

by the total power consumption of its IT equipment, witha score of 1 being optimal and the Japan average being

As networks and data centers evolved, companies began

1.7). To build data centers that would accommodate

installing the equipment needed for email services, web

cloud service platforms, IIJ developed the IZmo contain-

services, firewalls for security, and remote access. The

erized modules capable of providing 10KVA per rack in a

facilities attracted not only ISPs but also OTT (over-the-high-density environment with outside-air cooling for hightop) businesses such as content providers, and with theenergy efficiency, and this culminated in IIJ becoming theadvent of rental server services and hosting services,

first in Japan to operate a commercial outside-air-cooled

the facilities became not just places to locate equipment

containerized data center facility in 2011. At the time, IIJ’s

but places for providing the functionality to connectbusiness model involved reselling data center capacity

such systems to the Internet.

rented from other companies. We had no experiencebuilding our own data centers or developing electrical

And thus they came to be called Internet data centers.

and air conditioning equipment, so we had to start fromIn this era, floor load capacities exceeded one ton per

scratch. We visited the data centers of industry leaderssquare meter, multiple power feeds were installed, and(GAFA and the like) in North America and determined thatthe power supply and air conditioning equipment wasoutside-air cooling would be the most effective way to re-designed to facilitate 4–6KVA per rack. The Internet hasduce power consumption, and we settled on the concept

long been likened to a cloud, and it was from aroundof combining this with containerized modules that would

this time that Internet data centers would transform into

allow us to add capacity in stages. Working with our part-cloud service and connectivity hubs.

ners, we designed and built test facilities, and after a year

Table 2: Key Considerations when Constructing a Data Center
Consideration

Details

Building structure

Earthquake-resistant structure, seismic-isolation structure, seismic-isolation flooring

Power receiving system

3-spot network, receipt of power from substations

Power supply

UPS redundancy (standby redundancy, parallel redundancy, common standby method)

Communication circuits

Multi-carrier, carrier neutral, conduit redundancy

Disaster preparedness

Distance from active faults, avoidance of locations directly under air routes, hazard mapping

GroundGround strength (N value), liquefaction potential index

Indicators

PUE, WUE, PLM value, Tier level classification© Internet Initiative Japan Inc.

21

of testing, we were able to build the Matsue Data Center

fact that infrastructure is well developed by virtue of thePark (Matsue DCP).

area having long been a site for computing centers. Even
overseas, though, data centers do tend to cluster in certain

We named IZmo after the Izumo region, where it was firstareas, so Japan is no exception here. Other cluster sites

implemented, and the kanji for “cloud” (which is part of the

include Osaka’s Saito region and the Keihanna area, and withJapanese spelling of Izumo). IZmo’s excellent energy-saving

multiple large-scale development plans underway in Japan,

performance and ease of installation saw it not only used inthis trend can be expected to persist.

Japan but also exported overseas for use at a hydroelectric
power plant in Russia, a national data center in Laos, andAs the scale of IIJ’s business expanded, in 2019, we also

elsewhere. And a project is currently underway to provide

built and began operating a hyperscale data center, dubbed

data centers to Uzbekistan’s national telecommunications

Shiroi Data Center Campus (Shiroi DCC), in the Inzai district

carrier. With edge computing becoming more widespread,(Shiroi City is next to Inzai City). Located on a 40,000sqm site,

we now offer the modules both in Japan and internationallythe facility can be expanded to a maximum power receiving

under the DX edge moniker as a solution for easily andcapacity of 50MW. Matsue DCP can be expanded in container

rapidly building edge computing environments and digital/

increments (nine racks), but to allow for the expansion of

IT infrastructure.

Shiroi DCC’s service infrastructure in larger increments, wefirst constructed a system module building with a 1,000-

Matsue DCP is a medium-sized center with a power receiving

rack capacity, and this is divided into four modules to allow

capacity of 4MW, but from the late 2010s, the construction

for expansion in stages. The use of outside-air cooling systemsof multiple hyperscale data centers for foreign cloud providers

on a large scale also means that the facility achieves excellent

began in earnest. These were initially clustered in the Inzai

energy savings on par with Matsue DCP. In addition to out-

district around Chiba New Town Chuo Station. This areaside-air cooling, Shiroi DCC also uses side-flow systems,

is so well known as a data center hotspot that the namewhich are often used in hyperscale data centers. The low-

Inzai is recognized internationally. This clustering can be

speed, high-capacity airflow of this system is more energy

attributed to a confluence of factors, including the area’s

efficient than the conventional method of vigorously blowing

solid ground, the fact that it is within 30km of Otemachi,

cold air out from under the double floor and creates a com-easy access to a submarine cable landing station, and thefortable working environment.

AHU : Air Handling Unit

Outside-air cooled container module concept at initial planning stage in 2009
(excerpt from internal documents)

Shiroi Data Center Campus, launched in 2019

Figure 1: Creating a New Form of Data Center, Something from Nothing

22© Internet Initiative Japan Inc.

Jun.2024

Vol.

62

3. Focused Research (2)In the 2010s, as cloud services became widespread, the■ Carbon Neutrality

data center industry saw substantial change, including the

The use of data centers helps to save energy across society

ongoing build up of hyperscale data centers and the entry of

as a whole through the centralized, efficient operation offoreign players into the market. In response, IIJ developed

IT equipment, but even so, the energy consumption of thecontainer modules from scratch (Figure 1) and was eventuallyfacilities themselves is a focus of attention. Japan’s revisedable to build its own hyperscale data center. In 2023,

Energy Saving Act was enacted in 2023, introducing a

the Shiroi DCC Phase 2 building went into operation, and IIJ

benchmark system for the data center industry (PUE of 1.4

is currently working on plans for the facility’s Phase 3

or less), and requiring operators to submit a plan for convertingbuilding with a view to putting it into operation in 2026.

to non-fossil energy. Companies listed on the Tokyo Stock

We continue to explore new data center forms as we look

Exchange’s Prime market are also now effectively required

further into the 2020s and the prospect of even more drastic

to disclose information on climate change risks based onchange than we have seen so far.

the TCFD recommendations, and thus carbon neutrality is an
issue of some urgency for data centers. At IIJ, data centers

3.4 2020s—Next-generation Data Centersaccount for around 80% of greenhouse gas emissions (Scope

After all the changes so far, what will we ask of data centers

1 and 2)*1, and we have set and are working toward goals

going forward? The data center equipment life cycle is overconcerning the use of renewable energy and improvements

10 years, longer than the 3–5 years for IT equipment, so

in energy-efficiency (Figure 2).

societal and technological trends must be assessed from a
medium- to long-term perspective. As calls for sustainability

The use of renewable energy is a new initiative for IIJ, and takes

intensify and technological innovations like generative AI,

into account not only the quantity of power but also types oflarge language models (LLMs), and post-5G move forward,

power generation that do not produce CO2 emissions. When

IIJ will be focusing on the following three themes around

it comes to IIJ data centers, we evaluate power procurement

next-generation data centers.

methods based on delivery times, cost trends, and the

Scope1.

381(4%)• Usage of renewable energy*1
The target is to increase the renewable energy usage rate of data
centers (Scope 1 and 2) to 85% in FY2030
Scope2. 10,129(96%)• Improvements in energy conservation
The target is to keep the PUE of data centers at or below the industry’s

Roughly 80% comesfrom data centershighest level (1.4)*2 until FY2030 through continuous technologicalinnovation.

(t CO2)

FY2022 unconsolidated data for IIJ

Disclosures based on TCFD (Task Force on Climate-related Financial Disclosures) recommendations

Calculation method: “Basic Guidelines for Calculating Greenhouse Gas
Emissions Throughout the Supply Chain (Ver. 2.3)” (Ministry of the Environment,
Ministry of Economy, Trade and Industry)*1. Renewable energy usage includes substantial renewable energy through the use of non-fossil fuel certificates.
*2. As of April 2022, the Agency for Natural Resources and Energy set the benchmark index and target level for the data center industry to be a PUE of 1.4 or lower; businesses that achieve thisare regarded as energy-saving leaders.

Figure 2: IIJ’s Carbon Neutrality Initiatives*1

Scope 1, 2 (company’s own greenhouse gas emissions): Direct emissions from fuel used or industrial processes, and indirect emissions from energy/heat purchased (GHG
Protocol definition).

© Internet Initiative Japan Inc.

23

like, and we are combining multiple methods in an effort

We are also working on initiatives to develop the environmental

to increase our renewable energy usage rates, increase ourvalue trading business, with plans to start providing digital

proportional usage of power with high “additionality,” andassets (tokens) representing environmental value in Julystabilize our procurement costs (Figure 3). We have joined

2024 (Figure 5). And we are developing new roles for thethe Japan Electric Power Exchange (JEPX)’s renewable energy

data center to help solve societal issues, such as power

value trading market, obtaining certificates directly fromgrid stabilization and resilience bolstering. For instance,

the market, and we have installed solar power generation

we are part of a VPP (virtual power plant) project to use

systems on our data center premises, reducing our renew-

storage batteries in data centers (launched in FY2023),

able energy procurement costs. We aim to achieve carbon

and we are working with Matsue City, Shimane Prefecture

neutrality by combining our traditional approach to energy

as a co-proposer in the Ministry of the Environment’s

savings with offsite power purchasing and the like. We have

Decarbonization Leading Regions program, with plans toalso launched a new service that utilizes these resources touse storage batteries to supply power to local communities

provide environmental value to data center users (Figure 4).

in the event of disasters.

Increasing renewable energy usage rates early while also working to increase additionality*1 ratio and stabilize costs
Step 1. Raise renewable usage rates early through use of non-fossil certificates, green power certificates, etc.
Shiroi DCC: Directly procured non-fossil certificatesin FY2023

Matsue DCP: Achieved 100% renewable energy usage
in Apr. 2022

Non-fossil certificates
Bidding

Electricity retailer

Japan Electric Power Exchange (JEPX)

Shiroi DCC

Procuring directly from the market resultsin cost reductions (equiv. to commissions)

Step 2. Increase ratio of high-additionality renewable electricity and stabilize renewable energy costs
Shiroi DCC: Introduced onsite private
power generation in Feb. 2023
Matsue DCC: Introduced onsite private
power generation in Mar. 2023

Matsue DCP: 293kWp

Cost effective but only covers
a few percent of needs

Procure renewable electricity via offsite PPAs*(incl. wheeling for own use)

Shiroi DCC: 336kWp(400kW planned for Phase 2
building)*1. Has the effect of encouraging growth and investment in new renewable energy equipment.
*2. Power Purchase Agreement. Electricity sales agreement between an electricity user (consumer) and an electric power company (PPA operator) that sells electricity to the consumer.

Figure 3: IIJ’s Efforts to Use Renewable Energy

24© Internet Initiative Japan Inc.

2

Data center

Jun.2024

Vol.

62

3. Focused Research (2)■ AI-based Control and Automation

operator workloads at data centers. We are, for instance,

Reducing energy usage is crucial to achieving carbon neutrality

working to automate the various procedures and tasks thatin the data center. The air conditioning systems used to cool

are crucial to providing data center services, a key example

IT equipment account for the greatest share of energy

being the use of automated reception systems, which auto-

consumed by equipment other than IT equipment like servers.

mate the procedure of admitting people into data centerTo achieve data center PUEs close to the optimal value of 1.0,

buildings. We are moving in this direction because, amid

we are actively using outside air to cool IT equipment, andthe decline in Japan’s working population and the country’s

we select highly efficient equipment. At Shiroi DCC, we arework-style reforms, we expect it to become more and morealso using AI technologies to further improve energy-saving

difficult to maintain the quality of our product offerings as theperformance. We aim to achieve more effective control ofscale of our data centers expands if we continue to rely on

the overall air conditioning system by training AI on datasolely on human resources through the hiring and training offrom sensors and IT equipment. In addition to automating

operations personnel. We are looking to progressively expand

air conditioning systems, we are also taking steps to reduce

automation to other areas. For example, with respect to our

Pursuing initiatives to realize carbon-neutral data centers
Using the resources to return new value to customers and society
Conventional data centers

IIJ’s carbon-neutral data centers
Offset renewable power

Offsite power derived
from fossil fuels
Generators + UPS

Non-fossil
certificates

PPA, wheelingfor own use

Generators + UPS

Air conditioning systems

Onsite renewable power

High capacity storage batteries
Outside-air cooling systems
IT demand control

July 2022
Joined VPP (Virtual Power Plant) project,contributing to power grid stabilization

October 2023
Began procuring non-fossil certificates onbehalf of customers to support theirdecarbonization efforts
Revised Enery Saving
Act mandates regular
reporting to gov’t*1

Shiroi DCC
Electricity retailer

Certificates

For details, see IIJ’s July 28, 2022 press release, “IIJ to Participate in‘Virtual Power Plant (VPP)’ Project of Kansai Electric Power”(https://www.iij.ad.jp/en/news/pressrelease/2022/0728.html).
Source: Agency for Natural Resources and Energy website(https://www.enecho.meti.go.jp/category/saving_and_new/advanced_
systems/vpp_dr/about.html).

Renewable energy
value trading market(JEPX)

Data center
(IIJ)

User

July 2024 (planned)
Link environmental value to digital assets +digital currency payments to fuelcustomers’ GX/DX efforts
FutureEnvironmental
value+
payments

Data center(IIJ)

Secondarytrading

User

Company B

Electricity
Environmental value*1. Agency for Natural Resources and Energy, Key points for regular
reporting under the Energy Saving Act—inclusion of IT equipment
energy usage of tenant businesses, May 2023
(https://www.enecho.meti.go.jp/category/saving_and_new/saving/
enterprise/factory/faq/pdf/a1-18.pdf).

DCJPY Network

Figure 4: Beyond Carbon-Neutral Data Centers© Internet Initiative Japan Inc.

25

plans to supply environmental value-linked power to data

are set to come out with TDP (thermal design power) ratingscenter users, as described in the previous section, we hopein excess of 300W, and this is only expected to rise ahead

to go beyond simple automation to provide more sophisticated,in response to demand for AI. According to the American

higher-quality offerings by developing an electricity supply/Society of Heating, Refrigerating and Air-Conditioning

demand matching platform, introducing digital currency

Engineers (ASRAE), once TDP exceeds 300W, facilities will

payments, and the like.

need to adopt water-cooled systems instead of conventional
air-cooled systems (Figure 6).

■ Supporting Higher Server Densities and Water Cooling
As the use of technologies like generative AI and LLMs in

Because they will also house network devices and other sucha range of fields continues to advance and the processing

IT equipment that is air coolable, future data centers will need

power of CPUs and GPUs increases, next-generation data

to adopt hybrid cooling systems that combine air and water

centers will need to be able to efficiently accommodate hugecooling while also providing excellent energy-saving perfor-

numbers of these CPUs and GPUs. Ahead, data center CPUs

mance in order to achieve carbon neutrality. IT equipment

Source: Ashrae Emergence and Expansion of Liquid Cooling in Mainstream Data Centers(https://www.ashrae.org/file%20library/technical%20resources/bookstore/emergence-and-expansion-ofliquid-cooling-in-mainstream-data-centers_wp.pdf).

Figure 6: TDP (CPU Power) and Cooling Methods

Electricity supply/demand matching platform
Matching
payments

Grid (retailers,
generation firms, power
trading market, etc.)

Matching
payments

DC operator (IIJ)

Onsite PV
Ph-3-a: Use in otherindustries, applications(commercial buildings,
condos, etc.)

Power

Fossil-derived

Power
Wind
power

Solar
power

DC user
User-specified
renewable energy

User A
Power
StoragebatteryPower

Generation firmElectricity retailer
Environmental value

Data center

Low-cost, no
renewables
Additionality
renewable energy

Renewableenergy trading
market(JEPX)

User C

Ph-2: Offsite grid links

Ph-1: Data center onsite

DCJPY Network
Figure 5: Addressing the Rising Need for Decarbonization

26

User B© Internet Initiative Japan Inc.

Ph-3-b: Purchase of other
IIJ services using
environmental value
tokens (network, cloud,security, MVNO, etc.)

Jun.2024

Vol.

62

3. Focused Research (2)and facilities will become more closely intertwined, and wedevelopment of a high-density data center reference model,

believe this will be an area in which IIJ will be able to leverage

establishing hybrid cooling methods that combine air-coolingits strengths, given the company’s experience building andand water-cooling technologies, and formulating and developing

operating large-scale data centers and cloud services. IIJ,

evaluation methods for energy-saving indicators geared toPreferred Networks Inc., and the Japan Advanced Institute

AI computing platforms.

of Science and Technology (JAIST) jointly put forward a
project, which was subsequently accepted, for research

3.5 Conclusionand development into ultra-high-efficiency AI computation

While the users of IT systems may not often give much thought

infrastructure for inclusion in the Ministry of Economy, Tradeto data centers, there is growing recognition that data centers

and Industry and the New Energy and Industrial Technology

are a key component of the infrastructure underpinning a

Development Organization (NEDO)’s program of research

digitalized society. Drawing on our data center journey so far,and development of enhanced infrastructure for post-5G

from computing centers and communications stations,information and communication systems, specificallythrough to colocation services, the cloud, and hyperscaling,

addressing the “Development of post-5G information andwe will continue to pursue new forms of data centers andcommunication systems (commissioned project)” category. IIJ

as we go about operating the infrastructure that supports

is responsible for research and development of basic technol-

our society.

ogy for high-density data centers. We will be working on the

Yoshikazu Yamai
Managing Executive Officer and Director of the Infrastructure Engineering Division, IIJ

Isao Kubo
General Manager, Infrastructure Services Department, Infrastructure Engineering Division, IIJ
Mr. Kubo joined IIJ in 2008. He oversees the data center business and the construction of Matsue DCP and Shiroi DCC. His aim is toachieve carbon neutrality as soon as possible.

Yusuke Tsutsumi
Data Center Infrastructure Engineering Section, Infrastructure Services Department, Infrastructure Engineering Division, IIJ
Mr. Tsutsumi joined IIJ in 2015 and works on the construction of data centers in Japan and abroad. He is engaged in the development oftechnologies for next-generation data centers, which includes studying new technologies in the field of electric power.

Takahiro Mimura
Data Center Infrastructure Engineering Section, Infrastructure Services Department, Infrastructure Engineering Division, IIJ© Internet Initiative Japan Inc.

27