3. Network Technology

We illustrate the technical overview of 100 Gigabit Ethernet, which will become necessary in the near futurewith continuous growth in traffic. We also share the knowledge gained through our joint interoperability test.

3.1 Introduction

Network Technology

100 Gigabit Ethernet

In this report we first illustrate the technical overview of 100 Gigabit Ethernet (henceforth “GbE”), then report on the 100GbE
IX (ISP Internet exchange point) joint interoperability test conducted with Internet Multifeed Co. and NTT Communications
Corporation, and finally comment on 100GbE optical transceiver standards and future trends.

3.2 100GbE
100GbE was set out in IEEE 802.3ba*1, and ratified as the standard in June 2010. Here we discuss important points forunderstanding 100GbE. Because much of the technology established in 10GbE is reused, we assume that the reader has
detailed knowledge of 10GbE*2.
n Basic Principles of 100GbE
The technical hurdles involved in the serial transmission of a signal 10 times that of 10GbE are high, and implementationrelated costs are also an issue. For this reason 100GbE is implemented using parallel low speed data transmission such as10Gbps or 25Gbps (Figure 1). The technology that makes this parallel data transmission possible is one of the key featuresof 100GbE.
n Comparison of 100GbE and Link Aggregation
This parallel data transmission technology is called MLD (Multi Lane Distribution), and it is implemented in the physical layerof the OSI reference model (Figure 2). Link aggregation is a similar technology for implementing parallel data transmissionthat is set out in IEEE 802.3ad. It involves bundling multiple interfaces and presenting them as a single virtual interface, but
it differs from 100GbE in the following ways.
1. Because 100GbE conducts parallel data transmission over the physical layer, users do not need to care about parallel datatransmission settings or behavior, but for 802.3ad close attention must be paid to link aggregation settings and behavior.
2. Because 100GbE data transmission breaks Ethernet frames down into fixed length to transmit data down parallel channels
evenly, there is no risk of a specific channel carrying a disproportionate amount of data. In contrast, 802.3ad, has theproblem that the frame distribution method may not disperse frames into channels evenly*3.

MAC
RS

Interface
Module Board Side

Physical Media Transmission Section

OSI Reference Model
Data Link Layer
Layer 2

PCS (64B/66B, MLD)

100GbEInterface

100GbEInterface
Supports 1, 2, 4, 5, 10
Parallel Transmissions

Parallel Transmission Used• 10 x 10Gbps: uses multiple optical fibers• 4 x 25Gbps: uses a pair of optical fibers with wavelength multiplexing

Figure 1: Basic Principles of 100GbE

20:10 Lane
Conversion
CAUI 10 x 10.3Gbps Electric Signal
Optical Transceiver

OSI Reference Model
Physical Layer
Layer 1

10:10, 10:4 Lane
Conversions
Electric-OpticConversion
Physical Media

Figure 2: 100GBASE-SR10/LR4/ER4*1

IEEE802.3ba document (http://standards.ieee.org/about/get/802/802.3.html)*2

We recommend that readers look over the 2002 first edition of the “10 Gigabit Ethernet Textbook,” which was written by Osamu Ishida and edited by
Koichiro Seto, and published by IDG Japan.

*3

The distribution method of 802.3ad is frame-based and does not take frame length into consideration. This leads to the uneven utilization of links. 802.3ad also
does not specify any detailed algorithm to implement frame-based distribution, so distribution among links may be uneven depending on the equipment.

31Network Technologyn MLD
The RS (Reconciliation Sub-layer) receives Ethernet frames from the data link layer (MAC), and passes them on to lowerlayers after breaking them down into 64-bit units. In the PCS (Physical Coding Sub-layer), 64B/66B blocks used in physicalmedia transmission are made at first by adding a 2-bit header to the 64-bit data. The MLD function then sends these blocksin rotation down parallel data transmission channels called virtual lanes*4 (Figure 3).
n Multiplexing and Demultiplexing of Virtual Lanes
In the course of transmission, virtual lanes can be aggregated in stages, and in this case data between lanes is multiplexed at
bit level (Figure 4). If it is possible to use multiplexing for lane aggregation, the number of lanes can be changed as necessary,
meaning transmission adapted to a variety of physical media can be supported*5. However, when multiplexing is used at
bit level partway down the transmission channel, the lanes no longer align between the sender and recipient (see lane 5 inFigure 4). For this reason an alignment marker system is used to inform the recipient of the alignment of lanes between thesender and recipient.
n Alignment Markers
For every 16383 64B/66B blocks of data sent, data block transmission is temporarily suspended, and each alignment marker
is sent down all lanes at the same time (Figure 5). For example, the alignment marker for physical lane 5 on the sender side
contains information identifying it as lane 5. If this is received over physical lane 1 on the recipient side, it is interpreted asdata corresponding to virtual (sender side physical) lane 5 by checking the ID.
n Bit Error Monitoring for Virtual Lanes
When conducting parallel transmission using multiple lanes, data may be transmitted over different physical media foreach lane. For this reason, the BIP (Bit Interleaved Parity) function is implemented to monitor the line quality of each lane
(Figure 6). The BIP function calculates the bit parity of the 16384 64B/66B blocks including the previous alignment marker for

MAC

Ethernet Frame/w FCSPreambleRS

Frames are broken
into 64-bit units and a2-bit header is addedto form blocks

PCS (64B/66B, MLD)20:10 Lane
Conversion
10:10, 10:4 Lane
Conversions
Electric-Optic
Conversion
Physical Layer

64B/66B

Lane1

64B/66B

Lane2

RS

64B/66B

Lane20Parallel transmission is carried out using MLD- 64B/66B blocks are sent down each virtual lane
- During this process, blocks are sent in rotation downeach lane

20:10 Lane
Conversion
10:10, 10:4 Lane
Conversions
Electric-Optic
Conversion

1 2 3 4 5AM
LaneN

64B/66B

64B/66B

64B/66B

64B/66B

64B/66B

64B/66B

AM
Lane1

AM
Lane2

AM
LaneN

64B/66B

64B/66B

64B/66B

Lane1

Lane2

LaneNAM
Lane1

AM
Lane2

64B/66B

64B/66B

19 20

2:1 bit
MUX

64B/66B

64B/66B

10:4 bit
MUXAM
Lane1

AM
Lane2

4:10 bit
DEMUX

64B/66B

64B/66B

Lane1

Lane2

1:2 bit
DEMUX

Physical Layer

1 2 3 4 5

Data transmission is suspended
temporarily and AM inserted
into all lanes simultaneously
Roles of AM- Identifying the alignment ofsender/recipient lanes- Absorbing the skew in arrival times foreach lane- Detecting bit errors of each lane

Figure 5: Alignment Markers (AM)It is not known which recipient lane that data sentfrom lane 5 will arrive at due to lane
multiplexing/demultiplexing until it is actually sent.
Recipients are notified of lane alignment usingalignment markers.

PCS (64B/66B, MLD)AM
Lane216383 block

Figure 3 RS+PCS (64B/66B, MLD)

MAC

AM
Lane1

19 20

Figure 4: R elationship Between Sender/Recipient Lanes
During Lane Multiplexing/Demultiplexing

01

M0

M1

M2

BIP3

M4

M5

M6

BIP7

The bit parity value (BIP) for 64B/66B
16383+1 blocks including AM is calculated
and embedded in the AM
The BIP for the 64B/66B block is alsocalculated on the recipient side andcompared with the BIP in the received AM toenable detection of bit errors
01: 64B/66B block header (2 bits)
M0, M1, M2: lane identifiers (8 bits each)
M4. M5, M6: bit inversion of M0, M1, M2
BIP3: bit parity value for sent data (8 bits)
BIP7: bit inversion of BIP3

Figure 6: Relationship Between Alignment Markers and BIP*4100GbE uses 20 lanes.

*5

The current IEEE standard stipulates that data transmission between the interface board and the optical transceiver module uses 10 lanes, so 1, 2, 4, 5,
or 10 lanes are available for physical media. In actual practice, the 4-lane 100GBASE-LR4/ER4 and the 10-lane 100GBASE-SR10, 10x10 MSA are used forphysical media in 100GbE.

32

and this is compared with the BIP value in the alignment marker to confirm whether any bit errors occurred in the 16384
blocks. The BIP function makes it possible to determine whether bit errors occurring in the transmission channel affect allparallel transmission channels, or only transmission channels carrying some of the lanes.
n SummaryHere we have explained the features important for achieving parallel data transmission using 100GbE. We recommend thatNetwork Technology

each lane, and sends this value. The same method is used on the recipient side to calculate the bit parity of the data received,

those of you who would like to learn more look over the 802.3ba document.

3.3 100GbE IX Joint Interoperability Test
On June 1 we issued a press release regarding the 100GbE IX (ISP Internet exchange point) joint interoperability test
conducted by three companies*6. We also presented some of the details of this joint interoperability test at the JANOG28
Meeting on July 15*7. In this report we provide a brief summary of the information that has been released regarding thedetails of this joint interoperability test.
The purpose of the test was to confirm stability and verify operational issues in preparation for implementing 100GbE.
We also examined interconnection issues for IX connections in the multi-vendor equipment environment. We gained thefollowing information from this test.
1. There were no serious interconnection issues in the multi-vendor equipment environment
2. Care must be taken regarding some operational differences to conventional 10GbE- Be aware that the optical power for sending and receiving with 100GBASE-LR4 is strong due to the wavelength multiplexing- When you want to measure the optical level of each 100GBASE-LR4 wavelength, you must use a dedicated power
meter or a command line interface if it is supported.
- For many vendors there is no function for checking the BIP counter to monitor the quality of each lane
- 100GBASE-LR4 is highly susceptible to errors from even minor contamination, so it was necessary to use a cleaningproduct to clean both the CFP optics and fiber ends

3.4 100GbE Optical Transceivers
n CFP MSA and 10x10 MSA Standards
Currently, the two types of optical transceivers can be usedwith 100GbE. One is the CFP modules prescribed in the CFP

4 x 25.8Gbps
Wavelength
Multiplex
Signal4 x 25.8Gbps
Optical Signal

Wavelength
Multiplexing/Demultiplexing

MSA*8 in compliance with IEEE standards, and the other is thetransceivers built to 10x10 MSA*9 independent standards thatare not in compliance with IEEE standards. Figure 7 compares
the brief architecture of these optical transceivers.
There is an obstacle to 100GBASE-LR4/ER4 CFP opticaltransceivers being popularized, because they include two
expensive components: 4:10 lane conversion GearBox chips
and 4 x 25.8Gbps high speed laser diodes. In light of this,*6

4 x 25.8Gbps
Electric Signal

OpticalElectric
Conversion

10 x 10.3Gbps
Electric Signal

GearBox
4:10Interface
ModuleBoard

IEEE 100BASE-LR4/ER4 CFP Module Overview
10 x 10.3Gbps
Wavelength
Multiplexing
Signal10 x 10.3Gbps
Optical Signal

Wavelength
Multiplexing/Demultiplexing

10 x 10.3Gbps
Electric Signal

OpticalElectric
Conversion

10 x 10.3Gbps
Electric Signal

ElectricSignal
Reshaping
10:10Interface
ModuleBoard

10x10 MSA Module Overview

Figure 7: C omparison of 100GBASE-LR4/ER4 CFP and10x10 MSA

IIJ Press Release “Success of Industry’s First High Speed 100 Gigabit Ethernet Joint Interoperability Test at IX (Internet Exchange Point)” (http://www.iij.
ad.jp/en/news/pressrelease/2011/0601-02.html)*7

JANOG28 “IX and 100 Gigabit Ethernet” (http://www.janog.gr.jp/en/index.php?JANOG28%20Programs#baef9550)*8

An abbreviation of “C Form-factor Pluggable Multi Source Agreement.” Because the IEEE does not provide detailed specifications for the configurationor functions of 100GbE optical transceiver modules, optical module vendors came together to define common specifications and manufacture products
based on these. (http://www.cfp-msa.org/)*9

Ten by Ten Multi Source Agreement. Equipment vendors and their users came together to draw up independent optical transceiver standards that makelow cost production possible. (http://www.10x10msa.org/)

33

Network Technology

10x10 MSA reuses inexpensive 10GbE technology that is already produced in volume to directly convert 10 x 10.3Gbps electricsignals into 10 x 10.3Gbps optical signals, creating an independent specification that reduces component costs. However, 10x10
MSA is not in compliance with IEEE standards, so it must be noted that it is only supported by some vendor equipment.
n Usable Physical Media and Operating Distance
Table 1 shows the interface standards that can be used with 100GbE. The entries in red are optical transceivers that wereusable at the time of the joint interoperability test. The three main types of physical media that are usable are copper cable,multi-core multi-mode fiber (MMF), and single mode fiber (SMF). However, for actual connection between core network
equipment, the only options are 100GBASE-LR4/ER4 and 10x10 MSA, which can be used with existing single mode fiber
as-is. It is also difficult to construct metro area networks using 100GbE and dark fiber because the operating distance of the
optical transceivers currently usable cannot be extended very far, and at this point medium and long distance transmissions
are problematic without carrier-class transmission equipment. In the future it should become possible to use 100GBASE-ER4(30km) and 10x10-40km optical modules that enable transmission over medium distances, but considering the current priceof CFP it is not clear whether this will be feasible from a cost perspective.
n Future Transceiver Standards
Current CFP-type optical transceivers are extremely large at 78 x 13.6 x 144 (mm), and it is not possible to increase thedensity of ports that can be installed in an interface module board. However, in the near future CFP2 and CFP4 modules withsignificantly reduced size are planned to be released. After CFP2 the speed of electric signals between interface moduleboard and CFP2/4 will be changed to 4 x 25.8Gbps (Figure 8). This is expected to remove the need for GearBox chips in CFP
as with 10x10 MSA, and bring prices down below CFP.

3.5 Conclusion
At present there is very little comprehensive information regarding 100GbE in Japanese, so we hope that this report serves
to aid the understanding of 100GbE. Currently 100GbE products are extremely expensive and there are few products that
support it, so we believe it will still be some time before it spreads to common use. IIJ will actively engage in assessment ofincorporating 100GbE into our backbone network in preparation for future increases in traffic.

IEEE 100G

10x10 MSA 100G

Backplane 1 m

100G BASE-KR4 (Planned)

N/A

Copper Cable 5 m

100G BASE-KR4 (Planned)

N/A

Copper Cable 7 m

100G BASE-CR10

N/A

4 x 25.8Gbps
Wavelength
Multiplexing
Signal4 x 25.8Gbps
Optical Signal

Wavelength
Multiplexing/
Demultiplexing

4 x 25.8Gbps
Electric Signal

OpticalElectric
Conversion

4 x 25.8Gbps
Electric Signal

ElectricSignal
Reshaping
4:4Interface
ModuleBoard

IEEE 100BASE-LR4/ER4 CFP2 Module Architecture
MMF (OM3) 100 m100G BASE-SR10100G BASE-SR4 (Planned)

N/A

SMF 2 km

100G BASE-FR4 (Planned, WDM)

10x10-2 km (WDM)

SMF 10 km

100G BASE-LR4 (WDM)

10x10-10 km (WDM)

SMF 30 km/40 km

100G BASE-ER4 (WDM)

10x10-40 km (WDM)

Figure 8 100GBASE-LR4/ER4 CFP2

Table 1: List of Interfaces Defined in IEEE/10x10 MSA

Author:Munenori Ohuchi
Ph.D. Ohuchi works in the Network Service Department of the IIJ Service Division. Since joining IIJ, Ph.D. Ohuchi has been engaged in the testing ofequipment used in the IIJ backbone network, and the survey, research, and development of new Internet-related technologies.

34