Requirements for Switch Fabricand Expectations for Trellis
April 20, 2018
NTT Network Service Systems Lab.
Takayoshi Hirasawa

Copyright©2018 NTT corp. All Rights Reserved.

NTT GroupNTT Group Holding Company

R&D

12 Laboratories 2,500 researchers
Information Network Laboratory Group
Network Service Systems Lab.
Transport Network Innovation Project(Core network development team) ★ I am here

NTT East NTT WestNTT
Communications Dimension

NTT
DOCOMOLong distance andinternational
communications Business

Mobile

DataRegional
Communications
Business

NTT DATA

Communications

business

Systemintegration

Other
group
companies

Othersreal estate, finance,
construction powerbusiness etc.

Copyright©2018 NTT corp. All Rights Reserved.

2Requirements for current network
 NTT groups provide various services
with high scalability and high reliability.
 These requirements have been realized in a dedicated high-end routers.

 Example of NTT Regional Communications BusinessService
Basic service
Scalability

Internet(IPv4, IPv6), telephone, telecast
Route

Over a few hundreds of thousands routes

Traffic

Over tens of Tbps

Additional functions

PPPoE, IPv6 native etc.

Quality

High-endcore routers

Reliability

Redundancy of each function
Rapid switching for link failure

Recovery operation

Internet: within 2 hours
VoIP: within less time than internet

Copyright©2018 NTT corp. All Rights Reserved.

3

Disaggregation of high-end routers On the other hands, especially in OTT, a merchant silicon-based switch
has great demands.
 CAPEX/OPEX savings and flexibility can be expected for a carrier with
commodity products.
 CurrentManagement

High-endcore routers Future
IA server

Management

RoutingRouting

Forwarding

Forwarding
Merchant silicon-based switch clusters

Disaggregation of the unified functions
into commodity products
Copyright©2018 NTT corp. All Rights Reserved.

4

Activities
Our main activity Promotion of network development using commodity products• https://github.com/multi-service-fabric/msf• https://github.com/beluganos/beluganos* Detail in reference Discussion with an open community about carrier requirements• https://www.opennetworking.org/• https://telecominfraproject.com/

Our working about ONOS/CORD• ONOS/CORD verification (especially Trellis)• Trying to close the gap between the results and our requirements

Today’s topic
1.
2.

The verification results of the CORD-3.0 physical PoD
Proposal of new Trellis architecture for improvement of reliability
Copyright©2018 NTT corp. All Rights Reserved.

51. CORD physical PoD verification

Existing network architecture There are two architectures regarding the deployment of the routing function.
 Autonomous control architecture – Deploy routing functions on each switch
 Centralized control architecture– Deploy routing functions on central controller

 Autonomous control architecture
NOS control

Management

NETCONF etc.

NOS

Routing/Forwarding control Management
Routing
OpenFlow

NOS
RoutingNOS Centralized control architecture

NOSNOS

Forwarding

Forwarding

Advantage
Autonomous control => Conventional C-plane processing reliability.
Centralized control => Handling multiple switches as a single logical node.
* NOS: Network operation system

Copyright©2018 NTT corp. All Rights Reserved.

6

1. CORD physical PoD verification

Verification condition (1/2)
Period Installation – 2017/6 - 2017/7
 Verification – 2017/8 - 2017/12

Verification points
Router
Hardware

Component

Software
Internal IF

Redundancyof processor
Compatibility
Functionality
Maintenance
Scalability

Forwarding control
Routing control
Route update link
A single logical nodeL2 switching
IP routing
VPN
In-service operation
RouteTraffic

Dedicated
vendor Hardware
Dedicated
vendor NOS
Dedicated
vendor IF
AutonomousCentralized

MSF (NTT Lab.)

Trellis/CORD (ONF)

Commodity multivendor products
Multivendor NOS
Open NOS+ Open NOS
StandardStandard(Routing protocols)(OpenFlow)


N/A
Verification target(especially compatibility)
Copyright©2018 NTT corp. All Rights Reserved.

7

1. CORD physical PoD verification

Verification condition (2/2)
Components
 5 switches (2 Leaf and 3 Spine) and 4 servers

Profile A residential profile, R-CORD (Each profile applications were not main target.)
HeadNode Component
XOS
ONOS Controllers
QuaggaONOS_fabricvRouter Underlay
Control Control

Managementswitch

Whitebox switch
AS6712-32X

Whitebox switch
AS6712-32X

OpenstackApp.

Whitebox switch
AS6712-32X
Whitebox switch
AS6712-32X

OVS

OVS

OVS

OVS

PowerEdge
R630
HeadNodePowerEdge
R630

PowerEdge
R630

PowerEdge
R630

VNF

VNF

Main target “Trellis”

Whitebox switch
AS7712-32X

ONOS_cord
OverlayApp.
Control

External
RouterVNF

Copyright©2018 NTT corp. All Rights Reserved.

8

1. CORD physical PoD verification

Summary of results

 We confirmed basic operation, but there ware some lack of function.
 Especially, from viewpoint of reliability, we confirmed it indispensable toimprove network architecture.
RouterDedicated
vendor Hardware
Dedicated
Component
Softwarevendor NOS
Dedicated
Internal IF
vendor IF
Forwarding control

Redundancy
Routing controlof processor
Route update linkCompatibility A single logical nodeL2 switchingFunctionality
IP routing
VPN
Maintenance In-service operation
Route
Scalability
Traffic
HardwareAutonomous

Centralized

MSF (NTT Lab.)

Trellis/CORD (ONF)

Commodity multivendor products

Multivendor NOS+ Open NOS
Standard(Routing protocols)
N/A


Open NOSStandard(OpenFlow)N/AN/A
N/AN/ANot enoughCopyright©2018 NTT corp. All Rights Reserved.

91. CORD physical PoD verification

CORD PoD reliability

 Some essential functions are through the management switch.
 ARP request/reply, Flow rule update, ONOS clustering connection etc.

 When the management switch failed, all connections were disconnected.
HeadNode Component
XOS

When management switch failed, bothof overlay by VXLAN and underlay
becomes unconnectable. It means
PoD can’t provide service anymore.

Managementswitch

It is difficult to make management
switch redundantbecause a white-box switch has
generally only one management port.

ONOS Controllers
QuaggaONOS_fabricvRouter Underlay
Control Control

Whitebox switch
AS7712-32X

ONOS_cord
OverlayApp.
ControlWhitebox switch
AS6712-32X

Whitebox switch
AS6712-32X

OVS
HeadNode

OpenstackApp.

Whitebox switch
AS6712-32X

Whitebox switch
AS6712-32X

OVS
ComputeNode

OVS
ComputeNode

OVS
ComputeNode

Copyright©2018 NTT corp. All Rights Reserved.

102. New architecture for Trellis

Improvement of reliability
Characteristic of centralized control
 Advantages of compatibility with existing network design.
 Disadvantages of reliability at single point of failure on the management switch.

 Current architecture

ONOS
ServerONOS
ServerMgmt SW (SPOF)

Spine
Leaf

Single point of failure

Spine
Leaf

We first challenged to improve reliability withkeeping the advantage of the current architecture.

A single logical node

Copyright©2018 NTT corp. All Rights Reserved.

112. New architecture for Trellis

Solutions for improvement
1. Expansion of management network• An increase of management port is difficult to control by us.
(It is hardware requirement)• Management network design becomes complicated.

Mgmt SW1

Mgmt SW2

Spine

Spine

Leaf
Leaf
nd2 mgmt port

2. Implementation of back up routing function to switch• An architecture changes from centralized to autonomous.

Mgmt SW1

Spine• Changing the NW design will affect allover network.

IGP

Spine

Leaf
Leaf
Back up routing function

3. Distribution of centralized control via in-bound connection• There are no influence outside PoD• There are no additional hardware requirements

Spine

Spine

Leaf

Leaf

Copyright©2018 NTT corp. All Rights Reserved.

122. New architecture for Trellis

Proposed architecture overview
 Eliminate the management switch that becomes SPOF and connects it
via D-plane port.
 We considered following implementation on a switch.
Implementation on a switch
Base function

B. Pick up OF channel from D-plane port

ONOS2
ServerC. Establish OF channel and update flow rules

Spine2Spine1

A
B,C

Leaf1

D

Leaf2

Mgmt port

OpenFlow channel

D-plane port
Additional
function

ONOS clustering

D. Switching management connection

Verification point
Basic function

Connection establishment

Step 1

RecoveryStep 2

ONOS1
ServerA. Connect management port to D-plane port

Packet forwarding
Flow rule update
Failure scenario ONOS controller failure
Internal link failure

Switch node failure

Physical wiring

Copyright©2018 NTT corp. All Rights Reserved.

13

2. New architecture for Trellis

Initial study (Step1)

 We conducted initial study of in-bound packet control using OSS.
 Component Two white-box switches (AS5812-54X)
 Additional software in external VM (It will be put on switch in future.) OpenvSwitch=> for abstraction of indigo agent and management port.
 Ryu framework=> for control OF channel on D-plane

External VM2External VM1

Initial scenario
A sends packets to B

AONOS

Additional
function

Additional
function

Switch1

Switch2

In an ideal model
deploy it on the switch

BCopyright©2018 NTT corp. All Rights Reserved.

14

2. New architecture for Trellis

Detailed packet processing (1/2)
 Role of each component
 OVS1 : Abstraction of indigo agent and storing flow rules from ONOS controller
 OVS2 : Connection between virtual interfaces and D-plane port on ASIC Ryu : Control of port mapping between OVS2 and ASIC: Sync of flow rules between OVS1 and ASIC
2. OVS1 ask ONOS on VM2
about unknown packet
processing via port-M on VM1External VM 1

456123456M

OVS2:M/D-plane integration

45612345645

OF channel about ONOS

Dst.

Ryu

Switch2
6M

OVS2:M/D-plane integration

PacketFlow rule33

4. Pick up packets to itself by flow
rules (MAC/IP of port-M on VM1)

Indigo/OF-DPA
2

2Packet1PacketFlow rule

Switch1

Src.

1

3. Ryu maps
port-M and D-plane port

Ryu

1. Ryu picks up anunknown packet tomapped port.

OVS1:Indigo abstraction

Flow rule

3PacketFlow rule

2ONOSSrc : Port-M on VM1
Dst : Port-M on VM2

OVS1:Indigo abstraction
1

External VM 2

1

OF channel about Ryu

Indigo/OF-DPA
2

3

4

5

6

Data packet
Copyright©2018 NTT corp. All Rights Reserved.

15

2. New architecture for Trellis

Detailed packet processing (2/2)
 Role of each component
 OVS1 : Abstraction of indigo agent and storing flow rules from ONOS controller
 OVS2 : Connection between virtual interfaces and D-plane port on ASIC Ryu : Control of port mapping between OVS2 and ASIC: Sync of flow rules between OVS1 and ASIC
5. ONOS send new flow
rules to OVS1 on VM1External VM 1

External VM 2ONOS

8. OVS1 stores flow rules
OVS1:Indigo abstraction
34561

2

34

56

OVS1:Indigo abstraction
MOVS2:M/D-plane integration

12

3

4561

2

3

456Ryu1

7. Pick up packets to itself by flow
rules (MAC/IP of port-M on VM1)

Indigo/OF-DPA
23456

OF channel about ONOSPacket
Flow rule

Switch1

RyuPacket
Flow rule

9. Ryu copies flow rules
from OVS1 to ASICM

OVS2:M/D-plane integration

PacketFlow rule2

PacketFlow rule1

Switch2
1

OF channel about Ryu

6. Ryu maps
port-M and D-plane port

Indigo/OF-DPA
2

3

4

5

6

Data packet
Copyright©2018 NTT corp. All Rights Reserved.

16

2. New architecture for Trellis

Packet walkthrough After flow update, wideband transmission by hardware processing canbe performed without any software processing.
Switch Ryu OVS2Port 1
Port 1

OVS1

Switch D-plane SwitchOVS2OVS2Port M Ryu Port 6 connection Port 1 Ryu Port M ONOS

Switch1

OVS1

SwitchOVS2Port 6 Ryu Port 6

Switch2

Data
1

Packet A(Unknown) Packet-in

Virtual link

Packet-out 2

Port 1OF channelin OF channel

3
Packet-in

Packet-in
Packet-out

D-planeOF channelin OF channel
4

OF packet-in with
Packet-in
unknown packet
Packet-out

5 Packet-out

6 Flow mod.
 Storing flow rules
on OVS1

9Stats Req.

Data

Packet A(Known)

87

Packet-in
Packet-out

Flow mod.

Packet-in
Packet-out

Port 6Packet-in
Packet-outFlow mod.

Flow mod.

Hardware processing
OF channel about ONOS
1

Hardware processing

OF channel about Ryu

Icon of detailed packet processing steps

Data packet
Copyright©2018 NTT corp. All Rights Reserved.

17

2. New architecture for Trellis

Verification and results
 We experimentally demonstrated basic operation in two pattern.
 Test item 1 – OF channel connection via D-plane port
 Test item 2 – ONOS clustering connection via D-plane port
 Test item 1 – OF channel

OpenFlow channel
ONOS clustering Test item 2 – ONOS clustering

Ex-VM2

Ex-VM1

Ex-VM2

Ex-VM1
Additional
function

ONOS
Additional
function

ONOS
Additional
function

ONOS
Additional
function

Switch1

Switch2

Switch1

Switch2

Packet forwarding

Packet forwarding

Verification point
Basic function

Test item 1 Test item 2Connection establishment

Packet forwarding

Flow rule update–

Failure scenario ONOS controller failure

Successfullycompleted!

Copyright©2018 NTT corp. All Rights Reserved.

18

2. New architecture for Trellis

Next steps

 Implementation of Step 2 (Switching management connection)
 Expansion to CLOS topology (undergoing)
ONOS
Switch1

ONOS1

ONOS2

Spine1

Spine2Switch2
Leaf1

 Verification of failure scenario
 ONOS controller failure (done) Internal link failure Switch node failure

Leaf2

ONOS1

ONOS2

Spine1

Spine2

Leaf1

Leaf2

 Further study Add L2/L3 VPN function
 Redundancy of routing function (Quagga)
Copyright©2018 NTT corp. All Rights Reserved.

19

Expectations for Trellis We expect Trellis to expand further functions and the scale in the future.
ONOS server

High-endcore routers

RouterRedundancyof processor
Compatibility
Functionality
Maintenance
Scalability

Route update link
Forwarding control
Routing control
A single logical nodeL2 switching
IP routing
VPN
In-service operationRouteTraffic


Merchant silicon-based
switch clusters

Centralized

Trellis/CORD (ONF)

N/A => 
N/A*


N/A
N/A
Not enoughArchitecture design
RIB/FIB control software• FRR (Quagga) enhancement• ONOS enhancement

Handling ASIC
• OF-DPA• Native SDK• P4 (Stratum)* There is no redundant technique to keep a single logical node not.
Copyright©2018 NTT corp. All Rights Reserved.

20

For a high-reliable network
 In order to realize the high-reliable network without single point of failure,
we believe that some functions should be implemented on a switch device.
 Conventional architecture– Autonomous D-plane control by network OS)
Protocol control

Routing information

Routing information

Network OS

Network OS

Network OS

SwitchSwitchExternal router The proposed architecture– Autonomous OpenFlow/ONOS clustering control on D-plane by mini controller
Route control for ONOS

QuaggaONOS

Mini-CTL

SwitchRouting information

Forwarding information

Network OS

External router

Mini-CTL

Switch We believe it is important to utilize lightweight platform that can run on a switches.
That’s why we applied Ryu framework this time.
Copyright©2018 NTT corp. All Rights Reserved.

21

Expectation for Stratum
 We’re also interested in Stratum to realize a widely adaptable network.

Thin model
for more flexible D-plane control• Build forwarding table directly• C/D separated architecture
Thick model
for more an autonomous network• Autonomous route construction• Distributed processing

Focusing on both models, we will continue to research a better network

Copyright©2018 NTT corp. All Rights Reserved.

22

Thank you for listening
We are looking for collaborators who can discuss such an approach.
Please contact us if you are interested.

Copyright©2018 NTT corp. All Rights Reserved.

23

Reference: Multi Service Fabric An activity of our carrier-grade SDN architecture.
MSF is the architectural design of carrier-grade SDN
that provides wide logical network slices
with commodity products including a white-box switch.

 Newly updated on 4/16 !!!
https://github.com/multi-service-fabric/msf/tree/msf2017
https://github.com/multi-service-fabric/fabric-controller/tree/msf2017
https://github.com/multi-service-fabric/element-controller/tree/msf2017
https://github.com/multi-service-fabric/element-manager/tree/msf2017

Supported functions• Controller-independent autonomous control architecture• L2(VXLAN)/L3(MPLS) VPN control• Multi clusters control• In-service operation• High-reliability (There is no SPOF to stop services.)• Switch set-up automationetc.

Copyright©2018 NTT corp. All Rights Reserved.

24

Reference: NTT’s activity about network OSMPLS

IP

FRR

GoBGP

YourApp.

Management of routing database

RoutingEngine

 An activity of our open network OS

The prototype of carrier-grade NOS
based on open technology(OSS/Open API)
with IP/MPLS capability

Routing database of each slice (LXC)
Conversion route information to white-box’s format (RIBFIB conversion）

ForwardingAbstraction(netlink abstractor)

Control for ASIC Driver (Indigo)[Legend]
Existing Software(OSS)

Developed(OSS)ASIC Driver (OF-DPA)
Base OS (OpenNetworkLinux)

HardwarecontrolCreation of Forwarding entry (Ryu)

Copyright©2018 NTT corp. All Rights Reserved.

25