APAN Backbone
Network Operation
NICT/APAN-JP/KDDIJin Tanaka

2nd Asia Telemedicine Symposium
5th Dec. 2008
Seoul, Korea

Who am I ?
• Jin Tanaka tanaka”at”kddnet.ad.jp• KDDI Cooperation- Japanese Telecommunication Carrier- OTC/Otemachi Technical Center• Network Operation Engineer- Worked as network engineer of Commercial ISP for 2 years- Currently working as network operator of R&E Networks• JGN2plus NOC• APAN-JP NOC : Chief Engineer• TEIN2-JP NOC• JP-NOC member• NICT/SPARC Guest Researcher

Agenda• R&E Network and Requirements ofApplication• Backbone Network Operation• New Network Technologies• Conclusion

R&E Network andApplication

R&E Network
for Innovation of Technology• -Testbed
New Internet, Future Internet

FIND(US),FP7(EU), ASIA-FI(Asia), AKARI(JP)GENI(US)- Research of Network Technology- New/High-volume application conscious network JGN2plus(JP), GLIF

for Cutting-edge Research• -Infrastructure
No limitation on research field-Promote the science, technology, and education
Make full use of new network technology to correspond with various
requirement of researchers
Premise providing the stable network to users

Our Target!
TEIN2、APAN、Internet2(US)、GEANT2(EU)Research Activities over NetworkResearchers

Research Application

Requirements for NetworkGeneral
Researchers

Submission/Referee of paper
Database of paper, Polycom, Google...

Stable connectivity with decent BW Cover awide area
Commercial ISP
Easy

Co-researchers

Large data transfer, Grid
Real-Time data transfer, Remote Meeting
Experiment of Multicast

Request an advanced network
High-Speed, High Bandwidth
Strict about Delay, Packet loss

Data Transfer on Long Distance Network(High Energy, Space/Earth monitoring
InternationalGrid, e-VLBI)
Co-researchers
Video Application(Telemedicine,
Telemedicine Digital Cinema, HDTV,Distance learning)

High-Speed/Bandwidth International Circuit
Network coordination over domains
Very strict about delay, Packet loss and Traffic
from other countries

Our Target!

Researches forInter-connection

HardP2P/Overlay, DCN, UCLP, GMPLS
Interoperability

Scaled-up Point-to-Point communication
Requirement Layer-1/2 E2E network
Lightpath/SONET

Backbone NetworkOperation

Asia-Pacific Physical Backbone Topology

Last update
Sep. 2008

Our Main Roles in Network
Operation
Routing• Collecting routing
information• Usage of BGP
Technique• Set up optimum
path applicationconscious- Bandwidth- Connectivity- Latency- Traffic Pattern• Routing Policy

Performance

Coordination

Monitoring• NetworkPerformanceCheck• Information-sharing
among users andNOCs• 24H/7D monitoring- Circuit- Equipment• Finding Packet Loss
Point in Backbone• Network Parameter- VLAN ID- IP address/Network- AS-path• Ticket- Trouble- Maintenance• Avoiding crosstraffic• Latency Monitoring• Traffic Usage• With circuit carrier
about maintenance• 24/7D Help desk

• Securing network
resource

Campus
NetCampus
Net

Backbone Networks
End System

End System

- Routing -R&E International routing
Pre-1999 modelCurrent R&E International routing
155M

CSTNET

RBNET

2.5G
622M

155M

10G

GLORIAD1G

KREONET21G

1G
10G

KOREN
10G

SINET

155M

10G

10G

CERNET
1G

1G

10G

1G

NY

1G
1G

LA

ST CH
CH

CH

2G

APAN
10G

2G

TEIN210G
622M

155M

1G

LA

Internet2

1G

JGN2DCCHNYASCCTHAIREN

SINGAREN

ASGC

CH
LA

2.5G
ST
1G

1G

ST
LA

LA

NY

1G

1G

UNINET

10G

ST

622M

10G

SV
1G

1G

1G

THAISARN

1G

TWAREN
1G

155M

10G

1G
2.5G

CLARA

CH

1G

622M

45M

LA
LA
10G

10G

GEANT

1G

155M

45M

U HAWAIIERNETAARNET

622M

622M

Wrong routes detection system• Background- The connectivity of R&E networks are improving.
- Most user are expecting the high performance to the networks.
- Network operators have to support such user’s research or demonstration.
-And if there is any problem with networks, they have to resolve the problem as soon
as possible.
Layer3 routings are becoming complex. Also, router configuration tend to becomplex. it increase the burden on network operators with routing control.
Unexpected routings or asymmetric routings are generated.
It is difficult to collect network’s or organization’s information.
APAN-JP -> CERNETCERNET
AS 4538

1G155MAsymetric
Path

KOREN
AS 9270

10GInternet2
AS11537

Unexpected
Path
10G

1G

TEIN2-HK
AS 24489

622M

APAN-JPAS 7660

10G

TransPAC2
AS 22388

Wrong routes detection system• System-illustration-

KREONET2

GEANT2
Internet2

SingARENAARNET
TransPAC2

CERNET

TEIN2
KOREN

APAN-JPBGP Peering

R&E Route View Server

Route View Server- Save Routing Information

Collecting routing information

APAN-JPInternet2

TEIN2

All Route

All Route

All Route

Wrong Routes List

Extracting wrong routes
based on check list

Save Routing Information
by Each Network

Path Check Script
Wrong Routes List

Displaying and saving theresult on the route view
server- Save Checked Result- Check Routes
by every 1 hours

Route View Server- Display Checked Resulton the Web Page

Wrong Routes

ComPATH Tool

- Performance -Network Usagesnmpget• SNAPP- High resolution Traffic Graph- Collect Traffic usage data per 10 seconds- Enable to display more detailed/accurate traffic
pattern and peak value than general MRTG
which collects data 5 minutes interval- Make graph every ten seconds- Provide Quasi-real-time traffic monitoring

SNAPP

800Mbps

MRTG/Cricket（5min）
2.5GpsSNAPP（10sec）APAN Backbone Weather MAP• Traffic Weather MAP- Display link usage on topology MAP- Enable us to confirm current link usage overmultiple links- Inflow and outflow traffic are very obvious- Display the graph every ten seconds- Convenient for the detection of cross traffic at
the demonstration or event

10 sec MIBdata

Weather MAP
For SC08

Network Status
ARBOR Peakflow• Peakflow- Production of ARBOR Networks- Collect network data, SNMP, BGP routing,Flow (src/dest IP), and analyze it integrally- Basically security tool
- Detect DDos, Warm, Botnet and alert- Easily analyze an unexpected traffic- Effective for traffic confirmation atdemonstrations or experiments.

• Smoke Ping

- Keep track network RTT(round trip time)latency by ping
- Best of breed latency visualization- Wide range of latency measurement
From Toyko✓ To KR, CH, TH, SG, US, PH, HK.....
✓Smoke Ping Snapshot

Performance Measurement+ BWCTL• Iperf- Iperf – A network performance measurement tool
- BWCTL - A command line client application and a scheduling andpolicy daemon that wraps Iperf.

PerformanceMeasurement

Iperf+BWCTLPortable
Iperf + BWCTL

ClientPerformance Problem

Server- Coordination -Grasp the Network Domain
and Person• Think about a rough network

composition, and propose "Whichrouting is optimum pathfor researchers”

Backbone Network

Info for domain• Necessary-Campus network for source- Domestic network for source- International Net1... International Net2...
- Domestic network for destination- Campus network for destination

DomainA

DomainC
DomainB

DomainD

person for support• Contact-User (naturally)
Cooperative Network Engineers
NOC operators

NOC Collaboration

Information Sharing
Router Proxy
Advanced Scheduler

Wiki pageOperation Software and Tools in APAN-JP (As of May. 2008）
Subject/Function

Tools/Software

Level of importanceComment

RRDTool & Cricket
RRDTool & SNAPP

Indispensable in the dailyoperation
Indispensable in the daily operation

For demonstration support

Weathermap4RRD

Indispensable in the daily operation

For demo and trouble-shooting, verify the routing

MRTG

Less-useful

NagiosIndispensable in the daily operation

Primary monitor tool is OpenView NNM

BGP Status Check(Original)

Helpful in daily operation

For Efficiently improvement

Router Proxy

Indispensable in the daily operation

Trouble-shoot from other NOCs

Performance/ThroughputiPerf

indispensable in the daily operation

BWCTL

Partial use

Performance/delaySmokePing

Indispensable in the daily operation

Grasp the latency of international links

OWANP/One-Way Latency

Under developing and installation

Not worked on the actual operation

Performance/Inter-domain

PerfSONAR

Under developing and installation

Flow Monitoring

FlowTools

Helpful in daily operation

Security/Flow

PeakFlowSP

Helpful in operation

Useful for Trace back the traffic flow

Number of BGP route check(Original)Indispensable in the daily operation

Useful for check the BGP routs variation

BGP Path Choice(Original)

Under developing and implementation

Enable to confirm the path choice policy

BGP Compath(Original)

Under developing and implementation

Alert NOC to asymmetric routes

APAN Route View/Quagga

Helpful in operation

Enable to confirm the routing table in trouble

Request Tracker

Low utilization

Advanced Monitoring & Schedule(Original)Indispensable in demo support

Manage the network resources and lead to
improvement of performance

Trouble Ticket Issue

Indispensable in the daily operation

For Efficiently improvement

Advanced Trouble Ticket(Original)

Under developing

Syslog Check(Original)Indispensable in the daily operation

IPaddress Manage(Original)

indispensable in the daily operation

Config Backup(Original)

indispensable in the daily operation

Config Diff(Original)

indispensable in the daily operation

VLAN View(Original)

Helpful in operation

Traffic Graph

Network Monitoring
Advanced looking glassRouting

ProgressResource ScheduleTrouble-ticket

Operation Management

New NetworkTechnologies

DCN-Dynamic Circuit Network• Provides connection type of service• VLAN path is created by user request• Inter-domain connection is possible to be established by IDC

Proposal: Telemedicine demo over DCN
Web InterfaceIDC•XML Request•Authentication•Schedule Reservation•Request VLAN
•Request Bandwidth

IDCIDC•XML Request•XML Topology Exchange•XML Topology Exchange•Path Creation•Path Creation•Path Creation
DC/vlsr

DC/vlsr

DC/vlsr

DC/vlsr

DC/vlsr

Network2

DC/vlsr

Network3

Network1

User

End-to-End Ethernet VLAN Path

• Better Performance • Symmetric• Secure • Shorten Provisioning time
• Efficient usr of Network Resource
End System

End System

PerfSONAR
is• PerfSONAR
perfSONAR is an infrastructure for network performance monitoring,-making it easier to solve end-to-end performance problems on paths
crossing several networks.
Joint project - ESnet, GEANT2, Internet2, RNP and other participants.
http://www.perfsonar.net/to• It -enables
Do performance measurement across the multi-domain network.

- Exchange the network status data with standardized format

NOCs and users are able to- monitor the network status across the domain- visualize the network data for their own purpose

Diagram of PerfSONAR
User

NOC A

NOC B

NMWG format data
Service A

Service forMetric A

Domain A

MetricA

Domain A

MetricANOC C

Request/Obtain
the data

Service B

Domain B

Service forMetric B

MetricB

Domain B
End-to-End performance

Domain C

MetricA

MetricB

Domain CConclusion• Telemedicine is one of the key applications foradvanced networks.

• APAN NOCs are making effort to provide theoptimum backbone network with advanced tools andcooperation with other NOC.

• More joint efforts are required among network

administrators, network operators and advanced
application scientists such as medical doctors.

• New technology, such as DCN and perfSONAR, will
create a better network environment for thedemonstration of telemedicine.

Thank you!
APAN-JP NOC Web Site: www.jp.apan.net/noc
APAN-JP NOC ML: ops”at”jp.apan.net