Broadband measurement in Japan
Broadband Performance and QoE:
Measurement Technologies Improving the Marketplace

Kenjiro Cho, IIJ
2018-06-23

Kenjiro Cho• Research Director at IIJ• a board member, mawi-wg chair of WIDE Project• involved in measurement research for 20 years- data collection and analysis at ISP/academic
networks, especially on traﬃc- large-scale content distribution: traﬃc management- data-center networking and cloud services

2broadband traﬃc in Japan
12

Traffic volume [Tbps]

108

A1 (in)A1 (out)
Mobile (in)Mobile (out)64
20
11/2004 11/2006 11/2008 11/2010 11/2012 11/2014 11/2016
Dateestimated total broadband traﬃc in Japan
http://www.hongo.wide.ad.jp/InternetTraﬃc/data/3

overall traﬃc trends• steady increase driven by diﬀerent major factors- web (late 90ʼs), broadband deployment/P2P-ﬁle sharing (early2000ʼs), short-videos (late 2000ʼs), video subscriptions (mid 2010ʼs)• recent trends• video streaming by subscriptions• software updates: become larger and more frequent• cloud uploads: SNS, multiple-devices, auto-backups• faster access: NGN, 1G/10G access, LTE+• closer content: CDN, cache, peering• advertisement and analytics
4possible bottlenecks in broadband•

many potential bottlenecks: hard to distinguish!

ad & analytics

mobile carrier
Internet(transit)

ISP

home
applications
WiFi interference
buffer bloat

CDN
peering

content

congestion

server overload
ad & analytics

NTE
access network
interference
congestion

DNS
accounting
5

various measurements•client-side•
content-provider
side•
ISP-side6

ISP-side measurements•
ﬂow
exports•
DNSresolvers•

interface counters7

traﬃc between ISPs
• transit services: metered billing (e.g., by 95th-percentile
over 5-min volumes)• peering: bilateral settlement-free customer-traﬃcexchange

providerprovidercustomercustomerpeering

Regional ISP
providercustomer
Access ISP

Regional ISP

providerproviderprovidercustomercustomercustomer

Access ISP

Access ISP

8traffic (Mbps)

Tier-1 ISP

600
500
400
300
200
1000IN
OUT

05/03

05/10

05/17
time

05/24

05/31

diurnal broadband traﬃc2011.112017.119

traﬃc usage per customer• daily traﬃc usage distribution- mean of a week as weekdays and weekends diﬀer

daily traﬃc usage per user: broadband (left) and mobile (right)
10per-user traﬃc over 12 years
broadband
IN (MB/day)• broadband downloads over
12 years- mean: x5.9, mode: x39• heavy user traﬃc hasnʼt
increased much• mode of mobile traﬃc: about

OUT (MB/day)

year

mean

medianmode

mean

median

mode

2005

430

3

3.5

447

30

32

2007

433

54

712

58

66

2008

483

65797

73

94

2009

556

7

6971

88

114

2010

469

87910

108

145

2011

432

9

8.5

1,001

142

223

2012

410

12

14

1,026

173

282

2013397

14

18

1,038

203

355

2014

437

22

28

1,287

301

447

2015

467

33

40

1,621

430

708

2016

475

48

56

2,081

697

1,000

2017

520

63

79

2,624

835

1,260

mobilebroadband 10 years ago

IN (MB/day)

11OUT (MB/day)

year

mean

medianmode

mean

medianmode

2015

6.0

2.7

5.5

46.6

1940

2016

7.8

3.6

763.0

27

63

2017

12.0

4.3

777.4

3579

broadband 2005 vs. 2017
1110InOut

0.5

Daily inbound traffic (byte)

Probability density(a) Total (2005)
0.4
0.3
0.2
0.1
0 4

1010

5

6

789101010 1010Daily traffic per user (bytes)

1010102005

10109810101076510410 410111251067891010101010
Daily outbound traffic (byte)

101011

traﬃc usage skew among users• heavy users: statistically distributed, not outliers- broadband: top 10% users account for 60% of OUT, 85% of IN- mobile: top 10% users account for 48% of OUT, 62% of IN

CCDF of daily traﬃc usage per user: broadband (left) and mobile (right)
13

port usage•port no: min(sport, dport)- increasing TCP 443, UDP 443
broadbandmobile

14

port usage over a week

TCP port usage over a week: broadband (left) and mobile (right)
15

redundancy matters•

example: Japan earthquake in 2011a failed link:
one of the US-JP submarine cables damaged

a survived link:
immediately, traﬃc rerouted to available links

an alternate link:
one-day after, a new link brought up
16

IPv6 in Japan (by APNIC Labs)

https://stats.labs.apnic.net/ipv6/JP17

hyper-giants observed from IIJʼsbroadband traﬃc•by mapping TCP 80 and 443 src addresses to ASes
ASes

traffic volume (%)

google30.1

akamai

7.1

apple

6.6

limelight

4.8

amazon4.5

yahoo japan

3.4

microsoft

3.1

netflix

2.4

facebook

2.3

edgecast

1.9

peak traﬃc in North America
Sandvine Global Internet Phenomena Report 201618

client-side measurements•
probe
deployment
andsamplingissues
•crowd-sourcing:promisingfor
scalability•
more accurate views from users- e.g., RIPE Altas, CAIDA ark

19

RIPE Atlas• started in 2010, 10k+ probes worldwide, 300+ in Japan• periodic ping and traceroute + custom measurements• abundant but noisy data, requiring statistical ﬁltering

20inferring access delays from RIPE
Atlas traceroutes: preliminary results• mean diﬀerential-RTT of Japanese access links over 5 weeks- between home routers and ISPʼs ﬁrst hop- aggregated by the ﬁrst hop address• diurnal congestion observed in evening hours- no such congestion observed in non-FLETS access118.23.22.236
OCN (Tokyo)
6-8ms

203.180.8.11
IIJ (Tokyo)
5-32ms21221.110.198.225
Softbank (Tokyo)
2.5-15ms

content-provider measurements• http video-streaming- video as series of segment ﬁles- video replay under clientʼs control•

adaptive bit-rate: selected by client- pros: cost and scalability as web technologies
are used (but traﬃc behavior is diﬀerent)- cons: extra-delay, diverse clientsʼ behaviors
22

Netﬂix ISP Speed Index (Japan)

23

source: https://ispspeedindex.netﬂix.com/country/japan/

Netﬂix ISP Speed Index (US)24source: https://ispspeedindex.netﬂix.com/country/us/on-demand vs. live-streaming• challenges for live-streaming- client: to shorten delays (fewer buﬀering) still avoiding
rebuﬀering- server: large-scale bursty requests

video segment requests arriving at a server

S. Alcock, R. Nelson. Analysis of YouTube Application Flow Control. 2011.
http://www.wand.net.nz/~salcock/youtube/

25

summary•

steady increase in traﬃc- driven by diﬀerent technical and non-technical factors- dominated by videos (short and long)•technical trends- rise of https, QUIC, IPv6

• QoE
- better access networks, closer content, better algorithms
26