Internet
Infrastructure
Review
Nov.2022

Vol.

56

Periodic Observation Report

Broadband Traffic Report:
COVID’s 3rd Year Brings Lull in Traffic
Focused ResearchLive-Streaming the Spring Festival
in Tokyo—Our ReportInternet Inf rast ruc ture Review
November 2022 Vol.56

Executive Summary ........................ ................................................ 3
1. Periodic Observation Report .......... ................................................ 4
1.1
1.2
1.3
1.4
1.5

Overview .. ................................................................................ 4
About the Data ............................................................................ 5
Users’ Daily Usage . ....................... ................................................ 6
Usage by Port ............................. ................................................ 9Conclusion ................................ ............................................... 11

2. Focused Research ...................................................................... 12
2.1
2.2
2.3
2.4

2

Introduction .............................................................................. 12
Live-Streaming Specifications and Solutions ............................................... 12
Live-Streaming System Configuration ..................................................... 14
Conclusion ................................ ............................................... 17© Internet Initiative Japan Inc.

Nov.2022

Vol.

56

Executive Summary

Executive Summary

Nearly three years have now passed since the initial outbreak of COVID-19 at the end of 2019. Over that time,
the whole gamut of our daily lives across work and play has moved online, partly in response to social
restrictions designed to stop the spread, and we have thus observed a global increase in Internet traffic.
While many countries are lifting restrictions, the virus continues to mutate, and we are certainly aware thatthe future remains unpredictable.
International tensions between democratic and authoritarian regimes also continue to escalate, and half a year
has now passed since Russia began its military invasion of Ukraine. From the perspective of economic security,
this has cast into the spotlight the importance of supply chain risks relevant to the communications infrastructurethat underpins the Internet.
An end to the global semiconductor shortages that have emerged since autumn 2020 remains elusive.
The shortages have been attributed to a confluence of factors, including the COVID-19 pandemic, hostility
between democratic and authoritarian regimes, and the invasion of Ukraine. Some are of the view that thesituation will start moving toward a resolution from the latter half of this year, but the prospects remain
unclear for some of these factors, so the situation will continue to bear close watching.
As the world grows increasingly uncertain, I feel that the Internet will play an increasingly important rolein terms of the underlying infrastructure of our society and daily lives. Development of the Internet beganback in the 1960s, and to this day, new Internet technologies are still being developed, and the discussion
around rule-making and governance on the use of the Internet is ongoing.
Two important conferences related to the Internet will take place in Japan in 2023. IETF 116, which will
focus on technology development, is slated for March, and IGF 2023, on Internet governance, is slated
for November or December. Given our involvement with the Internet, we will be paying close attention tothese events.
The IIR introduces the wide range of technology that IIJ researches and develops, comprising periodic observation
reports that provide an outline of various data IIJ obtains through the daily operation of services, as well
as focused research examining specific areas of technology.
Our periodic observation report in Chapter 1 provides our analysis of IIJ’s fixed broadband and mobile traffic.
We have been performing this analysis for some time now, and the numbers show that traffic continues
to increase and that fixed broadband continues to shift from PPPoE to IPoE. Our analysis on usage by port
also bears out the shift from HTTP (TCP/80) to HTTPS (TCP/443) and to QUIC (UDP/443).
Our focused research report in Chapter 2 describes our work live-streaming the Spring Festival in Tokyo,of which IIJ is a sponsor. This music festival has been running since 2005, but since 2020, the COVID-19
pandemic has resulted in some performances not going ahead and measures such as attendance limits being
imposed. As such, we have been live-streaming the festival on the Internet so that as many people as possible
can enjoy the music on offer. This is a real-life report from members of the team that set up systems for
efficiently broadcasting from multiple venues and implemented features to make the experience enjoyablefor viewers.
Through activities such as these, IIJ strives to improve and develop its services on a daily basis while
maintaining the stability of the Internet. We will continue to provide a variety of services and solutions that
our customers can take full advantage of as infrastructure for their corporate activities.

Junichi Shimagami
Mr. Shimagami is a Senior Executive Officer and the CTO of IIJ. His interest in the Internet led to him joining IIJ in
September 1996. After engaging in the design and construction of the A-Bone Asia region network spearheadedby IIJ, as well as IIJ’s backbone network, he was put in charge of IIJ network services. Since 2015, he has been
responsible for network, cloud, and security technology across the board as CTO. In April 2017, he became chairmanof the Telecom Services Association of Japan’s MVNO Council, and in June 2021, he became a vice-chairman of theassociation.

© Internet Initiative Japan Inc.

3

1. Periodic Observation Report

Broadband Traffic Report:
COVID’s 3rd Year Brings Lull in Traffic
1.1 Overview

up again, traffic has been hovering roughly around the same

In this report, we analyze traffic over the broadband accesslevel. Over the past year, broadband IN traffic increased 13%

services operated by IIJ and present the results eachand OUT traffic increased 17%. These growth figures are

year*1*2*3*4*5. Here, we again report on changes in traffic

down a little from the year-earlier figures of 20% and 23%.

trends over the past year, based on daily user traffic andusage by port. Now in the third year of the COVID-19 pandemic,

The broadband figures include IPv6 IPoE traffic. IPv6

the solid increase in traffic we reported on last year has

traffic on IIJ’s broadband services comprises both IPoE

continued, albeit with the growth rate slowing slightly, andand PPPoE traffic. As of June 2022, IPoE accounted forso far we see no noticeable changes in that overall trend.

roughly 40% of all traffic, at 39% of IN and 41% of OUT
broadband traffic overall, year-on-year increases of 8 and

Figure 1 plots the overall average monthly traffic trends for

11 percentage points, respectively. With PPPoE congestion

IIJ’s fixed broadband services and mobile services. IN/OUT

having become quite noticeable amid COVID-19, users areindicates the direction from the ISP perspective. IN represents

increasingly shifting to IPoE, and use of IPoE thus continues

uploads from users, and OUT represents user downloads.

to rise.

Because we cannot disclose specific traffic numbers, wehave normalized the data, setting the OUT observations for

Mobile services traffic had remained range-bound amid

January 2020, just before the pandemic, for both services to 1.

COVID-19 up until we reported last year, but it subsequently
began rising from summer. It fell a little when case numbers

Broadband services traffic fell a little from autumn last year

increased again in January–February, but it has been risingwhen Japan’s COVID-19 case numbers settled down, andsince. Over the past year, mobile IN traffic increased 23%

after rising in January–February when case numbers picked

and OUT traffic increased 8%.

1.8
1.6
1.4

Traffic volume

1.2

OUT (broadband)
OUT (PPPoE Only)
OUT (mobile)
IN (broadband)
IN (PPPoE Only)
IN (mobile)

1.0
0.8
0.6
0.4
0.2
0
2016

2017

2018

2019

2020

2021

Month/Year

Figure 1: Monthly Broadband and Mobile Traffic

4*1

Kenjiro Cho. Broadband Traffic Report: Broadband Traffic Report: COVID-19’s Impact in its 2nd Year. Vol. 52. pp4-11. September 2021.

*2

Kenjiro Cho. Broadband Traffic Report: The Impact of COVID-19. Vol. 48. pp4-9. September 2020.

*3

Kenjiro Cho. Broadband Traffic Report: Moderate Growth in Traffic Volume Ongoing. Vol. 44. pp4-9. September 2019.

*4

Kenjiro Cho. Broadband Traffic Report: Download Growth Slows for a Second Year Running. Vol. 40. pp4-9. September 2018.

*5

Kenjiro Cho. Broadband Traffic Report: Traffic Growth Slows to a Degree. Internet Infrastructure Review. Vol. 36. pp4-9. September 2017.

© Internet Initiative Japan Inc.

2022

Nov.2022

Vol.

56

1. Periodic Observation Report

We now look at broadband traffic by time of day on weekdays

1.2 About the Dataamid COVID-19. Traffic volume here is the sum of PPPoE

As with previous reports, for broadband traffic, our analysis

and IPoE. Figure 2 shows traffic for the following sevenuses data sampled using Sampled NetFlow from the routersweeks. To illuminate the initial changes taking place underthat accommodate the fiber-optic and DSL broadband

COVID-19 in 2020, we show the week of February 25,

customers of our personal and enterprise broadband access

2020, before Japan’s school closures, the week of April 20,

services. For mobile traffic, we use access gateway billing

2020, corresponding to Japan’s first state of emergency,

information to determine usage volumes for personal andand the week of June 22, 2020, after the state of emergency

enterprise mobile services, and we use Sampled NetFlow

was lifted. And to illuminate changes that occurred later

data from the routers used to accommodate these services

on, we then jump forward about half a year to the week

to determine the ports used.

of January 18, 2021, and then to the weeks of July 5,
2021, January 17, 2022, and July 4, 2022. We plot hourly

Because traffic trends differ between weekdays and weekends,

average traffic volume figures for Monday–Friday for each

we analyze traffic in one-week chunks. In this report, we lookof these weeks. The lines in the lower part of each plotat data for the week of May 30 – June 5, 2022, and compare

represent uploads, but we again focus on download volume.

those data with data for the week of May 31 – June 6, 2021,
which we analyzed in the previous edition of this report.

Comparing February and April 2020 to see the impact of
the first state of emergency, we see that traffic was up

Results are aggregated by subscription for broadband traffic,

substantially in the daytime and that it also increased

and by phone number for mobile traffic as some subscriptions

during evening peak hours. When the state of emergency

cover multiple phone numbers. The usage volume for each

was lifted in June, the additional daytime traffic fell to lessbroadband user was obtained by matching the IP address

than half what it had been, but peak hours saw almost no

assigned to users with the IP addresses observed. We gathereddecline. Over the last two years, traffic during the peakstatistical information by sampling packets using NetFlow.

hours of 20:00–22:00 has seen a fairly consistent increase.

The sampling rate was set to around 1/8,192, taking into

Meanwhile, in both 2021 and 2022, daytime traffic was up

account router performance and load. We estimated overallsignificantly in January but not much in July. This probably

usage volumes by multiplying observed volumes by thereflects that in January of both years, people spent more

reciprocal of the sampling rate. Note that IPoE traffic istime at home amid increases in COVID-19 case numbers,

not included in the analysis of traffic by port, as detailed

whereas the COVID-19 situation was relatively calm at thedata is not available because we use Internet Multifeed

start of both Julys. So over the past two years, traffic has

Co.’s transix service for IPoE.

seen a solid increase during the nighttime peak hours, while
increases during daytime hours have been influenced by the

Weekday Traffic

rates of people staying at home in response to COVID-19.

0
0：00

6：00

12：00

18：00

Jul 4–8, 2022
Jan 17–21, 2022
Jul 5–9, 2021
Jan 18–22, 2021
Jun 22–26, 2020
Apr 20–24, 2020
Feb 25–28, 2020

Figure 2: Hourly Average Broadband Traffic on Weekdays© Internet Initiative Japan Inc.

5

1.3 Users’ Daily Usage

Figures 3 and 4 show the average daily usage distributionsFirst, we examine daily usage volumes for broadband and(probability density functions) for broadband (PPPoE andmobile users from several angles. Daily usage indicates theIPoE) and mobile users. Each compares data for 2021 andaverage daily usage calculated from a week’s worth of data

2022 split into IN (upload) and OUT (download), with user

for each user.

traffic volume plotted along the X-axis and user frequency
along the Y-axis. The X-axis shows volumes between

Since our 2019 report, we have used daily usage data

10KB (10 4) and 100GB (1011) using a logarithmic scale.

only on services provided to individuals. The distribution is

Most users fall within the 100GB (1011) range, with a few

heavily distorted if we include enterprise services, where

exceptions.

usage patterns are highly varied. So to form a picture ofoverall usage trends, we determined that using only the

The IN and OUT traffic distributions in the figures are

personal user data would yield more generally applicable,

close to a log-normal distribution, which looks like a normal

easily interpretable conclusions. Starting with this report,

distribution on a semi-log plot. A linear plot would show

we also include IPoE user data. Note that the analysis ofa long-tailed distribution, with the peak close to the left

usage by port in the next section does include enterprise

and a slow gradual decrease toward the right. The OUT

data because of the difficulty of distinguishing between

distribution is further to the right than the IN distribution,individual and enterprise usage.

indicating that download volume is more than an order ofmagnitude larger than upload volume.

1

Probability Density

0.8

2021(IN)
2021(OUT)
2022(IN)
2022(OUT)First, we look at the broadband PPPoE distributions in Figure
3. Both the IN and OUT distributions have barely changed
between 2021 and 2022. A close look reveals that the

0.6

2022 OUT distribution peak is ever so slightly lower while
the midrange of the left side of the distribution is com-

0.4

mensurately a little higher, indicating that the proportion ofrelatively low-usage users has risen a bit.

0.2

The broadband IPoE distributions in Figure 4 are shifted
0 4
10(10KB)

5

10(100KB)

6

7

8

9

10
10
10
10
(1MB)
(100MB)(1GB)(10MB)
Users' Daily Traffic Volume (Bytes)

1010(10GB)

11

10(100GB)

Figure 3: Daily Broadband (PPPoE) User Traffic Volume Distribution
Comparison of 2021 and 2022

further to the right than the PPPoE distributions, and overall
usage is much higher than for PPPoE. The distributions
are also narrower than for PPPoE and almost symmetrical.

1

1
IPoE(2022)

0.6

0.4

0.2

0 4
10
(10KB)

0.6

0.4

0.2

5

10
(100KB)

6

7

8

9

10
10
10
10
(1MB)
(100MB)(1GB)(10MB)
Users' Daily Traffic Volume (Bytes)

1010(10GB)

11

10(100GB)

Figure 4: Daily Broadband (IPoE) User Traffic Volume Distribution
Comparison of 2021 and 2022

62021(IN)
2021(OUT)
2022(IN)
2022(OUT)

0.8

Probability Density

Probability Density

0.8

2021(IN)
2021(OUT)
2022(IN)
2022(OUT)

0 4
10(10KB)

5

10(100KB)

6

7

8

9

10
10
10
10
(1MB)
(100MB)(1GB)(10MB)
Users' Daily Traffic Volume (Bytes)

1010(10GB)

Figure 5: Daily Mobile User Traffic Volume Distribution
Comparison of 2021 and 2022

© Internet Initiative Japan Inc.

11

10
(100GB)Nov.2022

Vol.

56

1. Periodic Observation Report

While the proportion of low-usage users on the left side ofTable 1 shows trends in the mean and median daily traffic

the distributions is smaller, there is not much difference forvalues for broadband PPPoE users as well as the mode

high-usage users on the right of the distributions. The IPoE(the most frequent value, which represents the peak of

distributions do not look much different from last year, but

the distribution). When the peak is slightly off the center

the peaks are slightly higher, and in contrast to PPPoE, theof the distribution, the distribution is adjusted to bring the

proportion of observations near the mode increased.

mode toward the center. Comparing 2021 and 2022, the IN
mode fell from 200MB to 178MB and the OUT mode was

So the overall broadband distributions for both PPPoE andunchanged at 3,981MB, translating into growth factors of

IPoE have hardly changed. Meanwhile, with the migration

0.9 for IN and 1 for OUT. Meanwhile, because the meansfrom PPPoE to IPoE progressing, the proportion accounted

are influenced by heavy users (on the right-hand side of thefor by IPoE is rising, so overall traffic volume is increasing.

distribution), they are significantly higher than the corresponding modes, with the IN mean at 698MB and the OUT

The peaks of the mobile distributions in Figure 5 havemean at 4,291MB in 2022. The 2021 means were 684MB

moved a little to the right since last year, indicating thatand 4,225MB, respectively.

overall traffic has increased. Mobile usage volumes are
significantly lower than for broadband, and limits on mobile

As Table 2 shows, IPoE usage is much higher than PPPoE

data usage mean that heavy users, which fall on the right-

usage. This is probably because relatively high-usage users

hand side of the distribution, account for only a small

are moving to IPoE first, or in other words, because many

proportion of the total. There are also no extremely heavyof the low-usage users still linger on PPPoE. In 2022, theusers. The variability in each user’s daily usage volume is

IN mode was 398MB and the OUT mode was 6,310MB,

higher for mobile than for broadband owing to there being

while the means were IN 1,007MB and OUT 7,700MB. The

users who only use mobile data when out of the home/2021 modes were IN 447MB and OUT 6,310MB, and theoffice as well as limits on mobile data.

means were IN 1,110MB and OUT 7,169MB.

Table 1: Trends in Mean and Modeof Broadband (PPPoE) Users’ Daily Traffic Volume
IN(MB/day)

OUT(MB/day)

Year

Mean

MedianMode

MeanMedian

Mode

2007

436

5

5

718

59

56

2008

490

6

6

807

75

79

2009

561

6

6

973

91

100

2010

442

7

7

878

111

126

2011

398

9

9

931

144

200

2012

364

11

13

945

176

251

2013

320

13

16

928

208

355

2014

348

21

28

1124

311

501

2015

351

32

45

1399

443

708

2016

361

48

63

1808

726

1000

2017

391

63

79

2285

900

1259

2018

428

66

79

2664

1083

1585

2019

479

75

89

2986

1187

1995

2020

609

122

158

3810

1638

3162

Year

Mean

2021

684

136

200

4225

1875

3981

2021

2022

698

130

178

4291

1778

3981

2022

Table 2: Trends in Mean and Modeof Broadband (IPoE) Users’ Daily Traffic Volume
IN(MB/day)© Internet Initiative Japan Inc.

OUT(MB/day)MedianMode

MeanMedian

Mode

1110

312

447

7169

4285

6310

1007

336

398

7700

4935

6310

7

All of the mobile traffic metrics were up, as Table 3 shows.

the upper right of the diagonal, but this is now no longer

In 2022, the IN mode was 10MB and the OUT mode wasdiscernible. Variability between users in terms of usage

89MB, while the means were IN 13MB and OUT 114MB.

levels and IN/OUT ratios is wide, indicating that there is

The 2021 modes were IN 8MB and OUT 71MB, and thea diverse range of usage styles. On IPoE, there is less vari-means were IN 10MB and OUT 86MB.

ability between users than with PPPoE, and the proportionof low-usage users is small. For mobile traffic, the patternFigures 6, 7, and 8 plot per-user IN/OUT usage volumes

of OUT being an order of magnitude larger also applies,

for random samples of 5,000 users. The X-axis shows

but usage volumes are much lower than for broadband. For

OUT (download volume) and the Y-axis shows IN (upload

both broadband and mobile, there is almost no difference

volume), with both using a logarithmic scale. Users with

between these plots and those for 2021.

identical IN/OUT values fall on the diagonal.
Traffic is heavily skewed across users, such that a small
The cluster spread out below and parallel to the diagonal

proportion of users accounts for the majority of overall

in each of these plots represents typical users with download

traffic volume. For example, the top 10% of broadband

volumes an order of magnitude higher than upload volumes.

users account for 50% of total OUT and 78% of total IN

For broadband PPPoE traffic, there was previously a clearlytraffic, while the top 1% of users account for 16% of OUT

recognizable cluster of heavy users spread out thinly about

and 53% of IN traffic.

11

10

IPoE(2022)

Table 3: Trends in Mean and Modeof Mobile Users’ Daily Traffic Volum

Year

Mean

MedianUser’ s Daily Upload Volume (Bytes)

IN(MB/day)

10

10

OUT(MB/day)
Mode

Mean

Median

Mode

2015

6.2

3.2

4.5

49.2

23.5

44.7

2016

7.6

4.1

7.1

66.5

32.7

63.1

2017

9.3

4.9

7.9

79.9

41.2

79.4

2018

10.5

5.4

8.9

83.8

44.3

79.4

2019

11.2

5.9

8.9

84.9

46.4

79.4

2020

10.4

4.5

7.1

79.4

35.1

63.1

10
10
10
10
10
10

9

8

7

6

5

4
6

10
(1MB)

7

10
(10MB)

8

10(100MB)

9

10
(1GB)

10

10
(10GB)

2021

9.9

4.7

7.9

85.9

37.9

70.8

User’ s Daily Download Volume (Bytes)

12.8

6.0

10.0

113.7

49.2

89.1

Figure 7: IN/OUT Usage for Each Broadband (IPoE) User

11

11

Mobile(2022)

10

10

10
10
10
10
10

10
User’ s Daily Upload Volume (Bytes)

10
10

11

10
(100GB)

10

Total(2022)

User’ s Daily Upload Volume (Bytes)

10(100KB)

2022

10

9

8

7

6

5

4
4

10
(10KB)

8

5

4

10(10KB)

5

10(100KB)

6

10
(1MB)

7

10
(10MB)

8

10(100MB)

9

10
(1GB)

10

10
(10GB)

11

10(100GB)

10
10
10
10
10
10

9

8

7

6

5

4
4

10
(10KB)

5

10(100KB)

6

10
(1MB)

7

10
(10MB)

8

10(100MB)

9

10
(1GB)

User’ s Daily Download Volume (Bytes)

User’ s Daily Download Volume (Bytes)

Figure 6: IN/OUT Usage for Each Broadband (PPPoE) User

Figure 8: IN/OUT Usage for Each Mobile User

© Internet Initiative Japan Inc.

10

10
(10GB)

11

10(100GB)Nov.2022

Vol.

56

1. Periodic Observation Report

The skew has increased just slightly from last year. On

than 1024, the traffic is likely to be from a client/server

IPoE, the skew is smaller than for PPPoE, with the top

application. In light of this, we take the lower of the source

10% of users accounting for 39% of OUT and 64% of IN

and destination port numbers when breaking down TCP

traffic and the top 1% of users accounting for 11% of OUT

and UDP usage volumes by port.

and 36% of IN traffic. As for mobile, the top 10% of users
account for 50% of OUT and 49% of IN traffic, while theTable 4 shows the percentage breakdown of broadband

top 1% account for 13% of OUT and 16% of IN traffic.

users’ usage by port over the past five years. In 2022, 72%

The skew in OUT traffic has increased just slightly fromof all traffic was over TCP connections, largely unchanged

last year.

from 2021. The proportion of traffic over port 443 (HTTPS)
was 56%, a 2-point increase from last year. The proportion

1.4 Usage by Portof traffic over port 80 (HTTP) fell from 12% to 9%. The figure

Next, we look at a breakdown of traffic and examine usage

for UDP port 443, which is used by the QUIC protocol, waslevels by port. Recently, it has become difficult to identify

largely unchanged at 16%.

applications by port number. Many P2P applications use
dynamic ports on both ends, and a large number of

TCP dynamic port traffic, which had been in decline, looks

client/server applications use port 80, which is assigned toto have leveled out at 6%. Individual dynamic port numbers

HTTP, to avoid firewalls. Hence, generally speaking, when

account for only a tiny portion, with the most commonlyboth parties are using a dynamic port numbered 1024 orused port 31000 only making up 0.9%. Port 1935, whichhigher, the traffic is likely to be from a P2P application, andis used by Flash Player, makes up 0.2%, but almost allwhen one of the parties is using a well-known port lower

other traffic is VPN related.

Table 4: Broadband Users’ Usage by Port

year

2018

2019

2020

2021

2022

(%)

(%)

(%)

(%)

(%)

TCP

78.5

81.2

77.2

71.9

71.6

(< 1024)

68.5

73.3

70.5

65.8

65.4

443(https)

40.7

51.9

52.4

53.5

55.7

80(http)

26.5

20.4

17.2

11.6

8.9

183

0.0

0.0

0.0

0.1

0.2

protocolport

22(ssh)

0.1

0.2

0.2

0.2

0.1

993(imaps)

0.2

0.3

0.2

0.1

0.1

(>= 1024)

10.0

7.9

6.7

6.1

6.2

31000

0.1

0.2

0.4

0.6

0.9

8080

0.3

0.5

0.4

0.4

0.3

1935(rtmp)

0.7

0.3

0.4

0.2

0.2

UDP

16.4

14.1

19.4

24.5

24.3

443(https)

10.0

7.8

10.5

15.9

16.3

4500(nat-t)

0.2

0.3

0.6

0.8

0.8

8801

0.0

0.0

1.1

0.9

0.6

ESP

4.8

4.4

3.2

3.3

3.8

GRE

0.1

0.1

0.1

0.2

0.2

IP-ENCAP

0.2

0.2

0.1

0.1

0.1

ICMP

0.0

0.0

0.0

0.0

0.0

© Internet Initiative Japan Inc.

9

Table 5 shows the percentage breakdown by port for mobile

Figure 9 compares overall broadband traffic for key port

users. The figures are close to those for broadband on the

categories across the course of the week from whichwhole. This is likely because apps similar to those for PC

observations were drawn in 2021 and 2022. We breakplatforms are now also used on smartphones, and because

the data into four port buckets: TCP ports 80 and 443,

the proportion of broadband usage on smartphones is rising.

and dynamic TCP ports (1024 and up), and UDP port 443.
The data are normalized so that peak overall traffic volume

Broadband data only include PPPoE, not IPoE, and soon the plot is 1. The overall peak is around 19:00–23:00.

do not necessarily reflect the trend in fixed broadband

When compared, there is almost no difference between

overall. Comparing IPv4 and IPv6 on mobile, port 443

2021 and 2022, and it appears that the migration fromaccounts for a higher proportion of both TCP and UDP

HTTP to HTTPS has largely run its course for now.

usage on IPv6, and there is probably a similar trend in thecase of IPoE.

Figure 10 shows the trend for TCP ports 80 and 443 andUDP port 443, which account for the bulk of mobile
traffic. Here too, there is almost no change from 2021. When
compared with broadband, we note that mobile traffic levels

Table 5: Mobile Users’ Usage by Portyear

remain high throughout the day, from morning through

2018

2019

2020

2021

2022

(%)

(%)

(%)

(%)

(%)

TCP

76.6

76.9

75.5

70.3

71.6

443(https)

52.8

55.6

50.7

44.4

42.3

protocolport

80(http)

16.7

10.3

7.4

5.0

4.1

993(imaps)

0.3

0.3

0.2

0.2

0.1

1935(rtmp)

0.1

0.1

0.1

0.1

0.1

UDP

19.4

17.3

18.0

23.8

24.4

443(https)

10.6

8.3

9.3

16.3

17.9

4500(nat-t)

4.5

3.0

1.8

3.7

2.7

1701(12tp)

0.0

0.4

0.9

0.0

0.8

8801

0.0

0.0

1.4

0.7

0.3

3480

0.0

0.0

0.4

0.3

0.1

ESP

3.9

5.8

6.4

5.8

3.9

GRE

0.1

0.0

0.1

0.1

0.0

ICMP

0.0

0.0

0.0

0.0

0.0

Figure 9: Broadband Users’ Port Usage Over a Week
2021 (top) and 2022 (bottom)

10night. The plot shows that usage times differ from those forbroadband, with three separate mobile traffic peaks occurring
on weekdays: morning commute, lunch break, and eveningaround 17:00–22:00.

Figure 10: Mobile Users’ Port Usage Over a Week
2021 (top) and 2022 (bottom)© Internet Initiative Japan Inc.

Nov.2022

Vol.

56

1. Periodic Observation Report

1.5 Conclusion

traffic volume has been solid, driven by the migration toIn summary, the trends in traffic we reported on last year

IPoE, but beyond that, there has not been much change

have continued to hold over the past twelve months,

in individual users’ usage volumes. Hence, while it iswhich marked COVID-19’s third year, with no major

evident that online conferencing and video streamingchanges apparent. Daytime broadband traffic does lookare establishing themselves as the norm, looking at theto have been shaped by the rate at which people stayed

past year in isolation, we do not really see any major

home in response to infection case numbers, but it ischanges in the services users use or how they use them.

steadily rising during the peak hours. Growth in overall

Kenjiro Cho
Research Director, Research Laboratory, IIJ© Internet Initiative Japan Inc.

11

2. Focused ResearchLive-Streaming the Spring Festival in Tokyo—Our Report
2.1 Introduction

2.2 Live-Streaming Specifications and Solutions

The Spring Festival in Tokyo is one of Japan’s largest

In 2022, the music festival organizers asked us if weclassical music festivals, held once a year at music halls,

could offer an increased range of payment options while

museums, and other venues around Ueno. Originally the

keeping ticket prices reasonable to boost live-stream

Tokyo Opera Nomori, which began in 2005, the festival inviewership. We added in-app smartphone (iOS/Android)

its current format has been adding color to Ueno’s spring

payment support to make it easier to buy tickets. And

scene since 2009. The 2019 event was huge, featuring

we had to do a full review of costs to ensure ticket prices

208 performances across all venues, but most of thewould be reasonable. Our goal with the live-streaming

performances were canceled in 2020 when COVID-19

specifications was to keep a lid on costs everywhere we

hit, leaving only 14 performances that year.

could, including how we deployed equipment and staff
and how we configured our systems, while not impairing

In 2021, the festival went ahead under restrictions tothe user experience.

prevent the spread of COVID-19—a maximum attendanceof 5,000 or a 50% venue capacity limit—so to make up for■ Consideration 1: Minimizing equipment

the limitations on physical attendance, we live-streamed

Up until 2021, we brought several cameras to each venue

all 60 performances held across the 14 venues.

so we could switch camera angles according to what was
happening in the performance, but our uniform approach

In 2022, as in 2021, we live-streamed almost all of thein 2022 was to install a single 4K camera in the back row

56 performances across 14 venues. With over 10 venuesof each venue to capture the entire scene. This madebeing streamed and some performance or other taking place

it possible to greatly reduce the amount of equipment

every day over the festival’s month-or-so running time, on

brought into the venues. On the other hand, it meant thatsome days we streamed as many as four performances

the viewer would also be seeing a full-angle image of thesimultaneously. To stream multiple performances at a time,

performance. So to make up for the video feed not being

in 2021 we carted a whole lot of equipment to the Tokyo

switched to focus on the highlights of each performance,

Bunka Kaikan, one of the venues, and set up a temporary

we incorporated zoom and pan features into the video

streaming center to record and broadcast the events. This

player to give viewers control over the viewport. Also,

year, however, we established our streaming center at IIJ’s

although the song list is posted on the player page, it

headquarters in Iidabashi and greatly increased the levelis not visible in fullscreen mode, so we added a featureof operations performed remotely via FLET’S and mobile

to the player to let viewers check the song list while inconnectivity in an attempt to reduce streaming costs,

fullscreen mode as well.

including those associated with personnel resources.
■ Consideration 2: On-site staff deployments

12This report takes you behind the live-streaming scenes

Since the performance venues change daily and are publicby explaining our process for deciding on system con-facilities, we are not able to set and leave our broadcast

figurations and the IP technology we used, along withequipment within the venues for the whole duration of thesome added color on the trials and tribulations of making

festival. Hence, we had to set things up before the start

everything work.

of every performance, and time was very tight at times© Internet Initiative Japan Inc.

Nov.2022

Vol.

56

2. Focused Researchdepending on what time performances were scheduled to

to stream four performances at once. Setting the streaming

start on the day in question. Laying cables within the venue

center up at IIJ’s headquarters in Iidabashi in 2022 obviated

was particularly challenging. Stability is important when

the need for people to travel, transport equipment, and build

transmitting video, so we were using wired connections,and operate a separate center. Staff responsible for feed

and this meant laying cables 100m long in large venues,

switching no longer needed to travel to Ueno, which

which required several staff to work together. To reduce

reduced not only the costs but also the workload on staff.

the workload for on-site staff, tasks such as focusing the

To prevent problems arising with the remote setup, wecamera, adjusting brightness, and performing audio level

decided on how staff would communicate with each other,and LR tests were accomplished remotely rather than bywhat the procedures would be for installing and checking

staff at the venue.

equipment, and so on before the festival, and we set rules
on updating our operating procedures as needed and swiftly■ Consideration 3: Remote production

communicating these changes to staff. These efforts are

Up until 2021, we set our streaming center up at the Tokyo

what made the remote production setup work.

Bunka Kaikan in Ueno, from where we performed the work
involved in transmitting the video to viewers, which included

Based on these three considerations, we decided on therecording, video feed switching, and subtitling, and we had

following specifications: the event would only be live

to bring a huge amount of equipment to Ueno to enable us

streamed with no archives kept, video would be at most■2021 setup♫

Steam ticket sales

IIJ WAN Service

Streaming
center

Ueno

Data Center

4K multi-angle■2022 setup♫

Remote control
Stream ticket sales

IIJ WAN Service

Ueno

Streaming
centerData Center

4K FreeViewpoint

Iidabashi

https://www.harusailive.jp/test/
Figure 1: Live-Streaming System Configuration Comparison (2021 & 2022)© Internet Initiative Japan Inc.

13

4K (with some 2K feeds), and sound would be recorded at■ Transmitting video from Ueno to IIJ headquarters

AAC 256kbps. In 2021, streaming tickets cost ¥1,500–

For the main 4K transmission circuit, we used the IIJ

2,500 (tax inclusive, same below), while in 2022 we set a

Multi Product Controller Service (https://www.iij.ad.jp/

relatively affordable price of around ¥1,100 per performanceen/biz/mpc/), which enables centralized management(with some available for ¥730).

and integrated operation of networked devices, andbuilt a Layer 3 VPN using IPv6 loopback on the FLET’S

2.3 Live-Streaming System Configuration

Hikari Next service. The high-performance IIJ router

We now describe how we set up the live-streaming system.

SEIL/X4, which we used as the VPN adapter, retrieves
its own configuration data from its management server■ Cameras

when powered on and wired up, so there is no need to

We used Panasonic’s AW-UE100K 4K integrated cameras.

have engineers on site to configure and check the de-We selected this model because its size would be unob-

vice. And because the status of all devices is available attrusive at the venues and because it supported remote

glance on the router’s control panel, it is very useful inPTZ (pan, tilt, and zoom). The AW-RP60G controller

situations, such as the festival, when there are multiple

equipped with a PTZ joystick allowed us to move thedevices to manage.

camera through the horizontal and vertical planes as wellas control zoom in/out, image brightness, and so forth

When we measured network bandwidth ahead of theeasily from a remote location (IIJ headquarters). This let

performances, there were differences depending on thethe on-site staff concentrate on the physical installationof the cameras, cables, and so on, thus reducing the on-site
staffing requirement.
1. Set up tripod and camera at venue
2. Check camera level/tilt remotely
and ask venue staff to adjust tripod
3. Use controller to remotely adjust
view angle and brightness
4. Send L and R audio signals fromvenue and check remotely for
CameraAW-UE100K

Controller
AW-RP60GJ

left/right stereo alignment
Tripod and camera placed at back of venue

Figure 2: Camera and Controller

Ueno

Figure 3: Equipment Setup Procedure at Venue

FLET'S IPv6 loopback

Iidabashi

Figure 4: Transmitting Video from Ueno to IIJ

Mobile networks

Ueno

IIJ Mobile
Docomo, au,
Softbank

Figure 5: Illustration of LiveU Backup Connection

14© Internet Initiative Japan Inc.

Iidabashi

Nov.2022

Vol.

56

2. Focused Researchtime of day, but we were getting around 200Mbps over the

2017, and the SRT Alliance had over 500 members as

VPN tunnel, which is sufficient bandwidth for effectivelyof June 2022; IIJ is also a member (https://www.srttal-transmitting video.

liance.org/). SRT is based on UDP, and while maintaining
low latency, it also provides more packet recovery features

We used LiveU (https://www.liveu.tv/ja) video transmission

than TCP, and supports encryption using AES. As one ofdevices as a backup and virtually bundled together

the leading protocols for transmitting video over broadband

multiple mobile carriers’ connections, over which weservices and the like where bandwidth is not guaranteed, ittransmitted video in HD. The LiveU system allows multiple

can be expected to be adopted on an increasing number ofSIMs to be installed to enable the transmission of more

devices going forward.

video data over multiple mobile connections, and is evenused by broadcasters for live coverage. We used LTE■ Transmitting and buffering

contracts for the SIMs because 5G was not yet available

Our connection had an effective bandwidth of 200 Mbps,

at the venues.

so we figured we would be able to transmit using SRT
at around 30Mbps without any problems, but when we

Reception was poor at some venues, and some of themactually tried it out, the system was generating annoying

jammed mobile signals during the performances, so welittle noises every few minutes. The error counter did notneeded to know the quirks of each individual venue torise when this happened, so we were unable to identify thedetermine where to install the LiveU units. Some trial andcause, but we were able to resolve it by setting a larger

error was involved in selecting installation spots, but we

buffer size on the receiving device.

were generally able to transmit in HD without issue at bit
rates of around 10–15Mbps.

A larger buffer increases the broadcast delay, but thiswas not a problem because the content we were dealing■ Transmission protocols

with was concerts, so strictly real-time coverage was not

Our video resolution and frame rate was 4K at 30p.

really a major imperative. When live-streaming sports andUncompressed, this would take as much as 6Gbps, soother such events, however, low latency is a must, so thewe compressed it to around 30Mbps using H.265 andsettings need to be adjusted according to the characteristics

transmitted it to IIJ headquarters using the SRT protocol.

of the connection being used.

SRT stands for Secure Reliable Transport and is a video■ Subtitles

transport protocol published by HaiVision in 2014 to fa-

The operas were not performed in Japanese. Lohengrin

cilitate the transmission of video via the Internet. A range

is sung in German and Turandot in Italian, so in the caseof vendors have adopted it since it was open-sourced inof large concert halls, Japanese translations appeared

Figure 6: Player with Subtitle Support© Internet Initiative Japan Inc.

15

on displays on both sides of the stage. Although there

stage. These timing instructions are issued in real time,

were no such in-venue subtitles for songs performed at

but the video is transmitted with a few seconds’ delay,

art galleries and museums, we had to include subtitles

so our subtitling system includes an appropriate delay tofor users watching via stream. In 2021, we hardcoded

ensure the text appears at the right time.

the subtitles into the video, but in 2022, we provided
subtitles using the setup shown in Figure 7. This meant■ Streaming center at IIJ headquartersthat users were able to easily turn subtitles on and off

We were in charge of coordinating the video feeds ofduring the performances.

the performances coming in from the venues in Ueno,
inserting on-screen text before the performance, during

To display the subtitles at the right time, the operator

intermission, and at the end, and transcoding the video

needs to keep the subtitling system in time with theinto a format viewable by end users and streaming it out.

song. But the engineers alone cannot handle the timing.

As the photo shows, we had displays as well as switchers

To deal with this issue, music festival staff who properly

and other equipment lined up in our workspace to enable

understood the content helped our engineers to display

us to process up to four performances simultaneously.

the right subtitles at the right time. We built a system

We used a network monitoring tool (Zabbix) to visualize

to insert the subtitles into the video data in the form of

network communications and bandwidth (video bitrate),

ID3 metadata in real time in accordance with the instructions

letting us easily determine venue status and see the statusissued, and we set our operations up so that we couldof our communications systems at a glance, and thiscontrol this subtitling system remotely in response

turned out to be an important means of telling what was

to instructions from staff positioned at the side of thehappening from a remote location.

Venue

Subtitle server
Subtitling
instructions

Instruction
issuerSubtitle
operator

Delay added

Subtitles inserted in ID3 tag

Streaming system
Video sent
over SRT

Streamed
broadcast

CameraFigure 7: Subtitling System Diagram

16

Figure 8: Streaming Center at IIJ Headquarters© Internet Initiative Japan Inc.

Nov.2022

Vol.

56

2. Focused Research■ Communication tools

team members posted and how granular the information

Communication among remotely located team members

they provided was. This was a reminder that with any

is key to making remote production work. It is common totool, we need to make sure everyone is fully aware ofuse an intercom system at live events, but on this occa-

why we are using the tool and what the communication

sion, we used Slack (chat room tool) and set up a channel

rules are.

for each venue. Although this created the extra work oftyping text messages, it also let us check on progress

2.4 Conclusion

at other venues at a glance and made it easy to go backPreparations for the Spring Festival in Tokyo 2023 arethrough the timeline and follow our workflow history, so

already underway, and we have started selecting andit provided excellent visibility. It also proved to be very

testing new equipment to push remote production even

useful in enabling the festival organizers to share certain

further. In light of the results and lessons learned thisdetails with us, such as information on encores, which

year, we will be working to provide an even better

is often sorted out on the day of the performance. That

streaming experience ahead.

said, we did note individual differences in terms of whenOn-site engineers

Engineerat venue C
Engineerat venue B

Progress report

Progress report

Engineerat venue A

Performance updates
Slack

Microsoft Teams
Voice call

Progress report(channel for each venue)• Progress Chart• Start time

• Intermission time

Stagingmanager• Encore: yes/no• Information aggregated in Slack so venue status is available at a glanceStreaming center engineers• Microsoft Teams to be used for voice calls so that group calls can be set up

Figure 9: How We Communicated Depending on the Situation

Hiro OkadaxSP System Services, Network Division, IIJ
Intrigued by the Internet, Mr. Okada joined IIJ in April 2000. After working in sales, he went on to work in business customer connection
services. Since 2015, he has worked in streaming/broadcast-related business. He is involved in starting up paid streaming platforms,
studios, etc.

Fumitaka Watanabe
Streaming Businesses Section, xSP System Services, Network Division, IIJ
Having previously worked in the digitization and development of broadcasting services, from the launch of BS digital broadcast channels
through to advanced BS broadcasting, Mr. Watanabe currently focuses on developing and proposing solutions for increasingly diverse video
streaming/broadcasting services, including simulcasting and VOD/live-streaming.

© Internet Initiative Japan Inc.

17

56
November 2022 Vol.
About Internet Initiative Japan Inc. (IIJ)
IIJ was established in 1992, mainly by a group of engineers who
had been involved in research and development activities related
to the Internet, under the concept of promoting the widespreaduse of the Internet in Japan.
IIJ currently operates one of the largest Internet backbones
in Japan, manages Internet infrastructures, and provides
comprehensive high-quality system environments (includingInternet access, systems integration, and outsourcing services,etc.) to high-end business users including the government andother public offices and financial institutions.
In addition, IIJ actively shares knowledge accumulated throughservice development and Internet backbone operation, andis making efforts to expand the Internet used as a social
infrastructure.­­The copyright of this document remains in Internet Initiative Japan Inc.
(“IIJ”) and the document is protected under the Copyright Law of Japanand treaty provisions. You are prohibited to reproduce, modify, or makethe public transmission of or otherwise whole or a part of this document
without IIJ’s prior written permission. Although the content of thisdocument is paid careful attention to, IIJ does not warrant the accuracy andusefulness of the information in this document.
Initiative Japan Inc. All rights reserved.
© Internet
IIJ-MKTG020-0054

Internet Initiative Japan Inc.
Address: Iidabashi Grand Bloom, 2-10-2 Fujimi, Chiyoda-ku,
Tokyo 102-0071, Japan
Email: info@iij.ad.jp URL: https://www.iij.ad.jp/en/