2. Focused ResearchLive-Streaming the Spring Festival in Tokyo—Our Report
2.1 Introduction

2.2 Live-Streaming Specifications and Solutions

The Spring Festival in Tokyo is one of Japan’s largest

In 2022, the music festival organizers asked us if weclassical music festivals, held once a year at music halls,

could offer an increased range of payment options while

museums, and other venues around Ueno. Originally the

keeping ticket prices reasonable to boost live-stream

Tokyo Opera Nomori, which began in 2005, the festival inviewership. We added in-app smartphone (iOS/Android)

its current format has been adding color to Ueno’s spring

payment support to make it easier to buy tickets. And

scene since 2009. The 2019 event was huge, featuring

we had to do a full review of costs to ensure ticket prices

208 performances across all venues, but most of thewould be reasonable. Our goal with the live-streaming

performances were canceled in 2020 when COVID-19

specifications was to keep a lid on costs everywhere we

hit, leaving only 14 performances that year.

could, including how we deployed equipment and staff
and how we configured our systems, while not impairing

In 2021, the festival went ahead under restrictions tothe user experience.

prevent the spread of COVID-19—a maximum attendanceof 5,000 or a 50% venue capacity limit—so to make up for■ Consideration 1: Minimizing equipment

the limitations on physical attendance, we live-streamed

Up until 2021, we brought several cameras to each venue

all 60 performances held across the 14 venues.

so we could switch camera angles according to what was
happening in the performance, but our uniform approach

In 2022, as in 2021, we live-streamed almost all of thein 2022 was to install a single 4K camera in the back row

56 performances across 14 venues. With over 10 venuesof each venue to capture the entire scene. This madebeing streamed and some performance or other taking place

it possible to greatly reduce the amount of equipment

every day over the festival’s month-or-so running time, on

brought into the venues. On the other hand, it meant thatsome days we streamed as many as four performances

the viewer would also be seeing a full-angle image of thesimultaneously. To stream multiple performances at a time,

performance. So to make up for the video feed not being

in 2021 we carted a whole lot of equipment to the Tokyo

switched to focus on the highlights of each performance,

Bunka Kaikan, one of the venues, and set up a temporary

we incorporated zoom and pan features into the video

streaming center to record and broadcast the events. This

player to give viewers control over the viewport. Also,

year, however, we established our streaming center at IIJ’s

although the song list is posted on the player page, it

headquarters in Iidabashi and greatly increased the levelis not visible in fullscreen mode, so we added a featureof operations performed remotely via FLET’S and mobile

to the player to let viewers check the song list while inconnectivity in an attempt to reduce streaming costs,

fullscreen mode as well.

including those associated with personnel resources.
■ Consideration 2: On-site staff deployments

12This report takes you behind the live-streaming scenes

Since the performance venues change daily and are publicby explaining our process for deciding on system con-facilities, we are not able to set and leave our broadcast

figurations and the IP technology we used, along withequipment within the venues for the whole duration of thesome added color on the trials and tribulations of making

festival. Hence, we had to set things up before the start

everything work.

of every performance, and time was very tight at times© Internet Initiative Japan Inc.

Nov.2022

Vol.

56

2. Focused Researchdepending on what time performances were scheduled to

to stream four performances at once. Setting the streaming

start on the day in question. Laying cables within the venue

center up at IIJ’s headquarters in Iidabashi in 2022 obviated

was particularly challenging. Stability is important when

the need for people to travel, transport equipment, and build

transmitting video, so we were using wired connections,and operate a separate center. Staff responsible for feed

and this meant laying cables 100m long in large venues,

switching no longer needed to travel to Ueno, which

which required several staff to work together. To reduce

reduced not only the costs but also the workload on staff.

the workload for on-site staff, tasks such as focusing the

To prevent problems arising with the remote setup, wecamera, adjusting brightness, and performing audio level

decided on how staff would communicate with each other,and LR tests were accomplished remotely rather than bywhat the procedures would be for installing and checking

staff at the venue.

equipment, and so on before the festival, and we set rules
on updating our operating procedures as needed and swiftly■ Consideration 3: Remote production

communicating these changes to staff. These efforts are

Up until 2021, we set our streaming center up at the Tokyo

what made the remote production setup work.

Bunka Kaikan in Ueno, from where we performed the work
involved in transmitting the video to viewers, which included

Based on these three considerations, we decided on therecording, video feed switching, and subtitling, and we had

following specifications: the event would only be live

to bring a huge amount of equipment to Ueno to enable us

streamed with no archives kept, video would be at most■2021 setup♫

Steam ticket sales

IIJ WAN Service

Streaming
center

Ueno

Data Center

4K multi-angle■2022 setup♫

Remote control
Stream ticket sales

IIJ WAN Service

Ueno

Streaming
centerData Center

4K FreeViewpoint

Iidabashi

https://www.harusailive.jp/test/
Figure 1: Live-Streaming System Configuration Comparison (2021 & 2022)© Internet Initiative Japan Inc.

13

4K (with some 2K feeds), and sound would be recorded at■ Transmitting video from Ueno to IIJ headquarters

AAC 256kbps. In 2021, streaming tickets cost ¥1,500–

For the main 4K transmission circuit, we used the IIJ

2,500 (tax inclusive, same below), while in 2022 we set a

Multi Product Controller Service (https://www.iij.ad.jp/

relatively affordable price of around ¥1,100 per performanceen/biz/mpc/), which enables centralized management(with some available for ¥730).

and integrated operation of networked devices, andbuilt a Layer 3 VPN using IPv6 loopback on the FLET’S

2.3 Live-Streaming System Configuration

Hikari Next service. The high-performance IIJ router

We now describe how we set up the live-streaming system.

SEIL/X4, which we used as the VPN adapter, retrieves
its own configuration data from its management server■ Cameras

when powered on and wired up, so there is no need to

We used Panasonic’s AW-UE100K 4K integrated cameras.

have engineers on site to configure and check the de-We selected this model because its size would be unob-

vice. And because the status of all devices is available attrusive at the venues and because it supported remote

glance on the router’s control panel, it is very useful inPTZ (pan, tilt, and zoom). The AW-RP60G controller

situations, such as the festival, when there are multiple

equipped with a PTZ joystick allowed us to move thedevices to manage.

camera through the horizontal and vertical planes as wellas control zoom in/out, image brightness, and so forth

When we measured network bandwidth ahead of theeasily from a remote location (IIJ headquarters). This let

performances, there were differences depending on thethe on-site staff concentrate on the physical installationof the cameras, cables, and so on, thus reducing the on-site
staffing requirement.
1. Set up tripod and camera at venue
2. Check camera level/tilt remotely
and ask venue staff to adjust tripod
3. Use controller to remotely adjust
view angle and brightness
4. Send L and R audio signals fromvenue and check remotely for
CameraAW-UE100K

Controller
AW-RP60GJ

left/right stereo alignment
Tripod and camera placed at back of venue

Figure 2: Camera and Controller

Ueno

Figure 3: Equipment Setup Procedure at Venue

FLET'S IPv6 loopback

Iidabashi

Figure 4: Transmitting Video from Ueno to IIJ

Mobile networks

Ueno

IIJ Mobile
Docomo, au,
Softbank

Figure 5: Illustration of LiveU Backup Connection

14© Internet Initiative Japan Inc.

Iidabashi

Nov.2022

Vol.

56

2. Focused Researchtime of day, but we were getting around 200Mbps over the

2017, and the SRT Alliance had over 500 members as

VPN tunnel, which is sufficient bandwidth for effectivelyof June 2022; IIJ is also a member (https://www.srttal-transmitting video.

liance.org/). SRT is based on UDP, and while maintaining
low latency, it also provides more packet recovery features

We used LiveU (https://www.liveu.tv/ja) video transmission

than TCP, and supports encryption using AES. As one ofdevices as a backup and virtually bundled together

the leading protocols for transmitting video over broadband

multiple mobile carriers’ connections, over which weservices and the like where bandwidth is not guaranteed, ittransmitted video in HD. The LiveU system allows multiple

can be expected to be adopted on an increasing number ofSIMs to be installed to enable the transmission of more

devices going forward.

video data over multiple mobile connections, and is evenused by broadcasters for live coverage. We used LTE■ Transmitting and buffering

contracts for the SIMs because 5G was not yet available

Our connection had an effective bandwidth of 200 Mbps,

at the venues.

so we figured we would be able to transmit using SRT
at around 30Mbps without any problems, but when we

Reception was poor at some venues, and some of themactually tried it out, the system was generating annoying

jammed mobile signals during the performances, so welittle noises every few minutes. The error counter did notneeded to know the quirks of each individual venue torise when this happened, so we were unable to identify thedetermine where to install the LiveU units. Some trial andcause, but we were able to resolve it by setting a larger

error was involved in selecting installation spots, but we

buffer size on the receiving device.

were generally able to transmit in HD without issue at bit
rates of around 10–15Mbps.

A larger buffer increases the broadcast delay, but thiswas not a problem because the content we were dealing■ Transmission protocols

with was concerts, so strictly real-time coverage was not

Our video resolution and frame rate was 4K at 30p.

really a major imperative. When live-streaming sports andUncompressed, this would take as much as 6Gbps, soother such events, however, low latency is a must, so thewe compressed it to around 30Mbps using H.265 andsettings need to be adjusted according to the characteristics

transmitted it to IIJ headquarters using the SRT protocol.

of the connection being used.

SRT stands for Secure Reliable Transport and is a video■ Subtitles

transport protocol published by HaiVision in 2014 to fa-

The operas were not performed in Japanese. Lohengrin

cilitate the transmission of video via the Internet. A range

is sung in German and Turandot in Italian, so in the caseof vendors have adopted it since it was open-sourced inof large concert halls, Japanese translations appeared

Figure 6: Player with Subtitle Support© Internet Initiative Japan Inc.

15

on displays on both sides of the stage. Although there

stage. These timing instructions are issued in real time,

were no such in-venue subtitles for songs performed at

but the video is transmitted with a few seconds’ delay,

art galleries and museums, we had to include subtitles

so our subtitling system includes an appropriate delay tofor users watching via stream. In 2021, we hardcoded

ensure the text appears at the right time.

the subtitles into the video, but in 2022, we provided
subtitles using the setup shown in Figure 7. This meant■ Streaming center at IIJ headquartersthat users were able to easily turn subtitles on and off

We were in charge of coordinating the video feeds ofduring the performances.

the performances coming in from the venues in Ueno,
inserting on-screen text before the performance, during

To display the subtitles at the right time, the operator

intermission, and at the end, and transcoding the video

needs to keep the subtitling system in time with theinto a format viewable by end users and streaming it out.

song. But the engineers alone cannot handle the timing.

As the photo shows, we had displays as well as switchers

To deal with this issue, music festival staff who properly

and other equipment lined up in our workspace to enable

understood the content helped our engineers to display

us to process up to four performances simultaneously.

the right subtitles at the right time. We built a system

We used a network monitoring tool (Zabbix) to visualize

to insert the subtitles into the video data in the form of

network communications and bandwidth (video bitrate),

ID3 metadata in real time in accordance with the instructions

letting us easily determine venue status and see the statusissued, and we set our operations up so that we couldof our communications systems at a glance, and thiscontrol this subtitling system remotely in response

turned out to be an important means of telling what was

to instructions from staff positioned at the side of thehappening from a remote location.

Venue

Subtitle server
Subtitling
instructions

Instruction
issuerSubtitle
operator

Delay added

Subtitles inserted in ID3 tag

Streaming system
Video sent
over SRT

Streamed
broadcast

CameraFigure 7: Subtitling System Diagram

16

Figure 8: Streaming Center at IIJ Headquarters© Internet Initiative Japan Inc.

Nov.2022

Vol.

56

2. Focused Research■ Communication tools

team members posted and how granular the information

Communication among remotely located team members

they provided was. This was a reminder that with any

is key to making remote production work. It is common totool, we need to make sure everyone is fully aware ofuse an intercom system at live events, but on this occa-

why we are using the tool and what the communication

sion, we used Slack (chat room tool) and set up a channel

rules are.

for each venue. Although this created the extra work oftyping text messages, it also let us check on progress

2.4 Conclusion

at other venues at a glance and made it easy to go backPreparations for the Spring Festival in Tokyo 2023 arethrough the timeline and follow our workflow history, so

already underway, and we have started selecting andit provided excellent visibility. It also proved to be very

testing new equipment to push remote production even

useful in enabling the festival organizers to share certain

further. In light of the results and lessons learned thisdetails with us, such as information on encores, which

year, we will be working to provide an even better

is often sorted out on the day of the performance. That

streaming experience ahead.

said, we did note individual differences in terms of whenOn-site engineers

Engineerat venue C
Engineerat venue B

Progress report

Progress report

Engineerat venue A

Performance updates
Slack

Microsoft Teams
Voice call

Progress report(channel for each venue)• Progress Chart• Start time

• Intermission time

Stagingmanager• Encore: yes/no• Information aggregated in Slack so venue status is available at a glanceStreaming center engineers• Microsoft Teams to be used for voice calls so that group calls can be set up

Figure 9: How We Communicated Depending on the Situation

Hiro OkadaxSP System Services, Network Division, IIJ
Intrigued by the Internet, Mr. Okada joined IIJ in April 2000. After working in sales, he went on to work in business customer connection
services. Since 2015, he has worked in streaming/broadcast-related business. He is involved in starting up paid streaming platforms,
studios, etc.

Fumitaka Watanabe
Streaming Businesses Section, xSP System Services, Network Division, IIJ
Having previously worked in the digitization and development of broadcasting services, from the launch of BS digital broadcast channels
through to advanced BS broadcasting, Mr. Watanabe currently focuses on developing and proposing solutions for increasingly diverse video
streaming/broadcasting services, including simulcasting and VOD/live-streaming.

© Internet Initiative Japan Inc.

17