6‑3

6 Games Management Operation Centres

System Integration and Games‑time Observation
Remastered for Pandemic Response
Michèle HYRON

An inside look at system integration and operations for the Olympic and Paralympic Games Tokyo 2020 a very special
edition run with one year postponement and under pandemic restrictions Teams from Atos the IOC the Tokyo 2020
Organising Committee and technology partners worked as one to implement innovative ways to work under thesecircumstances Their rapid response─on a foundation of digital excellence─mitigated the impact on people processes andsystems This article shows how the pandemic directly affected integration operational readiness and operations By the
Olympic and Paralympic Games Tokyo 2020 time the only noticeable impact was the elevated importance of observation
and IT service management
Keywords：Atos Integration Testing Operations Observation

one year postponement and under pandemic restric-

1．Introduction

tions

Atos as Worldwide IT Partner to the Olympic andParalympic Games has overseen system integration for
critical systems since 2002 This covers everything from
infrastructure to applications whether theyʼre deliveredfrom the cloud dedicated data centres or servers at

2．System Integration Impact of the State of theEmergency Declaration and Postponement
The integration team began as usual by building astrong presence in the host city Tokyo supported byvenuesAs integrator we ensure that the full system meetsfunctional requirements meaning that all functionalities
work as expected from a userʼs perspective We also

centralized remote resources and the Atos Integration
Test Lab (ITL) in Spain
The team in Tokyo shared a building with the Tokyo

non‑functional

2020 technology departments and technology partners

requirements such as performance stability availability(blue in Figure 1) Working in person without having toand of course security Additionally the integrator

coordinate time zones was key to facilitating communica-

supports the OCOG in preparations and operations

tions while addressing multiple aspects of systemproviding processes and tools

integration

ensure it meets the more technicalThis article provides an inside look at system

The centralized/remote teams (gray in Figure 1)

integration and operations for the Olympic and Paraly-

remain in place from Games to Games contributing vast

mpic Games Tokyo 2020 (hereinafter referred to as

experience to each edition The ITL team extensively“Tokyo 2020 Games”) a very special edition run withtests all critical systems and their interfaces Testing
includes delivery acceptance standalone end to end

Michèle HYRON
Major Events Atos

multisport

The Journal of The Institute of Electronics Information and Communication Engineers Vol 105 No 8 Supplement pp 313‑319 August 2022

recovery The full technology team peaked at around© 2022 The Institute of Electronics Information and Communication Engineers

technical

security

failover and disaster3 500 staff at Games time

Special Issue : Technology and Innovation in the Olympic and Paralympic Games Tokyo 2020313

Figure 1

Integration Team StructureBlue : on‑site teams Gray : centralized/remote teams

When the Tokyo 2020 Games were postponed in

Delivery” for details about OMS and ODS )March 2020 we activated several mechanisms in parallel

Because of the one‑year postponement some plat-

to streamline the on‑site presence Some teammatesforms became obsolete and new versions of browsers

were reassigned to other projects including Beijing

operating systems and other base software had to be2022 Others were furloughed and there was also someconsidered

natural attrition Then at the beginning of 2021 we
progressively re‑staffed to reach full capacity in March

22

2021

To cope with the postponement impact the results

System Integration

system altogether was frozen until the beginning of
21

2021 At unfreeze time the challenge was to absorb theBusiness Requirements

The decision to postpone came just weeks before the
Uniform and Accreditation Centre opening in Tokyo
with all Olympic Management System (OMS) modules

impact on the planning and integration of the results
systemPostponement also forced Tokyo 2020 preparations to

our custom-

overlap with Beijing 2022 and Paris 2024 preparations

ers─Tokyo 2020 functional areas press broadcasters

While the OCOG is dedicated to a single Olympic Games

International Sports Federations (IFs)─reviewed theiredition ; Atos the IOC and other technology partners

business plans and requirements All had to adapt to theare organized and staffed according to Olympic cycles

new schedule cascading significant changes in configu-

This overlap combined with the changes and uncer-

ration and business processes in the six months before

tainty of the pandemic generated a high level of risk

live with external users

Immediately

The full rescheduling of integration milestones

the Tokyo 2020 GamesBig organizations were limiting travel to Tokyo or(marked by each testing event) was monitored closely

reducing their venue presence Therefore requests forthrough strong integration governance involving Atos

remote access to the information systems for press andthe IOC Tokyo 2020 and technology partners

modifying the

A continuous risk analysis was run materialized byservice‑level agreement (SLA) and testing targets

periodic stakeholder meetings Key performance indica-

Notably the video press conference system an innova-

tors (KPIs) including defect tables testing progress andtion for Tokyo became a critical feature preferred tomilestone achievements were available weekly support-

physical presence at the conferences

ing the definition of mitigation actions

broadcasters increased significantly

The IFs defined competition format changes for somedisciplines impacting Olympic Diffusion System (ODS)
outputs and the results system (See in this special issue

23

Integration Testing

The ITL the heart of integration testing occupies a“3‑6 Olympic Diffusion and Management Systems

1 000‑square‑meterTokyo 2020 Bringing Agility into Critical Application

operates 24/7 It is divided into cells for virtual sports

314

access‑controlled

facilitythat

THE JOURNAL OF THE INSTITUTE OF ELECTRONICS INFORMATION AND COMMUNICATION ENGINEERS

Games venues ODS testing and OMS testing

to continue the integration test of the results system

Having the ITL in Spain strengthened relationships

with few exceptions The integration team was able towith providers partners and clients because most of

test from home and still run testing milestones withthem are based in Europe When travel outside the EU

clients by accessing the machines remotely without

was restricted they could easily visit the ITL One of the

stopping the project

most beneficial aspects of this location was the frequency

acceptance tests and the Paralympic results andwith which broadcasters were able to visit They could

information system validations with the International

regularly check the status of sports review require-

Paralympic Committee and the IFs also happened

ments with the testing team follow end‑to‑end testing

remotely with successful outcomes

Paralympic broadcastersʼ user

closely and perform validation tests on‑site without
travelling

232

Games Venue Cells (GVC)

On the other hand this generated a challenge to the

In the ITL several cells were equipped with hardware

integration team as a whole with one part based inprovided by Tokyo 2020 specifically for the results

Japan and the other in Spain Working in different timesystem They were called Games venue cells (GVCs)

zones we ensured close communication through a strict

The main goal of the GVCs was to install as many

underlying governance but meetings at odd hours for

sports as possible to validate that the results systemboth teams were still unavoidable

would run properly on the actual hardware used at thevenues Simulating the sports competition in a GVC also

231

Virtual Environment

allowed validation of base images and actual capacity of

The virtual sport cells occupied a space (as shown in

ODS physical venue servers as well as touchscreen

Figure 2) with physical terminals to access virtual

laptop functionalities for the Commentator Information

environments that simulated the results system in the

System and printed results reports

cloud There was also an additional desktop machine per

The ITL followed government instructions in terms ofcell with dual graphic cards to reproduce the results

maximum number of staff hygiene rules etc Proce-scoreboard graphic screens From any virtual sport cell

dures were adapted to allow access to the GVC forit was possible to simulate and test any sport competition

minimal necessary activities while respecting safety

by accessing each ODS and result application in therules This included rotation of staff to limit the number

cloud This enabled enormous flexibility when testingof people in the space at any given time

When the pandemic started in March 2020achallenge was promptly overcome : setting up enough233

Increase of Volume

equipment at each testerʼs home to test remotely in the

Another challenge arising very late in the project life

virtual environment Fortunately most of the applica-

cycle was to accommodate a much bigger volume oftions could be tested virtually and we had all the means

customers using ODS applications remotely due to thecoronavirus disease 2019 (COVID‑19) situationsIt

happened late in the project and required much moreemphasis on nonfunctional validation This increased thecomplexity of the cloud test and required setting up a
number of extra simulators in the lab to generate load

3．Operational Readiness Impact of the Stateof the Emergency Declaration andPostponement
31

Technology Rehearsals

Technology rehearsals 1 and 2 (TR1 and TR2) are key
exercises to check the technology teamʼs operational
readiness taking place in the ITL and the venues The
Figure 2Volleyball Cell in Integration Test Lab（©2021‑International Olympic Committee‑All Rights Reserved）

busiest days of Olympic competitions are simulated
while Technology Rehearsal Officials (TROs) trigger

Special Issue : Technology and Innovation in the Olympic and Paralympic Games Tokyo 2020315

scenarios to check team reactions

effectiveness ofpolicies and procedures and team understanding of thepolicies and procedures
Travel restrictions and the state of the emergency

4．Tokyo 2020 Games Operations Impact of
the State of the Emergency Declaration
and Postponement

declaration hit just before TR1 in March 2020 prevent-41

ing some teams from traveling to Tokyo or to ITL In a

Games operations are structured in 2 layers :Communication Tools

few days the Atos team proposed and prepared to run
the results system TR1 virtually by triggering scenarios･

Central provided by the TOC in Tokyo and its

remotely from the ITL That way TR activities among

satellites such as the Central Technology Opera-

the on‑site teams took place in the venues as planned

tions Centre (CTOC) for Atos mission‑criticalsystems based in Barcelona

These digital TRs brought many learnings that
improved communication and operations between the･

Venue‑based with a technology team in thecompetition and non‑competition venues

on‑site Technology Operations Centre (TOC) andvenues It was our first technology success to overcome
the pandemic impact and progress our Tokyo 2020

In all previous Games editions one of the challenges

preparations It sent a strong message that technology

has been to ensure fluid well balanced communication

would adapt and be ready to deliverand coordination :TR2 took place from May 31 to June 4 2021 During
the pandemic the venues operated with different teams･

Inside the TOC where 130 staff are working per
shift

in bubbles that could not mix Bubbles at the venues forTR2 prevented the TROs from fully observing the･Between the TOC and its satellites

results activities･

Between the TOC and the venues

Atos came with an innovative approach : One of thetwo Venue Results Managers located inside the results

For the Tokyo 2020 Games the use of communication

bubble was selected to become the Results TRO

knowledge and collaboration tools spiked These tools

complementing the initially planned TRO This allowed

were always used during preparations test events andthe results technology partners who could go to Tokyo

TRs ; but they were a lifesaver during the pandemic

to go and have a meaningful exercise
4232

Virtual Meetings

The need to respect social distancing in the TOC led toTraining
our training strategy is built on a

replacing traditional standup meetings held at shift

variety of delivery means : hands‑on classroom trainingchange with virtual meetings where the key Duty

face‑to‑face classroom training table‑top exercise andManagers in the TOC were reporting twice per day

TraditionallyThis was very successful

e‑learning

also allowing a broader

During and after the COVID‑19 suspension period all

audience to join the meetings Duty Managers who weretraining was delivered by e‑learning or online meetings

not on shift and executives from the IOC and technology

Some presentation courses such as the Atos Venue IT

partners could join virtual meetings to easily increase

Manager (VITM) in‑person presentation courses were

their dynamic understanding of the operations

converted into e‑learning courses
While the number of new resources without experience was increased by the postponement the effort

43

Planning and Policies

COVID‑19‑positive cases within the Atos TOC teamrequired to convert classroom training to e‑learnings

were immediately isolated andpaid off with a good level of training

confinement rules moved to a specific accommodation

following Japanese

Luckily all of our cases were either asymptomatic orlightly symptomatic Infected teammates could workfrom the government accommodations using our monitoring tools

chats

calls and online documentation

related to their specific areas

To allow thatwe

implemented a policy of TOC members taking their
316

THE JOURNAL OF THE INSTITUTE OF ELECTRONICS INFORMATION AND COMMUNICATION ENGINEERS

laptops with them after every shift

This policy

increased the planned equipment requirements from one
laptop per TOC role to one laptop per person

NTT
The initial Flash‑based solution was nearing the end ofits life but would have supported Tokyo 2020 The one‑

We also had a few positive cases in the venue teams

year postponement allowed us to migrate it to a modern

where presence at the venues is mandatory due to thesoftware‑as‑a‑service platform between Q3 2020 anddynamic of the results activity When evaluating theQ1 2021 before the obsolescence could block the new

impact of the pandemic

schedule The new solution provided quick value andwe had anticipated extra

resources available to cover temporary attrition This
allowed us to efficiently mitigate the impact

supported a smooth transition for all parties involved
System monitoring was projected onto walls in the
TOC and CTOC (shown in Figure 3) and each key team

44

Crisis Communications

The traditional bridge opened to manage high‑se-

had its own dashboard displaying the main resources
supporting its services

verity incidents and crises involves the TOC Duty
Managers and executives of the concerned parties For

512

Tokyo our bridge was enhanced with a chat that

Virtual user monitoring (VUM) was a new addition toremained open for each high‑severity incidentʼs dura-

the monitoring solution for Tokyo 2020 It leverages thetion

W3C WebDriver protocol to orchestrate a web browser

allowing updates between formal calls to all

involved stakeholders This was extremely efficient

Virtual User Monitoring

to monitor web applications

helping to meet two goals : limiting the number of calls

It does so by executing functional scenarios (e gwhich allows teams to focus on resolution and providing

login to application click on menu check displayed text)

the required level of visibility

to assess the health of the service If the scenario fails(e g

5．Tokyo 2020 Games Operations Observation
51

Monitoringa button does not show or page shows an error

message) or if health indicators are bad (e g an HTTP
500 error) an alert is raisedThis solution was put in place to address the growing

Considering the extent of adaptation required tocomplexity of IT services particularly modern applica-deliver this edition of the Olympic and Paralympic

tions that use JavaScript frameworks and REST APIsGames

instead of static pages Assessing the health of suchobservation and monitoring were relatively

unchanged‑but never more criticalapplications via traditional system monitoring can beinefficient as it cannot detect complex failures

511

System Monitoring

System monitoring covered all systems hosted in Atoscloud and was heavily used by Tokyo 2020 as well as key

VUM monitors all OMS/ODS applications as well as
ITSM and documentation management It was the key
monitoring tool shown on the TOC display wall

Competition Network (CPN) and back‑office OCOG
Network (BON) providers such as Fujitsu NEC and

513

Venue Device Monitoring

Venue device monitoring (VDM) consisted of anagent that regularly (every 5 minutes) executed probes
on a device to collect KPIs or perform sanity checks
This was then reported to a central management serverthat published a dashboard made available to teams inthe venues TOC and CTOC
The VDMʼs simple red/amber/green status dashboard was heavily used during operations to efficiently
monitor venue device health It also covered the venue
data centres for basic monitoring in complement of theirsystem and application monitoring tools
Figure 3 Monitoring in the TOC
sharing simple and effective
dashboards on the display wall （©2021‑International Olympic
Committee‑All Rights Reserved）

514

Application Monitoring

Detailed application monitoring was provided to theSpecial Issue : Technology and Innovation in the Olympic and Paralympic Games Tokyo 2020

317

Figure 4 Tickets Logged Per Day and Per Severity
July 23 2021

Note : The Opening ceremony took place

CTOC team in Barcelona using an open‑source analytics
and monitoring solution

6．IT Service Management
Atos provided the IT service management (ITSM)
tool used by all technology teams in the TOC CTOC andvenues as well as in the technology call centre ITSM isthe backbone of the technology operations supporting
the key processes that federate necessary evolutions andincident resolutions
61

Incident Management

Our incident management activities focused on
logging

sharing and tracking the progress of allincidents and service requests until resolution High‑severity incidents severity 1 and 2 (shown in Figure 4)
were displayed on the TOC wall and made available
Figure 5

online for support staff alerting all teams about urgent

Change Requests

actions to be taken An SLA is in place for each severity
62

Change Management

alternative workarounds were provided to ensure

Although stability is a token of quality for Gamessystems stability

services there are always changes that need to becarefully evaluated and managed

The Tokyo 2020

63

Interruption of Services

Games operations teams followed a stringent change‑

For each approved change that needs to be applied inmanagement process supported by the ITSM tool When

production (illustrated in Figure 5) a formal Interrup-

the risk was too hightion of Service (IOS) notice must be issued and approved

318

changes were rejected and

THE JOURNAL OF THE INSTITUTE OF ELECTRONICS INFORMATION AND COMMUNICATION ENGINEERS

7．Conclusion
Technology delivery was outstanding for the Tokyo
2020 Games under an extremely difficult situation
There may not have been spectators in the venues but
the technology teams allowed far more fans across theglobe to enjoy the events and celebrate the athletesʼ
determination through this challenging exercise
Atos has delivered the highest levels of integration
services and innovative methods to operate under the
Figure 6

state of the emergency declaration and implemented

Logged Problems

strong governance to mitigate risks generated byoverlapping the Tokyo 2020 Games with Beijing 2022 andby all stakeholders including end users

Paris 2024 preparation This is paving the way for futureThis was broadly in place for the Tokyo 2020 Games

editions The lessons learned during this public safety

and successfully ensured awareness among all stakehold-crisis will help to streamline the preparation anders including the support teams about implementation

operations of big and complex projects with morechanges This process accelerated the troubleshooting offlexible and efficient ways to work

the rare unexpected side effects(Received March 4 2022 ; Revised March 24 2022)

64

Problem Management

When the service is restored following an incident but
the root cause is still not clear then the resolution is
flagged and followed through as a problem

The

distribution of problem severity at the Tokyo 2020
Games is shown in Figure 6 We followed our normal
standards to ensure all problems were closely followed
up and resolved by the end of the operations

Michèle HYRONMichèle HYRON as Atos Chief Integrator
for the Olympic Games leads the consortium oftechnology partners that designs builds andoperates its mission‑critical IT infrastructureand systems She has managed various aspectsof the Gamesʼ integration and operations for 20
years dedicating the last 10 to her current role
Michèle graduated from Ecole PolytechniqueFeminine in France with a degree in Computer
Science㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇

Special Issue : Technology and Innovation in the Olympic and Paralympic Games Tokyo 2020319