Deliverable
Deliverable nameD5.1 – Report on Immersive Content Production
Part 1

Project Acronym:
Grant Agreement number:
Project Title:

IMMERSIFY
762079
Audiovisual Technologies for Next Generation
Immersive Media

Revision:
Authors:
Delivery date:Dissemination level (Public / Confidential)

0.4
Maciej Glowiak (PSNC)
M12
Public

Abstract
The subject of deliverable 5.1 is to deliver video clips for project needs (type is “DEC: Websites,
patents filing, press & media actions, videos, etc.”). This report focuses on Immersive Content
Production and shortly summarizes work done in ​“Task 5.1 Content Production” ​and partially in“Task 5.2 Interactive Media Authoring”.​ The document describes how T5.1 and T5.2 addressed
objectives defined for “WP5 Creative Development and Demonstrations” and describes set ofimmersive content material created using different production methods for the Immersify project.
The document is iterative and will be extended by project results after first reporting period in
M24.

This project has received funding from the European Union’s Horizon 2020 research and innovation
programme under grant agreement 762079.

REVISION HISTORY
Revision

Date

Authors (Entity)

Description of changes

0.1

1 Sep 2018

Initial content of the document, integration of PSNC content
descriptions

0.2

17 Sep 20180.3

18 Sep 2018

0.4

21 Sep 2018

1.0

28 Sep 2018

Maciej Glowiak
Maciej Strozyk
Eryk Skotarczak
Christian Södergren
Erik Sunden
Anna Kuthan
Sergio Sanz-Rodriguez
Maciej Glowiak
Maciej Strozyk
Maciej Glowiak

Content/immersive production
Content + review of document
Adding missing sections, first complete draft for review
Final review

Contributors
Contributor’s nameEntityContact e-mailMaciej Glowiak
Eryk Skotarczak
Maciej Strozyk
Sergio Sanz-Rodriguez
Anna Kuthan
Christian Södergren
Erik Sundén

PSNC
PSNC
PSNC
SD
AE
NVAB
NVAB

mac@man.poznan.pleryk@man.poznan.plmackostr@man.poznan.pl
sergio@spin-digital.com
anna.kuthan@ars.electronica.art
christian.sodergren@visualiseringscenter.se
erik.sunden@liu.se

Deliverable 5.1

Revision 1.0

2/25

TABLE OF CONTENTS

Introduction

5

Results5

Addressing objectives

5

Movie clips

6

Summary

6

Clips produced by Immersify project8

Dziekanowice Ethnographic Park

8

Baltic Legend Opera

960 years of Polish Television in Poznań Anniversary Gala

10

Follow Car

11

Karkonosze

12

Cathedral of Poznan

13

Cathedral of Poznan

14

Adventure Dome

15

OpenSpace

16

Clips produced with third parties

17

Atacama Desert

17

Pano LA 10K

18

Prima Materia 8K

19

Ancient Invisible Cities: The Great Pyramid20

Cinematic rendering – Virtual Anatomy

21

Clips obtained from external sources

22

Preparing the content for final showcase

23

Conclusions

24

LIST OF ACRONYMS
Deliverable 5.1

Revision 1.0

3/25

Acronym

Description

HMD
Mxx
CGI
VR
UHD

Head Mounted Display
Month xx
Computer Generated Imagery
Virtual Reality
Ultra High Definition (4K or 8K)

Deliverable 5.1

Revision 1.0

4/25

1 Introduction
This report focuses on Immersive Content Production and involves tasks “Task 5.1 Content
Production” and “Task 5.2 Interactive Media Authoring”.
Task 5.1 is dedicated to the creation of new immersive content that can be used to demonstrate thecreative and immersive capabilities of the new formats. It also determines how to improve andassess the tools to be developed in the project. We assumed that the content for the project could
be acquired and produced in several ways: by using existing content; by producing new immersive
video content using the infrastructure and equipment from the project partners or by partnerships
with external partners.
Content for the project was to be produced using different techniques in order to explore andcompare different production methods: time-lapse photography, 3D laser scan based renderings, CGI
and animation, 360° panoramic high-resolution video, and real-life 8K footage. It was also assumed
that a part of the content creation and production will be subcontracted to leading content
producers. The new test content was also needed to validate the codec technologies developed in
the project and optimize the media creation workflows with regard to different content genres incollaboration with the content producers.
Task 5.2 focuses on the creation process of interactive media for VR environments and VR
applications. Using the playback infrastructures created in WP3 (mostly T3.5 Unity3D integration) andthe content from T5.1 several prototypes for interactive non-linear narratives will be created. The
prototypes may target different VR systems (dome, deep space, VR glasses), target audiences(children - adult, single user - multi user groups), interaction forms (gesture based, positional
tracking, head tracking, direct and indirect manipulation), presentation forms (guided, unguided),
media forms (video, audio, real-time CGI) and rendering techniques (monoscopic - stereoscopic,
augmented virtuality). The results will be evaluated in T5.3 and integrated in T5.4.

2 Results
2.1 Addressing objectives
The WP5 defines 7 major objectives, but only some of them are related to T5.1 and T5.2
Objective 1​: Create new 360° content that is able to demonstrate the capabilities of the project’s
outcomes on different environments: Head-mounted-display, multi-screen, multi-projection.
During the first reporting period (until M12) we created several video clips and gained some fromother sources (as part of subcontracting). The full list is enlisted in Section 2.2. As the Task 5.1 is notfinished yet, some clips are there in production phase or in planning and will be reported in D5.2.
Objective 2​: Explore different methods for producing and presenting beyond UHD-4K content
including time-lapse photography, 3D laser scanning, CGI and animation, panoramic video, andreal-life 8K footage.

Deliverable 5.1

Revision 1.0

5/25

A number of video clips were produced in various technologies and different conditions: indoor andoutdoor, fast movement, negative temperatures, stereoscopic, timelapse, laser scanning, CGI etc.
Objective 3​: Experiment and prototype on new forms of interactive, non-linear storytelling buildingon the technologies created in WP3 and WP4 and unique high-resolution content.
We started work on interactive non-linear storytelling, however this work is still in progress and will
be a subject of Deliverable 5.2.
Objective 4​: Encode the immersive content using the HEVC encoder developed in WP4, and find theappropriate quality-rate points for target applications and demonstrations. and ​Objective 5​: Find
optimal configurations of the encoder for high-quality 8K delivery and playback. and ​Objective 6​:
Perform an informal subjective test on target display environments to validate the quality
enhancements developed in the encoder, and give feedback for potential encoder improvements.
All video contents are accessible for all the partners using common repository. Each movie is
encoded by Spin Digital HEVC encoder in several ways and optimal parameters are set. Parameters
are selected based on experience gained so far, as well as after screening on different devices andsubjective quality assessment (e.g. 8K video wall, 8K monitors, Deep Space and CAVE projection).
Objective 7:​Establish and document best practice guidelines for media production workflows
During production of each movies, we document the whole process, equipment used. The making-of
photos and video clips will be provided on ​www.immersify.eu webpage and will be a contribution tobest practice guidelines in the future.

2.2 Movie clips
2.2.1 Summary
In the project proposal, we planned production (or gaining from external sources) of several video
clips in various technologies:

Count

Resolution

Video

Technique

Duration

2 clips

8K or higher

Immersive 360°,2D

360° Time lapse or
360° camera

1 minute or longer

Results​: ​Partially done
So far PSNC produced several 8K 360° video clips using Orah 4i and Insta360 Pro camera, allrecorded in Poznan. Although Insta360 Pro is one of most advanced 360° integrated camera, thequality of panoramas are not sufficient for advanced use with high-end visualisation
infrastructures. Partners of the project try different methods of obtaining better content:
timelapse 360°, collaboration with external vendors such as Arri (which provided 12K content forthe project) and renting 360° rig.
Content finished​:● Cathedral 8K 360° (PSNC)● Poznan 8K 360° (PSNC)● Arri demonstration reel 12K (SD)

Deliverable 5.1

Revision 1.0

6/25

2 clips

8K or higher

Stereo 3D

360° camera
3D scan CGI

1 minute or longer

Results: ​Partially done
There is currently no available 360° 3D 8K camera on the market. We have already made 3D laserscans of Cathedral church and the 8K 3D clips will be generated soon (October 2018).
Point cloud renderings of the Great Pyramid in Cairo (BBC /Scanlab) are in progress (3 clips in
12K).
Content finished:● in progress
2 clips

8k or higher

Stereo 3D

3D animatedrendered content

1 minute or longer

Results:​​Partially done
16K animation using point cloud rendering was provided from PSNC in order to present for IBC
fairs. The 3D version will be available soon (October 2018). Next laser scans are planned forupcoming months.
The stereoscopic piece Prima Materia from Istanbul based studio Nohlab was rendered in 8K x 8K
in 3D.
Content finished:● Cathedral scans in 16K, 12K and 8K (PSNC) - 2D version● Prima Materia 8K (AE)
2 clips8K

2D
60-120 fps

SONY F65

1 minute or longer

Results​: ​Done
PSNC already recorded 12 video clips in 8K 60p using its modified F65 camera in different
conditions. Work will be extended in the future in order to have wider portfolio of movie clips, as
there is big interest from stakeholders.
Content finished:● Dziekanowice Ethnographic Park (PSNC) - 3 clips● Poznan Race Track (Followcar) (PSNC)● Karkonosze - Szklarka Waterfall (PSNC)● Karkonosze - Kamieńczyk Waterfall (PSNC)● Karkonosze - Szrenica Peak (PSNC)● Karkonosze - Wang Temple in Karpacz (PSNC)● Karkonosze - Glassworks in Szklarska Poręba (PSNC)● Cathedral (PSNC)● Baltic Legend Opera (PSNC)● 60th TVP Anniversary Gala
Moreover, PSNC started to create content for Immersify project purposes even before official
starting date, which gave us the possibility to test tools and promote Immersify at the very
beginning of the project. These 4 clips were:● Clock 8K ​https://www.youtube.com/watch?v=IB25yQtsA9g● Fish 8K ​https://www.youtube.com/watch?v=wOecjp5GolU● Magda 8K (PSNC) ​https://www.youtube.com/watch?v=vi4tTu5EEjg● Palm House 8K (PSNC) ​https://www.youtube.com/watch?v=cRgLlVDgosg
2 clips
Deliverable 5.1

8K

3D

SONY F65 + 3D rig

Revision 1.0

1 minute or longer
7/25

60-120 fps
Results​: ​Partially done
PSNC recorded one 3D 8K video clip in Kornik Castle. This video content is still in post-production,
which is more demanding and requires more time for preparation and display.
Content finished:● in progress
2 clips

8K or more

2D

Timelapse

1 minute or longer

Results​: ​Done
AE has subcontracted two timelapse videos in 8K and 10K.
Content finished:● Atacama Desert (8K) - Martin Heck / Timestorm Films (AE)● Pano LA 10K - Joe Capra / Scientifantastic (AE)The task T5.1 has been not finished in M12, so it means not all the content is already completed.
Some video clips need to be processed in post-production and color-grading, and some need morework related to software development in other tasks (e.g. point cloud). We also intend to produce
new clips for whole duration of the project depending on particular needs of the dissemination
activities.
Production of new clips in novel technologies require a lot of technology testing, integration andresearching new ways of filmmaking (e.g. how to expose small details for ultra high resolution, how
to make good 360° content, etc).

2.2.2 Clips produced by Immersify project2.2.2.1

Dziekanowice Ethnographic Park

General description
Title

Dziekanowice Ethnographic Park

Date

Autumn 2017

Place

Dziekanowice, Poland

Number of clips

3

Content Summary
Video clip for folk music in the open-air museum
Technology
8K 60p 12bpp - 8K camera
Introduction
The idea of the movie is to show a typical cultural place in form of a music clip. Near Poznan there
Deliverable 5.1

Revision 1.0

8/25

is ​Wielkopolski Ethnographic Park Dziekanowice​which presents the reconstruction of historical
Wielkopolska village architectures from the 17th to early 20th centuries. This place in combination
with an invited local folk band performing on contrabass, hurdy gurdy and singing ethno songs
created scenic background for the three extraordinary 8K video clips.
Reference to the content
https://www.youtube.com/watch?v=wc_gyf4H68I&feature=youtu.be
https://www.youtube.com/watch?v=Tzl7hDktr50&feature=youtu.be
https://www.youtube.com/watch?v=Kc1mDEv5dIM&feature=youtu.be

2.2.2.2

Baltic Legend Opera

General description
Title

Baltic Legend Opera

Date

Winter 2017

Place

Poznań, Poland

Number of clips

1

Content Summary
8K clips from the Opera
Technology
8K 60p 12bpp - 8K camera
Introduction and story
December 2017 PSNC was contracted to produce live TV content and streaming from premiere of“Baltic Legend” an opera of Feliks Nowowiejski performed in Poznań Opera House. During
rehearsals of the spectacle, there were opportunities to shoot some additional 8K footage
together with immersive 360° videos. By this, we created demonstration content for two H2020
projects: Immersia TV (synchronized 4K TV content and 360° 4K movie clips) and Immersify (8Kcontent). Some parts of 8K clips were made behind the stage showing the work that is not visible
to the audience, making the final content more interesting.

Deliverable 5.1

Revision 1.0

9/25

Reference to the content
https://www.youtube.com/watch?v=SwGmPmTbldM&feature=youtu.be

2.2.2.3

60 years of Polish Television in Poznań Anniversary Gala

General description
Title

60 years of Polish Television in Poznań Anniversary Gala

Date

Winter 2017

Place

Poznan, Poland

Number of clips

1

Content Summary
8K event and news documentation
Technology
8K 60p 12bpp - 8K camera
Introduction and story
PSNC and Polish Television (TVP) are long-term partners. In 2017 TVP had its 60th Anniversary ofoperating in Poznań. PSNC as an organisation researching new media technology was invited to
document this very special moment using the most up to date video techniques. The outcome is adetail-rich content with vivid colors and wide dynamics.
Reference to the content
https://www.youtube.com/watch?v=UqvaL_pfs5Y&feature=youtu.be

Deliverable 5.1

Revision 1.0

10/25

2.2.2.4

Follow Car

General description
Title

Follow Car

Date

Autumn 2017

Place

Poznan and Bednary, Poland

Number of clips

1

Content summary
Race cars in action on the track with 8K camera stabilisation and smooth moving shots.
Technology
8K 60p 12bpp - 8K camera mounted on car
Introduction
PSNC film crew in cooperation with two other companies did an 8K experimental shooting on
Racetrack Poznan and old military airport in Bednary. The choice was made on racetrack, mainly
because one of our Polish broadcasters was shooting “​Automaniak​”, a Polish version of wellknown TV show “​Top Gear”​ and they brought a few nice cars which we could use for our
experimental 8K shooting with big gimbal installed on the follow car. During two day shooting
session mainly sports car were filmed, but also professional cyclists were caught “on tape” during
their training.
Reference to the content
https://www.youtube.com/watch?v=SCw9j3T8Ff4&feature=youtu.be

Deliverable 5.1

Revision 1.0

11/25

2.2.2.5

Karkonosze

General description
Title

Karkonosze Mountains

Date

March 2018

Place

Szklarska Poręba, Poland

Number of clips

5

Content summary
Winter shooting of 8K in mountain area and negative temperatures
Technology
8K 60p 12bpp - 8K camera
Introduction
Due to the fact that this year winter in Poland was quite cold and snowy we had a chance to try 8K
real winter shootings in Polish mountains. Our choice was Szklarska Poręba, a beautiful small town
located at the foot of the mountain range of Karkonosze. Particularly, five nice places wereselected, three in the mountains and two just close to town:● Szklarka Waterfall● Kamieńczyk Waterfall● Szrenica Peak
● Wang Temple in Karpacz● Glassworks in Szklarska Poręba
Frozen waterfalls and winter mountain panoramas from the Szrenica peak (1362m above sea
level) were the main goal, but additionally we recorded the Wang Temple, built originally frompine logs in Norway at the turn of the 12th and 13th centuries. The building was transported in
1842 to Karpacz (Poland) and is now considered the oldest surviving wooden church in Poland.
Another place that was visited and filmed was a glassworks called “Leśna Huta”, where our filmcrew was very warmly welcomed and the glass-specialists showed our camera their work.
Reference to the content
https://www.youtube.com/watch?v=Jo0BdHwy7xY&feature=youtu.be
https://www.youtube.com/watch?v=rCeau4OrH3w&feature=youtu.be

Deliverable 5.1

Revision 1.0

12/25

https://www.youtube.com/watch?v=O0iTjZvOvcQ&feature=youtu.be
https://www.youtube.com/watch?v=4t4Ol6WIGPw&feature=youtu.be
https://www.youtube.com/watch?v=YiMZKS1TLr4&feature=youtu.be

2.2.2.6

Cathedral of Poznan

General description
Title

Cathedral

Date

May 2018

Place

Poznań (Ostrów Tumski), Poland

Number of clips

3

Technology
1) 8K 60p 12bpp - 8K camera, Aerial photos, 360° camera, binaural audio, ambisonics audio2) 16K 60p 8bpp - laser scanning3) Insta360 Pro - 360 content
Introduction
In May 2018, thanks to the courtesy of the Poznan Archdiocese, the PSNC team was able to shoot
8K clips in the Poznan Cathedral. A few photographing days were preceded by making laser scans,
thanks to which the main nave and the Golden Chapel were accurately mapped. We recorded thesurroundings of the cathedral, the church interior, aerial photos (these in 6K due to limitation ofdrone load capacity – 8K camera was too heavy) and 360° video clips. Also binaural andambisonics audio during a special organ concert were recorded. The source material will be used
to contribute several tasks of the Immersify project related to high quality content, interactive
content, immersive sound as well as encoding 16K and VR. By scanning point clouds, we will beable to render movies up to a resolution of 8K (i.e. 16K) and stereoscopic films. So far, two film
clips have been produced: 8K from SONY cameras and 8K + 16K from point clouds.
Reference to the content
https://www.youtube.com/watch?v=zdoKocv5cek&feature=youtu.be

Deliverable 5.1

Revision 1.0

13/25

2.2.2.7

Cathedral of Poznan

General description
Title

Poznan 360

Date

August 2018

Place

Poznań - different places

Number of clips

1

Technology
Insta360 Pro
Introduction
In order to create 360 content, we have used Insta360 Pro camera and recorded several clips from
Poznan city center in good light conditions. The final clip is in 8K resolution.
Reference to the content
https://www.youtube.com/watch?v=n6Ks-Ks_HnI&feature=youtu.be

Deliverable 5.1

Revision 1.0

14/25

2.2.2.8

Adventure DomeGeneral description
Title

Adventure Dome

Date

2018-2019

Place

Norrköping, Sweden

Number of clips

Work in progressContent summary
Animated interactive prototype for children
Technology
Offline movie / real time Unity application, fulldome format
Introduction
The planetarium industry is craving new types of immersive content and there are great demands
for new interactive non-linear content.
NVAB’s idea is to develop an interactive movie prototype telling a non-linear story where theaudience is able to participate and interact simultaneously in real time.
We lacked references from previous productions of this type. Our solution was to work with an
iterative process of a prototype. We tested our way through with the help of audiences to gain
knowledge about how we could create the right feel and interactivity.
The real-time version had problems with load times between scenes which was solved by putting
up a code framework that avoided large amounts of data to load simultaneously.
The FPS was another issue where the goal was to never fall below 30 fps in 4K resolution.
Optimization of the scenes was the solution. E.g. reducing number of objects in the scene, terrain
with fewer polygons, downscaling of textures and changing texture format from PNG to JPEG,
Deliverable 5.1

Revision 1.0

15/25

reducing the number of objects that were controlled by code by grouping them and then control
them by code.
NVAB has a small team of 3 people, AD, programmer and 3D-artist working together with script,
planning and concept.
The interactive prototype is created in Unity3D as a real-time application to be converted tooffline video for playback on media players developed from the Immersify project. The prototype
can be converted and made available for other types of screens and platforms in 8K resolution.

2.2.2.9

OpenSpace

General description
Title

OpenSpaceDate

2018-2019

Place

Norrköping, Sweden

Number of clips

Work in progressContent summary
Platform for interactive rendering of scientific space data
Technology
Real-time application with offline rendering support and various immersive camera models
Introduction
Theme “Data evolution in space”. We will create an interactive production which primarily will be
Deliverable 5.1

Revision 1.0

16/25

a visit to Mars (optionally also Pluto), where we discover how much data of the planet we havegathered over the years, and how it was gathered. While the final production can be runinteractively, it should also be possible to make an offline version for a specific immersive space(with multiple projectors/screens) such that any site can run it through their video playback
system and that it will fill the whole immersive space with content. The production is scheduled to
be started in October 2018 (planning is now complete), with an initial two month production
phase to implement functionality needed to support offline rendering to any immersive setup.
Within this project one key task is measuring rendering and distribution time, i.e. how long it takes
to deploy it to the various immersive environments (online and/or offline), with various
parameters such a camera/output resolution and number of camera views, which often dependon the setup in the actual immersive theatre/space.

OpenSpace rendering of Ganges Chasma, a deep canyon on Mars

2.2.3 Clips produced with third parties
2.2.3.1

Atacama Desert

General description
Title

Atacama Desert

Subcontractor

Martin Heck / Timestorm Films

Date

July 2017

Number of clips

1

Content summary
8K timelapse video of Atacama Desert in Chile, taken at night (2am) showing stars spreading upon
Deliverable 5.1

Revision 1.0

17/25

the night sky
Technology
Timelapse with sound
8K@29,97fps, 1 min
Sony A7RII camera
Introduction
Upon receival of the raw material from the artist at first the material was screened. Then a clip
was selected from the total material. A fitting audio was selected that emphasizes the video
material and at last the final compound was made.

2.2.3.2

Pano LA 10K

General description
Title

Pano LA 10K

Subcontractor

Joe Capra / Scientifantastic

Dateshot over 2 years

Number of clips

1

Content summary
10K timelapse video of Los Angeles, showing panoramic views of the city of Los Angeles
Technology
Timelapse with sound
10k@23.967fps, 2 min 30 sec
shot over a period of 2 years entirely in true panoramic form
using 2 synced DSLR cameras (Canon 5DIII) side by side
Introduction
Upon receival of the raw material from the artist at first the material was screened. Then a
Deliverable 5.1

Revision 1.0

18/25

selection of several scenes of 2min 30sec was made from the total material. The final cut wasdone, as well as final camera movements were rendered in raw material. A fitting audio wasselected that emphasizes the video materia. At last the final compound of the material was madeas well as video encoding.

2.2.3.3

Prima Materia 8K

General description
Title

Prima Materia 8K

Subcontractor

NohlabDate

2018

Number of clips

1

Content summary
A stereoscopic piece taking the audience on an audiovisual journey.
Technology
Computer generated stereoscopic video with soundFormat is side by side stereoscopic 3D
Rendered 8K @60 fps for each eye, 7 min 28 sec
Introduction
Story: A stereoscopic audiovisual journey of the primitive formless base of all matter Prima
Materia. Alchemical authors have compared the “Prima Materia” to everything, to male andfemale, to the hermaphroditic monster, to heaven and earth, to body and spirit, chaos,microcosm, and the confused mass; it contains in itself all colors and potentially all metals; there isnothing more wonderful in the world, for it begets itself, conceives itself, and gives birth to itself.
Nohlab delivered 4 video files (for wall, floor, left eye, right eye). On the basis of the raw material(delivered as single frame sequences) the final compound was made by AE, rendering material
with audio and credits.

Deliverable 5.1

Revision 1.0

19/25

2.2.3.4

Ancient Invisible Cities: The Great Pyramid

General description
Title

Ancient Invisible Cities: The Great Pyramid 3D Immersive Experience

Subcontractor

BBC / Scanlab

Date

June 2018 - March 2019

Number of clips

3 videos embedded within a framework allowing interactive use
Work in progressContent summary
Point cloud renderings of the Great Pyramid in Cairo
Technology
3 clips in 12K@60fps, Stereo 3D, 360° camera based on 3D scan CGI
Exploring the millimetre accurate full-colour scans in spectacular VR gives you more freedom of
movement and access to this remarkable structure's secrets than a visit to the pyramid itself. Thisscanning technology creates point-clouds of data that allow us to recreate the Pyramid in thedigital world allowing us to move through the space in a way that is impossible in real life.
Introduction
The Great Pyramid in 3D: From the BBC series, Ancient Invisible Cities will offer users access to aspectacular VR experience inside the oldest wonder of the ancient world: The Great Pyramid ofGiza. With a choice of three starting points users can explore the stunning signature scansfeatured in the Ancient Invisible Cities series in astonishing 12K. Enhanced by surround sounddesigns, specially crafted soundtracks, and content-packed commentary, international venues caninvite their visitors to embark on their own immersive journey around, under, and above the Great
Pyramid of Giza. It can also be experienced using VR headsets.

Deliverable 5.1

Revision 1.0

20/25

2.2.3.5

Cinematic rendering – Virtual Anatomy

General description
Title

Cinematic rendering – Virtual Anatomy

Subcontractor

Siemens Healthineers /JKU

DateWork in progress (production in 2019)

Content summary
On the basis of physical fundamentals, Siemens has developed an innovative rendering algorithm
that provides superbly detailed, photorealistic images.
A game controller can be used to operate Cinematic Rendering software so that it can generateinteractive real-time depictions of human anatomy on the basis of CT and MR data sets.
Technology
8K Stereo with 60/120 fps
Introduction
Imaging procedures such as computer tomography (CT) and magnetic resonance (MRI) deliver
realistic, extraordinarily detailed pictures of the interior of a patient’s body. This opens up new
possibilities for medical education and the planning of surgical procedures, and also facilitatescommunication with referring physicians and patients.
On the basis of physical fundamentals, Siemens has developed an innovative rendering algorithm
that provides superbly detailed, photorealistic images. Driven forward by progress in the animatedfilm industry, new rendering techniques bring forth results that are increasingly realistic andbeautiful.

Deliverable 5.1

Revision 1.0

21/25

2.2.4 Clips obtained from external sources
In order to have wider range content for testing the codec and for screening purposes at
dissemination events, the partners of the consortium obtained several video clips from external
partners. These clips are not official Immersify content and may be available for only selected
partners due to licence limitations (but were provided free of charge).
-

Devil’s Lungs: This 10-min 360° piece was shooted by the Loikka Dance Film Festival (Finland)with the Ozo camera. This camera produces spherical videos in 4Kx4K (stereo) at 30 frames/s.
The content about musicians and dancers moving around the camera and playing accordion.
- https://www.imdb.com/title/tt8385026/- https://www.danceinfo.fi/en/news/finnish-3d-vr-360o-dance-film-in-cannes-devils-l
ungs-celebrates-the-mystical-power-of-accordion-and-women/
For the purposes of Immersify, this piece has been upscaled to 8K resolution using the Spin
Digital high precision pre-processing filters.

-

-

-

NHK “Beyond ​the frame: 8K Future Project” offers a glimpse into the joint research project
between the Ars Electronica Futurelab and NHK, Japan’s largest public broadcastingorganization. ​Tokyo – The Luminescent Megacity t​akes the audience on a flight in 8K over the
megacity at nighttime, providing spectacular views in astounding resolution. The stop-motion
animated ​DOMO WORLD i​nvites viewers to discover a complex, miniature world. For ​8K
TECHNE five artists were asked to express themselves artistically in 8K, producing a series of
unique high-resolution videos: ​1024 ​by F​uyu Arai, ​Cosmos under Foot ​by Koki Sone, ​The
Window ​by Haruka Furuya, ​88888888​•••​K​by Koichiro Tsujikawa and ​mimesis ​by Eiji Tanigawa.
Pandarama: It is a 3-min 360° piece shooted by ARRI using their Alexa Omnicam. The FOV is
360° x 120°, the resolution is 13.7K x 4.8K, and the frame rate is 24 frames/s. More details
about this omnicam camera can be see in Deliverable 4.1 on “Report on HEVC Encoder for
Immersive Content”. This content is about a Chinese town celebrating a dragon boat race.
16K clips: Spin Digital received from Azlab three video clips in 16K resolution at 60 frames/s
for showcase at major Trade Fairs including IBC (Amsterdam), ISE (Amsterdam), NAB (Las
Vegas), InterBEE (Tokyo).

Deliverable 5.1

Revision 1.0

22/25

2.2.5 Preparing the content for final showcase
All the content produced or acquired in this task are master files in very high quality (4:2:2/RGB
format, 10/12-bit). In order to show the content at the envisioned events or installations (Trade Fairs,
Deep Space 8K, large screen displays, etc.), they must be encoded and played using the Spin Digital
HEVC codec. The encoding parameters also have to be carefully selected according to the applicationor use case. In other words, not all the target applications demand the same level of quality or
resolution. In the table below we specify the encoding options we proposed for each use case. These
options are the result of subjective and objective quality analysis performed by the project partners.
Use Case

Encoding parameters

8K HLS streaming(distribution)- Format: 8Kp60, 4:2:0, 10-bit- IP1: 64 frames (1 second)- GOP2: 16 frames- Bitrate: 70 Mbps

HLS streaming for distribution
to end users. The bitrate must
be as low as possible.

8K RTP streaming(contribution)- Format: 8Kp60, 4:2:2, 10-bit- IP: 64 frames (1 second)- GOP: 16 frames- Bitrate: 160 Mbps

RTP streaming for real-time
contribution links from the live
event to the broadcast studio.
High bitrate is required.

Master file quality
playback- Format: 8Kp60, RGB, 12-bit- IP: 64 (1 second)- GOP: 16 frames- Bitrate: 600 Mbps

Visually lossless / original
quality distribution format(same quality level as theoriginal).

Ultra-high resolution
playback on large
screen displays- Format: 16Kp60, 4:2:0, 10-bit- IP: 64 (1 second)- GOP: 16 frames- Bitrate: 400 Mbps

As coding artefacts (blocking,ringing, etc.) can be very visible
on large screen displays, highbitrate is required in thisapplication scenario.

Point cloud

8K- Format: 8Kp60, 4:2:0, 10-bit- IP: 64 frames (1 second)- GOP: 4 frames- Bitrate: 1600 Mbps

A point cloud is a set of data
points in space. The picturesgenerated by 3D scanners are
the most difficult andchallenging for any video
encoder including HEVC, as
they present very highfrequency components (thepoints). As a result, the content
must be encoded at the
maximum bitrate that theworkstation can process.

12K- Format: 12Kp60, 4:2:0, 10-bit- IP: 64 frames (1 second)- GOP: 4 frames- Bitrate: 1000 Mbps

Comments

16K- Format: 16Kp60, 4:2:0, 10-bit

1

IP refers to Intra Period: the distance between two consecutive Intra frames. The longer the intra period, thefewer seeking points in the bitstream.
2
GOP refers to Group of Pictures: The GOP defines the length of the coding structure (see Section 4.5 of
Deliverable 4.1 for more details).

Deliverable 5.1

Revision 1.0

23/25

- IP: 64 frames (1 second)- GOP: 4 frames- Bitrate: 400 Mbps
Deep Space 8K

6K- Format: 6Kp60, 4:2:2, 10-bit- IP: 128 frames (2 seconds)- GOP: 16 frames- Bitrate: 100 Mbps
8K
Format: 8Kp60, 4:2:2, 10-bit- IP: 128 frames (2 seconds)- GOP: 16 frames- Bitrate: 160 Mbps
12K stereoscopic side-by-side- Format: 12Kp60, 4:2:2, 10-bit per view
(wall, floor)- IP: 128 frames (2 seconds)- GOP: 16 frames- Bitrate: 30 Mbps

High-res 360° video
playback on screen

8K- Format: 8Kp30, 4:2:0, 10-bit- IP: 128 frames (2 seconds)- GOP: 16 frames- Spherical format: Cubemap- Bitrate: 70 Mbps

A Windows 10 machine with
Spin Player was used to play 8K
videos.
Since the AE playback system isstill under development toenable 8K and beyond
playback, downscaled versionsof the content in 6K werecreated for the AE Festival
2018 to ensure stableperformance.
Because of its spatio-temporalcharacteristics, Prima Materia
in 12K stereoscopic
side-by-side was compressed ata relative low bitrate (30Mbps), which is a goodtrade-off between quality anddecoding performance.
In the case of the Pandarama
12Kx8K sequence, there was noneed to encode it at highbitrate. A bitrate of 50 Mbps
was sufficient to ensure highquality playback.

12K- Format: 12Kp24, 4:2:0, 10-bit- IP: 128 frames (2 seconds)- GOP: 16 frames- Spherical format: Cubemap- Bitrate: 50 Mbps

3 Conclusions
In the first reporting period (until M12) we focused on delivery of nice 8K videos using 8K cameras forpresentation and dissemination purposes. In this time we promoted the project at several fairs andDeliverable 5.1

Revision 1.0

24/25

festivals and some 8K clips were required (the full list of dissemination events is collected in D2.3
Dissemination and Exploitation Report). We started also laser scanning, which enables rendering 16K
resolution images as well as 360° panoramas in high resolution or 8K/16K stereoscopic content. We
will go on with laser scanning, stereoscopic recording and timelapse technologies in next months.
The cooperation with other companies such as NHK or studios and individual media creators allows
us to produce or get licences for several other interesting video clips.

Declaration
The information, documentation and figures available in this deliverable, is written by the membersof ​Immersify (Audiovisual Technologies for Next Generation Immersive Media) project consortiumand does not necessarily reflect the views of the European Commission. The European Commission is
not liable for any use that may be made of the information contained herein.
This document contains original unpublished work except where clearly indicated otherwise.
Acknowledgement of previously published material and of the work of others has been madethrough appropriate citation, quotation or both.

Deliverable 5.1

Revision 1.0

25/25