8K Cultural Heritage Project Initiatives

Ichitaka Takaki

Technical Producer
Media Development & Strategy Center
General Media Administration
Japan Broadcasting Corporation (NHK)

1. NHK and Tokyo National Museum joint
project: “Our 8K Cultural Heritage”

In September, 2020, the Tokyo National Museum and NHK
began a joint research project using 8K technology titled, “Our8K Cultural Heritage.” The joint research uses 8K and the latest
scanner and photogrammetry technologies to focus on someof the national treasures and cultural properties held by theTokyo National Museum, creating 8K cultural properties using
ultra-high-definition 3D computer graphics, far exceeding video
experiences created in the past.
Under restrictions on gatherings due to the COVID-19
pandemic, art galleries and museums around the world aresearching for new ways to appreciate art. By combining these 8K
cultural properties with various digital tools, we are pursuing ways
to appreciate art in this new era, so that everyone can enjoy thecultural assets and experience a “wealth of heart and mind”. We
are also presenting the results of this work at events and throughbroadcast programs.
NHK has produced and broadcast a television series related tothis project titled “Fresh Encounters with our Cultural Heritage,”and the Tokyo National Museum plans to hold an exhibition
featuring a compilation of results from this joint research in 2022,
the 150th anniversary of the museum.

Takao Tsuda

Senior Manager
Systems Solution Center
Engineering Administration Department
Japan Broadcasting Corporation(NHK)■ Figure 1: Pre-rendered image of the Kuze Kannon 8K
cultural property■ Figure 2: Golden stamp 8K cultural property

2. Created 8K cultural properties and content

8K cultural properties are extremely detailed 3D computergraphics (CG) models created using software to combine datafrom photogrammetry and various other sensors. So far, we havecreated 8K cultural properties including the seven listed below (inno particular order). For each of them, we prepared high-quality
pre-rendered movies and/or interactive CG content rendered inreal time, based on the original 8K cultural property CG models.
・The Kudara Kannon (held at Horyu-ji temple)
・The Goggle-eyed dogu (clay figurine)・The Funakibon byobu screen with scenes in and around Kyoto
・The Kuze Kannon (held at Horyu-ji temple)
・A Kin-in (golden stamp, held at the Fukuoka City Museum)・A Ko’omote-type Noh mask (Kawauchi)
・A Denyamanba-type Noh mask
▼ Scanning tools
The main tools used for scanning the cultural properties werethe following.

・3D laser scanner: FARO Focus350, ScanArm 2.5C・Optical 3D scanner: AICON SmartScan
・Digital single-lens camera: SONY α7R Mark IV▼Pre-rendered movies
To create the 3D CG models, Autodesk Maya and Foundry
Modo software was used, and rendering was done using Arnold.
We aimed to achieve high quality, assuming the results would beappreciated on an 8K-HDR display, striving to achieve near-real
reproduction of both detail and color. Rendering high-quality,
high-resolution CG is time consuming, and in some cases,
production of these 8K cultural properties required more than anhour of processing per frame. For the four programs discussed
below, NHK performed in-house rendering of 179,020 frames in a
total of 160 cuts.

New Breeze Winter 2023

13■ Figure 3: Taking photos for photogrammetry

capable of displaying high-resolution CG for the HMD.
The interactive content was all produced using the SDR color
space. The photogrammetry photos are recorded as RAW data,
so it would be possible to produce it with a richer color space, but
considering issues such as production time and the processing loadon the PC used for real-time CG rendering, we decided to use
SDR. Support for HDR will be an issue for consideration in thefuture.
■ Figure 5: Performer manipulating the Goggle-eyed clay
figurine projected on a 450-inch screen■ Figure 4: Pre-rendered image of the Kudara Kannon 8K
Cultural property

3. BS8K program, “Fresh Encounters with ourCultural Heritage”

We received support from Mr. Shuichi Narita of AfterImage
Inc. in producing these 3D CG models.
▼ Interactive content
With CG images rendered in real-time (real-time CG), it isdifficult to achieve image quality better than pre-rendered CG
images, but real-time CG has the advantage of interactivity,
which cannot be achieved with pre-rendered CG. Observers canmanipulate the item themselves, viewing it at whatever angle orsize they desire. It is also useful in terms of efficiency, with lower
production cost in many cases, depending on the length andnumber of cuts required.
The interactive content was produced using UnrealEngine,with collaboration from historia Inc. To avoid degrading theimmersive quality of the content for viewers, we focused on
maintaining resolution and a frame rate of 60 fps. When
importing the original 8K cultural property data into theUnrealEngine environment, we optimized to minimize any visualeffect. We used a workstation equipped with an NVIDIA Quadro
RTX 8000 GPU to create the real-time CG images.
We output the interactive content to 8K displays andprojectors, manipulating it with a game controller, and wealso developed VR/AR content using a head-mounted display(HMD). To utilize the quality of the 8K cultural properties andfacilitate immersive virtual-space content, we used a Varjo XR-1

14

New Breeze Winter 2023

NHK is also producing a TV series for the BS8K channel
titled “Fresh Encounters with our Cultural Heritage” (https://www4.nhk.or.jp/P6741/). The program features various culturalproperties, with the host and guests examining them from angles
and close-ups that would never be possible normally, gaining athorough appreciation of the items, speculating on the intentionsand values of the creators as they do so, and enabling viewers toalso share in this appreciation.
As of January, 2022, the following four programs have beenproduced. We created pre-rendered movies and interactive content
for each of the programs, to clearly present features of each item,
as required for production. Each of the 8K cultural properties isdiscussed more specifically below.
▼ Goggle-eyed clay figurine
The 3D CG model of the clay figurine is composed ofapproximately 7.85 million polygons, using 13 8K-resolution
textures. The interactive content can be manipulated using a gamecontroller, with zoom, pan and tilt operations to rotate the figure
and change the viewpoint. A user-friendly interface is provided,
which can be used easily even by users not accustomed to 3D CG
operations. It has a simple viewpoint-reset operation to reset theposition to facing the figure, and a “B-dash” function to movethe figure quickly, by holding the “B” button and operating thejoystick. Another function enables the user to go inside the figure
with the push of a button.
We also developed VR content that enlarges the figurine by500-times and lets the user move around inside the figurine.
■ Figure 6: P erformer wearing an HMD and the clay
figurine VR content

210 8K-resolution textures (which are actually divided into 840
4K-resolution images). This comprises over 100 GB of data, so
it was impossible to load the entire set of textures into the GPUfor display. By implementing careful high-speed processing toswap textures according to the camera distance and the area
being displayed, we were able to implement seamless operation
without losing detail, regardless of the magnification of the byobu
screen (so-called level of detail (LOD) processing). With simple
LOD processing alone, it is difficult to avoid noticeable texture
switching, so before recording the TV program, we adjusted
various parameters such as display timing on a trial and error basis
to find the best settings.

■ Figure 7: Pre-rendered image of the 8K cultural property,“Funakibon byobu screen with scenes in andaround Kyoto”▼ Kudara Kannon
The first program in the series dealt with the Kudara Kannon,and preparation involved was really a learning process. The 3D
CG model consisted of approximately 4.8 million polygons and21 8K-resolution textures. In the studio we displayed the prerendered CG on a 1.6 m by 3.6 m LED display built on the set,to create a photo-real, actual-size display. We also developed
interactive VR content that gives a sense of scale while wearing theHMD by showing the Kudara Kannon image over the Japanesearchipelago, and AR content that allows very close observation
by integrating the Kudara Kannon image with live images inside
the HMD. In the program a Buddha sculptor attempts to carve areplica using this AR content.
Normally with VR and AR, some method is used to detect
the location of the HMD and its motion is linked to a camerain the CG space to display the CG at the corresponding positionin the HMD. In this case, we developed a function using asimilar mechanism, using a controller attached to the HMD
to move the camera in CG space, so the controller can be usedinstead of the HMD to set the camera position for the CG
shown on an 8K display connected to a workstation. Using thisfunction, after recording the program, the presenter’s movements
were reproduced using the controller, and the video seen by thepresenter in the HMD was simulated as 8K video. This wasoutput and recorded on P2. This arrangement was also used later■ Figure 8: Interactive content: Zooming to see people’s
facial expressions■ Figure 9: Kudara Kannon 8K cultural property built into
the studio set▼ Funakibon byobu screen with scenes in and around Kyoto
The Funakibon byobu screen portrays more than 2,700 peopleof influence, such as samurai, as well as shopkeepers and othercommoners. To be able to show each of the people on the screen
magnified on an 8K display, the CG model required textureswith extremely high resolution. The 8K cultural property of theFunakibon byobu screen with scenes in and around Kyoto uses

New Breeze Winter 2023

15

for the Goggle-eyed clay figurine content, and was effective for
events during the COVID-19 pandemic.
▼ Kuze Kannon
The 3D CG model of the Kuze Kannon consists ofapproximately 6.65 million polygons and 17 8K-resolution
textures. For interactive content, we developed two features: 1)
Content that enables the Kuze Kannon to be appreciated onan 8K display or 8K screen from any viewpoint, using a gamecontroller, and 2) AR content that enables the operator to pickup the CG Kuze Kannon and move it around using the HMDcontroller. In the TV program, a researcher in the studio uses
this AR content, attempting to place the Kuze Kannon image
appearing in the HMD onto a virtual pedestal. We also developeda system described below, to share the Kuze Kannon CG withremote locations (Tokyo, Nara), and conducted a “Digital researchconference”.
■ Figure 10: I nteractive content video of the Kuze Kannon
8K cultural property

4. Non-broadcast exhibitions▼ Tokyo National Museum exhibit event
In March, 2021, an event exhibiting our 8K culturalproperties, together with a press conference, was held at the Tokyo
National Museum. The interactive content for the Funakibon
byobu screen with scenes in and around Kyoto was projected in
8K onto a large screen. Also in the exhibit were real byobu screensand PV of the 8K cultural property pre-rendered movie, on an8K display playing on a loop. The Goggle-eyed clay figurineinteractive content was also shown and demonstrated on an 8Kdisplay.
▼ Inter BEE 2020/ 2021 exhibits
For Inter BEE 2020, which was held online, a 3D scan wastaken of the artistic set in the “Fresh Encounters with our CulturalHeritage” studio and presented as VR content under the topic,“8K production tour of studio and relay vehicle”.
For Inter BEE 2021, the interactive content for the Goggleeyed clay figurine was exhibited, and many visitors were able toexperience this 8K cultural property.
▼ Kuze Kannon image digital research conference
In June, 2021, we held a “Digital research conference” asa special part of the 8K Cultural Heritage Project. For theresearch conference, researchers from the NHK Science andTechnology Research Labs (NHK STRL) and from the NHK
Nara broadcasting station gathered to study the Kuze Kannon,a national treasure. The Kuze Kannon interactive content wasinstalled on workstations at the two locations and projected onto
8K screens. At each location, a game controller was used to zoom
in and out, and to move the 3D CG model around, and the■ Figure 11: Projecting the Funakibon byobu screen with scenes in and around Kyoto on a large screen

16

New Breeze Winter 2023