1

2

3

Gran Turismo is “AAA” racing game series on PlayStation platform, andGran Turismo SPORT is a latest title.
Gran Turismo SPORT was released in October 2017.
This title has a high reputation for its utilization of HDR and wide color
technology.

4

Watch our trailer

5

We’ll talk about past days of real-time rendering.
The quality of computer graphics imagery has improved significantly overthe past few years.
However, final output quality is restricted by the limitations of conventional
output devices, mainly their limited color space and limited dynamic range.

6Recently, HDR and wide color technology has expanded these limitations.
Using wide color space and high dynamic range standard, we can achieve
more attractive images effectively.

7However, using HDR and Wide Color technologies are not trivial and arestill difficult problems.
It’s because consistent interpretations of both hardware behavior andsoftware specification is difficult.

8

We are game developers.
So, our solution is from the game industry and it’s a consistent, theorybased approach for each aspect of the workflow.
Such asasset collecting and editing,Interchangeable formats,
Encoding,
Working environment,
Verification,
Rendering pipeline,
And so on.
As a result, high quality output is achieved.

9

From our course, what you will learn are “fundamental theory” and“practical implementation” about a HDR and wide color technology.
These knowledge are based on our case studies in game developing and“real” knowledge.
We believe your will get knowledge about developing HDR and wide color
applications effectively and efficiently.

10This is a overview of this course.
Chapter one is “Introduction of This Course” and we are talking now.
Chapter two is “Introduction of Color and HDR “.
In this chapter, we’ll talk about the fundamental theory about color system
and HDR.
Chapter three is “Practical HDR Output Techniques”.
In this chapter, we’ll talk about the practical techniques such as tone
mapping and gamut mapping.
Chapter four is “Asset Creation”.
In this chapter, we’ll talk about our HDR asset creation workflow.
Chapter five is “Implementations and Results”.
In this chapter, we’ll explain about case studies in our real game.
Implementations and results about problems we encountered.
Chapter six is “HDR Working Environment”.
In this chapter, we’ll discuss about exploring HDR working environment
based on our experience.

11

Our course materials are here.

12

To discuss about color and dynamic ranges we have to start from somebasics of color science.
We will talk about human eye mechanism and the basics of modern display
technologies.

13

First, we’d like to talk about human eye color sensitivity.

14

The human eye sees Electro Magnetic Wave as a light.

15

Light contains multiple different frequency electro magnetic waves.
This mixture is called spectrum.

16The spectrum from 360 to 800 nm is visible to the human eye.
Each frequency of spectrum has unique color stimulation.

17

Because human eye has three cones that are sensitive to colors.

18

Scientists wanted to measure human eye sensitivity.
In 1931 CIE, a International Commission on Illumination did a research on it.

19

To measure a human eye sensitivity, first we prepare a white diffuse board
and a separator.

20

Then light up each part.
Upper part is a mixture of red, green, blue lights.
Bottom part is illuminated by target light which we want to measure.

21

Then, adjust illumination balance to match two colors.

22

Keep Adjusting…

23

Adjust until two colors matches.

24

If these colors matched, the balance of the primary light represents humaneye color sensitivity of that target spectrum.
CIE, a international commission on illumination repeated this process from
380 to 780 nm.

25This is a result.
These graph is called CMFs, Color Matching Functions.

26

Many CMFs were proposed in history.
CIE 1931 XYZ is the most popular one.
CIE 1931 XYZ Judd Vos modified is closer to original XYZ functions and it isbetter at blue color representation.
CIE 2012 XYZ or CIE 2006 is relatively new and used in some TV
manufacturer.

27

As mentioned before, human eye has three cones.
Due to that, three degrees of freedom is sufficient to describe color.
Technically all the colors can be described within three dimensions vector,
which is called tristimulus value.

28

For a given spectrum C lambda and Color Matching Function x, y, z,
We can calculate tristimulus values by just simple integral like this.

29

In some case three dimensions are too much complicated to draw on apaper or 2D surface.
And thus a smaller, two-dimensional xy coordinate is introduced.
This coordinate is used to discuss about any light’s absolute color.

30

Using xy coordinate we can plot any color onto 2d diagram.
This diagram is called chromaticity diagram.
Once we plot all visible colors, we get this horse shoe shape, which iscalled “Color Locus”.
No one can see colors outside this locus. Those colors are called imaginary
color.

31

y function of CMF CIE 1931 XYZ is exactly a same function of Luminosity
function of human eye.
Any luminous intensity of any light wave can calculated with this formula.
In TV specification, 1 nit is commonly used unit and 1 nit equals to 1
candela per meter square.

32With two colors c1 and c2 we can write color matching simply using equal
operator.
Due to CMF shape different lights spectrum can be matched.
This phenomenon is called metamerism.

33

For tristimulus value pair c1 and c2 these rules are established fromexperience.
These rules are called Grassman ’s law.

34

Color can be interpolated. For two colors c1 and c2 we can calculate
blended color just by alpha interpolation like this.

35

When we select three primary color pr, pg and pb arbitrarily, these primarycolors can act as a basis.
Three primary colors are enough to describe other color by linear
combination.

36

When the linear combination weight is limited to positive, the range
becomes a triangle on the chromaticity diagram.
If we allow negative weight, any three primary colors can describe allvisible color, but usually we don’t do that.

37

Therefore, three primary colors define one color region.
This region is called Gamut.
Describing color using gamut is called the “colorimetric system”.

38Basically, we have to know which gamut is used when any tristimulus value
is present.
Otherwise, the color is not determined exactly.
For example, the XYZ tristimulus normally used with CIE 1931 XYZ gamut,
but no one guarantees that.

39Different gamut describes same color in different tristimulus.
The conversion between any gamuts can be described in a simple matrix
transformation.
This matrix is 9 degree of freedom and a gamut has three tristimulus, so
the answer exists.

40

To wrap up this section,
We can say, light is electro magnetic wave spectrum.
Human eye perceives spectrum as a color.
Color can be treated as a three dimensional vector.

41

From here we will talk about High Dynamic Range display.

42

The motivation for HDR TV comes from a fact that human eye is really
good sensor.

43

Human eye can see a light from very dark to very bright with pupil
adaptation.
It has 10 powered by 14th of dynamic range, which is equivalent to 46 bitsof precision.
This dynamic range is too wide to reproduce but recently
Dolby Imaging research team found 90 % of non professional people aresatisfied with
1:10,000 of dynamic range in a picture.

44

Human eye is also good at seeing colors.
For example, human eye can discover 1nm difference of spectrum.
But the conventional TV system can cover less than 80% of all visible
colors.

45

Thanks to the technology evolution, devices nowadays are getting better.
To use advantages, HDR TV specification is established.
“HDR10” is the most popular HDR standard used in games.
HDR TVs means TVs with High Dynamic Range capability.
They still new and expensive, but it’s getting more common.
They can output more vivid pictures with wider gamut and brighter peakluminance.

46

On the other hand, conventional monitors are called SDR TV.
SDR means Standard Dynamic Range.
They supports limited gamut and limited peak luminance so the color
reproductivity is limited compared to HDR TVs.

47

In gaming industry, the term LDR is commonly used to describe a value
which range is limited.
In TV Broadcasting industry, the term SDR is commonly used instead.

48

This is a visualization of color volume of SDR TV.

49

This is HDR color volume.
It’s big and covers more colors.
This is the benefit of HDR TV.

50

There is a small problem in the HDR TV market these days.
There are too many variations of specs between products andmanufacturers.
Some TV monitor are really close to SDR TV and picture is notcomparatively better.
Some TV monitor can output more than the professional monitor.
The market of HDR TV is full of dynamic range.
The most problem is that, the pictures are different.
We need to measure or calibrate the actual performance of each display inuser environment.

51

From game industry point of view, we need to provide equal experience foreach user regardless of environment.
For example, a scene like this, the track line in the bright white will be very
important.
As recent games gets more competitive, proper specification of device
calibration becomes more important.

52

There are two variations in HDR TV mechanism.
One is non emissive and one is self emissive.

53

Non emissive display is the most common display these days.
The emission from backlight goes through color filter.
Then Liquid crystal polarized filter makes picture out of filtered light.

54

OLED is another popular display.
OLED has self emissive pixels made from organic chemicals.
It has good picture quality thanks to the perfect dark pixel.
But OLED has a weak point.
It causes burn in when picture is not moving or pixel is too bright, heatdamages the pixels.
Future games must think about it on OLED monitor.

55

Micro LED array is the future of self emissive displays.
It has great picture quality and no burn in effect but very expensive right
now.

56

Let’s talk about mechanism of image transfer.

57

Human eye has nonlinear sensitivity.
Eyes are more sensitive at the dark picture than the brighter picture.
This nonlinearity is very useful to reduce bandwidth of the data.

58

As you may already know of it, gamma curve is very popular in imagetransfer.
The original idea of gamma curve was an tonal reproduction
characteristics of photographic film.
This nonlinear curve fits human eye sensitivity.

59

The gamma curve was introduced to TV system to reduce radio bandwidth.
So the gamma curve was a compression technique at that time.
But not only that, TV display tube also had nonlinear characteristics.
Gamma curve solved everything at once.

60

Recently, we call that nonlinear curve, OETF and EOTF.
OETF means Opto-Electric Transfer Function.
Camera uses OETF to convert optical signal to electric signal.
EOTF is opposite side of OETF and it means Electro-Optic Transfer
Function.
TV monitor uses EOTF to calculate the power of output light.

61

First, camera captures linear optical signal.
Then converts the picture using OETF.
Then the image is transferred.
Finally the TV monitor recovers output picture using EOTF.

62

This is sRGB Gamma curve.
A de facto standard OETF curve for non HDR monitors.
Please check ITU-R BT.1886 information for more detail.

63

There are two variations for SDR Gamma. One is BT.1886, and one is sRGB
gamma.
The difference between these two is just some parameter values.

64But the real world is more complicated.
For example, some TV monitors do not comply with the regulation anduses simple 2.2 gamma curve.
So we decided to use the “sRGB gamma” as a standard for SDR TV OETF.

65

And we will use “sRGB” as a standard name for SDR TV gamut.
Also defined 100 nits as a SDR TV maximum brightness.

66

There are two variation of OETF for HDR TV.
One is HDR10. HDR10, PQ curve, Dolby Vision and BT.2100, all of this uses
same curve.
In HDR10, pictures can contain absolute luminance of 10,000 nits
maximum.
One another is Hybrid Log-Gamma, HLG.
It uses gamma-ish curve to have compatibility with SDR TV.

67

This is PQ Curve.
PQ Curve is a standard EOTF for HDR10 system and it is based on Dolby’s
research.
PQ curve has another names.
ITU-R Recommendation BT.2100 HDR10 EOTF, ST2084.
We use “PQ curve” and “Inverse PQ curve” in this slide.

68

This is Hybrid Log Gamma curve.
Hybrid log gamma curve is designed to compatible with SDR TVs.
So the dynamic range is not as wide as HDR10.

69

From game graphics point of view, we usually use OETF to encode scene
linear image.
But PQ curve defines an EOTF.
So, Inverse PQ Curve will be more used in game graphics output.
In this slide, we use a word “Inverse PQ Curve” as a OETF for HDR TV.

70

Next to OETF and EOTF, we will talk about color.

71

Wide Color Gamut is one of key technology of HDR TVs.
So what is wide color gamut?
There is no exact define of it but
We will use Wide Color Gamut, in meanings of a color gamut that is wider
than conventional sRGB.

72

The main motivation to use wide color gamut can be found in thesepictures.
The whole world is full of brilliant colors.
sRGB is not sufficient.

73

As mentioned before again and again, the gamut of sRGB is narrow.
We cannot represent actual color of tropical flowers or racing cars using
sRGB.

74

So wide color gamut is proposed and getting popular.
Of course, each of them has pros and cons.

75

This is Adobe RGB.
The most popular wide color gamut.
This gamut has just green wider than sRGB.
But a lot of devices supports this gamut.
So this gamut will be very good first step.

76

This is BT.2020 gamut.
A standard gamut of HDR TVs.
It has different names like ITU-R Recommendation BT.2020, or Rec.2020
but means same.
All primary colors are on color locus.
That means this gamut is very difficult to be implemented on real monitors.
So in 2018, no TV monitor can output this gamut 100% yet.

77

This gamut is DCI-P3.
Widely used in Cinema grading.
Recently iPhone and some apple product starts to supports this gamut.
Some good PC monitors and TV monitors also supports this gamut.

78

This is an ACEScg gamut.
ACES is Academy Color Encoding System, a group of colorimetry experts.
ACEScg is designed for rendering and compositing so some games infuture will use this.

79

Lastly this is ACES2065-1 gamut.
Also known as ACES AP0.
This gamut is designed to store image and covers full locus.
Therefore loss in color precision can happen.

80

Wide color gamut has a lot of benefits.
WCG can represent most of artificial lights such as traffic signals and neonlights.
WC image keeps quality after heavy post process.
Some wide color gamut has better lighting accuracy than sRGB.
These benefits works fine even if the output device is sRGB.

81

From here we will talk why wide color gamut is better in lighting accuracy.
In Wide color gamut, saturated colors such as 255,0,0 a vivid red are veryrare.
Plus, wide color gamut is designed to have more evenly distributed hues.
That’s why ACEScg and BT.2020 has better accuracy in lighting calculation.

82

To test lighting accuracy, we will have side-by-side comparison.
Tristimulus lighting is a usual lighting calculation used in computer
graphics.
Lighting is calculated by per channel multiplication.
Ground truth is calculated in spectrum domain.

Finally we will calculate color difference using CIE Delta E 2000 formula.

83

This graph is a CIE 1931 XYZ Color Matching Functions.
X, Y and Z are the red, green, blue of the CIE 1931 XYZ color space.

84

By this point you’ve become very acquainted with color science, but letme introduce some basic calculation of lighting in spectrum domain.
Let’s think about a simple scenario.
There is a illuminant.
There is an surface.
And Human eye sees a color of reflected light.

85

We’d like to calculate a color of this.

86

First we have to know a illuminant spectrum.
With no light, nothing is visible. Here I will call this C lambda.

87

Second we have to know reflectance spectrum of surface.
Let this be A lambda.

88

So, the reflected light is C lambda multiplied by A lambda

89

We can calculate the color of the reflected light from convolution of theresult spectrum and Color Matching Functions.

90For real-time lighting, we usually use tristimulus values instead of full
spectrum convolution.
Lighting calculation in Tristimulus values are very simple and fast but it will
not match to ground truth result calculated in spectrum.

91

We repeated this process for popular 24 colors from xrite color checker.
This is a cropped sample from results.
In this case we used sRGB gamut for tristimulus calculation.
The left-bottom triangles are results of tristimulus calculation.
And the top right triangles are ground truth.

So the color reproduction error can be seen as a line between them.

92This is a full visualization of results calculated in bt.2020.

93And this is a result of sRGB.

94

Here is a plot of color reproduction error in case of illuminant CIE D65.
Color differences are calculated using CIE LAB Delta E 2000.
Horizontal axis is patch index and vertical axis is color difference.
The line in Blue is sRGB, orange is BT.2020 and grey is ACEScg.

95

This is the average error.
Each gamut has color reproduction error but BT.2020 and ACEScg are
significantly better than sRGB.

96

97

Before talking about practical HDR output techniques we have to talkabout HDR10 itself a short.
We make games for PlayStation®4 platform and playstation4 supports
HDR10,
So we decided to use HDR10 as a HDR output format.
In addition to that, HDR10 has a metadata system so here we introduce
them very short.

98

HDR10 uses BT.2020 compared to sRGB.

99

HDR10 uses PQ curve for EOTF and it has 100times larger dynamic range.

100

HDR10 format can have some metadata.
These metadata is used by TV monitor to optimize their settings.
MaxFALL is a maximum frame average luminance, it describes content
average luminance.
MaxCLL is a maximum content luminance, it describes content peakluminance.
Some TV just ignores these values, but some TV uses these value.
Some Game console can not send these metadata correctly.
Complicated.

101

This is a calculation of MaxFALL value.
Just an average of maximum luminance per pixel in each frame.

102And this is the formula of MaxCLL metadata.
Very simple.
It’s just a maximum luminance for the content’s whole frame.

103

From here we will talk about our HDR output strategy.

104

As talked before, HDR and SDR uses different OETF and Gamuts.
To output images for both TV, we have to apply different conversion
functions.
Some of cinema industry uses different grading and post process to
optimize image quality for each of them.

105But we used single source multiple output style.
We prepare single beauty source and apply single grading on it.
Then convert it to the output format.
Movies uses multiple grading because they are static and predictable somultiple gradings are possible.
But Games are dynamic and unpredictable so static conversion or multiple
grading is not able to applied.

106

This is a conventional flow of static content.
It applies multiple grading independently designed and optimized for each
output devices.
This style needs a lot of time to prepare multiple source and difficult in real
time.

107

This is single source multiple output style.
First we render single source beauty image.
Using most wide dynamic range and most wide gamut.
And then apply proper output conversion for each device in real time.
HDR and SDR compatibility is achieved.

108

To achieve this we created two techniques.

109

110

Variable tone mapping is created to fill the gap of different peak brightnessof different TV models.
Some TV can output 4000 nits, but some TV outputs only 300 nits.
So we need an adaptive mapping function for luminance.

111

There are three popular tone mapping functions introduced.
Both of them has pros and cons, but none of them fit our application.
So we developed our own tone mapping function.

112

One popular tone mapping is presented from ACES.
It is called RRT and ODT.
This tone mapping is based on fuji film’s physical film characteristic.
This tone mapping is designed by professionals and battle proofed but
some color effects are baked in.
Our game is a car game so embedded color effects doesn’t fit for us.

113This is Kawase’s variable tone mapping function.
First presented at CEDEC2016, he introduced an idea of variable tone
mapping.

114

But there is no actual formulation and implementation information.

115

Lottes from AMD also presented variable tone mapping in GDC2016.

116

His tone mapping is open and flexible.
But the function is little bit difficult to adjust.
Also this function has no linear part.

117

So we developed our own tone mapping function, GT Tone Mapping.
Design philosophy of our tone mapping are learnt from our artist and kept
it simple in mind.

118

This is our tone mapping design philosophy.
There are three important points learnt from our artist.

119

One is a contrast adjustable toe,120

One is a middle linear section.
This makes whole picture photorealistic.
This section is the most important part.

121

And a smooth shoulder.

122

GT Tone mapping smoothly connect these three sections.

123

We presented this function first at CEDEC 2017.
Formulation is on desmos.

124

By the way, we have made one mistake during GT Tone Mappingdevelopment.
It is inverse tone mapping function.
We used smoothstep function to connect sections.
But it made inverse tone mapping a little bit difficult.

125

Next we’d like to talk about gamut mapping.

126

When converting vivid colors into narrower gamut,
simple gamut conversion is not enough to achieve visual quality.
In this pictures the yellow part actually is out of gamut,
so the simple conversion results in just saturated red.

127

Here is a illustration why it happens.
The color outside sRGB gamut saturates and clipped.

128

At the development time, there was only one method is proposed.
Alex Fry from EA proposed his method at GDC 2017.
But that method doesn’t matched to our game,so we decided to use our own method.

129

This is a frostbite method, presented at GDC 2017.
They used ICtCp color space to separate chromaticity and lightness of
HDR framebuffer.
Then, they apply different curve to each channel.
For lightness, a simple tone mapping function is applied.
For chromaticity, a shoulder function is applied to have desaturation effect.
We also tested same idea, but the result doesn’t fir for us.

130We used other method.
We apply GT Tone mapping function to each RGB channel independently.
This gives us gamut mapping and tone mapping, two effects at once.
The figure below shows the gamut mapping effect of our method.
From left to right, the colors gets brighter, but gamut mapping gives us
natural look.

131

By the way, our method has some minor issue. That is Hue shifting.
While applying tone mapping function for RGB channel independently,
we don’t care about hue of the pixels.
So the hue shifts when the color is very bright and vivid.

132

But we accepted it.
Take a close look at this real photography.
Hue shifting occurs even in the real world.

133

This section is just a small tips but we’d like to introduce some basic tips
for moving SDR to HDR.

134

To make your game looks good at HDR TV,
you need to know what is “good” in HDR.
A high-end HDR TV will be required.
Netflix has a lot of great HDR contents.
I recommend to see “Chef’s table France” in HDR.
It has some really good HDR picture is in it.
The reason is the show is not so much graded and has very bright specular.

135

First step of move is converting your already owning assets.
Change OETF, apply color conversion.
Then check how your assets looks like on HDR TV.

136

Color expanding is also a good idea.
Some inverse tone mapping and chroma saturation can expand your assets.
These are not best answer but still is a good start line.

137

138

From here we will talk about asset creation.
Especially for capturing HDR, wide color images.

139

To capture the high dynamic range image we need to use image bracketing
technique.
Image sensors in DSLR has 14 or more steps of dynamic range, it isequivalent to 10^4 of dynamic range.
So we need multiple exposure to cover up the dynamic range of human eye.

140

From each pixel on multiple exposure pictures, we can estimate actual
irradiance using Paul Debevec’s weighted sum method.

141

But naïve implementation will hit the problem.
Dark pictures contains a lot of noise by nature and Debevec method
boosts the noise and, thus, the result will be very noisy.
To solve this we created our own weighting function that takes care ofdark noisy image.
Also we implemented wavelet denoiser.

142

To used in game, image assets must be neutral.
No look and develop is necessary.
So the image calibration process is very important.

143

We used Mitsunaga’s response curve calibration to measure image sensor
linearity.

144

We used OpenCV to remove lens distortion.
Computer graphics basically has no distortion on it.
So lens distortion calibration is very important.

145

146

We also calibrated image sensor’s native color gamut.
We used X-Rite color checker and Sekonic C-700 spectrometer.

147

We take photo of color checker using high color rendition light to capturecolor sensor characteristics.

148

Xrite color checker has nicely controlled color patches.
Spectrum reflectance of each patches are well known.
We also measured spectrum reflectance of each patch by ourselves, butthere is no significant difference.

149

From these reflectance and captured illuminant spectrum, we cancalculate ideal absolute color of each patches.
Then we can solve gamut conversion matrix from color relation between
patches and camera raw image using simple Gauss-Seidel solver.
For more detail please check my talk at CEDEC 2016.

150

We used SONY ILCE-7 series DSLR.
We have measured all distortions of lenses

151

Here are some results.
Actually this is a rendering from game itself, background images and lightreflections are captured using the camera system we presented.

152

To achieve photorealistic blending between background images and car
model, Lens distortion calibration is required.

153

Captured in High dynamic range and wide color gamut, these images also
works well in HDR TV.

154

Next, we’d like to talk about handling HDR still images and videos in ourdevelopment.

155

These are examples of our HDR still images.
This is an image of a texture used for light emission.

156

These are images used in our sky dome.

157

And this is an example background image used in Scapes, an in-gamefeature.
Scapes will be described in detail later.

158

We use a few HDR image formats for each purpose. Mainly, we use
different formats for asset creation and for runtime.
For asset creation, HDR images are mainly used by artists during asset
creation and editing.
On the other hand, during runtime, HDR images are directly used by thegame itself.

159

First, we would first like to talk about HDR image formats for asset
creation.
There are many image formats that support high dynamic range images,
and we mainly use two formats: Radiance HDR and OpenEXR.
We would like to explain some of the advantages and disadvantages ofeach format.

160

Radiance HDR is a widely known image format.
The format specification is easy to understand, and many public
implementations exist for it.
In addition to that, many existing software can handle Radiance HDR, so it
has good portability.
However, it has some weakness.
First, it cannot store complex data such as multi channel data, and due toits shared exponent method, the precision is not so good. (For our
purposes, this precision is almost sufficient.)
Its compression method is also simple and inefficient, and high resolution
images consume a large amount of storage.

161

Open EXR was developed by Industrial Light & Magic, and is the de facto
standard HDR format in the computer graphics industry.
Open EXR can store complex data and better precision data such as 32 bit
floating point values.
Major DCC software support and it has a better compression methods
than Radiance HDR. Also, a lossy compression method is available.
Basically, Open EXR is a very good format, but lossy compression quality is
not as good as JPEG XR.
In any case, artists can use these two formats for their work.

162

There are other HDR image formats of course, but we are using these twoformats because of portability and rich features.

163

We also need to use HDR images during the game’s runtime. There aresome requirements for runtime HDR image formats.
First, this format needs to be very compact, as we need to store it in thegame’s disk, which has a limited amount of space.
Second, it needs to have a fast decode time, as decoding performance
directly impacts loading times.
Finally, it needs to be GPU friendly, as runtime memory efficiency andsampling cost is very important.

164

So, we use BC6H format for runtime images.
This is a block compression format and introduced in DirectX11.
This format can be decoded very fast and it’s designed for GPU so it’s very
GPU friendly.
Because of that, this format is appropriate for runtime.

165

We use the BC6H format for HDR textures when performancerequirements are severe, such as race sequences.

166

BC6H is a good format, but its compression rate leaves much to be desired,as it can only achieve one byte per pixel.

167

Instead, for Scapes, we need to handle a lot of HDR images, and thus
another format was introduced.

168

By the way, let me explain a bit more about Scapes.
Scapes is an in-game feature that allows you to position a car in multiple
scenes from around the world and take photos.

169

In Scapes, all background photos are HDR and resolutions are very high,and each photo has a resolution between 6000x4000 and 8000x5000)
In addition to that, we had over 1000 background images at launch time.

170And because of our requirements, BC6H is inefficient.
If we used BC6H for Scapes, the estimated total size for all images would
be over 30GB, which is half the size of a game disk.

171

So we decided to use JPEG XR, which was developed my Microsoft.
This format can store complex data and support not only HDR format but
many other color formats.
For our purposes, the most important aspect of JPEG XR is that JPEG XR
has high quality, and an efficient lossy algorithm.
Unfortunately, there are few software that support JPEG XR appropriately,
but the image preview in Windows 10 support it.

172Our implementation is based on a public domain version of JPEG XR.
We improved the decoding performance, and applied code optimizations
using SIMD and added parallel decoding features.
Using these optimizations, we achieved about 10 times to 20 times fasterdecoding times than the original code.

173

On average, compared to BC6H, the disk consumption decreased fromabout one half to one third of the size.
Therefore, we use this modified JPEG XR format for Scapes backgroundimages and in-game photos captured by users.

174

Next, we will explain our HDR video formats.
As a pre-processing step in-game videos, such as the opening and ending
videos, are graded for 10,000 nit targets.
These videos are manually graded to expand the dynamic range.

175

Then, graded videos are stored as files after the PQ curve is applied.

176

At runtime, before playing the video, inverse PQ curve is applied to thevideo file to play, and then the buffer is linearized177

Next, GT tone mapping is applied for each output device.

178

Finally, OETF is applied for each output device.
In SDR, sRGB gamma is applied and in HDR, inverse PQ curve is applied.
With this process, we can use the same source video for SDR and HDR
output device.

179

Our runtime video format is HEVC.
HEVC is a high quality video format.
This format can store 10 bit format pixels, which is useful for HDR outputs,
and has better quality than H.264.
On the other hand, HEVC is expensive to encode and decode. In addition tothat, it has a high licensing fee.
We needed a high quality runtime video, which is why we are using HEVC.

180

Next, we’d like to talk about Implementations and Results in Our Game.

181

First we will talk about implementation of GT Tone Mapping for HDR
Output.

182

Our runtime tone mapping method is called “GT Tone Mapping”.
We already explained the theory behind it in this course, but how did we
implement it in our game?

183

First we had an issue with the performance.
The simple implementation has a high computational cost because GT
Tone Mapping is a complex function.
It requires the use of power and exponential functions, which are heavy tocompute on the GPU.

184

Approximations such as rational function fitting need to be calculated foreach tone mapping parameter.
But because GT tone mapping varies with to a device’s peak brightness,
we can’t simply apply rational function fitting.

185

So we used a lookup table for computing GT Tone Mapping at runtime.
Using a lookup table means we applied a piecewise linear approximation to
the function
By using a lookup table, approximation error can be reduced by just using alarger size table.
Sampling cost tend to be smaller than the analytic calculation, but in ourcase ALU pressure was the main bottleneck.
Of course, memory consumption is larger than using an analytic function,
but it didn’t matter for our case.
By using a lookup table, quantization and discretization error happen, so
we need to decide appropriate parameters.

186

Let us begin by explaining about how usual linear lookup table mapping
works.

188

In this mapping, we uniformly sample a target function.

189

Example values are shown in the figure.

190

Then store sampled values onto a lookup table.

191

At runtime, we need to adjust offsets and scale for lookup table coordinate.

192

The figure shows a naïve bilinear approximation using lookup table.
As you can see, the UV range zero to one doesn’t match the range of thetarget function.

193

So, we need to adjust offsets and scale of lookup table UV coordinates.
With this process, we can find valid positions to sample the lookup table.

194

Next, we will explain about our nonlinear lookup table mapping.
We do not uniformly map the input value to the lookup table coordinates.

195

Accuracy is more important the near the origin, because there are low andmiddle parts of the function near the origin.
These parts decide the whole behavior of the function.
So we want to assign more texels near the origin to improve the accuracy
and assign less texels far from the origin.

196

This is our nonlinear lookup table mapping.
We non-uniformly sample the target function to assign more texels near
the origin.

197

We use a square mapping because the inverse function, the square root, iseasy to calculate at runtime.

198

We will show the visualization of mapping.
The whole input range is 0 to 50,000 nits and the lookup table resolution is
32.

200

And this is a comparison.
Vertical lines indicates sampling points.
As you can see, in our nonlinear mapping, there are more sampling points
near the origin compared to the usual linear mapping.

201

This is a comparison of the accuracy between linear mapping andnonlinear mapping.
The black line is the original function, the green line is the lookup tableapproximation, and the red line is the root squared error.
The LUT resolution used here is 8,192.
The left figure shows the error of linear mapping is much larger thannonlinear mapping.

204

Finally, our results.
On the top table you can see the specification for the lookup table in ourgame.
We used a 32-bit one dimensional texture, with 8,192 texels, using a totalof 32kilobytes.
As for the performance, on the PlayStation®4, outputting a Full HD image,the analytic implementation takes 0.40 milliseconds per frame.
On the other hand, our lookup table approximation takes 0.24 milliseconds
per frame.

206

Next, we will talk about tone mapping calibration for output devices.

207

We want to output appropriate HDR images and for that purpose, we need
information about the output HDR device.
Therefore, we need information about peak brightness of the device.

208

The reason we need peak brightness is because tone mapping uses thisparameter to adapt rendered images to output devices.
We want to output optimal images adapted to the dynamic range of theoutput device.
The detail has been already explained in this course.

209

If you want to know something about devices, you should measure it.
Recent games often have their own calibration processes. When starting
games for the first time, users need to calibrate their screens.

210

Next we will explain our own calibration process.
First, make sure users are facing straight at the device.
In HDR devices, viewing angle affects the visual appearance greatly, so
this process is important.

211

Second, move the slider to adjust the brightness of the checker board
patterns to the point where they are almost invisible.

212

This is our calibration process movie.

213

Then, how can this process estimate the peak brightness? Let us explain.
In the fixed white area, the input signal at the device is 10,000 nits so thedevice tries to output 10,000 nits pixels.

214

However, the typical device can’t output such bright pixels, so the device
will actually output its peak brightness.

215

On the other hand, in the darker area, when moving the slider, the input
signal is adjusted and brightened.

216

Then, the output pixels also become brighter.

217

When the user adjusted output is indistinguishable from the output of the
fixed white area, the peak brightness can be estimated.

218

If the output is indistinguishable, it can be thought that both area have thesame brightness.
On the other hand, the brightness of the adjusted darker area is obviously
a known value, so we can conclude that the device’s peak brightness is thesame as the adjusted brightness.

219

There are however some important points to consider.
Our method assumes that devices map the input signal directly to the
output brightness. The figure on the slide illustrates this assumption.

220

However, an actual behavior is often the right figure.
In this case, so called display mapping is applied and an estimated peakbrightness is overestimated.
But indeed the overestimation doesn’t matter because it doesn’t reduce
the effective dynamic range, so our method works.

221

Also, this process assumes that the peak brightness is constant across allareas of the target device, and that the same input signal will output thesame brightness.

222

However, the actual behavior depends on the entire output image.
This behavior happens because of each device’s power management
features.
Whenever a bright area becomes wider, the power available will be
distributed onto a wider area.
Because of that, the power per pixel will be reduced and pixel brightness
will also be reduced.

223

In our case, the peak brightness of the full screen is not necessary toestimate.
And this is why we use a checker board pattern on a part of the screen andnot full screen.

224

This slides shows some comparison results between measured brightness
and estimated value after calibration.
Our method overestimate the peak brightness in ordinary TVs, but works
very well in master monitors.
It’s because the behavior of a master monitor is very simple.
In a middle-end OLED TV, estimated value is much higher than themeasured value.
Because their target can be thought of Dolby Vision, the estimated value isnear 4,000 nits. (In Dolby Vision, the maximum brightness is 4,000 nits.)227

Next, we will talk about rendering in a wide color space and HDR.

228

For HDR rendering, we render frames in a wide color space and apply
inverse PQ curve such as OETF.
So we will focus on wide color rendering and inverse PQ curve in thissection.

229

Recently, HDR rendering has become the means to generating output for
HDR ready devices.
In the past, HDR rendering we used to render on a high dynamic range
linear space.

230

Let us begin with the conservative conservative approach for HDR
rendering.
The bottom figure shows this approach.
In this approach, we will render in a narrow, sRGB-like color space, thenconvert them for wide, BT.2020 color space to output.
In the BT.2020, every sRGB color can be represented, so it’s very simple
and easy to implement the color space conversion, as we can just apply aconversion matrix.

231

In this approach, the rendering process itself doesn’t need to be changed alot.
Every sRGB assets or texture can be used and doesn’t need to beconverted.
Some assets may be expanded to a wide color such as particles, lights.

232However, this approach can’t make full use of the wide color and HDR
output because of limited assets and rendering color space.

233

Here are some examples of wide color materials in the real world.
Wide color materials are very common, so it is useful if we can render in a
wider color range as well.

234

Like this.

235

And like this.

236

We will talk about our approach for HDR rendering.
In our approach, we render scenes in a high dynamic range, linear space.
And, we render scenes in a wide color space. Bt.2020.
Our frame buffer is RGB11 11 10 formats to reduce bandwidth.

237

We used BT.2020 but there were other candidates.
Next, we will talk about the reason we used BT.2020 instead of other color
spaces.

238

scRGB is similar to sRGB color space and uses negative values.
This color space has good compatibility with sRGB, but our frame buffer
can’t store negative values, so we don’t use scRGB.

239

CIE XYZ is a widely known color space.
This color space is a basic building block for color science, and it’s a very
wide color space.
However CIE XYZ is too wide for us, and the error during lighting or asset
conversion can be large.
In addition to that, white point specification differs from BT.2020, so color
shifting can easily occur.
Because of that, we don’t use CIE XYZ.

240

ACES is has become popular recently.
In ACES color space, standard color transformation is well defined such as
Reference Rendering Transform and Output Device Transform.
In addition to that, due to ACES popularity it has a good portability.
But in our opinion, many magic numbers exist in the RRT and ODT, andcolor grading processes are included in the ODT.
We want to use a neutral color space, so we don’t use ACES.

241

BT.2020 is the color space used in HDR10.
So, there is no need to convert for HDR10 outputs.
And the white point in BT.2020 is the same as sRGB. Thanks to that, color
space conversion between sRGB and BT.2020 is stable.
On the other hand, there was no previous work using BT.2020 for therendering color space at the time.
We didn’t have any knowledge.

242

But, we thought BT.2020 was the best solution, so we decided to use
BT.2020 for the rendering color space.

243

Next we will talk about the inverse PQ curve.
After rendering scenes and tone mapping, we need to apply the inverse PQ
curve to the HDR10 output.
Inverse PQ curve is shown below.

244

For runtime, we need a fast and accurate approximation of inverse PQ
curve.
Why fast?
It’s because inverse PQ curve is applied every frame, so we want to reduce
computational costs as much as possible.
Why accurate?
We wanted to use functions following the standard as much as possible.
We wanted to isolate problems easily and properly in whole workflow

245

Some useful approximations are already proposed.
[Patry 2017] is a fast analytic approximation but it is only valid under about
4,000 nits. And the accuracy is not so good.
[Malin 2018] is a lookup based approach and it looks fast, but we believe it
is not accurate enough. We didn’t know about it during development, so we
couldn’t use it.

246And this is our approximation.
First, we introduce a pow term, then we used Mahetmatica for fitting after
that.

247

This is a absolute error comparison between our approximation and [Patry
2017].
The purple line is our approximation and the red line is [Patry 2017].
As you can see, in our approximation, the bright input part is much moreaccurate.

248

Finally, we will talk about performance.
The reference implementation takes 0.36 ms on PlayStation®4, full HD.
And our approximation takes 0.33ms which is 10% faster than thereference.
[Patry 2017] is faster but not so accurate, so we used our approximation.

249

Next, we will talk about HDR color grading in Gran Turismo SPORT.

250

Color grading is the process of modifying input images in various ways toenhance final look.
Some of the modifications include brightness, contrast, hue, and so on.
In the past, color grading was directly applied to the film itself such as
bleach bypass.
Today, we can use a digital processing workflow, so can apply various
effects much more easily.

251

In Gran Turismo SPORT, there are two purposes for color grading.
The first is for artists.
Artists can enhance appearances in cut scene for attracting users withimpressive images.
The second is for users.
Users can edit in-game taken photos by in-game color grading tools.

252

We will show an example of color grading.
This is the original image.

254

This is the color graded image.

255

And this is a side-by-side comparison.

256

And this is our in-game color grading tool.
The right column is the editing tool.

257

User can use lookup table color grading filters.

258

And curve based, parametric color grading.

259

In SDR, color grading is important for a “more” artistic looking. And ofcourse, we also want to apply color grading in HDR.

260

So, we had two goals.
The first was support both lookup table based color grading andparametric based grading.
And the second was to support a compatible solution between SDR and
HDR.
Lookup table based grading is commonly used but hard to create andcurve based grading is limited but easy to use.
And we want to reduce the grading cost.

261

First, we will talk about a general method of lookup based grading in SDR.
In SDR, lookup based grading is commonly used.
Grading itself can be authorized by any grading software such as DaVinci
Resolve and Adobe Photoshop.
Then, why lookup table?
We wanted to reproduce certain grading processes easily.
Grading processes can be very complex, but by baking a grading process
into lookup tables, we can easily apply the same color grading.

262

As you know, using three dimensional lookup table, arbitrary grading
functions can be approximated in a piece-wise linear manner.
For that purpose, graded RGB values are stored into a LUT texture, andinput RGB values are interpreted as positions on the LUT texture.
By this process, computing original function can be approximated by single
3D texture fetching.

263

In SDR, 3D lookup table is very simple and efficient.
The resolutions don’t need to be so large. 16 by 16 by 16 is enough.
And the texture bit-depth is 8 bit and it’s compact.
These figures are examples of SDR lookup table.

264

In HDR, what can be solution?
Indeed, the main difference between SDR color grading and HDR color
grading is an input and output format.
In SDR, color space is sRGB and OETF is also sRGB.
But in HDR, color space is BT.2020 and OETF is inverse PQ.
So applying process itself is the same but grading process is different.
We need to do color grading and create 3D lookup table independently for
HDR.
In addition to that, for each HDR device and each device peak brightness.

265

That is why, we need to prepare many color grading settings, for both SDR
and HDR and for each HDR device.
It is hard to maintain a similar appearance in both SDR and HDR manuallyThis has a high cost.

268

So we want to use a single lookup table for SDR and HDR and for all
devices.
For that purpose we developed our new compatible solution.

269

Next, we will talk about SDR and HDR compatible lookup table grading
method.
In our method, once color grading is done, we can use it everywhere.
For that purpose, the grading target is set for the most genericenvironment.
The color space is BT.2020, OETF is Inverse PQ (ST.2084) and the target
nits is 10,000 nits.
And we are using DaVinci Resolve for grading.
This is a de facto standard in the movie and game industry.
For preview, we apply an appropriate display lookup table for DaVinci
Resolve.

270

We will explain about our compatible solution.
First, we will show an ordinary color grading process.
In this process, color grading is applied after OETF is applied.

271

And, this is an overview of our compatible grading process.
The latter stage is similar to the ordinary grading process, but the firststage is the key process.

272

We will explain about the first stage more.
First, pre temporal tone mapping is applied to a linear, rendered buffer.
We apply GT Tone Mapping and then inverse PQ curve in this process. The
target of tone mapping is 10,000 nits.
At this point, the buffer is completely valid for 10,000 nits target HDR10
format.

273

Second, HDR 3D lookup table is applied in an ordinary way. This lookup
table is for 10,000 nits.
This process is also valid because the input buffer’s target and color space
are the same as grading lookup table.

274

Third, post temporal tone mapping is applied.
We apply the PQ curve and then inverse GT tone mapping in this process
to reconstruct a linear buffer.

275

Finally, we can gain the linearized and color graded buffer.
Regardless of output devices, this process works so we achieve acompatible solution.

276

After grading, it’s simple.
First, apply tone mapping to the linear graded buffer.
This tone mapping parameter varies for each output device as described
before.

277

Second, apply color space conversion. In SDR, BT.2020 color is converted
to sRGB color. In HDR, nothing happens.

278

Finally, apply OETF. In SDR, sRGB and in HDR, inverse PQ curve.

279

Here are some results of our compatible solution.
In this slide, the left is SDR and the right is the HDR output.
These are the same scene and images are directly captured from a TV.

280

As you can see, in HDR, the dynamic range is preserved more than SDR.
In SDR, wider area is saturated because of its narrow range.

281

And these are images lookup table is applied.
As you can see, the whole impression looks similar, so our compatible
solution works well.

282

off

283

Next we will talk about parametric curve based grading.
Lookup table is great but is often hard to create, because there are manymethods to adjust and it’s not easy to learn the grading process.
So, we need a simple, parametric curve based grading. We want a method
like the “Curve” tool in Adobe Photoshop, but we also need it to becompatible with both SDR and HDR.

284

There was a request from artists. They want to enhance contrasts easily!
In SDR, “S” curve like filmic tone mapping can be a solution.
In HDR, we apply directly in a linear space and only the dark part needs tobended to enhance contrast.
The bright part can't be bended because the range is not between 0 and 1
and tone mapping will bend.

285

This is our solution.
The dark part is a simple gamma curve and the bright part is just a line andsimple scaling function.
This curve has 3 parameters. DarkPartGamma, MidPoint controller, andBrightPartScaling.

287

These are the behavior of the curve.
The center is strong emphasizing a dark part.
The right is strong emphasizing a bright part.

288

The dark part and bright part isn’t connected smoothly, but artists liked
this curve, so it’s OK!
The smoothness of the function is not so important in this situation.

289

The same grading curve is implemented in a photo mode in Gran Turismo
SPORT for photo editing tools.
World-wide players are grading their photos in a linear space using a
SDR/HDR compatible parametric curve!

290

We will show an example of curve based grading.
This is the original image.

291

And this is the graded image.

292

And this is a side-by-side comparison.

293

To conclude.
We achieved support of both lookup table based grading and simple
parametric curve based grading.
And both methods are SDR/HDR compatible.

294

Thanks to these methods, we were able to significantly enhance our HDR
rendering output.

295

296

For final section we’d like to cover how we built our HDR working
environment.
I can say verifying collect HDR output is really hard problem.
The reason is there are so many variations of TV monitors.
To debug where the problem is,
Keeping output image as neutral as very important.

297

Of course you can find specification numbers on internet,
But their number are sometimes not true.
So touching, watching and testing actual monitor is really important.
We have created our own testing suite.

298

Here is the list of the tests.
We will introduce what these tests do and what we check.

299

To test and verify, we collected a lot of monitors from cheaper to expensive.
Price range varied.

300

In small window test, we draw small white box in the tv monitor.
What we want to check here is the behavior of monitor tone mapping
functions and backlight static specifications.

301

Changing size you can determine how peak brightness and tone mapping
curve changes.

302

We have found some TV monitor has interesting behavior.
When the size of the box is really small, the TV suddenly shuts down.
Maybe TV decided my precious white box is just noise.

303

Next up, moving small window test.
In this test, the white small box moves around.

304

This test is to check the behavior of local dimming backlight.
Local dimming is a technology used to enhance TV contrast.
Local dimming uses an array of small backlights.
But how this array is designed and located is hidden secret.
So we have to test the actual performance.
In local dimming monitor, halo artifact is well known phenomenon.
Halo is a blurry artifact around the bright pixels caused by light leaking.

305

Next is moving small window test.
In this test, the white small box moves around.

306With this test you can check the halo effect and behavior of the backlights.

307This sample is to show how our actual test looks like on TV monitor.
This TV is a 100 inches SONY Z9D.
This TV monitor is actually the best in line up, no leaking can be noticed.

308

Compared to previous TV, this PC monitor has very interesting behavior.
From this test, we can say, the image quality may vary.
Imagine a situation that game player plays your game with this display, andhe/she writes a review on amazon.
What can we do is, at least, is knowing.

309

Next test is full locus test.
We draw full color locus image on a display.
This test is to check how color signals are interpreted.

310

This test is very simple but, a lot of thing can be determined.
These pictures are taken from actual TV monitors.
The left picture shows how colors are skewed and enhanced.
The right picture shows how colors are saturated and how narrow thismonitor gamut is.

311

Next test is HSV scroller.
In this test we output HSV bars.
This test is to check how colors changed related to the luminance.
This is also very simple test, but works really well.

312With this test we can find some TVs performs very well,
But some PC monitors have too much enhancement.
We can find which monitor is good for contents creation.

313

For conclusion, HDR world is getting complicated like SDR.
To build your believable environment, testing is important.

314

This is future work.
We’d like to build more robust tone mapping by much wider verify.
Also we’d like to build more precise tone maping calibration process.
We’d like to start full sptctra rendering.
Next, we’d like to build more effective implementations and asset
capturing.
There are still lots of room to optimized.

315

We established consistent theory-based approach for each aspect of the workflow.
As a result high quality output is achieved.
We talked about fundamental theory and practical implementation about HDR and
Wide Color. Based on our case study
We believe you have gotten knowledge about developing HDR and wide color
applications effectively and efficiently.

316

317

318

Thank you for our acknowledgements.

319

320

321

322

323

324

325