Practical Development for Vulkan
Dan Ginsburg, ValveBaldur Karlsson, Unity
Dean Sekulic, Croteam

Session Overview

 Vulkan Status Update, Dan Ginsburg
 Vulkan – Care and Feeding, Dean Sekulic
 Debugging with Vulkan Renderdoc, Baldur Karlsson
 Q&ASession Overview

 Vulkan Status Update, Dan Ginsburg
 Vulkan – Care and Feeding, Dean Sekulic
 Debugging with Vulkan Renderdoc, Baldur Karlsson
 Q&A

Vulkan Status on Desktop

 Vulkan 1.0 has shipped Windows 7, 8, 10
 NVIDIA – GeForce 600-series+ (Kepler, Maxwell) AMD – HD 7700+ (GCN 1.0, 1.1, 1.2)
 Intel – Skylake (Beta) Linux
 NVIDIA – same GPUs as Windows AMD – unreleased Intel OTC – Broadwell, Skylake

Steam Survey Data

DX12 Support
DX12, 36.34%
Not
Supported,
63.66%

Steam Hardware Survey, Feb 2016

Vulkan Support

Not
Supported,48.96%

Vulkan,51.04%

Vulkan Adoption

 Vulkan Steam Overlay complete Linux Vulkan Loader included in Steam Linux runtime
 SteamOS 2.64 – Vulkan NVIDIA
 Working with Linux distros (Canonical, RedHat) to include
Vulkan Loader

 Windows IHV Driver installers including VulkanRT installerVulkan Source 2

 Dota 2 running on Vulkan
 Seeded to all GPU vendors Up on NVIDIA, AMD, and Intel Scaleform Autodesk working on Vulkan support
 Will release as soon as this is integrated

Vulkan Status

 Wide platform support
 Larger share of the desktop than DX12

 Drivers rolling out quickly
 NVIDIA already released WHQL non-beta Vulkan drivers

 Tools LunarG SDK Loader Validation Layers
 vktrace Samples RenderDoc glslang All Open Source on github

HLSL Translation

 HLSL translation continues to be impediment fordevelopers from DirectX
 Work has begun on HLSL -> SPIR-V
 Part of glslang project – hlsl-frontend branch Leverages existing SPIR-V code generation

HLSL -> SPIR-V

HLSL -> SPIR-V – Get Involved!

 Develop https://github.com/KhronosGroup/glslang/commits/hlsl-frontend

 Discuss https://github.com/KhronosGroup/glslang/issues/200

Session Overview

 Vulkan Status Update, Dan Ginsburg
 Vulkan – Care and Feeding, Dean Sekulic
 Debugging with Vulkan Renderdoc, Baldur Karlsson
 Q&A

VULKAN – care and feeding
Dean Sekulić
graphics programmer

And now...

Conclusion!

• Vulkan IS hard to code for• Not for everyone– small projects or not-CPU bound - go with OpenGL (ES)(but lots of objects on screen -> Vulkan!)– quick prototyping is faster with OpenGL• Vulkan is fast and portable! what more do you want?
– so answer is YES - go for it, we did!

Conclusion (2)• You can "just port" your engine right away (like we did)
– will get speedups and less stuttering– do proper way later or...
• ... right from the start!
– change paradigm! forget about state machine - it's so 90's!
– code gfx part of engine around Vulkan, do a wrapper for OpenGL(ES) and/or Direct3D 11• And now, onto some problems we stumbled upon along the way...

The Talos Principle GFX design• We ported The Talos Principle to Vulkan as proof of concept– took us 4 men/months• at least a month of that time was because of "hitting a moving target“(lots of API changes, because of work-in-progress state)• Our gfx wrapper– API agnostic (of course!)– all old-fashion functions are inlined!
– at Draw time, first call
[D3D9|D3D11|OGL|VLK...]::UpdateStates()

Handling Pipeline state objects•

Quick’n’simple way– load/cache/create them per request, do hashing, binning, sorting, fast search(we use linear search with more frequently used sorted to begining of array is fast enough)– not so optimal, could produce stuttering in frame rate
• yes, use PSO caches!
– CreatePipeline is more expensive than vkCreateShader!
(shader is actually optimized during pipeline creation)•Don't forget to destroy some when frame-buffer is destroyed(to keep count to minimum for faster searches!)Descriptor sets•

We're currently emulating old bind model– several "predefined" layouts•

Do everything like for PSOs (create, hash, bin, sort...)(much faster to create them than PSOs, no caching needed here)– we update update only once, at creation time
– later on, just bind•Don't forget to destroy some when a texture that DS references is destroyed(same as for PSOs, to keep count to minimum)

Memory management•

Rewrote memory management code four times! :((personal record for rewrites)– critical for performance•Don't use Vulkan vkAllocateMemory() for every object(Vulkan is not designed this way!)•

Have your own memory manager - it's a must!
– watch out for fragmentation!
(should have some form of anti-fragmentation system)•

But don't have just one big pool of device memory(otherwise, OS will have hard time swapping it, if/when needed!)– have several small(er) pools– we also have separate pools for linear and optimal allocations!
(if you forget about VkPhysicalDeviceLimits.bufferImageGranularity, hell awaits!)

Memory management (2)•Basically, there are 3 types of memory...

•Device, host (mappable) and shared––•

Keep host memory mapped all the time–•

have several pools for each of these types
preallocate and do additional allocation as needed

but be careful with CPU-GPU sync!

If system (also) has shared memory(device mappable; like VkPhysicalDeviceType.VK_PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU)
forget about all the copies and updates; just keep it mapped all the time and write/copy into that memory directly•

Watch out!
––

there could be some hidden internal memory allocations(host allocations exposed, but not driver’s device memory allocations!)
using host allocations call-back might hurt performance (so use them only for debugging!)Uploading resources•

Not simple as with older gfx APIs :(•Have a staging resource class that has own command buffers, fences and buffers
used as source for transfer from host to device (video) memory––––•have one command pool for all that (or one per thread)
allocate command buffer when needed for upload, begin, add copy/update commands,
submit and free CB when its fence is done
don't wait end of frame to submit and/or free those(it might fill up your whole host memory when reloading a lot of textures!)submit from other threads, have a spin-lock to avoid calling vkQueueSubmit() concurrently!

Also reduce number of submits that you have per resource–––

there's a certain amount of “call”-overhead attached to it (regardless of command buffersize!)
don't use one staging resource per each mipmap of each face of a texture!
use one for all mipmaps and all faces!

Destroying resources• We're lazy, no fences per resource :)• We just wait for some frames to pass
– put them at "delay" array or list
(some frames > swap-chains_count+1, to be on the safe side)• But be careful not to overflow memory– new resources are uploaded before old have been really
destroyed! (the same applies to reusing resources)

Barriers•

Critically important for cross-vendor correctness– and you are now required to insert them yourself!

•Read-after-Write access (shadow maps, for example)– definitely requires a barrier• otherwise you’ll get artifacts, but only sometimes/somewhere (Yikes!)•

if you have too many per frame (say, >20) - batch them!

•strategies– change early• better performance, might require high-level changes– change "on-time“• might cause GPU stallOcclusion culling•

we use OC for high level rendering work decisions•

our pipeline looks like this– visiblity system (trivial rejection and acceptance) -> distance culling -> frustum culling -> occlusion culling(!)-> animations -> bones’ transformations -> material modifiers -> render (could be several batches!)•old API ordering:– begin query -> draw -> end query -> swap-buffers -> get query result•

vulkan ordering:– reset query? (must be outside of render pass, so this might not be the place for it!) ->-> begin query -> draw -> end query-> swap-buffers -> get query result -> reset query now (one by one)•query reset might be executed by GPU too late, after get result in next frame == wrong result!
– might require extra frame delay!
(cannot reset query outside of render pass, nor directly via CPU!) :(•track which queries were tested (begun/end) in a given pool, to avoid infinite waits inside the driver!

Thank you•special thanks goes to all the great folks at nVidia, AMD, Valve, LunarG, Intel andBaldur (RenderDoc!)•and Alen (our CTO) who started all this by being really (pro)active on Vulkan Advisory
Panel, while I was busy on other fronts•

also all friends and colleagues at Croteam who helped me with this port and gave me
courage with their kind words ("You're never gonna finish this", "Vulkan 'tillretirement", "Vulkan programmer work is never done", "Drop it while you're young...
oh sorry, you're not young anymore"...;)•and of course, LunarG's Vulkan Validation Layer!

Wasn't enough time for... •Round-robin buffers•

what if you ran out of memory?

•other queue for copying resources to device memory (haven't experimented with that yet)•you (usually?) can't have staging linear image for mipmapped (cube-)textures used as source because these are not supported- just use vkCmdCopyBufferToImage() and that's it!

•Validation layers•

Tools!
- RenderDoc is great- but we need something that can look inside vkQueueSubmit() - IHVs?!

•

Further in time- "unwrapping" everthing- needs lots of recode in high-level rendering path- real MT renderer, not wrapper: command buffers on other threads!
- precreate PSOs (for each loaded material, i.e. high-level shader)- precreate command buffers! (geometry plus material, for whole models)RenderDoc for Vulkan
Baldur Karlsson (@baldurk)

Brief History
Started as Hobby/spare-time project, mid 2012
Born out of need, other debuggers weren’t working
First public release, early 2014 with source
Initially D3D11 support, then OpenGL, now...

RenderDoc Vulkan support
Began work late 2015
Developed with help from Unity and LunarG
Launched simultaneously with Vulkan 1.0 Spec
Available on Windows bundled in SDKLinux is coming Real Soon Now™

Initial Capture screen

In-game overlay

Event Browser & API Inspector

Markers from
VK_EXT_debug_marker

Texture Viewer

Pipeline State

Pipeline StateShader View

Mesh View

Mesh View

But wait, there’s more!
Timeline bar showing intraframe resource reads and writes
Drawcall microsecond timings
Texture/buffer export to file (.dds/.exr)
Python Shell with access to all underlying data

Conclusion
RenderDoc is available and working now - today
May even already be installed if you have Windows Vulkan SDKOpen source on github - MIT license
Active development & improvement all the time
Talk to me! - I will help fix issues, improve workflows, add features. Always
happy to talkThank you!
baldurk@baldurk.org@baldurk
https://github.com/baldurk/renderdoc

Appendix: Coherent persistent maps!
Keeping pointers to mapped memory in Vulkan can be very usefulBe wary of doing this for coherent memory types when debugging
Requires a very slow memcmp() on each frame’s submission to detect changes
Instead either prefer non-coherent types, or call vkFlushMapped... anyway todenote modified regions to the debugger
Only do this while debugging though!