WebTransport + WebCodecs
at W3C Games Workshop 6/19

Problem 1: WebRTC not great for cloud gamingProblem 1: WebRTC not great for cloud gaming
Problem 2: MSE not great for cloud gamingProblem 1: WebRTC not great for cloud gaming
Problem 2: MSE not great for cloud gaming
Problem 3: WebSocket not great for gamingProblem 1: WebRTC not great for cloud gaming
Problem 2: MSE not great for cloud gaming
Problem 3: WebSocket not great for gaming
Solution: WebTransport + WebCodecs

HTTP + MSE

WebRTC

Cloud gaming

HTTP + MSE

WebTransport + WebCodecs!

WebRTC

TransportCodecs

WebTransport

WebCodecs

UDP isn't secure!
● encryption● congestion control● CORS/consent

If only there were a way to add that to UDP but keep
all the good parts of UDP...

HTTP/3

What is QUIC?
●
●
●
●
●
●
●
●
●

Fast connection setup (1-RTT, or sometimes 0-RTT)
SecureLow-latency congestion control (pluggable)
Many reliable streams (like TCP/TLS * N)
Unreliable/unordered datagrams (like UDP)
Can be p2p (with ICE)
Basis of HTTP/3
Widely deployed
Many implementations comingBenefits for games● Faster game loading(particularly with many components).
● More network resilience: making bad networks usable(particularly for mobile network with high RTT/loss)

But what about cloud gaming?

Streaming with WebRTC stack

ICE, DTLS, SRTP
ServerEncode/Forward, Packetize

Browser(WebRTC stack)Depacketize, Buffer, Decode, Render

Streaming with WebRTC stack

ICE, DTLS, SRTP
ServerBrowser(WebRTC stack)

Encode/Forward, Packetize

Depacketize, Buffer, Decode, Render"Hard to use in a client-server architecture"
Not a lot of control in buffering, decoding,rendering. All controlled by browser.
Limited by RTP (no generic data)

Streaming with WebRTC Data Channel + MSE

JSdata

ICE, DTLS, SCTP
ServerEncode/Forward, Serialize

ContainerizedMediaBrowser(RTCDataChannel + MSE)
Decontainerize, Buffer, Decode, Render

Streaming with WebRTC Data Channel + MSE

JS"Hard to use in a client-server architecture"
Low-latency mode is implicit magic
Have to containerize media just to get it indata

ICE, DTLS, SCTP
ServerEncode/Forward, Serialize

ContainerizedMediaBrowser(RTCDataChannel + MSE)
Decontainerize, Buffer, Decode, Render

Streaming with WebSocket + WebAssemblywasm Deserialize, Buffer, Decode
data

WebSocket, TCP
ServerEncode/Forward, Serialize

Raw Media

Browser(WebSocket)
Render

Streaming with WebSocket + WebAssembly
Head-of-line blocking

wasm Deserialize, Buffer, Decode

Performance/latency
QualitydataBattery life

WebSocket, TCP
ServerEncode/Forward, Serialize

Raw Media

Browser(WebSocket)
RenderWe can do betterWe can do bettertransport

WebTransport is a better RTCDataChannel and a better WebSocket
Reliable/ordered and unreliable/unordered
Easy to use in a client/server architecture
Client/server and p2pProvides datagram support
Same security properties as RTCDataChannel and WebSockets(encryption, congestion control, CORS)● Faster!
● Better API (support for back pressure)
●●
●
●
●

We can do betterWe can do better
codecsMediaRecorder

RtpSender

MSE

Encoders

Decoders

Like mobile APIs,
but hidden"built-in encoder makes sense so we
don't have to ship WebAssembly
modules to do the same thing"

RtpReceiverStreaming with WebTransport + WebCodec

JS

Easier to deploy server
Low latency
More web developer control...
dataQUICServer
Encode/Forward, Serialize

Deserialize, Buffer

Raw MediaBrowser
WebTransport + WebCodec

Decode, RenderOther use cases/benefits
●
●
●
●
●
●
●
●
●

Push game state (with low latency)
Stream game assets (with low latency)
Communication during a game (with same APIs)
Upload media to server for ML (with low latency)
Transcoding (faster than real-time)
Get new codec support faster (less for a browser to implement)
Get new/wider container support (from JS library)
Better support for spatial/temporal scalability
e2e encrypted group communication

WebTransport Status:● has origin trial (p2p version)● looking for customers
WebCodecs Status:● has proposal/explainer● looking for interestMore Info
• RTCQuicTransport Origin trial• Announcement: https://developers.google.com/web/updates/2019/01/rtcquictransport-api• Documentation: https://github.com/shampson/RTCQuicTransport-Origin-Trial-Documentation• Sample code: https://webrtchacks.com/first-steps-with-quic-datachannel/• WebTransport• Proposal/Explainer: https://discourse.wicg.io/t/webtransport-proposal/3508• Spec: https://wicg.github.io/web-transport/• WebCodecs• Proposal/Explainer: https://discourse.wicg.io/t/webcodecs-proposal/3662• Repository for future spec: https://github.com/pthatcherg/web-codecs

Topics for discussion

We are looking for customers●●●●

What use cases do you have for secure datagrams or low-level codecs?
What would you like to see in these APIs?
What do you care more about? client/server? p2p? cloud gaming? normal gaming?
communication in games?
Use the RTCQuicTransport origin trial! (If you run into issues with client/server, letme know... I've got a demo server almost working at
https://github.com/pthatcherg/quic-go/tree/gquic/example/webtransport)Other things I couldn't fit into the slides-HouseParty/EPIC acquisition
Data with audio/video at the same time
Spatial audio
Temporal/spatial scalability
Study of origin trial vs SCTP:
https://docs.google.com/document/d/1F0lfp62blYTiqDrbvcRewkKBr20a2RsnoFOwcexKUWk/edit#