Bringing the BBC’s World Cup & Wimbledon online coverage to you
By: Nik Rahmel
Tags: BBC, Broadcasting, World Cup, AWS, TV

This post is coauthored by engineers across the different teams involved: Marc Burrows, Alex Chesters, Neil Craig, Sally Fletcher, Marc Littlemore, Nik Rahmel, and Lloyd Wallis.

There is a lot of collaboration across the BBC’s Design & Engineering teams to bring the World Cup and Wimbledon to the BBC. From the actual application, over scheduling behind the scenes, how the actual video transmission makes it way into iPlayer and BBC Sport and how this lets users actually watch a live broadcast.

This year we are also broadcasting for the first time in UHD (Ultra-High Definition) resolution for the World Cup and selected Wimbledon matches.

Let’s start with how most people watch a match — they tune in into the match. BBC One is a pretty good bet to try if you’re in the UK, and if you’re lucky might see one of our ‘UHD Call To Actions’ on the screen and one of the commentators tell you this:

You can watch this match in Ultra HD via the BBC iPlayer

Someone decided that some of the World Cup matches are broadcast on the BBC in HD and UHD — more on the production side, and how our UHD trials worked this summer in this blog post from BBC R&D (Research and Development).

Once we know the schedule, we co-ordinate the schedule for our Audio/Video encoders to deliver the correct streams for the correct events at the correct times. For the World Cup, the UHD streams are actually completely separate episodes in the BBC programme structure, but embargoed in the system so they do not show up anywhere, including programme guides.

For Wimbledon, we also can’t use the same episodes as on broadcast TV since the broadcasts sometimes switch between channels, and are split up, so the UHD broadcast is attached to the equivalent Webcasts.

For the UHD streams, the encoder, packagers and backups, as well as the CDN (Content Delivery Network) configuration are controlled manually. This means, that prior to the match BBC Sport staff in Salford co-ordinate with their Media Services and OTG (Online Technology Group) colleagues in London, who are responsible for the stream delivery, and the broadcast staff in Moscow responsible for the contribution feed coming into London to turn everything on in the right order and start the stream.

This needs to happen before the broadcast begins because as soon as the broadcast episode becomes ‘available’ on iPlayer, the TV app starts to offer the UHD version — and if the encoder isn’t turned on, that does not look pretty.

FIFA is very nice and puts their Central Equipment Room behind glass for some great photos (Picture: Quinn Cowper)

Once everything is running, our CDNs are warmed up and we see no issues, we are getting closer to the match. The ‘Call to Action’ (CTA) for users to switch to iPlayer for UHD actually comes from the Outside Broadcast truck in Russia.

When everything is prepared, we are ready to serve the live streams to the audience, and that is co-ordinated across the different iPlayer teams.

For the app on smart TVs and set-top boxes, the basic architecture is a pretty typical client-server one. We have a common client application which is used for all of the BBC’s Connected TV products and numerous backend services. These services often take the form of a Node.js application deployed to Amazon EC2.

Request rate for the iPlayer Homepage from TV devices for BRA-BEL on Jul 6 —The On-Screen CTA is clearly visible. This is where scheduled scaling comes in very handy.

Our usual scaling process is to scale in response to AWS CloudWatch alarms which are hooked up to appropriate metrics (e.g. memory utilisation, CPU utilisation, network throughput) for the service in question. This approach works well with our usual traffic patterns as they are reasonably gradual increases. When an on-air CTA is made, however, our traffic increases at a rate far too great for AWS AutoScaling to have an effect. We have seen the network traffic through our NAT Gateways increase by 150% in the space of 60 seconds and traffic we send to the API that powers the iPlayer homepage tripled in less than 20 seconds.

And even if matches are not on the BBC, we have to keep an eye out on our performances.

As soon as we find out which games are due to be broadcast on BBC One (and therefore in UHD) we start analysing the times to determine when the peak traffic points are expected to be. We liaise with our colleagues from across the whole of the BBC to determine if there are any on-air CTAs planned and if so, when they are likely to occur.

Once we’ve got all the necessary information we begin planning our scaling rules. Unfortunately this process is (for now) rather manual as we make use of AWS’ Scheduled Scaling. Internally we’ve started looking at various other solutions that would allow automated scaling to be effective in these situations — hopefully we can write another blog post about this in the future.

To enforce a limit on the number of UHD streams, we couldn’t rely on previously used measurement systems, as the latency is too high. At kick off and on-screen CTAs, we have seen the number of viewers increase by a third within 60 seconds. We need to ensure that we can “close the gate” reliably when we reach the limit, as there is a real possibility to reach the limits of the UK internet infrastructure due to the bandwidth requirements of UHD — roughly 8x that of HD.

There is a Counting Service built by our Live Sport team, which uses AWS Kinesis Firehose to collect the viewer-count statistics, which was linked to the “gate”, and heavily used in the Live Pages — Mark Woosey has blogged about the details separately here.

Counting Service figures for ENG-SWE on July 7 — Kick Off and corresponding CTA clearly visible

BBC Account