Handling advertiser bids using Scala & akka-streams
By: Marios Assiotis
Tags: Programming, Scala, Adtech, Akka

In a previous post, we already looked at how ad requests are received, enriched and subsequently go through a series of filters to determine which creatives are eligible. In this post, we resume with steps 4–8, looking at how different bids are received and paced, as well as how the final determination is made on which bids may win.

Step Four: Pacing

Pacing is one of the most important functions of any ad server. Advertisers want their campaigns to be spread evenly throughout the day, hoping to reach as big and diverse an audience as possible.

We maintain campaign spending and budget information in Akka actors. Since spending is a monotonically increasing operation, and we are OK with eventual consistency, CRDT operators provided by Akka Distributed Data, make for an excellent choice of tracking spent, even across data centers. After we verify that a campaign has not exceeded its budget, we load forecasts, previously calculated by a Spark Job, along with budget and spend information. The pacing graph then utilizes simple techniques derived from control theory to decide if we should pace the campaign out or allow it to participate in the auction.

Forecasts are used to determine how traffic will be shaped throughout the day. They are calculated across all major targeting criteria that we know, based on domain knowledge, to affect traffic patterns and available inventory. For example, traffic on our apps running on gaming consoles follows a different pattern than our mobile apps.

Step Five: Collecting Bids

Tubi ad inventory is sold both directly by our very own sales team (they’re hiring!), as well as programmatically through direct integrations with top-tier DSPs. These integrations typically involve calling into a third party, with all the gotchas that come with making network calls to third parties and accepting arbitrary responses.

Being able to capture each one of these into its own akka-streams flow allows us once again to write a common graph that takes care a lot of the boilerplate as well as the mechanics of async network calls, concurrency control, failure handling, throttling, circuit breaking, etc. We can then focus on the business logic and make sure the quirks for each third party integration are well coded and everyone’s individual interpretation of IAB specs is well tested. Similarly, since this is part of a per-request flow with strict upper bounds on concurrency, a misbehaving third party will not affect the rest of the normal operation of our ad serving platform.

Step Six: Ad Video Creative Transcoding

Once we have collected all eligible bids, we check if we have our own transcodes for each included video creative, and if not, take it through the transcoding process. We do this for a number of reasons:

As a video streaming company, we have deep expertise in video and we want to make sure ad playback works well across all devices where Tubi runs. Advertisers sometimes have inconsistent (read: loud) volume on their ad creatives, which we try to normalize so as to not hurt the user’s experience and ear drums. We want to make sure ad creatives are appropriately rated for the content they play against. For example, an advertisement for tobacco should not be shown against content rated PG-13 For advertisements coming through programmatic means, like DSPs, we want to identify the brand they are advertising, both for reporting as well as frequency capping purposes.

The steps above are relegated to a set separate services and machine learning models, worthy of their own blog post. Our ad server, like all services at Tubi, communicates with it using gRPC with a sub-10ms p99 SLA.

Step Seven: Auction

We are now at one of the most, if not the most, important steps in an ad request’s lifecycle. We have a number of bids at our disposal, from a number of different advertising partners and internally sourced campaigns, and we need to quickly decide which of those bids will fill our ad pod, which ranges from one to five slots. While most other parts of an ad request flow deal extensively with network I/O, caching, concurrency, etc, the auction is 100% CPU, solving a complex constraint optimization problem. A simplified version of the optimization problem is shown below. Let there be a matrix n x k, consisting of n advertiser bids vying for k advertising slots in the ad pod.

Each cell in the matrix can take values 1 or 0, indicating whether the bid in position was selected or not respectively. It is easy then to show how various constraints can be set as simple sums. For example, to make sure a particular advertiser bid is only selected once, all we need to do is add a constraint that the sum of each matrix row cannot be larger than 1. Similarly, to make sure that only a single bid is selected for each position in our ad pod, we add a constraint such that the sum of each matrix column must be less than or equal to 1. More constraints can be added, for example to enforce competitive separation, certain UX guarantees around competitive ads, and more. The solver then works tirelessly to optimize our objective function subject to the constraints we added. The objective function itself is a linear combination of revenue optimization as well as encompassing certain guarantees around delivery.

Step Eight: Render

In the last and final step, we collect all winning bids and render them as appropriate for each platform. Some of our client applications prefer VAST XML, some prefer custom JSON. Using the built-in akka-http rendering mechanisms, we can output as many predefined templates as we want.

Step Nine: Join Tubi

Tubi’s in-house AdTech stack is one of many areas where we leverage our own technology to deliver a superior product compared to using an off-the-shelf solution. If you are passionate about distributed systems and reactive programming in Scala, we are actively hiring. In addition to AdTech, we use the same technologies to build machine learning platforms, big data systems and much more.