The Algorithm Worth Billions: How YouTube’s Addictive Video Recommender Works
By: Andre Ye
Tags: Data, Statistics, AI, Data Science, Machine Learning

Recommending YouTube videos is an extremely challenging problem, according to Google researchers, for three reasons:

Scale. Most current recommendation videos are proven to work well on very small problems, but fail to operate on YouTube’s massive scale. Highly efficient and specialized learning algorithms are imperative for handling its massive user base and corpus.

Freshness. YouTube has several hours of video uploaded to its platform every second, meaning the recommendation algorithm will need to be responsive enough to handle the newly uploaded content and the latest user actions, balancing new content and well-established videos in an exploration-exploitation balance.

Noise. Historical user behavior on YouTube is inherently difficult to predict because of its sparsity (lack of known values), complex user behavior, and a variety of unobservable external factors. A recommender needs to be able to handle these particular data characteristics.

YouTube, along with other sectors of Google, underwent a fundamental shift in technology and an embrace of deep learning to build such an addictive and well-performing algorithm. The model used for recommending consists of approximately one billion adjustable parameters and are trained on hundreds of billions of examples.

YouTube’s algorithm comprises of two neural networks — one for candidate generation and the other for ranking. The first narrow down YouTube’s massive video corpus into a few hundred videos, and the second ranks them by their value to the user. The top few ranked videos are selected to be displayed. Think of the process like prelims and finals in sports — the best way to evaluate the best athletes is not in one free-for-all heat, but in several organized rounds of prelims and finals to narrow down the field in a structured manner.

The former network takes in events from a user’s activity history and retrieves a subset (hundreds) of videos from YouTube’s video corpus. These candidates for recommendation are intended to be relevant to the user with high precision. The candidate generation network uses collaborative filtering, which makes a recommendation not based on inherent video attributes (run time, genre, etc. — content filtering) but on what people with a similar rating history enjoyed.

The second neural network ranks the collection of filtered videos. This neural network must be able to represent each video’s value at a granular level to distinguish from the many worthy participants. The ranking model assigns each video a score according to a (secret) objective function that considers a rich set of features describing the video and the user.

The two-stage approach, as opposed to traditional one-stage methods, allows for greater accuracy and personalization whilst operating on the Internet’s richest pool of data. It is more suited to address YouTube’s unique problem than one-stage methods, which need to consider a wide variety of rich features for every single video, leading to long response times and a lower accuracy.

The goal of the models is to optimize expected watch time, instead of another of YouTube’s assortment of possible metrics, such as the chance of liking the video and number of interactions. This is why the model favors longer videos, but not at the expense of the user’s interest. Videos the model recommends to you are videos it believes you will watch for the longest out of all other YouTube videos, hence increasing the number of advertisements you watch. A ‘sub-model’ within the algorithm is used to determine a video’s expected watch time.

The algorithm also has a tendency to spread shocking and inflammatory content because it is told to optimize watch time, not user approval. Content that it is disagreeable and controversial maintains the viewer’s attention and gets longer watch times, hence gaining priority in the algorithm. It’s not YouTube’s fault: firstly, they have a business incentive to do so (greater advertisement revenue), but secondly, from the intensive research they have conducted, they find watch time to be the best metric to give the viewers content they want to see, opposed to likes or another measurement.

In this sense, the inflammatory content of YouTube’s algorithm seemingly propagates is a negative externality of what has been proven to work. Still, to address this mathematical problem, YouTube implements human checkers to add a human input into a purely mechanical decision. The platform also has added ‘recommended channels’ in addition to ‘recommended videos’ in an attempt to diversify user attention away from a single potentially inflammatory video to channels.

Source: SEOClerk

To test the final effectiveness of an algorithm or model, YouTube uses live A/B tests. These live tests seek to confirm an algorithm’s positive performance by measuring clickthrough rates and other metrics when users are presented with videos recommended by that model. YouTube performs countless A/B tests every day to perfect their algorithm — the next time you are scrolling through the ‘Recommended Videos’ options, you very likely are a participant in an experiment. This real-world feedback may not always correlate with the expected result from offline metrics, so A/B tests are essential for the final deployment of a new release of the recommendation algorithm.

YouTube’s ingenious algorithm represents a fundamental shift of the embrace of classical machine learning methods like tree and linear-based methods to deep learning, which can hold the complexity yet adaptability and robustness required for modern-day problems. Yet deep learning models need to be carefully used — not as a bashable cookie-cutter solution to everything but a delicate tool — which is purposed cleverly by YouTube to their unique problems.

The next time a video in the ‘Recommended Videos’ catches your eye, you will know how it made its journey from YouTube’s video corpus to your personal feed.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬.

To join our community Slack team chat 🗣️ read our weekly Faun topics 🗞️, and connect with the community 📣 click here⬇