Wikia/app — pull request #1844 by relwell (2013-10-23 22:59:38+00:00)
Title: Use numeric unsupervised topics in ad serving

This is an update to last week's pull request for using entities. Since we want to be economical about the size of the data we're sending to DFP, we're scrapping the string value for entities for a compressed, vectorized approach.

For the top 5,000 WAM wikis, we're extracting the top 50 wiki entities and top 50 sentence-scoped syntactic heads (usually verbs) from each wiki. Those values are transformed into a vector for each wiki. The vector dimensionality is then reduced using latent Dirichlet allocation. This is an unsupervised learning algorithm that has been shown to work excellently against natural language data sets. We explicitly asked for 999 topics, because a high number of topics has been shown to [work well for ad targeting](http://www.academia.edu/1616507/Topic_Models_for_Retrieving_Relevant_Ads).

These topics, which range from 0 to 1 as a measure of "intensity", as it were, are injected into a wikifactory variable as a serialized array. That array is basically pushed into the "wtpx" key.

The average number of topics each wiki has is 18 (out of 999), but since we're trying to be space-constrained, we're only transmitting the top 5.