Fluid concepts and creative probabilities
By: Jacopo Tagliabue
Tags: Machine Learning, Probabilistic Programming, NLP, AI, Semantics

Going forward, looking backward: probabilistic modeling 101

“Million to one odds happen eight times a day in New York.” — Penn Jillette

In our life as A.I. practitioners and startup founders, we found multiple times that a lot of ideas in Probabilistic Programming are still far from being mainstream in the data science community. Convinced by this anecdotal evidence, we decided to include an overview of PP and what makes it special for interesting inference problems, such as the ones you meet while teaching language to machines (the reader familiar with PP/WebPPL can quickly skip through the next two sections).

We will make heavy use of WebPPL examples and take full advantage of the possibility of running code in the browser: all examples are fully runnable from this simple HTML page.

Let’s start with a simple example (with dice, of course). If we have two fair dice, what is the probability of getting 8 when you throw them? This WebPPL program (remember: you can run it here) displays the distribution of outcomes in a nice histogram (if something is not clear, don’t panic: it will be obvious soon):

Frequency distribution for our dice model.

The cool thing about PP is how easy it is to do the inverse inference: given that we observe that the sum is 8, what is the chance of one die being a 2?

Frequency distribution for our dice model, conditioned upon observation.

As the reader may have suspected, there is a 20% chance of die #1 being a 2 if the total is 8 (quiz for the lazy reader: why there is no 1 in the histogram above?).

While this may look like an overtly simple example, it is really important to appreciate the “power of models”, i.e. simplified versions of the processes generating our data of interest. Models are a very compact representation of the complexity of the world: within a single model, you can in fact reason from cause to effect (forward, when you plan) and from effect to causes (backward, when you make educated guesses). Models are also at the core of our cognitive abilities: by having a “Tangram model”, kids know how to build a duck from scattered pieces (forward) and from the shape of a duck they can guess which pieces are used to build it (backward).

A Tangram duck — humans can easily reason from pieces to figures and from figures to pieces.

For many people, model-based reasoning is the key to intelligence, with applicability well beyond Tangram (such as intuitive physics and visual learning):

A generative model describes a process, usually one by which observable data is generated, that represents knowledge about the causal structure of the world. These generative processes are simplified “working models” of a domain (…); many different questions can be answered by interrogating the mental model.

For today’s more modest purpose than understanding human intelligence, it is just important to appreciate how PP can help. As we have seen, these models are often called “generative models”: without bothering too much with technical details, what is important for us are the following two considerations:

generative models tell a story on how data points are generated: they have a structure that can be inspected and they often embody causal assumptions that humans can understand, question, debate;

generative models can be used to, well, generate new instances of our target data, which is a feature not shared by every Machine Learning model out there. For example, if we train generative models to classify news articles in sport vs finance, we could use them to create new articles on these topics.

While people have been writing generative models by hand for a long time, the golden age of probabilistic modeling just started. The rise of PP solved two major problems for practitioners: first, PP brings to the table the expressivity of programming languages: by using familiar tools data scientists can now express arbitrary complex models instead of a few distributions; second, PP pushes the burden of inference down to the compiler (and related tools): researchers focus on modeling since the computer will solve the required inferential challenge.

Express generative models in Javascript: WebPPL 101

“The limits of my programming language mean the limits of my world.” — (almost) Ludwig Wittgenstein

Now that we have some shared context on why generative models are so freakin’ cool, we can go back to the question of how they are implemented in languages such as WebPPL. If you recall our first example, the model is specified as a simple function, wrapped in the Infer method:

model() {

var die1 = randomInteger(6) + 1;

var die2 = randomInteger(6) + 1; return die1 + die2;

}

What happens when it runs? Since enumerate is the inference method specified, the program will run the function once for all the possible values of the random variables in the model (i.e. random integers from 1 to 6 for each die) and collect the results in a distribution (that gets visualized through the handy viz utility). Since the model is generative, it should be possible to sample from it and create new values (i.e. simulate dice rolling): the sample function does exactly that, as you can see by adding the following statement to the cell block (it will print an array of 5 integers sampled from the model):

print(repeat(5, function() { sample(roll); }));

The second example introduces a key idea of PP: conditioning. The model is the same as before but with a new command:

condition(die1 + die2 == 8)

What happens when it runs? condition tells the program to ignore runs that do not meet the required condition: this is why no 1 shows up in the distribution — if one die is 1, there is no scenario where the sum is 8 (as 1+6=7).

Conditioning is important for at least three reasons: it lets you reason from observations to causes; it helps stating, in a principled way, information that may constrain the space of program runs to be considered; finally, it can be used to model a very important concept in contemporary A.I., learning, through an 18th century idea known as Bayes’ theorem: your confidence in an hypothesis H given some data D should be related to a) how much you believe in H in the first place and b) how much H can explain D.

We conclude our outrageously short introduction to WebPPL with a slightly juicier example on this topic. Consider a young field linguist — let’s call him Willard — in an exotic country with a very minimal world (a.k.a. ontology):

A toy universe with a tree-based ontology and 7 objects.

This exotic world has seven objects (named with integers 1–7) organized in a tree-like structure, with concepts on three levels of generality (e.g. everything is a mammal, dachshund is a type of dog, #3 is a dachshund). When #2 is present, Willard hears native speakers say bassotto (1000 bonus points to readers expecting the natives to say gavagai). What does bassotto mean: dachshund, dog or mammal? More generally:

given an unknown word W and a set of objects denoted by W, how can Willard learn the appropriate level of generality for W?

Unsurprisingly, probabilistic programming is perfectly suited to model this problem, as exemplified by the following sketch:

learningModel is where the magic happens. We can distinguish three main parts:

first, sampling from the hypotheses space: Willard has some prior ideas on what can be named by W (summarized in the distribution ontologicalTree ); in particular, Willard believes that “natural kinds” like dog and cat are more salient than the very generic mammal;

); in particular, Willard believes that “natural kinds” like dog and cat are more salient than the very generic mammal; second, the likelihood function: Willard knows how to prioritize hypotheses given observations (a set of objects); what matters in the formula is realizing that 1) lower-level concepts (retriever) will assign greater likelihood to the data than higher-level concepts (mammal); 2) as the number of consistent events increases ( observedData.length ), lower-level concepts get exponentially more likely;

), lower-level concepts get exponentially more likely; third, actually observing data, as mapData takes care of feeding what Willard observes to the scoring function.

[SMALL TECHNICAL NOTE: the original idea for this example comes from the work of Xu and Tenenbaum — see their paper for a bit more context on the psychological facts behind the model and an in-depth discussion on priors and likelihood.]

So, back to our example with #2 observed for the word bassotto, what is Willard thinking? Running the code gives us the following distribution:

Distribution over target concepts after one observation.

It seems a reasonable guess, doesn’t it? Willard prefers dachshund but he is still keeping his mind open; of course, hypotheses inconsistent with the data (such as retriever, whose extension does not include #2) are not considered at all. The scenario gets really interesting when Willard observes #2 + #3 and hear bassotto:

Distribution over target concepts after two observations.

What happened? If Willard was just excluding hypotheses inconsistent with the data, seeing two dachshunds instead of one would not have added meaningful information: simply put, Willard would have not learned anything new. Our probabilistic model allows instead to capture Willard’s learning: after seeing two dachshunds, he is much more confident that bassotto means dachshund, while he is almost ready to rule out mammal as a plausible interpretation (even if entirely consistent with the data, from a purely logical perspective). We leave it as an exercise to the reader to make sure all the code is clear: try changing priors, likelihood and observations to get a better grasp of what is going on behind the scenes!

If you want to toy around some more before continuing, some effective examples of the interplay between forward and backward inference can be found on probmods. While the field is still relatively niche in the data science community, things are moving fast: we discovered this excellent WIP last week and major players are getting into PP. For the convenience of non-lazy readers, we have included a bunch of additional books/links at the end.

Let’s get back to our sequence game now!