# Active Data Selection and Information Seeking
## 3 A Worked Example  
In Figure 2, we illustrate a simple worked example based on Equation (3) using a Bayesian network of normally distributed variables. The network includes three nodes. These are: the choice (\(\pi\)) we make about how to sample our data, the parameter (\(\theta\)) we seek information about, and our data (\(y\)). The Bayesian network is first transformed into a factor graph--with explicit representation of factors of the joint probability distribution as boxes--and then to a (normal) Forney factor graph [25] that omits the variable nodes (circles). The Forney factor graph formulation is perhaps the easiest to use when visualising the passing of messages along the edges of the graph. The messages can then be used to construct the two entropies we need to compute expected information gain.  
The model in this toy example has the following factors:  
\[\begin{split} p(y|\theta,\pi)&=\mathcal{N}(\theta \cdot\cos(\pi),1)\\ p(\theta)&=\mathcal{N}(0,1)\end{split} \tag{4}\]In effect, this model amplifies or attenuates the amplitude of the predicted data depending upon a periodic function of our data-sampling policy, \(\pi\). The upper-left panel of Figure 2 depicts the data sampled from this model for several possible values of \(\theta\). Computing the relevant messages for the information gain, we have (omitting normalising constants for simplicity and using \(\varnothing\) to denote the empty set)  
\[\begin{split}&\mu_{\theta}^{\varnothing}=e^{-\frac{1}{2}\theta^{ 2}}\\ &\mu_{y}^{\theta}=\Lambda_{\theta}\Big{[}\mu_{\theta}^{\varnothing }e^{-\frac{1}{2}(y-\theta\text{-}\cos(\pi))^{2}}\Big{]}=e^{-\frac{1}{2}(\frac{ y^{2}}{(\text{cos}^{2}(x)+1)})}\\ &\mu_{\theta}^{y}=e^{-\frac{1}{2}(y-\theta\text{-}\cos(\pi))^{2} }\end{split} \tag{5}\]  
The directions of these messages are illustrated in the bottom-right panel of Figure 2. We next substitute these into the first line of Equation (3) to find an expression for the information gain about the parameter \(\theta\). Once all terms that are constant with respect to \(\pi\) are eliminated, we are left with:  
\[I[\pi]^{\theta}=\frac{1}{2}\ln(\cos^{2}(\pi)+1) \tag{6}\]  
Equation (6) is a special case of the third row of Table 1, which highlights analytical expressions for expected information gain for a few common model structures. This function is plotted in the lower-left panel of Figure 2. As we might intuit, the most informative places to sample data align with those in which differences in \(\theta\) lead to large differences in the predicted data--i.e., in which our choice of \(\pi\) maximises the sensitivity with which \(y\) depends upon \(\theta\). Given that a periodic (cosine) function is used for the effect of how we chose to sample (\(\pi\)) the data (\(y\)), there are multiple peaks which coincide with the peaks and troughs of the periodic function.  
Figure 2: A worked example. This figure offers a worked example of how we compute expected information gain. It draws from the message passing perspective set out in Equations (3) and (4) which detail the analytic computations that lead us to expected information gain as a function of our choices, \(\pi\), for this generative model. The colours in the upper left plot represent different values of \(\theta\). The red arrows in the lower right factor graph indicate the directions of messages being passed.