A step-by-step guide in designing knowledge-driven models using Bayesian theorem. | by Erdogan Taskesen | Towards Data Science

Data is the fuel for models but you may have witnessed situations where there is no data but solely a domain expert that can very well describe or even predict “the situation” given the circumstances. I will summarize the concepts of knowledge-driven models in terms of Bayesian probabilistic, followed by a hands-on tutorial to demonstrate the steps of converting an expert’s knowledge into a Bayesian model with the goal to make inferences. I will use the Sprinkler system to conceptually explain the steps in the process: from knowledge to model. I will end with a discussion about the challenges of complex knowledge-driven models, and the systematic errors that can occur due to questioning and extracting knowledge. All examples are created using the python library bnlearn.
When we talk about knowledge, it is not solely descriptive knowledge such as facts. Knowledge is also a familiarity, awareness, or understanding of someone or something, procedural knowledge (skills), or acquaintance knowledge [1].
Whatever knowledge you have, or want to use, it needs to be presented in a computer interpretable manner if you want to build a computer aided knowledge model.
This means that you need to design a system that is built on a sequence of process stages. Or in other words, a sequence goes from the output of the process into the input in the next process. Multiple simple sequences can then be combined into a complex system. We can represent such a system as a graph with nodes and edges. Each node corresponds to a variable and each edge represents conditional dependencies between pairs of variables. In this manner, we can define a model based on the expert’s knowledge, and the best way to do that is with Bayesian models. To answer the question, ‘Can we get experts knowledge into models?’ Well, it depends on how accurate you can represent the knowledge as a graph and how precise you can glue it together by probability theorem a.k.a. Bayesian graphical models. Still, there are some restrictions though.
The use of machine learning techniques has become a standard toolkit to obtain useful insights and make predictions in many domains. However, many of the models are data-driven, which means that data is required to learn a model. Incorporating expert’s knowledge in data-driven models is not possible or straightforward to do. However, a branch of machine learning is Bayesian graphical models (a.k.a. Bayesian networks, Bayesian belief networks, Bayes Net, causal probabilistic networks, and Influence diagrams), which can be used to incorporate experts knowledge into models and make inferences. See below some bullet points with the advantages of Bayesian graphical models, which I will stress throughout this article.
The possibility to incorporate domain/expert knowledge in a graph.
It has a notion of modularity.
A complex system is built by combining simpler parts.
Graph theory provides intuitively highly interacting sets of variables.
Probability theory provides the glue to combine the parts.
To make Bayesian graphical models, you need two ingredients: 1. Directed Acyclic Graphs (DAG) and 2. Conditional Probabilistic Tables (CPTs). Only together it can form a representation of the expert’s knowledge.
At this point, you know that knowledge can be represented as a systematic process, and can be seen as a graph. In the case of Bayesian models, a graph needs to be represented as a DAG. But what exactly is a DAG? First of all, it stands for Directed Acyclic Graphs and is a network (or graph) with nodes (variables) and edges that are directed. Figure 1 depicts three unique patterns that can be made with three variables (X, Y, Z). Nodes correspond to variables X, Y, Z, and the directed edges (arrows) indicate dependency relationships or conditional distributions. The network is acyclic, which means that no (feedback) loops are allowed.
With a DAG, a complex system can be created by combining the (simpler) parts.
All DAGs (large or small) are built under the following 3 rules:
Edges are conditional dependencies.Edges are directed.Feedback loops are not allowed.
Edges are conditional dependencies.
Edges are directed.
Feedback loops are not allowed.
These rules are important because if you would remove the directionality (or arrows), the three DAGs become identical. Or in other words, with the directionality, we can make the DAG identifiable [2]. There are many blogs, articles, and Wikipedia pages that describe the statistics and causality behind the DAGs. What you need to understand is that every Bayesian network can be designed by these three unique patterns, and should be representative of the process you want to model. Designing the DAG is the first part of creating a knowledge-driven model. The second part is defining the Conditional Probabilistic Tables which describe the strength of the relationship at each node in terms of (conditional) probabilities.
Probability theory (a.k.a. Bayes theorem or Bayes Rule), forms the fundament for Bayesian networks. Check out this medium article about Bayesian structure learning [3] to read more about the specific parts of Bayes theorem. Although the theorem is applicable here too, there are some differences. First, in a knowledge-driven model, the CPTs are not learned from data (because there is none). Instead, the probabilities need to be derived from the expert by questioning, and subsequently being stored in so-called Conditional Probabilistic Tables (CPT) (also named Conditional Probability Distribution, CPD). I will use CPT and CPD interchangeably throughout this article.
The CPTs describes the strength of the relationship at each node in terms of conditional probabilities or priors.
The CPTs are then used with the Bayes rule to update model information which allows making inferences. In the next section, I will demonstrate, with the Sprinkler use case, how to exactly fill in the CPT with expert knowledge. But first, there are challenges in converting expert's knowledge into probabilities.
When we want to make a knowledge-driven model, it is crucial to extract the right information from the expert. The domain expert will inform about the probability of a successful process and the risks of side effects. It is utterly important that the message is understood as intended to minimize the risk of miscommunication. When talking to experts, many estimated probabilities are communicated verbally, with terms such as ‘very likely’ instead of exact percentages.
Make sure that the verbal probability phrase is the same for both sender and receiver in terms of probabilities or percentages.
In certain domains there are guidelines that describe terms such as ‘common’ in certain ranges, let say 1–10%. But without background knowledge of the domain, the word ‘common’ can easily be interpreted as a different number [4]. In addition, the interpretation of a probability phrase can be influenced by its context [4]. For instance, compare your numerical interpretation in the next two statements:
It will likely rain in Manchester (England) next June.
It will likely rain in Barcelona (Spain) next June.
Probably, your numerical interpretation of ‘likely’ in the first statement is higher than in the second. Be cautious of contextual misinterpretations as it can also lead to systematic errors and thus incorrect models. An overview figure of probability phrases is shown in Figure 2.
‘Impossible’ seems not always be impossible!
A few words about the bnlearn library that is used for all the analysis in this article. The bnlearn library is designed to tackle a few challenges such as:
Structure learning: Given the data: Estimate a DAG that captures the dependencies between the variables.
Parameter learning: Given the data and DAG: Estimate the (conditional) probability distributions of the individual variables.
Inference: Given the learned model: Determine the exact probability values for your queries.
What benefits does bnlearn offer over other bayesian analysis implementations?
Build on top of the pgmpy library
Contains the most-wanted bayesian pipelines
Simple and intuitive
open source
Documentation page
Let's start with a simple and intuitive example to demonstrate the building of a real-world model based on an expert's knowledge. In this use case, I will play the role of domain expert of the Sprinkler system.
Suppose I have a sprinkler system in my backyard and for the last 1000 days I have eye-witnessed how and when it works. I did not collect any data but I created an intuition about the working. Let’s call this an experts view or domain knowledge. Note that the sprinkler system is a well-known example in Bayesian networks.
From my expert’s view, I know some facts about the system; it is sometimes on and sometimes off (isn’t it great). I have seen -very often- that if the sprinkler system is on, the grass is -possibly- wet. However, I also know that rain -almost certainly- results in wet grass too and that the sprinkler system is then -most of the time- off. I know that clouds are -often- present before it starts to rain. Finally, I noticed a -weak- interaction between Sprinkler and Cloudy but I’m not entirely sure.
From this point on you need to convert the expert’s knowledge into a model. This can be done systematically by first creating the graph and then define the CPTs that connect the nodes in the graph.
There are four nodes in the sprinkler system that you can extract from the expert's view. Each node works with two states: Rain: yes or no, Cloudy: yes or no, Sprinkler system: on or off, and Wet grass: true or false.
A complex system is built by combining simpler parts. This means that you don’t need to create or design the whole system at once but first define the simpler parts. The simpler parts are one-to-one relationships. In this step, we will convert the expert’s view into relationships. We know from the expert that: rain depends on the cloudy state, wet grass depends on the rain state but wet grass also depends on the sprinkler state. Finally, we know that sprinkler depends on cloudy. We can make the following four directed one-to-one relationships.
Cloudy → Rain
Rain → Wet Grass
Sprinkler → Wet Grass
Cloudy → Sprinkler
It is important to realize that there are differences in the strength of the relationships between the one-to-one parts and needs to be defined using the CPTs. But before stepping into the CPTs, let’s first make the DAG using bnlearn.
The four directed relationships can now be used to build a graph with nodes and edges. Each node corresponds to a variable and each edge represents a conditional dependency between pairs of variables. In bnlearn, we can assign and graphically represent the relationships between variables.
pip install bnlearn
In figure 3 is the resulting DAG. We call this a causal DAG because we have assumed that the edges we encoded represent our causal assumptions about the sprinkler system.
At this point, the DAG has no knowledge about the underlying dependencies. We can check the CPTs with bn.print(DAG) which will result in the message that “no CPD can be printed”. We need to add knowledge to the DAG with so-called Conditional Probabilistic Tables (CPTs) and we will rely on the expert’s knowledge to fill the CPTs.
Knowledge can be added to the DAG with Conditional Probabilistic Tables (CPTs).
The sprinkler system is a simple Bayesian network where Wet grass (child node) is influenced by two-parent nodes (Rain and Sprinkler) (see figure 1). The nodes Sprinkler and Rain are influenced by a single node; Cloudy. The Cloudy node is not influenced by any other node.
We need to associate each node with a probability function that takes, as input, a particular set of values for the node’s parent variables and gives (as output) the probability of the variable represented by the node. Let’s do this for the four nodes.
The Cloudy node has two states (yes or no) and no dependencies. Calculating the probability is relatively straightforward when working with a single random variable. From my expert view across the last 1000 days, I have eye-witnessed 70% of the time cloudy weather (I’m not complaining though, just disappointed). As the probabilities should add up to 1, not cloudy should be 30% of the time. The CPT looks as following:
The Rain node has two states and is conditioned by Cloudy, which also has two states. In total, we need to specify 4 conditional probabilities, i.e., the probability of one event given the occurrence of another event. In our case; the probability of the event Rain that occurred given Cloudy. The evidence is thus Cloudy and the variable is Rain. From my expert's view I can tell that when it Rained, it was also Cloudy 80% of the time. I did also see rain 20% of the time without visible clouds (Really? Yes. True story).
The Sprinkler node has two states and is conditioned by the two states of Cloudy. In total, we need to specify 4 conditional probabilities. Here we need to define the probability of Sprinkler given the occurrence of Cloudy. The evidence is thus Cloudy and the variable is Rain. I can tell that when the Sprinkler was off, it was Cloudy 90% of the time. The counterpart is thus 10% for Sprinkler is true and Cloudy is true. Other probabilities I’m not sure about, so I will set it to 50% of the time.
The wet grass node has two states and is conditioned by two-parent nodes; Rain and Sprinkler. Here we need to define the probability of wet grass given the occurrence of rain and sprinkler. In total, we have to specify 8 conditional probabilities (2 states ^ 3 nodes).
As an expert, I am certain, let’s say 99%, about seeing wet grass after raining or sprinkler was on: P(wet grass=1 | rain=1, sprinkler =1) = 0.99. The counterpart is thus P(wet grass=0| rain=1, sprinkler =1) = 1 – 0.99 = 0.01
As an expert, I am entirely sure about no wet grass when it did not rain or the sprinkler was not on: P(wet grass=0 | rain=0, sprinkler =0) = 1. The counterpart is thus: P(wet grass=1 | rain=0, sprinkler =0) = 1 – 1= 0
As an expert, I know that wet grass almost always occurred when it was raining and the sprinkler was off (90%). P(wet grass=1 | rain=1, sprinkler=0) = 0.9. The counterpart is: P(wet grass=0 | rain=1, sprinkler=0) = 1 – 0.9 = 0.1.
As an expert, I know that Wet grass almost always occurred when it was not raining and the sprinkler was on (90%). P(wet grass=1 | rain=0, sprinkler =1) = 0.9. The counterpart is: P(wet grass=0 | rain=0, sprinkler =1) = 1 – 0.9 = 0.1.
This is it! At this point, we defined the strength of the relationships in the DAG with the CPTs. Now we need to connect the DAG with the CPTs.
All the CPTs are created and we can now connect them with the DAG. As a sanity check, the CPTs can be examined using the print_DAG functionality.
The DAG with CPTs will now be like Figure 4.
Nice work! At this point, you created a model that describes the structure of the data, and the CPTs that quantitatively describe the statistical relationship between each node and its parents. Let’s ask some questions to our model and make inferences!
How probable is having wet grass given the sprinkler is off? P(Wet_grass=1 | Sprinkler=0) = 0.6162How probable is a rainy day given sprinkler is off and it is cloudy?P(Rain=1 | Sprinkler=0, Cloudy=1) = 0.8
An advantage of Bayesian networks is that it is intuitively easier for a human to understand direct dependencies and local distributions than complete joint distributions. To make knowledge-driven models, we need two ingredients; the DAG and Conditional Probabilistic Tables (CPTs). Both are derived from the expert by questioning. The DAG describes the structure of the data, and CPTs are needed to quantitatively describe the statistical relationship between each node and its parents. Although such an approach seems reasonable to do, you should be aware of systematic errors that can occur by questioning the expert, and the limitations when building a complex model.
In the sprinkler example, we extracted domain expert’s knowledge through personal experience. Although we created a causal diagram, it is hard to fully verify the validity and completeness of the causal diagram. For example, you may have a different perspective on the probabilities and the graph, and you may also be true about that. As an example, I described that: ‘I did also see rain in 20% of the time without visible clouds’. It may be reasonable to argue about such a statement. In opposite, it can also occur that multiple true knowledge models can exist at the same time. In such a case, you may need to combine the probabilities or fight out who is right.
The knowledge being used is as rich as the experts experiences and as colored as the experts prejudices.
Or in other words, the probabilities we extract by questioning an expert are subjective probabilities [5]. In the sprinkler example, it is sufficient to accept that this notion of probability is personal, it reflects the degree of belief of a particular person at a particular location at a particular time. Question yourself this; will the Sprinkler model change if the expert lives in England compared to Spain?
If you want to use such a procedure to design a knowledge-driven model, it is important to understand how people (experts) arrive at probabilistic estimations. In literature it is described that people rarely follow the principles of probability when reasoning about uncertain events, rather they replace the laws of probability with a limited number of heuristics [6, 7], such as representativeness, availability, and adjustment from an anchor. Note that this can lead to systematic errors and to that extent incorrect models. Also, make sure that the verbal probability phrase is the same for both sender and receiver in terms of exact probabilities or percentages.
The presented sprinkler system has only a few nodes but Bayesian networks can contain many more nodes with multiple levels of parent-child dependencies. The number of probability distributions required to populate a conditional probability table (CPT) in a Bayesian network, grows exponentially with the number of parent-nodes associated with that table. If the table is to be populated through knowledge elicited from a domain expert then the sheer magnitude of the task forms a considerable cognitive barrier [8].
Too much levels of parent-child dependencies can form a considerable cognitive barrier for the domain expert.
For example, if m parent nodes represent Boolean variables, then the probability function is represented by a table of 2^m entries, one entry for each of the possible parent combinations. Be hesitant of creating large graphs (more than 10-15 nodes) as the number of parent-child dependencies can form a considerable cognitive barrier for the domain expert. If you have data for the system that you want to model, it is also possible to learn the structure (DAG), and/or its parameters (CPTs) using structure learning [3].
I will repeat my earlier statement: “Well, it depends on how accurate you can represent the knowledge as a graph and how precise you can glue it together by probability theorem.”
This is it. Creating a knowledge-driven model is not easy. It is not only about data modeling but is also about human psychology. Make sure you prepare for the expert interviews. It is better to have multiple short interviews than one long interview. Ask your questions systematically. First design the graph with nodes and edges, then go into the CPTs. Be cautious when discussing probabilities. Understand how the expert derives his probabilities and normalize when required. Check if time and place can differ the outcome. Make sanity checks after building the model. Smile occasionally.
Be Safe. Stay Frosty.
Cheers, E.
If you found this article helpful, help support my content by signing up for a Medium membership using my referral link or follow me to access similar ones.
bnlearn github/documentation
Let’s connect on LinkedIn
Follow me on github
Wikipedia, Knowledge
Wikipedia, Knowledge
2. Pearl, Judea (2000). Causality: Models, Reasoning, and Inference. Cambridge University Press. ISBN 978 – 0 – 521 – 77362 – 1. OCLC 42291253.
3. E.Taskesen, A Step-by-Step Guide in detecting causal relationships using Bayesian Structure Learning in Python, Medium, 2021
4. Sanne Willems, et al, Variability in the interpretation of probability phrases used in Dutch news articles — a risk for miscommunication, JCOM, 24 March 2020
5. R. Jeffrey, Subjective Probability: The Real Thing, Cambridge University Press, Cambridge, UK, 2004.
6. A. Tversky and D. Kahneman, Judgment under Uncertainty: Heuristics and Biases, Science, 1974
7. Tversky, and D. Kahneman, ‘Judgment under uncertainty: Heuristics and biases,’ in Judgment under Uncertainty: Heuristics and Biases, D. Kahneman, P. Slovic, and A. Tversky, eds., Cambridge University Press, Cambridge, 1982, pp 3–2
8. Balaram Das, Generating Conditional Probabilities for Bayesian Networks: Easing the Knowledge Acquisition Problem. Arxiv