Hybrid Rule-Based Machine Learning With scikit-learn | by Lukas Haas | Towards Data Science

TL;DR scikit-learn does not allow you to add hard-coded rules to your machine learning model, but for many use cases, you should! This article explores how you can leverage domain knowledge and object-oriented programming (OOP) to build hybrid rule-based machine learning models on top of scikit-learn.
Supervised machine learning models are great for making predictions under uncertainty; they pick up patterns in past data and accurately extrapolate them into the future. Machine learning has pushed the frontier in fields where determining the most likely outcome, whether a class or specific value, has historically been challenging, prone to error, or too time-consuming or expensive at scale.
Still, there exist many domains in which some of all possible outcomes are not ambiguous but certain by definition. You might encounter them in the form of rules embedded in business-specific processes or regulations. In such an environment, it seems inefficient to have an ML model asymptotically guess pre-formulated rules using implicit learning. Instead, we want the model to focus its attention on all cases where no pre-defined rules exist.
In this article, you will learn that incorporating pre-defined domain rules into machine learning models has many benefits. To get more hands-on, we will build a simple wrapper-class for scikit-learn estimators, which takes into account explicit rules and leaves the model to sort out the hard cases.
If you cannot wait, skip to the fully documented implementation in Python.
Any good machine learning project starts with the aggregation of domain knowledge — the process of gathering relevant information and expertise about a business problem. Typically, we talk with industry practitioners, research online, and conduct data exploration to expose specific trends, patterns, or hints which facilitate machine learning model building.
Domain knowledge is incredibly helpful for many reasons: it helps us balance stakeholder requirements, understand our target audience, but most importantly, it gives us vital clues about feature engineering. While these clues are self-explanatory when attempting to identify a cat in a photograph, in many sectors of industry such as law, insurance, or medical diagnosis, feature engineering is far from intuitive.
Illustrating this point, suppose your objective is to build an ML model to predict customer churn (plan cancellation rates) for a telecommunication corporate. Before engineering and iterating on probable features, it certainly helps to gather the opinions of experts in the field regarding factors influencing churn — the retention department being a logical starting point. Possibly, the retention department can even back up their opinions with data, using customer surveys conducted to expose specific pain points. In any case, data and industry practitioners can point you in the right direction, saving time and potentially unveiling previously unconsidered data sources and feature combinations.
In many domains of industry, you can derive simple deterministic rules from processes that are already in place. For example, in specific legal proceedings, a claim for reputational damages may never be granted because the legal code simply states so. Similarly, an insurance company might not pay out damage claims below $1.000 because, by contract with the insured, they are not liable to do so. If we wanted to predict say litigation outcomes or insurance losses, such simple rules could be built directly into machine learning models to enhance performance.
As machine learning is exceptional in figuring out ambiguous and challenging cases, it mostly only makes sense to incorporate deterministic rules into models if they are applicable in every circumstance and are not too numerous or complex. Yet, other use cases exist, so I compiled a complete list on when you might want to consider deploying hybrid, rule-based models:
Deterministic rules for the prediction process already existAs mentioned before, depending on what you are attempting to predict, deterministic rules might already exist for the evaluation process. If the rule is simple, applicable in every case, and there are not too many rules overall, hard-coding it into your machine learning model would guarantee you can already predict a subset of cases with perfect accuracy.
Lack of data for specific types of prediction casesIn cases where data is sparse for certain types of prediction cases, your model might struggle to develop correct implicit rules to classify or estimate the data points correctly. Frequently, this occurs if the model cannot accurately infer the target variable solely from other features. In the legal example from above, there might be infrequent claim categories, such as the reimbursement of expenses, for which only a few data points exist. As the claim category is of critical importance to determine the litigation outcome, it is unlikely the model can correctly predict it without further knowledge of reimbursement claims. In this case, performance might be enhanced by simply predicting the average target variable of all instances of the category, say the mean success rate of all expense claims.
High feature cardinalityHigh feature cardinality (the number of possible values for a feature) is a problem for almost all machine learning models. Especially for categorical data, which needs to be encoded, a high number of unique possible values hampers model performance. Thus, if an appropriate rule of thumb or statistical parameter exists, approximating the target variable might make for an attractive trade-off as it leaves the remaining data with lower cardinality to aid model training.
Actively fighting biases in the dataMachine learning predictions are inherently biased because of real-world patterns reflected in our training data. In some cases, we can forestall biased predictions by taking matters in our own hands and hard-coding rules which overwrite model behavior.
As mentioned before, machine learning models learn rules implicitly. The epitomes of such learning are decision-tree-based algorithms such as scikit-learn’s DecisionTreeClassifier or GradientBoostingRegressor, the latter being an ensemble of decision trees.
Decision-tree-based algorithms attempt to predict the target variable by learning decision rules inferred from the supplied data. The decision rules itself are incredibly simple; they are a sequence of splits on the data using only the basic logical operators =, <, >, ≤, ≥. Nevertheless, all splits only approximate any explicit rules and consequently may not be as accurate.
We can use the same approach to build simple deterministic rules as logical formulae which we can hence translate into code. For instance, let us again suppose we want to design a predictive model to estimate the total losses of an insurance company, and we know the company rejects claims less or equal to $1.000. One way of hard-coding this rule would be:
if claim_amount <= 1000:   # reject claimelse:   # use machine learning model
There are many ways in which we can integrate deterministic rules into our machine learning pipeline. Adding rules progressively as data pre-processing steps might seem intuitive, but this would not suit our goal. Preferably, we aim to leverage the concept of abstraction by adopting object-oriented programming (OOP) to generate a novel ML model class. This hybrid model will then encompass all deterministic rules, enabling us to train it like any other machine learning model.
Conveniently, scikit-learn provides a BaseEstimator class which we can inherit to build scikit-learn models ourselves without much effort. The advantage of building a new estimator is that we can blend our rules directly with the model logic while leveraging an underlying machine learning model for all data to which the rules don’t apply.
Let us start by building our new hybrid model class and adding an init method to it. As an underlying model, we will use the scikit-learn implementation of a GradientBoostingClassifier; we will call it the base_model.
import pandas as pdfrom typing import Dict, Tuplefrom sklearn.base import BaseEstimatorclass RuleAugmentedGBC(BaseEstimator):     def __init__(self, base_model: BaseEstimator, rules: Dict, **base_params):        self.rules = rules    self.base_model = base_model    self.base_model.set_params(**base_params)
We created the RuleAugmentedGBC class which inherits from BaseEstimator. Our class is not complete yet and is still missing some essential methods, but it is now technically a scikit-learn estimator. The init method initializes our estimator utilizing a base_model and a dictionary of rules. We can set additional parameters in the init method which are then directly passed to the underlying base_model. In our case, we will use a GradientBoostingClassifier as the base_model.
In this article’s implementation, we will supply rules to the model in the following format:
{"House Price": [    ("<", 1000.0, 0.0),    (">=", 500000.0, 1.0)], "...": [    ...    ...]}
As illustrated above, we format rules as a Python dictionary. The dictionary keys represent the feature column names to which we want to apply our rules. The values of the dictionary are lists of tuples, each tuple representing a unique rule. The first element of the tuple is the logical operator of the rule, the second the split criterion, and the last object is the value which the model should return if the rule is applicable.
For instance, the first rule in the example above would indicate that if any value in the House Price feature column is less than 1000.0, the model should return the value 0.0.
We proceed to code a fit method (within our RuleAugmentedGBC class) to allow our model to train on data. What is important to notice here is that we want to use our deterministic rules wherever possible, and train the base_model only on data which is not affected by the rules. We will decompose this step by formulating a private helper method called _get_base_model_data to filter out the data necessary to train our base_model.
def fit(self, X: pd.DataFrame, y: pd.Series, **kwargs):  train_x, train_y = self._get_base_model_data(X, y)  self.base_model.fit(train_x, train_y, **kwargs)
The fit method is pretty straightforward: it first applies the to be coded _get_base_model_data method to distill the training features and labels for our underlying base_model and then fits the model to the data. Similar to before, we can set additional parameters which we subsequently pass to the fit method of the base_model. Let us now implement the _get_base_model_data method:
def _get_base_model_data(self, X: pd.DataFrame, y: pd.Series) -> Tuple[pd.DataFrame, pd.Series]:  train_x = X    for category, rules in self.rules.items():    if category not in train_x.columns.values: continue    for rule in rules:      if rule[0] == "=":        train_x = train_x.loc[train_x[category] != rule[1]]      elif rule[0] == "<":        train_x = train_x.loc[train_x[category] >= rule[1]]      elif rule[0] == ">":        train_x = train_x.loc[train_x[category] <= rule[1]]      elif rule[0] == "<=":        train_x = train_x.loc[train_x[category] > rule[1]]      elif rule[0] == ">=":        train_x = train_x.loc[train_x[category] < rule[1]]      else:        print("Invalid rule detected: {}".format(rule))  indices = train_x.index.values  train_y = y.iloc[indices]  train_x = train_x.reset_index(drop=True)  train_y = train_y.reset_index(drop=True)  return train_x, train_y
Our private _get_base_model_data method iterates through the rule dictionary keys and finally through every unique rule. At every rule, depending on the logical operator, it narrows down the train_x pandas dataframe to only include the data points not affected by the rule. Once we have applied all rules, we match the corresponding labels via indices and return the residual data for the base_model.
The predict method works in like manner to the fit method. Wherever possible, rules should be applied; if no rules are applicable, the base_model should produce a prediction.
def predict(self, X: pd.DataFrame) -> np.array:    p_X = X.copy()  p_X['prediction'] = np.nan  for category, rules in self.rules.items():    if category not in p_X.columns.values: continue    for rule in rules:      if rule[0] == "=":        p_X.loc[p_X[category] == rule[1], 'prediction'] = rule[2]      elif rule[0] == "<":        p_X.loc[p_X[category] < rule[1], 'prediction'] = rule[2]      elif rule[0] == ">":        p_X.loc[p_X[category] > rule[1], 'prediction'] = rule[2]      elif rule[0] == "<=":        p_X.loc[p_X[category] <= rule[1], 'prediction'] = rule[2]      elif rule[0] == ">=":        p_X.loc[p_X[category] >= rule[1], 'prediction'] = rule[2]      else:        print("Invalid rule detected: {}".format(rule))  if len(p_X.loc[p_X['prediction'].isna()].index != 0):    base_X = p_X.loc[p_X['prediction'].isna()].copy()    base_X.drop('prediction', axis=1, inplace=True)    p_X.loc[p_X['prediction'].isna(), 'prediction'] = self.base_model.predict(base_X)  return p_X['prediction'].values
The predict method copies our input pandas dataframe in order not to change the input data. We then add a prediction column in which we gather all our hybrid model’s predictions. Just as in the _get_base_model_data method, we iterate through all rules and, wherever applicable, record the corresponding return value in the prediction column. Once we have applied all rules, we check whether any predictions are still missing. If this is the case, we revert to our base_model to generate the remaining predictions.
To get a working model that inherits from the BaseEstimator class, we need to implement two more simple methods — get_params and set_params. These allow us to set and read the parameters of our new model. As these two methods are not integral to the topic of this article, please have a look at the fully documented implementation below if you want to know more.
Below, you will find the full code of the scikit-learn wrapper class that we built in this article, complete with documentation. You might find it useful for one of your use cases.
Here is a short code snippet to illustrate how you could utilize the RuleAugmentedEstimator wrapper class to add rules to a GradientBoostingClassifier. This example assumes you have already initialized the variables rules, train_X, train_y, and test_X. Please refer to the section A Common Format for Rules to examine how any rules should be employed.
gbc = GradientBoostingClassifier(n_estimators=50)hybrid_model = RuleAugmentedEstimator(gbc, rules)hybrid_model.fit(train_X, train_y)predictions = hybrid_model.predict(test_X)
Congratulations on making it this far! I hope this article has helped you to leverage domain knowledge and object-oriented programming (OOP) to build hybrid rule-based machine learning models. As you have seen, the concept of abstraction can be incredibly helpful to directly incorporate rules into an ML model while keeping your data pipeline clean.
Writing this article has helped me to explore the topic in-depth as well as its applications. While I attempted to check my work for errors, please let me know should you find any.
I am always happy for feedback and open to discussions about topics in the fields of data science, machine learning, and technology in general. I would love to hear from you, so please feel free to connect with me on LinkedIn.