Machine Learning : Decision Tree using Spark For Layman | by Sandeep Khurana | Towards Data Science

We will go over the definition, intuition and algorithm of a Decision Tree in this article.
Then we will write our own code, without using any ML library (like tensorflow etc). This will give good understanding and hands on experience for implementing decision tree.
Definition from study.com
A decision tree is a graphical representation of possible solutions to a decision based on certain conditions. It’s called a decision tree because it starts with a single box (or root), which then branches off into a number of solutions, just like a tree.
Lets consider this data of some users visiting ecom website. Lets assume we know the profile of the users visiting the site like their age, job , credit_limit etc.
This table contains some sales records and the schema of the data is
First column is city where the purchaser was from.
The second column job indicates the job purchaser was doing.
Third column is credit limit of the person
Fourth column is the age of the purchaser
The last most important column indicates if the user ended up purchasing any item or not. ‘yes’ means the user bought an item and ‘no’ means user did not buy.
What we want to do it that based upon this data, we want to build a decision tree to further predict if any new user will end up buying any item or not on the website.
Lets look at the data and see which of the four factors (city, occupation, cc_limit and age) might be influencing the decision of user to buy or not.
Consider ‘age’, this property has different value for each row. Even for unique outputs — ‘yes’ and ‘no’ the values of this property are very random. So we can say age for sure is not the determining factor for user to buy the product.
But how can we prove mathematically that age is not a factor at least not a major factor in determining the outcome? Lets look at entropy.
Entropy is measure of randomness (or impurity). In the above sample, age is most random attribute.
But how do we measure the entropy? The formula to measure entropy is. This is also called Shannon Entropy
n is the number of values and
p is number of elements in the class divided by total elements.
Lets dig into details. Consider the column of our sample data above having the outcome values
The number of ‘yes’ (‘yes’ class) for this property (i.e. y) are = 4
The number of ‘no’ (‘no’ class ) are = 3
Total number of elements are = 7
p(0) is = number of elements in ‘yes’ class / Total number of elements = 4/7
p(1) is = number of elements in ‘no’ class / Total number of elements = 3/7
Entropy = ( — 4/7 * log2(4/7) — 3/7 * log2(3/7)) = ( — 0.57* — 0.81–0.43 * -1.22) = ( 0.4617 + 0.5246) = 0.9863
0.9863 is the entropy of the data.
Now we know the entropy of the dataset based upon the outcome, how do we mathematically find out most influential top level attribute which contributes to the decision. This will be root of the decision tree we are trying to build.
As discussed above, ‘age’ can not be the factor for any customer to decide buy decision as we are seeing people age is varying with almost each record. Too much varying’ness of the age makes it least factor in decision making so conversely the attribute whose varying ness is least would be the most influential attribute or rather the attribute which varies more inline with the outcome? Hint, its the latter. Lets see.
We already have entropy (impurity) of dataset which for above example is 0.9863. So, we do below to get the information gain for an attribute
Find out the entropy of each class in each attribute. Eg city has 2 classes in — Bangalore and Chennai in our example
Proportionate the entropy for each class wrt datasets classes.
For city attribute
(total records with Bangalore city / Total records) * Bangalore Entropy
(total records with Chennai city / Total records) * Chennai Entropy
Sum up above values for each class and subtract it from dataset’s entropy
Lets find out Bangalore city entropy. Check the records for Bangalore. 2 of these have ‘yes’ output and 1 has ‘no’ output. So, Bangalore entropy would be
Bangalore Entropy = -2/3log2(2⁄3) — 1/3log2(1⁄3) = 0.39 + 0.53 = 0.92
Similarly, Chennai Entropy = -2/2log2(2/2) — 2/2log2(2/2) = 0 (its fully uniform property, equal numbers of it belong to ‘yes’ and ‘no’ outcomes)
So, information gain of the city would be as discussed above = 0.9863 — { (3/7)*0.92 + 0 } = 0.592
Information gain for ‘job’ — 0.9863 — { 4/7 * Entropy of Employed + 2/7 * Entropy of the unemployed + }
Similarly we get the information gain for ‘credit_limit’ as well.
Lets add another attribute called ‘ownshouse’ which whose values are varying exactly the way output varies. So the dataset now looks like
‘owns house’ indicates if the customer owns a house or not. It has same values as the outcome attribute ‘y’. Lets fine the Info Gain for this attribute.
It has 2 classes ‘yes’ and ‘no’.
Entropy for ‘yes’ = — 4/4log2(4/4) — 0 (no item belonging to ‘no’ class) = 0
Entropy for ‘no’ = — 0–3/3log2(3/3) = 0
Information Gain for ‘ownshouse’ attribute = 0.9863–0 = 9.9863
What did just happen? Since this attribute’s ‘ownshouse’ values are varying exactly same as of the outcome then its information gain is same as the entropy of the dataset.
Once we have the information gain for attributes then
We get the attribute which has maximum information gain
Make that attribute as head of the tree
Get the values of that attribute and make these branches of this head e.g. lets assume “city” has maximum info gain (actually it does not max gain in out example but assume it for discussion sake).
Head will be city and its branches will be Bangalore and Chennai
In next iteration — remove city from the list of attributes.
For pending attributes, get the records which belong to city = Bangalore for each attribute for branch under Bangalore.
Calculate the Info Gain for attributes under Bangalore city , get the attribute with max gain and that becomes the node under Bangalore.
Keep doing it till either we exhaust all attributes or get entropy zero.
I have another example with code at the end which implements this logic for other dataset, so you can go over that to see how its implemented.
As mentioned above we stop when either
We have gone thru all the attributes or
Found attribute whose entropy is zero.
Actually in our sepecial case example, we had met the attribute which has entropy zero in the first iteration itself. That attribute was ‘ownshouse’.
We start hands on exercise from here. If you are not keen on coding then the article concludes for you here.
Tools used — pySpark, networkx python lib to visualize the decision tree
The code is on github as well at https://github.com/skhurana333/ml/blob/master/decision_tree/decision_tree_model_complex.py
Basically this datasets lists the conditions which impacts if a game of tennis can be played outside or not. The values of outlook, temperature, humidity and wind are described and outcome that the game was played or not under these conditions. We will build the decision tree now.
outlook;temp;humidity;wind;ySunny;Hot;High;Weak;noSunny;Hot;High;Strong;noOvercast;Hot;High;Weak;yesRain;Mild;High;Weak;yesRain;Cool;Normal;Weak;yesRain;Cool;Normal;Strong;noOvercast;Cool;Normal;Strong;yesSunny;Mild;High;Weak;noSunny;Cool;Normal;Weak;yesRain;Mild;Normal;Weak;yesSunny;Mild;Normal;Strong;yesOvercast;Mild;High;Strong;yesOvercast;Hot;Normal;Weak;yesRain;Mild;High;Strong;no
Lets assume the file having above data is saved at the location = /home/me/ml/practice/decision_tree/datasets/simple_dataset
We will go over the flow and the algorithm
Lets load the dataset.
data = sqlContext.read.format('com.databricks.spark.csv').option('header', 'true')\ .option('delimiter', ';') \ .load("/home/me/ml/practice/decision_tree/datasets/simple_dataset")
Since it contains header, we have set header as true in the option.
The delimiter is ;
And its in csv format.
We will register this as table as we will need to query this dataset with different where conditions.
data.registerTempTable('data')
Calculate the entropy of the dataset
Get the count of rows having outcome as ‘yes’ and ‘no’ . Below are pyspark queries
played = sqlContext.sql("select * from data WHERE y like '%y%' ").count()notplayed = sqlContext.sql("select * from data WHERE y like '%n%' ").count()
Since we need to get the information gain for each attribute and find the attribute with max info gain, so we will be applying same logic for info gain calculation for all attributes. Create ‘process_dataset’ function.
def process_dataset(excludedAttrs, data, played, notplayed, where_condition):
excludedAtttts will contain the list of attributes which are already processed so that we dont need to process again.
data is the spark dataframe for this file
played — count when match was played
notplayed — count when match was not played
Where_condition — condition used to select the data, as and when attributes are processed we will keep chaging this condition
Lets call this function from main function,
def main():   data = sqlContext.read.format('com.databricks.spark.csv').option('header', 'true')\       .option('delimiter', ';').load("/home/me/ml/practice/decision_tree/datasets/simple_dataset")   data.registerTempTable('data')   played = sqlContext.sql("select * from data WHERE y like  '%y%' ").count()   notplayed = sqlContext.sql("select * from data WHERE y like  '%n%' ").count()   process_dataset([], data, played, notplayed, '')
2) Processing Data
Process_dataset will calculate the entropy of dataset first and then get info gain for each attribute.
We will declare global level variable to store attributes’s info gain. And also will create directed graph to visualize the decision tree.
attr_name_info_gain = {}G = nx.DiGraph()
Lets also declare schema of the dataset. I have created 2 variables for storing the schema and types. You can create one if you wish and modify subsequent code accordingly.
attrs = ["outlook","temp","humidity","wind"]attrs_type = {"outlook":"string","temp":"string","humidity":"string","wind":"string"}
process_dataset function
def process_dataset(excludedAttrs, data, played, notplayed, where_condition):   total_elements = played + notplayed   subs_info = {"played" : played, "notplayed" : notplayed}   entropy = calculate_entropy(total_elements, subs_info)   print "entropy is " + str(entropy)   global attr_name_info_gain   attr_name_info_gain = dict()   for attr in attrs:       if attr not in excludedAttrs:           get_attr_info_gain_data_prep(attr, data, entropy, total_elements, where_condition)
It calls calculate_entropy function and then for each attribute it calls get_attr_info_gain function.
3) Calculate_entropy function
def calculate_entropy(total_elements, elements_in_each_class):   # for target set S having 2 class 0 and 1, the entropy is -p0logp0 -p1logp1   # here the log is of base 2   # elements_in_each_class is a dictionary where the key is class label and the   # value is number of elements in that class   keysInMap = list(elements_in_each_class.keys())   entropy = 0.0   for aKey in keysInMap:       number_of_elements_in_class = elements_in_each_class.get(aKey)       if number_of_elements_in_class == 0:           continue       ratio = number_of_elements_in_class/total_elements       entropy = entropy - ratio * np.log2(ratio)   return entropy
4) Attr info gain data preparation function
def get_attr_info_gain_data_prep(attr_name, data, entropy, total_elements, where_condition):   if not where_condition:       attr_grp_y = data.where(col('y') == 'yes').groupBy(attr_name).agg({"y": 'count'})\           .withColumnRenamed('count(y)','played_count')   else:       attr_grp_y = data.where(" y like '%yes%'  " + where_condition).groupBy(attr_name).agg({"y": 'count'})\           .withColumnRenamed('count(y)','played_count')   if not where_condition:       attr_grp_n = data.where(col('y') == 'no').groupBy(attr_name).agg({"y": 'count'})\           .withColumnRenamed(attr_name,'n_' + attr_name)\           .withColumnRenamed('count(y)','not_played_count')   else:       attr_grp_n = data.where(" y like '%no%'  " + where_condition).groupBy(attr_name).agg({"y": 'count'})\           .withColumnRenamed(attr_name,'n_' + attr_name)\           .withColumnRenamed('count(y)','not_played_count')   joined_df = attr_grp_y.join(attr_grp_n, on = [col(attr_grp_y.columns[0]) == col(attr_grp_n.columns[0])], how='outer' )\       .withColumn("total", col(attr_grp_y.columns[0]) + col(attr_grp_n.columns[0]))\       .select(attr_grp_y.columns[0], attr_grp_y.columns[1],\                attr_grp_n.columns[1]) \   gain_for_attribute = calculate_info_gain(entropy, joined_df, total_elements)   attr_name_info_gain[attr_name] = gain_for_attribute
Explanation of the function
It reads data from the file. As we build the tree, we will need to get data corresponding to that branch of the tree only. The ‘where_condition’ attribute will contain these predicates.
We group the records in the file which have outcome as ‘yes’ for the attribute names passed
For first time, the where_condition will be blank,
Second iteration onwards, after root of the tree is found, we will have where_condition
if not where_condition:   attr_grp_y = data.where(col('y') == 'yes').groupBy(attr_name).agg({"y": 'count'})\       .withColumnRenamed('count(y)','played_count')else:   attr_grp_y = data.where(" y like '%yes%'  " + where_condition).groupBy(attr_name).agg({"y": 'count'})\       .withColumnRenamed('count(y)','played_count')
Similarly It groups the records in the file which have outcome as ‘no’ for the attribute names passed.
if not where_condition:   attr_grp_n = data.where(col('y') == 'no').groupBy(attr_name).agg({"y": 'count'})\       .withColumnRenamed(attr_name,'n_' + attr_name)\       .withColumnRenamed('count(y)','not_played_count')else:   attr_grp_n = data.where(" y like '%no%'  " + where_condition).groupBy(attr_name).agg({"y": 'count'})\       .withColumnRenamed(attr_name,'n_' + attr_name)\       .withColumnRenamed('count(y)','not_played_count')
We rename the columns count(y) to played_count and count(n) to not_played_count
Rename the attr_name column for n
.withColumnRenamed(attr_name,'n_' + attr_name)\
Now we join these 2 dataframes, one having the count of ‘yes’ outcomes for this attribute and other having count of ‘no’ outcomes for same attribute
joined_df = attr_grp_y.join(attr_grp_n, on = [col(attr_grp_y.columns[0]) == col(attr_grp_n.columns[0])], how='outer' )\
We join on the attribute name. We need outer join as if an attribute does not has record for either ‘yes’ or ‘no’ outcome then we still want the present ‘yes or ‘no’ count.
Also, since we have the count of yes and no outcomes, we will need the total count also (total # of records having yes and no) to calculate the entropy. Hence we added a new column which has the total
.withColumn("total", col(attr_grp_y.columns[0]) + col(attr_grp_n.columns[0]))\
And we are not selecting the join column twice
.select(attr_grp_y.columns[0], attr_grp_y.columns[1],\        attr_grp_n.columns[1])
We did all this exercise to calculate the info gain for this particular attribute, so lets calculate that
gain_for_attribute = calculate_info_gain(entropy, joined_df, total_elements)
We will save info gain for this attribute in a global variable, attr_name_info_gain. We will later sort this dict to get the attr having max info gain. INvoked Calculate_info_gain function is described next.
gain_for_attribute = calculate_info_gain(entropy, joined_df, total_elements)  # attr_name_info_gain[attr_name] = gain_for_attribute
5) Info gain function
def calculate_info_gain(entropy, joined_df, total_elements):   attr_entropy = 0.0   for anAttributeData in joined_df.rdd.collect():       yes_class_count = anAttributeData[1]       no_class_count = anAttributeData[2]       if yes_class_count is None:           yes_class_count = 0       elif no_class_count is None:           no_class_count = 0       count_of_class = yes_class_count + no_class_count       # do the summation part e.g. if age is 56, 60, 45 then its sum of entropy for each of these element       classmap = {'y' : yes_class_count, 'n' : no_class_count}       attr_entropy = attr_entropy + ((count_of_class / total_elements) *\                                      calculate_entropy(count_of_class, classmap))   gain = entropy - attr_entropy   return gain
If you have read the earlier sections then this function is self explanatory
6) Coming back to main function again
We had done processing till this code line — ->”process_dataset([], data, played, notplayed, ‘’)”
def main():   data = sqlContext.read.format('com.databricks.spark.csv').option('header', 'true')\       .option('delimiter', ';').load("/home/me/ml/practice/decision_tree/datasets/simple_dataset")   data.registerTempTable('data')   played = sqlContext.sql("select * from data WHERE y like  '%y%' ").count()   notplayed = sqlContext.sql("select * from data WHERE y like  '%n%' ").count()   process_dataset([], data, played, notplayed, '')   # sort by info gain   sorted_by_info_gain = sorted(attr_name_info_gain.items(), key=operator.itemgetter(1), reverse=True)   processed_attrs = []   max_gain_attr = sorted_by_info_gain[0][0]   processed_attrs.append(max_gain_attr)   build_tree(max_gain_attr, processed_attrs, data, '')   nx.draw(G, with_labels=True)   plt.show()
We sort the info gain function and get the attr having max gain, make that as root node. Now we call build tree function. This is last and important function.
The input to this method is name of the attribute having max info gain. This becomes root node for the tree. We want to get the distinct values of this attribute. These values will become branches of this node.
This function will build complete tree recursively.
def build_tree(max_gain_attr, processed_attrs, data, where_condition):   attrValues = sqlContext.sql("select distinct " + max_gain_attr + " from data  where 1==1 " + where_condition)   orig_where_condition = where_condition   for aValueForMaxGainAttr in attrValues.rdd.collect():       adistinct_value_for_attr = aValueForMaxGainAttr[0]       G.add_edges_from([(max_gain_attr, adistinct_value_for_attr)])       if attrs_type[max_gain_attr] == "string":           where_condition = str(orig_where_condition + " and " + max_gain_attr + "=='" + adistinct_value_for_attr + "'")       else:           where_condition = str(orig_where_condition + " and " + max_gain_attr + "==" + adistinct_value_for_attr)       played_for_attr = sqlContext.sql("select * from data where y like '%yes%' " + where_condition).count()       notplayed_for_attr = sqlContext.sql("select * from data where y like '%no%' " + where_condition).count()       # if either has zero value then entropy for this attr will be zero and its the last attr in the tree       leaf_values = []       if played_for_attr == 0 or notplayed_for_attr == 0:           leaf_node = sqlContext.sql("select distinct y from data where 1==1 " + where_condition)           for leaf_node_data in leaf_node.rdd.collect():               G.add_edges_from([(adistinct_value_for_attr, str(leaf_node_data[0]))])           continue       process_dataset(processed_attrs, data, played_for_attr, notplayed_for_attr, where_condition)       if not attr_name_info_gain: # we processed all attributes           # attach leaf node           leaf_node = sqlContext.sql("select distinct y from data where 1==1 " + where_condition)           for leaf_node_data in leaf_node.rdd.collect():               G.add_edges_from([(adistinct_value_for_attr, str(leaf_node_data[0]))])           continue # we are done for this branch of tree       # get the attr with max info gain under aValueForMaxGainAttr       # sort by info gain       sorted_by_info_gain = sorted(attr_name_info_gain.items(), key=operator.itemgetter(1), reverse=True)       new_max_gain_attr = sorted_by_info_gain[0][0]       if sorted_by_info_gain[0][1] == 0:           # under this where condition, records dont have entropy           leaf_node = sqlContext.sql("select distinct y from data where 1==1 " + where_condition)           # there might be more than one leaf node           for leaf_node_data in leaf_node.rdd.collect():               G.add_edges_from([(adistinct_value_for_attr, str(leaf_node_data[0]))])           continue # we are done for this branch of tree       G.add_edges_from([(adistinct_value_for_attr, new_max_gain_attr)])       processed_attrs.append(new_max_gain_attr)       build_tree(new_max_gain_attr, processed_attrs, data, where_condition)
Explanation
We want distinct values for the node.
attrValues = sqlContext.sql("select distinct " + max_gain_attr + " from data  where 1==1 " + where_condition)
For each value for this node
for aValueForMaxGainAttr in attrValues.rdd.collect():
Get the attribute value under this iteration
adistinct_value_for_attr = aValueForMaxGainAttr[0]
Add this value as branch of the node
G.add_edges_from([(max_gain_attr, adistinct_value_for_attr)])
We need to get the data under this attribute value. If the tree is
And we are processing ‘Sunny’ value of the parent node then to proceed under same branch ( outlook -> sunny) we need to get data where outlook = sunny. So we add this as new where condition.
Also, if the attribute type is string when we need to add value under quotes and it is int type then we dont add quotes.
if attrs_type[max_gain_attr] == "string":           where_condition = str(orig_where_condition + " and " + max_gain_attr + "=='" + adistinct_value_for_attr + "'")       else:           where_condition = str(orig_where_condition + " and " + max_gain_attr + "==" + adistinct_value_for_attr)
We will again find out the attr having max info gain from the list of attributes excluding the attributes which are already processed. So for this value of the parent attr we want to know how many records outcome as ‘yes’ and ‘no’
played_for_attr = sqlContext.sql("select * from data where y like '%yes%' " + where_condition).count()notplayed_for_attr = sqlContext.sql("select * from data where y like '%no%' " + where_condition).count()
As the comment states, if we get zero entropy value then we have reached the leaf node of this branch, we will just add the outcomes to the tree under this node.
# if either has zero value then entropy for this attr will be zero and its the last attr in the tree       leaf_values = []       if played_for_attr == 0 or notplayed_for_attr == 0:           leaf_node = sqlContext.sql("select distinct y from data where 1==1 " + where_condition)           for leaf_node_data in leaf_node.rdd.collect():               G.add_edges_from([(adistinct_value_for_attr, str(leaf_node_data[0]))])           continue
If not then we keep processing. Code similar to below was already was processed in main function and explained. Maybe we should put that one iteration also as part of this recursion, let this be exercise for you.
process_dataset(processed_attrs, data, subscribed_for_attr, unsubscribed_for_attr, where_condition)if not attr_name_info_gain: # we processed all attributes   # attach leaf node   leaf_node = sqlContext.sql("select distinct y from data where 1==1 " + where_condition)   for leaf_node_data in leaf_node.rdd.collect():       G.add_edges_from([(adistinct_value_for_attr, str(leaf_node_data[0]))])   continue # we are done for this branch of tree# get the attr with max info gain under aValueForMaxGainAttr# sort by info gainsorted_by_info_gain = sorted(attr_name_info_gain.items(), key=operator.itemgetter(1), reverse=True)new_max_gain_attr = sorted_by_info_gain[0][0]if sorted_by_info_gain[0][1] == 0:   # under this where condition, records dont have entropy   leaf_node = sqlContext.sql("select distinct y from data where 1==1 " + where_condition)   # there might be more than one leaf node   for leaf_node_data in leaf_node.rdd.collect():       G.add_edges_from([(adistinct_value_for_attr, str(leaf_node_data[0]))])   continue # we are done for this branch of treeG.add_edges_from([(adistinct_value_for_attr, new_max_gain_attr)])processed_attrs.append(new_max_gain_attr)build_tree(new_max_gain_attr, processed_attrs, data, where_condition)
Now if we last 2 lines of main function are executed then you can visually see the decision tree built :-)
nx.draw(G, with_labels=True)plt.show()
Enjoy !!! And if possible, provide the feedback about wrong spelling or any other such edit (have not done any review or proof reading), about algo, optimization etc.