# Application of Decision Tree-Based Classification Algorithm on Content Marketing

Authors: Yi Liu; Shuo Yang
Journal: Journal of Mathematics
 (2022)

---

## Abstract

Traditional content marketing methods resort grossly to market requirements but barely obtain relatively accurate marketing prediction under loads of requirements. Machine learning-based approaches nowadays are widely used in multiple fields as they involve a training process to deal with big data problems. In this paper, decision tree-based methods are introduced to the field of content marketing, and decision tree-based methods intrinsically follow the process of human decision making. Specifically, this paper considers a well-known method, called C4.5, which can deal well with continuous values. Based on four validation metrics, experimental results obtained from several machine learning-based methods indicate that the C4.5-based decision tree method has the ability to handle the content marketing dataset. The results show that the decision tree-based method can provide reasonable and accurate suggestions for content marketing.


---

## Body

## 1. Introduction

Content marketing is normally regarded as a management process responsible for identifying, anticipating, and satisfying customer requirements profitably in the context of content via electronic channels [1]. The designation of research methods in content marketing has received plenty of attention for more than 100 years. Traditionally, content marketing strategies are determined based on different developments technologies, market requirements and expectations, growth in knowledge, and so on [2]. Machine learning-based methods have received more attention in the past several decades because they have the ability to analyze historical data and predicting future potential behaviors and activities more effectively [3, 4]. When there exist newly available data, managers can capture relatively accurate estimations from machine learning prediction models to tackle underlying challenges, such as missing outcomes and stimulus sampling [5]. Due to the fast development of machine learning, the current content marketing strategies tend to be digital-driven, meaning that companies put more attention to the consumer concentration on the Internet or historical habitats [6]. In this regard, the data scale and dimensions become huge, meaning that the traditional marketing methods are not suitable. The major motivation of this paper is therefore to investigate the potential of machine learning methods in dealing with the high dimensionality of attributes of marketing data.Based on different data requirements, machine learning methods can be divided into three parts [7]. The first part refers to supervised learning-based methods that require an independent training process and predict testing data using an already learned model. The representative supervised models include sparse/collaborative representation [8–10], support vector machine [11–13], ensemble learning [14–17], and so on. The second part refers to unsupervised learning-based models that do not demand training samples and determine entire classes by considering the correlations among samples. The unsupervised models are also called clustering methods, such as K-means [18, 19], ISODATA [20, 21], fuzzy C-means [22–24], and so on. The third part refers to semi-supervised learning-based models that simultaneously consider labeled and unlabeled samples in the training stage. Representative methods include semi-supervised support vector machines [25] and deep generative model [26].Among the above-mentioned machine learning methods, decision trees are the most discussed scalable multivariate methods in machine learning, which intrinsically follow the process of human decision making. Unlike emerging deep learning-based methods [27–31] that require strict support, decision trees consider the latent structure of training data. That is to say, they split the training samples into bins and each of them is verified after calculating the most prevalent split variable by means of a specific metric, such as information gain value, Gini index, and entropy metrics. Besides, decision tree-based methods show at least three advantages. First, the decision tree model to some degree is easy to follow and implement, meaning that customers do not have to know complex and latent background knowledge in the training procedures. Second, the decision tree-based learning models do not pay sufficient attention to the preparation process of samples as it to some degree is useless and time consuming. The main reason behind this fact is that the decision tree-based model equally treats the attributes of samples as the learning foundations, which makes it possible to handle a relatively huge data scale. Third, the process of verifying effectiveness and robustness of the decision tree model can be easily measured. Figure 1 visually illustrates the processing of classification using a decision tree.Figure 1
Visual descriptions of classification process in content marketing behaviors under the decision tree framework. Based on a splitting metric, each node will be quantified. When the leaf nodes satisfy the requirements, they will be selected as a category; otherwise, they will be allocated to different classes.Chou [32] proposed a classification and regression tree method in 1984, called CART. This method is a representative nonparametric learning strategy that produces classification and decision trees based on the status of dependent variables. In 1986, Quinlan proposed an iterative dichotomiser 3 method, called ID3 [33]. In this method, information gain is considered to determine the potential of the next nodes, meaning that the nodes with high information gain will be split. Xiaoliang et al. [34] extended Quinlan’s work and developed a well-known C4.5 method in 2009. Similar to ID3, C4.5 uses a well-known information gain metric. Besides, it also deals well with continuous inputs because it uses thresholds and splits attributes when the values are higher than the threshold. In 2012, Patil et al. [35] further extended C4.5 and ID3 and proposed C5.0 classification method. C5.0 not only has low time and memory consumptions compared to C4.5 and ID3 but also is efficient as it splits the nodes in terms of the field that has the maximum information gain. Recently, publications related to decision tree-based machine learning methods exist in different variants such as random forest [36, 37], gradient boosting decision tree [38], regression decision tree [39, 40], and so on.Among loads of decision tree-based methods, this paper specifically considers C4.5 and applies it to content marketing data analysis. This method is widely used because its classifying rules follow the human’s thinking process and it also has the ability to produce desirable results. Based on four validation metrics, we conduct several important experiments on the bank content marketing dataset for the purpose of verifying the performances of C4.5 and other comparison methods. The main contribution of this paper is that we introduce a decision tree-based method to the content marketing field to efficiently improve the marketing capability.


## 2. Materials and Methods

### 2.1. Validation Metrics

Validation metrics including entropy, information gain, Gini index, gain ratio, and so on are the most important standards to determine whether the current node should be split. Entropy verifies the uncertainties among a set of random variables. When random variables have high entropy values, it means that the uncertainty of the random variables is equally high and the current should be further divided into two nodes. The entropy information of such random variables is given as follows:(1)HX=−∑x∈Xpxlogpx,where log denotes a base 2 logarithm and they have the same meaning in the subsequent formulas.Information gain represents the difference between the entropy obtained before performing splitting and that after performing splitting. In this regard, the standard of splitting relies on the method that can produce maximum information gain as it means better classification performances. The information gain is(2)gainX,A=HX−HX|A,where HX|A means the entropy after excluding A attributes from X, given by(3)HX|A=∑a∈Apa×HX|A=a=∑a∈Apa×−∑x∈Xpx|A=alogpx|A=a.Gini index represents the uncertainty of data. High Gini values mean high uncertainties of samples, and it can be defined as follows:(4)GX=∑x∈Xpx1−px=1−∑x∈Xpx2.


### 2.2. C4.5-Based Decision Tree Method

C4.5 algorithm is normally seen as an important variant of the traditional ID3 algorithm, which contains the following representative improvements to ID3 algorithm: (1) C4.5 selects split attributes through information gain rate; (2) C4.5 can handle discrete-continuous attributes; (3) C4.5 involves pruning processing after finishing the construction of the decision tree; (4) when the problem of missing values occurs, C4.5 still can maintain its performances.When C4.5 constructs the decision tree, those attributes with high information gain rate are normally adopted for splitting the current node. Following this recursive process, the calculated information gain will get smaller. Finally, if the attribute contains high information gain rate, it will be split. Gain ratio can be given by(5)gain ratioX,A=gainX,AIVA,where IV⋅ denotes intrinsic values of attributes and pa|A denotes the ratio of a in A. The IV⋅ operator is defined as(6)IVA=−∑a∈Apa|Alogpa|A.When the attribute type is discrete, there is no need to discretize the data. When the attribute type is continuous, the data need to be discretized. C4.5 algorithm aims at the discretization of continuous attributes. The core idea is to arrange theN attribute values in ascending order. All attribute values are divided into two parts by dichotomy. Besides, the information gain corresponding to each division method is calculated and the threshold of the splitting rules is selected according to the largest information gain. The detailed process is given as follows:(1)
All data samples on the node are arranged from small to large according to the specific values of the continuous attribute to obtain its attribute value sequencex1A,…,xNA.(2)
There areN−1 possibilities in splitting of sequence x1A,…,xNA. For i-th possibility, its threshold is calculated by θi=xiA+xi+1A/2. Based on this threshold, two subsets are produced, i.e., x1A,…,xiA and xi+1A,…,xNA. Then, we calculate the information gain ratio and go to step (3).(3)
By calculating the entire information gain ratio under possibilities, we can obtain two optimal splitting subsets corresponding to maximal gain ratio results. We record the corresponding splitting threshold.


### 2.3. Pruning Process in C4.5

The establishment process of the decision tree depends heavily on training samples, meaning that the fitting effect on the given training data will occur. It is worth mentioning that, however, the decision tree-based learning model to some extent is complex for the testing data, triggering low classification accuracies, a.k.a. overfitting. Therefore, the process of simplifying the decision model is highly demanding, resulting in a crucial step, i.e., pruning.C4.5 algorithm adopts the BEP (Bayesian error pruning) method. The BEP method was proposed by Quinlan, which is a top-down pruning method. It determines whether to prune the subtree according to the error rate before and after pruning, so it does not need a separate pruning dataset.For a leaf node, assume that it involvesN samples with E errors. Then, the error ratio is E+p/N, where p denotes a penalty factor. Then, for a subtree with L leaf nodes, its error ratio is(7)error ratio=∑i=1LEi+pL∑i=1LNi,where Ei denotes the sample numbers with regard to i-th leaf and Ni denotes the total number of samples in i-th leaf node.Suppose a subtree misclassifies a sample with a value of 1 and correctly classifies a sample with a value of 0, then the number of misclassifications of the subtrees follows a Bernoulli distribution, so their statistical information, i.e., mean and standard deviation values, can be obtained by(8)error mean=error ratio×∑i=1LNi,error Std=error ratio×∑i=1LNi1−error ratio.After replacing the subtree by leaf nodes, their error values can be determined by(9)error ratio′=E′+pN′,where E′=∑i=1LEi and N′=∑i=1LNi.Note that the number of misclassifications of leaf nodes also follows a Bernoulli distribution. The mean values of the number of misclassifications of leaf nodes are defined as(10)error mean′=error ratio′×N′.Here, pruning can be performed if it satisfies the following formula:(11)error mean+error Std≥error ratio′.In order to visually describe the process of performing C4.5 decision tree on given data, this paper provides the corresponding flowchart of C4.5 (see Figure2).Figure 2
Flowchart of C4.5 decision tree. In this first step, N nodes are created. In the second step, the algorithm will start a training process. In this regard, the attributes and the gain ratio of attributes are verified. After obtaining gain ratios of each node, the several nodes with high gain ratio are selected for further split by iteratively executing the aforementioned step.


## 2.1. Validation Metrics

Validation metrics including entropy, information gain, Gini index, gain ratio, and so on are the most important standards to determine whether the current node should be split. Entropy verifies the uncertainties among a set of random variables. When random variables have high entropy values, it means that the uncertainty of the random variables is equally high and the current should be further divided into two nodes. The entropy information of such random variables is given as follows:(1)HX=−∑x∈Xpxlogpx,where log denotes a base 2 logarithm and they have the same meaning in the subsequent formulas.Information gain represents the difference between the entropy obtained before performing splitting and that after performing splitting. In this regard, the standard of splitting relies on the method that can produce maximum information gain as it means better classification performances. The information gain is(2)gainX,A=HX−HX|A,where HX|A means the entropy after excluding A attributes from X, given by(3)HX|A=∑a∈Apa×HX|A=a=∑a∈Apa×−∑x∈Xpx|A=alogpx|A=a.Gini index represents the uncertainty of data. High Gini values mean high uncertainties of samples, and it can be defined as follows:(4)GX=∑x∈Xpx1−px=1−∑x∈Xpx2.


## 2.2. C4.5-Based Decision Tree Method

C4.5 algorithm is normally seen as an important variant of the traditional ID3 algorithm, which contains the following representative improvements to ID3 algorithm: (1) C4.5 selects split attributes through information gain rate; (2) C4.5 can handle discrete-continuous attributes; (3) C4.5 involves pruning processing after finishing the construction of the decision tree; (4) when the problem of missing values occurs, C4.5 still can maintain its performances.When C4.5 constructs the decision tree, those attributes with high information gain rate are normally adopted for splitting the current node. Following this recursive process, the calculated information gain will get smaller. Finally, if the attribute contains high information gain rate, it will be split. Gain ratio can be given by(5)gain ratioX,A=gainX,AIVA,where IV⋅ denotes intrinsic values of attributes and pa|A denotes the ratio of a in A. The IV⋅ operator is defined as(6)IVA=−∑a∈Apa|Alogpa|A.When the attribute type is discrete, there is no need to discretize the data. When the attribute type is continuous, the data need to be discretized. C4.5 algorithm aims at the discretization of continuous attributes. The core idea is to arrange theN attribute values in ascending order. All attribute values are divided into two parts by dichotomy. Besides, the information gain corresponding to each division method is calculated and the threshold of the splitting rules is selected according to the largest information gain. The detailed process is given as follows:(1)
All data samples on the node are arranged from small to large according to the specific values of the continuous attribute to obtain its attribute value sequencex1A,…,xNA.(2)
There areN−1 possibilities in splitting of sequence x1A,…,xNA. For i-th possibility, its threshold is calculated by θi=xiA+xi+1A/2. Based on this threshold, two subsets are produced, i.e., x1A,…,xiA and xi+1A,…,xNA. Then, we calculate the information gain ratio and go to step (3).(3)
By calculating the entire information gain ratio under possibilities, we can obtain two optimal splitting subsets corresponding to maximal gain ratio results. We record the corresponding splitting threshold.


## 2.3. Pruning Process in C4.5

The establishment process of the decision tree depends heavily on training samples, meaning that the fitting effect on the given training data will occur. It is worth mentioning that, however, the decision tree-based learning model to some extent is complex for the testing data, triggering low classification accuracies, a.k.a. overfitting. Therefore, the process of simplifying the decision model is highly demanding, resulting in a crucial step, i.e., pruning.C4.5 algorithm adopts the BEP (Bayesian error pruning) method. The BEP method was proposed by Quinlan, which is a top-down pruning method. It determines whether to prune the subtree according to the error rate before and after pruning, so it does not need a separate pruning dataset.For a leaf node, assume that it involvesN samples with E errors. Then, the error ratio is E+p/N, where p denotes a penalty factor. Then, for a subtree with L leaf nodes, its error ratio is(7)error ratio=∑i=1LEi+pL∑i=1LNi,where Ei denotes the sample numbers with regard to i-th leaf and Ni denotes the total number of samples in i-th leaf node.Suppose a subtree misclassifies a sample with a value of 1 and correctly classifies a sample with a value of 0, then the number of misclassifications of the subtrees follows a Bernoulli distribution, so their statistical information, i.e., mean and standard deviation values, can be obtained by(8)error mean=error ratio×∑i=1LNi,error Std=error ratio×∑i=1LNi1−error ratio.After replacing the subtree by leaf nodes, their error values can be determined by(9)error ratio′=E′+pN′,where E′=∑i=1LEi and N′=∑i=1LNi.Note that the number of misclassifications of leaf nodes also follows a Bernoulli distribution. The mean values of the number of misclassifications of leaf nodes are defined as(10)error mean′=error ratio′×N′.Here, pruning can be performed if it satisfies the following formula:(11)error mean+error Std≥error ratio′.In order to visually describe the process of performing C4.5 decision tree on given data, this paper provides the corresponding flowchart of C4.5 (see Figure2).Figure 2
Flowchart of C4.5 decision tree. In this first step, N nodes are created. In the second step, the algorithm will start a training process. In this regard, the attributes and the gain ratio of attributes are verified. After obtaining gain ratios of each node, the several nodes with high gain ratio are selected for further split by iteratively executing the aforementioned step.


## 3. Result Analysis and Discussion

### 3.1. Experimental Parameters

The entire experiment involved in this paper was conducted on a PC with Intel Core i7-10700 processor at 2.90 GHz and 16 GB RAM. The GPU is NVIDIA GeForce GTX 1660 Super. The comparison algorithms in this paper involve nearest neighbors (the number of nearest neighbors is fixed at 3), linear SVM (the penalty parameter is fixed at 0.025), neural network (the regularization parameter is fixed at 1 and other parameters are fixed under default settings in sklearn package), Naive Bayes, and random forest (the maximum depth is fixed at 5).


### 3.2. Datasets

This paper considers a bank campaign content marketing dataset, which was originally derived from the Portuguese banking institution. This dataset has 50 user features by investigating multiple content marketing campaign activities based on traditional phone calls in order to find the real information that if users are interested in purchasing the products in the near future. Figure3 visually describes the correlations among several features using a heatmap.Figure 3
Heatmap of correlations among several representative features.


### 3.3. Validation Metrics

In order to accurately describe the experimental accuracy, this paper uses four standard evaluation metrics derived from the confusion matrix for evaluation purposes including accuracy, precision, recall, andF1 score. The above four evaluation criteria are calculated based on the confusion matrix. Figure 4 gives the visual explanations of the confusion matrix. For the classification results of a given binary classification model, true positive (TP) represents an outcome where the model correctly predicts the positive class; false positive (FP) represents the results that indicate a given condition exists when it does not; true negative (TN) is an outcome where the model correctly predicts the negative class; false negative (FN) is an outcome where the model incorrectly predicts the negative class. Based on four definitions, accuracy (acc), precision (prec), recall, and F1 score (F1) can be further defined by(12)acc=TP+TNTP+TN+FP+FN,prec=TPTP+FP,recall=TPTP+FN,F1=2×TP2×TP+FP+FN.Figure 4
Explanation of confusion matrix.


### 3.4. Experimental Results

Figure5 displays the resulting trend of the decision method when varying the max depth from 1 to 15. As can be seen from Figures 5(a) and 5(b), when the number of max depths is fixed as 4, the decision tree method shows the best training accuracies in Gini index and gain ratio metrics. Normally, the max depth of decision tree means the maximum depth to which a tree is allowed to grow. However, a deeper tree will trigger a complex model. Specifically, for the training stage, if we increase max depth, the training error will always decrease (or at least not increase). For the testing stage, if depth is set too high, the decision tree may result in overfitting problems, which increase test errors. On the contrary, if it is set too low, the decision tree barely captures latent information from training data. Hence, in this paper, we fix the number of max depths of the decision tree as 4.Figure 5
Accuracy tendency in different number of depths. (a) The method using gain ratio index. (b) The method using Gini index.
(a)(b)Figure6 visually illustrates the node construction process of the decision tree. Owing to the fact that the number of useful features is too large, the max depth of the decision tree is fixed at 3 for better visualization. As can be seen from Figure 4, the splitting manner highly depends on a determined metric and a threshold. When the Gini index of a node is higher than a threshold, then it will be split into two new nodes.Figure 6
Visualization of nodes of the decision tree method in max depth 3.Generally, each classifier pays different attention to features, which means that not entire features are equally important in model training. In order to visually display the importance of features in the content marketing dataset, this paper provides comparisons among linear SVM, decision tree, and random forest. It can be seen from Figure7 that top 10 important features that play a crucial role in model training are visually displayed. For the linear SVM, among 10 features, the first five features including pdays, poutcom_success, poutcom_failure, poutcom_nonexistent, and duration have higher impacts on linear SVM’s training process. In terms of decision tree, it is more obvious that only duration and nr.employed are the key features while other features such as euribor3m, cons.conf.idx, poutcome_success, and so on have lower importance. For the random forest method, the top 10 features seem equal to play an important role in the training model. The relatively lower weights also indicate that most of the features are highly used.Figure 7
Visualization of the importance of features in three methods.In this paper, 20% of total samples are selected as training samples and the rest of 80% samples are treated as testing samples. Table1 tabulates overall experimental results obtained from six classification models based on four validation metrics. As can be found from Table 1, the decision tree can capture the best predicting performances as it has the highest metric values. Compared to the decision tree, other comparison algorithms such as neural network and random forest, respectively, have the optimal results with regard to F1 score and recall. In addition, the nearest neighbor has the second-best results for F1 score, linear SVM can achieve the second-best results for recall and F1 score, the neural network has the suboptimal results for accuracy, Naive Bayes has the suboptimal results for precision, and random forest has the suboptimal results for F1 score. Execution time obtained from entire algorithms is displayed in Table 1. As can be found from this table, nearest neighbors and linear SVM require the highest computational time, while others cost much less. Figure 8 visually displays the decision boundaries of six classification methods. From those figures, we find that compared to Figures 8(a)–8(d), Figures 8(e) and 8(f) show more distinct boundaries. Figure 9 further adds the ROC curves obtained from four representative classification methods. As can be seen from Figure 9, even though the decision tree has lower area under the ROC curve than that of the nearest neighbors, it shows high intensity.Table 1
Comparison results obtained from six classification methods.
Nearest neighborLinear SVMNeural networkNaive BayesRandom forestDecision treePrecision0.910.910.920.930.890.94Recall0.970.990.980.7611F1 score0.940.940.950.870.940.95Accuracy0.88770.89810.90370.7820.88790.9135Time (s)3.633.070.480.070.110.10Figure 8
Visualization of decision boundaries of six methods. The red and green backgrounds, respectively, denote the decision boundaries of duration and employed attribute. The red and blue solid points denote two classes corresponding to the duration and employed attributes. (a) Nearest neighbors. (b) Linear SVM. (c) Neural net. (d) Naive Bayes. (e) Random forest. (f) Decision tree.
(a)(b)(c)(d)(e)(f)Figure 9
ROC curves of four methods. (a) Nearest neighbors. (b) Neural net. (c) Random forest. (d) Decision tree.
(a)(b)(c)(d)To verify the impact of different ratios of training samples on the experimental results of six algorithms, this paper performs an experiment on the content marketing dataset with varying the number of training samples from 5% to 25. Figure10 provides a visual comparison of six classifiers on the different ratios of training samples. Compared to other methods, the decision tree shows the best results in all situations of training sample number. The results obtained from Naive Bayes seem disappointing primarily because it assumes that the attributes of samples are independent. In this regard, if the sample attributes are related, the effect is not good. Besides, it cannot learn the interaction among features, which highly limits its experimental performances.Figure 10
Classification accuracy of six methods on different ratio of training samples.


## 3.1. Experimental Parameters

The entire experiment involved in this paper was conducted on a PC with Intel Core i7-10700 processor at 2.90 GHz and 16 GB RAM. The GPU is NVIDIA GeForce GTX 1660 Super. The comparison algorithms in this paper involve nearest neighbors (the number of nearest neighbors is fixed at 3), linear SVM (the penalty parameter is fixed at 0.025), neural network (the regularization parameter is fixed at 1 and other parameters are fixed under default settings in sklearn package), Naive Bayes, and random forest (the maximum depth is fixed at 5).


## 3.2. Datasets

This paper considers a bank campaign content marketing dataset, which was originally derived from the Portuguese banking institution. This dataset has 50 user features by investigating multiple content marketing campaign activities based on traditional phone calls in order to find the real information that if users are interested in purchasing the products in the near future. Figure3 visually describes the correlations among several features using a heatmap.Figure 3
Heatmap of correlations among several representative features.


## 3.3. Validation Metrics

In order to accurately describe the experimental accuracy, this paper uses four standard evaluation metrics derived from the confusion matrix for evaluation purposes including accuracy, precision, recall, andF1 score. The above four evaluation criteria are calculated based on the confusion matrix. Figure 4 gives the visual explanations of the confusion matrix. For the classification results of a given binary classification model, true positive (TP) represents an outcome where the model correctly predicts the positive class; false positive (FP) represents the results that indicate a given condition exists when it does not; true negative (TN) is an outcome where the model correctly predicts the negative class; false negative (FN) is an outcome where the model incorrectly predicts the negative class. Based on four definitions, accuracy (acc), precision (prec), recall, and F1 score (F1) can be further defined by(12)acc=TP+TNTP+TN+FP+FN,prec=TPTP+FP,recall=TPTP+FN,F1=2×TP2×TP+FP+FN.Figure 4
Explanation of confusion matrix.


## 3.4. Experimental Results

Figure5 displays the resulting trend of the decision method when varying the max depth from 1 to 15. As can be seen from Figures 5(a) and 5(b), when the number of max depths is fixed as 4, the decision tree method shows the best training accuracies in Gini index and gain ratio metrics. Normally, the max depth of decision tree means the maximum depth to which a tree is allowed to grow. However, a deeper tree will trigger a complex model. Specifically, for the training stage, if we increase max depth, the training error will always decrease (or at least not increase). For the testing stage, if depth is set too high, the decision tree may result in overfitting problems, which increase test errors. On the contrary, if it is set too low, the decision tree barely captures latent information from training data. Hence, in this paper, we fix the number of max depths of the decision tree as 4.Figure 5
Accuracy tendency in different number of depths. (a) The method using gain ratio index. (b) The method using Gini index.
(a)(b)Figure6 visually illustrates the node construction process of the decision tree. Owing to the fact that the number of useful features is too large, the max depth of the decision tree is fixed at 3 for better visualization. As can be seen from Figure 4, the splitting manner highly depends on a determined metric and a threshold. When the Gini index of a node is higher than a threshold, then it will be split into two new nodes.Figure 6
Visualization of nodes of the decision tree method in max depth 3.Generally, each classifier pays different attention to features, which means that not entire features are equally important in model training. In order to visually display the importance of features in the content marketing dataset, this paper provides comparisons among linear SVM, decision tree, and random forest. It can be seen from Figure7 that top 10 important features that play a crucial role in model training are visually displayed. For the linear SVM, among 10 features, the first five features including pdays, poutcom_success, poutcom_failure, poutcom_nonexistent, and duration have higher impacts on linear SVM’s training process. In terms of decision tree, it is more obvious that only duration and nr.employed are the key features while other features such as euribor3m, cons.conf.idx, poutcome_success, and so on have lower importance. For the random forest method, the top 10 features seem equal to play an important role in the training model. The relatively lower weights also indicate that most of the features are highly used.Figure 7
Visualization of the importance of features in three methods.In this paper, 20% of total samples are selected as training samples and the rest of 80% samples are treated as testing samples. Table1 tabulates overall experimental results obtained from six classification models based on four validation metrics. As can be found from Table 1, the decision tree can capture the best predicting performances as it has the highest metric values. Compared to the decision tree, other comparison algorithms such as neural network and random forest, respectively, have the optimal results with regard to F1 score and recall. In addition, the nearest neighbor has the second-best results for F1 score, linear SVM can achieve the second-best results for recall and F1 score, the neural network has the suboptimal results for accuracy, Naive Bayes has the suboptimal results for precision, and random forest has the suboptimal results for F1 score. Execution time obtained from entire algorithms is displayed in Table 1. As can be found from this table, nearest neighbors and linear SVM require the highest computational time, while others cost much less. Figure 8 visually displays the decision boundaries of six classification methods. From those figures, we find that compared to Figures 8(a)–8(d), Figures 8(e) and 8(f) show more distinct boundaries. Figure 9 further adds the ROC curves obtained from four representative classification methods. As can be seen from Figure 9, even though the decision tree has lower area under the ROC curve than that of the nearest neighbors, it shows high intensity.Table 1
Comparison results obtained from six classification methods.
Nearest neighborLinear SVMNeural networkNaive BayesRandom forestDecision treePrecision0.910.910.920.930.890.94Recall0.970.990.980.7611F1 score0.940.940.950.870.940.95Accuracy0.88770.89810.90370.7820.88790.9135Time (s)3.633.070.480.070.110.10Figure 8
Visualization of decision boundaries of six methods. The red and green backgrounds, respectively, denote the decision boundaries of duration and employed attribute. The red and blue solid points denote two classes corresponding to the duration and employed attributes. (a) Nearest neighbors. (b) Linear SVM. (c) Neural net. (d) Naive Bayes. (e) Random forest. (f) Decision tree.
(a)(b)(c)(d)(e)(f)Figure 9
ROC curves of four methods. (a) Nearest neighbors. (b) Neural net. (c) Random forest. (d) Decision tree.
(a)(b)(c)(d)To verify the impact of different ratios of training samples on the experimental results of six algorithms, this paper performs an experiment on the content marketing dataset with varying the number of training samples from 5% to 25. Figure10 provides a visual comparison of six classifiers on the different ratios of training samples. Compared to other methods, the decision tree shows the best results in all situations of training sample number. The results obtained from Naive Bayes seem disappointing primarily because it assumes that the attributes of samples are independent. In this regard, if the sample attributes are related, the effect is not good. Besides, it cannot learn the interaction among features, which highly limits its experimental performances.Figure 10
Classification accuracy of six methods on different ratio of training samples.


## 4. Conclusions

Traditional content marketing strategies rely heavily on empirical knowledge such as market requirements and expectations. However, when the content marketing process meets loads of diverse users in different societies or communities, the user data will be huge so that the traditional content marketing strategy will be difficult. Machine learning-based methods have the ability to analyze historical data and predict future potential behaviors and activities more effectively. Among plenty of machine learning-based methods, decision tree has received more attention from content marketing managers as it intrinsically follows the process of human decision making. To verify the performance of the decision tree on the content marketing data, this paper considers a well-known decision method, called C4.5, and simultaneously compares C4.5 with other five machine learning methods including nearest neighbor, linear SVM, neural network, Naive Bayes, and random forest. Based on four validation metrics, experiments were conducted on the bank content marketing dataset under different experimental scenarios and settings. Experimental results obtained from six methods indicate that the decision tree has the ability to handle the content marketing dataset, meaning that it can provide reasonable and accurate content marketing suggestions for managers.



---

*Source: 6469054-2022-03-04.xml*