# System for Recommending Telecommunication Packages Based on the Deep and Cross Network

Authors: Congming Shi; Wen Wang; Shoulin Wei; Feiya Lv
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

With the evolution of the 5th generation mobile network (5G), the telecommunications industry has considerably affected livelihoods and resulted in the development of national economies worldwide. To increase revenue per customer and secure long-term contracts with users, telecommunications firms and enterprises have launched diverse types of telecommunication packages to satisfy varied user requirements. Several systems for recommending telecommunication packages have been recently proposed. However, extracting effective feature information from large and complex consumption data remains challenging. Conventional methods for the recommendation of telecommunications packages either rely on complex expert feature engineering or fail to perform end-to-end deep learning (DL) during training. In this study, we propose a recommender system based on the Deep and Cross Network (DCN), deep belief network (DBN), embedding, and Word2Vec using the learning abilities of DL-based approaches. The proposed system fits the recommender system for telecommunication packages in terms of click-through rate prediction to provide a potential solution to the recommendation challenges faced by telecommunication enterprises. The proposed model captures the finite order interactional and deep hidden features. Additionally, the text information in the data is used to improve the model’s recommendation capability. The proposed method also does not require feature engineering. We conducted comprehensive experiments using real-world datasets, the results of which demonstrated that our proposed method outperformed other methods based on DBNs, DCNs, deep factorization machines, and deep neural networks in terms of the area under the ROC curve, cross entropy (log loss), and recall metrics.


---

## Body

## 1. Introduction

The telecommunications industry has become a critical support industry, serving almost all domains worldwide owing to the development of communication technology. Furthermore, with the evolution of 5G technology, the telecommunications industry has impacted livelihoods, and it contributes to the development of national economies. This new era of 5G networks will increase the competitive pressure among telecommunication enterprises. In particular, the implementation of number portability, which allows mobile users to switch from one operator to another without changing numbers, further enhances the competitive pressure of various service providers, such as China Unicom, China Mobile, and China Telecom, in the market. To retain existing customers and attract new ones, operators have launched various telecommunication packages, which often combine internet services, cable television and telephone services, rendering them more cost-effective than purchasing each service individually. However, telecommunication enterprises face challenges when recommending a suitable telecommunication package to users with similar needs.In other words, if packages recommended by telecommunication enterprises are regarded as recommended items by recommender systems, the challenge faced by such enterprises is transformed into the task of improving the probability of a user clicking on a recommended item. The recommender system is a crucial basic technology of machine learning (ML) used as a potential solution to the recommendation challenges faced by telecommunication enterprises. A recommender system aims to solve the problem of information overload, and it is used to assist users in making the best choice from multiple options based on their interests [1]. Most recommender systems, including content-based [2, 3], collaborative filtering (CF) [4, 5], and knowledge-based [6, 7] systems, rely on a single-criterion rating as a primary source to obtain the recommended results [8]. Although these models have been successfully applied in multiple instances, they are generally affected by issues such as data sparseness [9]. Moreover, despite being widely adopted, they fail to generate effective recommendations in cases of multidimensionality, such as when contextual information is available [10]. In other words, conventional recommender systems based on conventional item-based CF cannot satisfy the recommendation requirements of telecommunication enterprises offering high-dimensional features for customers.However, the recent success of deep learning (DL)-based applications in multiple fields has resulted in the combination of recommender systems with DL being considered a novel and potential solution to the recommendation challenges faced by telecommunication enterprises. Such combination captures non-linear features and non-trivial user or item relationships and enables the codification of complex abstractions as data representations in the higher layers. In recommender systems, product-based neural networks (PNNs) are used to learn high-order feature interactions [11], deep neural networks (DNNs) with more than three layers are used to obtain improved generalization capabilities for unseen feature combinations [12], and the wide and deep network aids in modeling low- and high-order feature interactions [13]. A deep belief network (DBN) is generated by stacking multiple restricted Boltzmann machines (RBM) and through the greedy training of the RBM [14, 15]. Because a DBN is a generative model based on unlabeled data with a predictive classification function, it is used in recommender systems [16]. Additionally, a factorization-machine based neural network (DeepFM) uses the power of factorization machines (FMs) for recommendation and DL for feature learning [17]. The eXtreme deep factorization machine (xDeepFM) combines the compressed interaction network (CIN) and a classical DNN, which learns specific bounded-degree feature interactions explicitly and arbitrary low- and high-order feature interactions implicitly [18]. The Deep and Cross Network (DCN) maintains the DNN’s benefit of generalization capability to unseen feature combinations and obtains specific bounded-degree feature interactions [19]. Although these DL-based models were developed to predict the click-through rate (CTR) on the Internet, their direct application to telecommunication package recommender systems remains a challenge.Therefore, we propose a novel recommender system for telecommunication packages based on DCN. The proposed model uses the DBN network to replace the “deep” segment of the original DCN exploiting unlabeled data with a predictive classification function, embedding [20] to map the discrete categorical features, and Word2Vec [21] to solve the problem of the DCN's inability to process textual information obtaining similarities between numerical features and data wrangling (imputation and drop). Additionally, raw dataset conversion is used to fit the format of the CTR paradigm.The remainder of this paper is organized as follows: Section2 presents a brief overview of the related work and the proposed model. Section 3 explains the comprehensive set of experiments performed to validate the proposed method by comparing its performance to that of other methods. Finally, Section 4 presents the conclusions.


## 2. Methods

### 2.1. Related Work

Owing to rapid advancements in smartphone technology, telecommunication service providers must establish comprehensive recommender systems to ensure the provision of high-quality and custom services to different customers. Before the development of DL, conventional ML-based recommender systems were used for telecommunication package recommendations.


#### 2.1.1. Conventional ML-Based Recommender Systems

TheK-Nearest Neighbor (KNN) [22] was used to separate the historical datasets of customers into several classes to predict the classification of a new customer. However, the computing cost was high for a large characteristic dimension. Moreover, KNN was sensitive to noise in the dataset, and it exhibited a low fault tolerance.CF, a popular technique for developing recommendation systems, was employed in several applications [23]. In such applications, the behaviors of users based on the data collected in the past were analyzed to determine the connection between users and their interests, which could aid in recommending items to users with similar preferences and support different users’ considerations. A hybrid recommendation approach that combined user- and item-based CF techniques was proposed for mobile product and service recommendations [23, 24]. Although CF achieved efficiency based on similar measurements of users’ interests and recommended items, it was difficult to exploit the cross features completely owing to the lack of deep and effective feature extraction. CF recommendation systems also ad problems associated with cold start and low recommendation accuracy.Although recommender systems are widely used in the telecommunications industry, the methods employed focused on conventional ML-based and CF techniques. Only a few practical or academic cases exist in which state-of-the-art deep CTR models were used for the recommendation of telecommunication packages.


#### 2.1.2. DL-Based Recommender Systems

However, the significant success of DL introduced DL-based CTR to the field of recommender systems, which were applied in websites.FMs [25] integrated with DL exhibited outstanding performance and achieved promising results in CTR prediction. However, the two types of DL models proposed, namely, FM-supported neural network and sampling-based neural network, could capture only high-order feature interactions [26]. Wide and deep learning [13], which trained wide linear models and DNNs simultaneously to combine the benefits of memory and generalization for recommender systems, was initially introduced for app recommendation in Google Play. However, it required designing cross-product transformations through expertise feature engineering.PNN [11] attempted to capture high-order feature interactions by involving a product layer between the embedding and hidden layers. Unlike the conventional embedding combined with multilayer perceptrons, PNN explicitly captured the second-order feature correlation in multiple fields.Influenced by wide and deep learning, the DeepFM [17] replaced the “wide” segment of the wide and deep model with FM to learn the low-order features. The “deep” segment was developed using a feedforward neural network to learn high-order features. Both the “wide” and “deep” segments shared identical raw feature inputs. In comparison with the wide and deep model, the DeepFM was an end-to-end learning model, i.e., independent of manual feature engineering. Moreover, the xDeepFM [18] explicitly modeled low- and high-order feature interactions using a novel CIN segment.In summary, the above-mentioned models cannot simultaneously overcome the recommendation challenges faced by telecommunication enterprises with high-dimensional textual features, low- and high-order feature interactions, and unlabeled data. Therefore, we proposed a novel recommender system for telecommunication packages based on DCN, DBN, embedding, and Word2Vec, which provided a potential solution to the recommendation challenges faced by telecommunication enterprises.


### 2.2. Proposed Model

In this section, we describe the architecture of the proposed model, and we explain the process of developing the dataset.


#### 2.2.1. Model Overview

Figure1 illustrates the architecture of the proposed model based on the DCN. The model began with an embedding and stacking layer in which the sparse, dense, and converted features were stacked using Word2Vec. Subsequently, the stacked value was fed into the cross and deep network layers in parallel inputs. The two results were combined in the combining layer, and the final output was computed through a single-layer neural network and activated using a sigmoid function. Unlike the original DCN, the Word2Vec model was combined with the DCN to train text features, which solved the problem of the DCN’s inability to process textual information. This enhanced the recommendation accuracy of the proposed model. Moreover, using the DBN network to replace the “deep” segment of the original DCN could exploit the advantages of DBN completely.Figure 1
Architecture of the proposed model based on the DCN.


### 2.3. Model Analysis

(i)
Input and embedding layerWe classified all the features of the raw dataset into three types: categorical features, continuous integer features (such as times of payment), and decimal numerical features. The categorical features were transformed into a low-dimensional space using embedding for dimensional reduction. These transformed features were represented asXembT. To avoid it being affected by the dimension, continuous integer features and some numerical features in the model were scaled into a fixed range between zero and one using normalization or min-max normalization, and these normalized features were represented as XnormT. Additionally, other numerical features in the model were converted into textual features to be more suitable. The textual features were transformed into embedding vectors using Word2Vec, which was represented as XtextT. Finally, we stacked XembT, XnormT, and XtextT into one vector as follows:
(1)x0=XembT,XnormT,XtextT,and we fed x0 to the cross and deep network layers simultaneously.
(ii)
Cross network layerThe cross network layer comprisesm cross layers, where each cross layer could be calculated as follows:
(2)xj+1=x0xjTwj+bj+xj=fxj,wj,bj+xj,wherej∈0,1,2,⋯,m−1,m∈1,2,3,⋯; xj and xj+1 represent the output of the j-th cross layer and j+1-th cross layer, respectively; and wj and bj denote the weight and bias parameters of the j-th cross layer, respectively. All the aforementioned variables were column vectors. Furthermore, the features of each layer were cross-combined with the previous layer and original features (x0) and then added back to the previous layer. This is similar to the structure of a residual network in which the function f of each layer fits the residual of xj+1−xj. Thus, the gradient dispersion problem caused by the DNN could be solved using this residual network.The output of the last cross layer in the cross network layer was the output of the cross network layer, which was calledxcross.
(iii)
Deep network layer based on the DBNThe deep network was a fully connected feedforward neural network that shared the same input with the cross network. It enabled the model to learn the combined features of higher-order nonlinearity. The DBN was generated by stacking a set of RBMs. The output of the input and embedding layer (x0) was the input of the first RBM in the DBN.Except for the first RBM, the output of each RBM was calculated as follows:(3)hi+1=FWihi+bi,where i∈1,2,⋯,l−2,l−1, l represents the number of RBMs in the DBN, hi+1 and hi represent the output of the i+1-th and i-th RBM, respectively, wi and bi denote the weight and bias parameters of the i-th RBM, respectively, and F indicates the activation function for enhancing the nonlinear capability of the network. The rectified linear unit (ReLU) [27] was used as the activation function in the proposed model owing to its calculation simplicity. Moreover, the convergence speed of ReLU significantly outperformed that of other activation functions, such as sigmoid [28] and tanh [29].For the first RBM,h1=FW1x0+b1, where h1,W1,andb1 represent the output, the weight, and bias of the first RBM.The output of the last RBM is the output of the deep network layer based on the DBN, which is calledxdeep.
(iv)
Combining layerWhenx0 flows through the DCN, xcross and xdeep are generated. In the combining layer, xcross and xdeep are stacked into xcom as follows:
(4)xcom=xcross,xdeep.(v)
Output layerHere, we calculated the output through a perceptron using a sigmoid activation functionδ, which is calculated as follows:
(5)out=δwxcom+b,where w and b represent the weight vector and bias parameter for the combination layer.The loss function was the log loss along with a regularization term:(6)loss=1N∑i=1Nyilogouti+1−yilog1−outi+λ∑lwl2,whereouti represents the value computed from equation (5), yi represents the true labels, N represents the total number of inputs, and λ represents the L2 regularization parameter.


### 2.4. Dataset

The dataset used in this study was provided by the China Unicom Research Labs, and it is available at DataFountain (https://www.datafountain.cn/competitions/311/datasets) in .csv format. The dataset comprises 743,990 samples. Each sample indicated a consumption record, thereby implying that a telecommunication package was currently used by a customer.Table1 shows the dataset comprising 27 fields. From the field description of the dataset used, six fields, i.e., service_type, is_mix_service, many_over_bill, is_promise_low_consume, gender, and complaint_level, are categorical features. Furthermore, through the statistical analysis of each field in the dataset used, three fields, i.e., current_type, contract_type, and net_service, are categorical features. Clearly, after excluding the USERID field and nine categorical features, the other 15 fields are numerical features.Table 1
Specifications of the dataset used in this study.
Field nameField data typeCommentUSERIDVARCHAR2(50)User identitycurrent_type/current_serviceVARCHAR2(500)/service_typeVARCHAR2(10)0: 23G, 1:212C, 2:2G, 3:3G, 4:4Gis_mix_serviceVARCHAR2(10)1: yes, 0: noonline_timeVARCHAR2(50)/1_total_feeNUMBERUnit: Renminbi yuan2_total_feeNUMBERUnit: Renminbi yuan3_total_feeNUMBERUnit: Renminbi yuan4_total_feeNUMBERUnit: Renminbi yuanmonth_trafficNUMBERUnit: megabytesmany_over_billVARCHAR2(500)1: yes, 0: nocontract_typeVARCHAR2(500)ZBG_DIM.DIM_CBSS_ACTIVITY_TYPEcontract_timeVARCHAR2(500)/is_promise_low_consumeVARCHAR2(500)1: yes, 0: nonet_serviceVARCHAR2(500)20AAAAAA-2Gpay_timesNUMBERUnit: timepay_numNUMBERUnit: Renminbi yuanlast_month_trafficNUMBERUnit: megabyteslocal_traffic_monthNUMBERUnit: megabyteslocal_caller_timeNUMBERUnit: minuteservice1_caller_timeNUMBERUnit: minuteservice2_caller_timeNUMBERUnit: minuteGenderVARCHAR2(100)01: male, 02: femaleAgeVARCHAR2(100)/complaint_levelVARCHAR2(1000)1: ordinary, 2: import, 3: significantformer_complaint_numNUMBERUnit: timeformer_complaint_feeNUMBERUnit: minuteFor the dataset to be compatible with the CTR prediction task, it was transformed into binary classification data. In the case of CTR, each sample represents historical click behavior of a specific user on a telecommunication package, which was considered a displayed advertisement. The raw dataset used in this study comprised 11 unique telecommunication packages in the “current_type” categorical feature.Figure2 shows the distribution of 11 unique telecommunication packages in the “current_type” categorical feature. The mapping relationships between the 11 unique telecommunication packages were as follows: {0: 99999825, 1: 90063345, 2: 90109916, 3: 89950166, 4: 89950168, 5: 89950167, 6: 90155946, 7: 99999828, 8: 99999826, 9: 99999827, 10: 99999830}. Evidently, the 11 unique telecommunication packages were not represented equally. In other words, the dataset used was imbalanced. To solve the problem of imbalance, we used the synthetic minority over-sampling technique [30]. We randomly chose five samples from the 11 unique telecommunication packages to construct negative samples by maintaining consistency with the user-side features. Based on the CTR model, a “clicked” column was created in the dataset. Additionally, the click values of the positive and negative samples were “1” and “0,”, respectively.Figure 2
Distribution of the 11 unique telecommunication packages in the “current_type” categorical feature.


### 2.5. Data Wrangling

(i)
Dealing with missing valuesBecause the raw dataset comprised missing values, we used two methods to determine such values from the existing data.(1)
For the numerical features, the datasets were grouped in terms of the current package by first calculating the mean of nonmissing values in a column and then replacing the missing values within each group separately, independent of the other groups. For these 15 numerical features, i.e., online_time, 1_total_fee, 2_total_fee, 3_total_fee, 4_total_fee, month_traffic, contract_time, pay_times, pay_num, last_month_traffic, local_traffic_month, local_caller_time, service1_caller_time, service2_caller_time, and age, their missing values were replaced with the mean corresponding to the feature column. For example, the mean of the nonmissing values of “online_time” in the group of zero telecommunication packages was 74.184570, which was used to replace the missing values within the group of zero telecommunication packages(ii)
In the case of the categorical features, the datasets were grouped by considering the current package and replacing the missing data with frequently occurring values within each group separately. For these 10 categorical features, i.e., service_type, is_mix_service, many_over_bill, contract_type, is_promise_low_consume, net_service, gender, complaint_level, former_complaint_num, and former_complaint_fee, their missing values were replaced with frequently occurring values corresponding to the feature column. For example, frequently occurring values of gender within the group, 0 telecommunication packages, was one; the number of one was 15,627, and the number of two was 4,723, which was used to replace the missing values within the group of zero telecommunication packages(iii)
In the case of scaling using min-max normalization on the numerical features unaffected by the dimension, 17 features, i.e., online_time, 1_total_fee, 2_total_fee, 3_total_fee, 4_total_fee, month_traffic, contract_time, pay_times, pay_num, last_month_traffic, local_traffic_month, local_caller_time, service1_caller_time, service2_caller_time, age, former_complaint_num, and former_complaint_fee, were scaled into a fixed range between 0 and 1 using normalization or min-max normalization(iv)
In the case of feature engineering on the nine categorical features, i.e., is_mix_service, many_over_bill, is_promise_low_consume, gender, service_type, contract_type, complaint_level, net_service, and current_service, and to ensure their compatibility with the recommender system models, the values of these nine categorical features were transformed into numeric labels. For each categorical feature, Scikit-learn [31] was leveraged to map each category to a numeric value


## 2.1. Related Work

Owing to rapid advancements in smartphone technology, telecommunication service providers must establish comprehensive recommender systems to ensure the provision of high-quality and custom services to different customers. Before the development of DL, conventional ML-based recommender systems were used for telecommunication package recommendations.


### 2.1.1. Conventional ML-Based Recommender Systems

TheK-Nearest Neighbor (KNN) [22] was used to separate the historical datasets of customers into several classes to predict the classification of a new customer. However, the computing cost was high for a large characteristic dimension. Moreover, KNN was sensitive to noise in the dataset, and it exhibited a low fault tolerance.CF, a popular technique for developing recommendation systems, was employed in several applications [23]. In such applications, the behaviors of users based on the data collected in the past were analyzed to determine the connection between users and their interests, which could aid in recommending items to users with similar preferences and support different users’ considerations. A hybrid recommendation approach that combined user- and item-based CF techniques was proposed for mobile product and service recommendations [23, 24]. Although CF achieved efficiency based on similar measurements of users’ interests and recommended items, it was difficult to exploit the cross features completely owing to the lack of deep and effective feature extraction. CF recommendation systems also ad problems associated with cold start and low recommendation accuracy.Although recommender systems are widely used in the telecommunications industry, the methods employed focused on conventional ML-based and CF techniques. Only a few practical or academic cases exist in which state-of-the-art deep CTR models were used for the recommendation of telecommunication packages.


### 2.1.2. DL-Based Recommender Systems

However, the significant success of DL introduced DL-based CTR to the field of recommender systems, which were applied in websites.FMs [25] integrated with DL exhibited outstanding performance and achieved promising results in CTR prediction. However, the two types of DL models proposed, namely, FM-supported neural network and sampling-based neural network, could capture only high-order feature interactions [26]. Wide and deep learning [13], which trained wide linear models and DNNs simultaneously to combine the benefits of memory and generalization for recommender systems, was initially introduced for app recommendation in Google Play. However, it required designing cross-product transformations through expertise feature engineering.PNN [11] attempted to capture high-order feature interactions by involving a product layer between the embedding and hidden layers. Unlike the conventional embedding combined with multilayer perceptrons, PNN explicitly captured the second-order feature correlation in multiple fields.Influenced by wide and deep learning, the DeepFM [17] replaced the “wide” segment of the wide and deep model with FM to learn the low-order features. The “deep” segment was developed using a feedforward neural network to learn high-order features. Both the “wide” and “deep” segments shared identical raw feature inputs. In comparison with the wide and deep model, the DeepFM was an end-to-end learning model, i.e., independent of manual feature engineering. Moreover, the xDeepFM [18] explicitly modeled low- and high-order feature interactions using a novel CIN segment.In summary, the above-mentioned models cannot simultaneously overcome the recommendation challenges faced by telecommunication enterprises with high-dimensional textual features, low- and high-order feature interactions, and unlabeled data. Therefore, we proposed a novel recommender system for telecommunication packages based on DCN, DBN, embedding, and Word2Vec, which provided a potential solution to the recommendation challenges faced by telecommunication enterprises.


## 2.1.1. Conventional ML-Based Recommender Systems

TheK-Nearest Neighbor (KNN) [22] was used to separate the historical datasets of customers into several classes to predict the classification of a new customer. However, the computing cost was high for a large characteristic dimension. Moreover, KNN was sensitive to noise in the dataset, and it exhibited a low fault tolerance.CF, a popular technique for developing recommendation systems, was employed in several applications [23]. In such applications, the behaviors of users based on the data collected in the past were analyzed to determine the connection between users and their interests, which could aid in recommending items to users with similar preferences and support different users’ considerations. A hybrid recommendation approach that combined user- and item-based CF techniques was proposed for mobile product and service recommendations [23, 24]. Although CF achieved efficiency based on similar measurements of users’ interests and recommended items, it was difficult to exploit the cross features completely owing to the lack of deep and effective feature extraction. CF recommendation systems also ad problems associated with cold start and low recommendation accuracy.Although recommender systems are widely used in the telecommunications industry, the methods employed focused on conventional ML-based and CF techniques. Only a few practical or academic cases exist in which state-of-the-art deep CTR models were used for the recommendation of telecommunication packages.


## 2.1.2. DL-Based Recommender Systems

However, the significant success of DL introduced DL-based CTR to the field of recommender systems, which were applied in websites.FMs [25] integrated with DL exhibited outstanding performance and achieved promising results in CTR prediction. However, the two types of DL models proposed, namely, FM-supported neural network and sampling-based neural network, could capture only high-order feature interactions [26]. Wide and deep learning [13], which trained wide linear models and DNNs simultaneously to combine the benefits of memory and generalization for recommender systems, was initially introduced for app recommendation in Google Play. However, it required designing cross-product transformations through expertise feature engineering.PNN [11] attempted to capture high-order feature interactions by involving a product layer between the embedding and hidden layers. Unlike the conventional embedding combined with multilayer perceptrons, PNN explicitly captured the second-order feature correlation in multiple fields.Influenced by wide and deep learning, the DeepFM [17] replaced the “wide” segment of the wide and deep model with FM to learn the low-order features. The “deep” segment was developed using a feedforward neural network to learn high-order features. Both the “wide” and “deep” segments shared identical raw feature inputs. In comparison with the wide and deep model, the DeepFM was an end-to-end learning model, i.e., independent of manual feature engineering. Moreover, the xDeepFM [18] explicitly modeled low- and high-order feature interactions using a novel CIN segment.In summary, the above-mentioned models cannot simultaneously overcome the recommendation challenges faced by telecommunication enterprises with high-dimensional textual features, low- and high-order feature interactions, and unlabeled data. Therefore, we proposed a novel recommender system for telecommunication packages based on DCN, DBN, embedding, and Word2Vec, which provided a potential solution to the recommendation challenges faced by telecommunication enterprises.


## 2.2. Proposed Model

In this section, we describe the architecture of the proposed model, and we explain the process of developing the dataset.


### 2.2.1. Model Overview

Figure1 illustrates the architecture of the proposed model based on the DCN. The model began with an embedding and stacking layer in which the sparse, dense, and converted features were stacked using Word2Vec. Subsequently, the stacked value was fed into the cross and deep network layers in parallel inputs. The two results were combined in the combining layer, and the final output was computed through a single-layer neural network and activated using a sigmoid function. Unlike the original DCN, the Word2Vec model was combined with the DCN to train text features, which solved the problem of the DCN’s inability to process textual information. This enhanced the recommendation accuracy of the proposed model. Moreover, using the DBN network to replace the “deep” segment of the original DCN could exploit the advantages of DBN completely.Figure 1
Architecture of the proposed model based on the DCN.


## 2.2.1. Model Overview

Figure1 illustrates the architecture of the proposed model based on the DCN. The model began with an embedding and stacking layer in which the sparse, dense, and converted features were stacked using Word2Vec. Subsequently, the stacked value was fed into the cross and deep network layers in parallel inputs. The two results were combined in the combining layer, and the final output was computed through a single-layer neural network and activated using a sigmoid function. Unlike the original DCN, the Word2Vec model was combined with the DCN to train text features, which solved the problem of the DCN’s inability to process textual information. This enhanced the recommendation accuracy of the proposed model. Moreover, using the DBN network to replace the “deep” segment of the original DCN could exploit the advantages of DBN completely.Figure 1
Architecture of the proposed model based on the DCN.


## 2.3. Model Analysis

(i)
Input and embedding layerWe classified all the features of the raw dataset into three types: categorical features, continuous integer features (such as times of payment), and decimal numerical features. The categorical features were transformed into a low-dimensional space using embedding for dimensional reduction. These transformed features were represented asXembT. To avoid it being affected by the dimension, continuous integer features and some numerical features in the model were scaled into a fixed range between zero and one using normalization or min-max normalization, and these normalized features were represented as XnormT. Additionally, other numerical features in the model were converted into textual features to be more suitable. The textual features were transformed into embedding vectors using Word2Vec, which was represented as XtextT. Finally, we stacked XembT, XnormT, and XtextT into one vector as follows:
(1)x0=XembT,XnormT,XtextT,and we fed x0 to the cross and deep network layers simultaneously.
(ii)
Cross network layerThe cross network layer comprisesm cross layers, where each cross layer could be calculated as follows:
(2)xj+1=x0xjTwj+bj+xj=fxj,wj,bj+xj,wherej∈0,1,2,⋯,m−1,m∈1,2,3,⋯; xj and xj+1 represent the output of the j-th cross layer and j+1-th cross layer, respectively; and wj and bj denote the weight and bias parameters of the j-th cross layer, respectively. All the aforementioned variables were column vectors. Furthermore, the features of each layer were cross-combined with the previous layer and original features (x0) and then added back to the previous layer. This is similar to the structure of a residual network in which the function f of each layer fits the residual of xj+1−xj. Thus, the gradient dispersion problem caused by the DNN could be solved using this residual network.The output of the last cross layer in the cross network layer was the output of the cross network layer, which was calledxcross.
(iii)
Deep network layer based on the DBNThe deep network was a fully connected feedforward neural network that shared the same input with the cross network. It enabled the model to learn the combined features of higher-order nonlinearity. The DBN was generated by stacking a set of RBMs. The output of the input and embedding layer (x0) was the input of the first RBM in the DBN.Except for the first RBM, the output of each RBM was calculated as follows:(3)hi+1=FWihi+bi,where i∈1,2,⋯,l−2,l−1, l represents the number of RBMs in the DBN, hi+1 and hi represent the output of the i+1-th and i-th RBM, respectively, wi and bi denote the weight and bias parameters of the i-th RBM, respectively, and F indicates the activation function for enhancing the nonlinear capability of the network. The rectified linear unit (ReLU) [27] was used as the activation function in the proposed model owing to its calculation simplicity. Moreover, the convergence speed of ReLU significantly outperformed that of other activation functions, such as sigmoid [28] and tanh [29].For the first RBM,h1=FW1x0+b1, where h1,W1,andb1 represent the output, the weight, and bias of the first RBM.The output of the last RBM is the output of the deep network layer based on the DBN, which is calledxdeep.
(iv)
Combining layerWhenx0 flows through the DCN, xcross and xdeep are generated. In the combining layer, xcross and xdeep are stacked into xcom as follows:
(4)xcom=xcross,xdeep.(v)
Output layerHere, we calculated the output through a perceptron using a sigmoid activation functionδ, which is calculated as follows:
(5)out=δwxcom+b,where w and b represent the weight vector and bias parameter for the combination layer.The loss function was the log loss along with a regularization term:(6)loss=1N∑i=1Nyilogouti+1−yilog1−outi+λ∑lwl2,whereouti represents the value computed from equation (5), yi represents the true labels, N represents the total number of inputs, and λ represents the L2 regularization parameter.


## 2.4. Dataset

The dataset used in this study was provided by the China Unicom Research Labs, and it is available at DataFountain (https://www.datafountain.cn/competitions/311/datasets) in .csv format. The dataset comprises 743,990 samples. Each sample indicated a consumption record, thereby implying that a telecommunication package was currently used by a customer.Table1 shows the dataset comprising 27 fields. From the field description of the dataset used, six fields, i.e., service_type, is_mix_service, many_over_bill, is_promise_low_consume, gender, and complaint_level, are categorical features. Furthermore, through the statistical analysis of each field in the dataset used, three fields, i.e., current_type, contract_type, and net_service, are categorical features. Clearly, after excluding the USERID field and nine categorical features, the other 15 fields are numerical features.Table 1
Specifications of the dataset used in this study.
Field nameField data typeCommentUSERIDVARCHAR2(50)User identitycurrent_type/current_serviceVARCHAR2(500)/service_typeVARCHAR2(10)0: 23G, 1:212C, 2:2G, 3:3G, 4:4Gis_mix_serviceVARCHAR2(10)1: yes, 0: noonline_timeVARCHAR2(50)/1_total_feeNUMBERUnit: Renminbi yuan2_total_feeNUMBERUnit: Renminbi yuan3_total_feeNUMBERUnit: Renminbi yuan4_total_feeNUMBERUnit: Renminbi yuanmonth_trafficNUMBERUnit: megabytesmany_over_billVARCHAR2(500)1: yes, 0: nocontract_typeVARCHAR2(500)ZBG_DIM.DIM_CBSS_ACTIVITY_TYPEcontract_timeVARCHAR2(500)/is_promise_low_consumeVARCHAR2(500)1: yes, 0: nonet_serviceVARCHAR2(500)20AAAAAA-2Gpay_timesNUMBERUnit: timepay_numNUMBERUnit: Renminbi yuanlast_month_trafficNUMBERUnit: megabyteslocal_traffic_monthNUMBERUnit: megabyteslocal_caller_timeNUMBERUnit: minuteservice1_caller_timeNUMBERUnit: minuteservice2_caller_timeNUMBERUnit: minuteGenderVARCHAR2(100)01: male, 02: femaleAgeVARCHAR2(100)/complaint_levelVARCHAR2(1000)1: ordinary, 2: import, 3: significantformer_complaint_numNUMBERUnit: timeformer_complaint_feeNUMBERUnit: minuteFor the dataset to be compatible with the CTR prediction task, it was transformed into binary classification data. In the case of CTR, each sample represents historical click behavior of a specific user on a telecommunication package, which was considered a displayed advertisement. The raw dataset used in this study comprised 11 unique telecommunication packages in the “current_type” categorical feature.Figure2 shows the distribution of 11 unique telecommunication packages in the “current_type” categorical feature. The mapping relationships between the 11 unique telecommunication packages were as follows: {0: 99999825, 1: 90063345, 2: 90109916, 3: 89950166, 4: 89950168, 5: 89950167, 6: 90155946, 7: 99999828, 8: 99999826, 9: 99999827, 10: 99999830}. Evidently, the 11 unique telecommunication packages were not represented equally. In other words, the dataset used was imbalanced. To solve the problem of imbalance, we used the synthetic minority over-sampling technique [30]. We randomly chose five samples from the 11 unique telecommunication packages to construct negative samples by maintaining consistency with the user-side features. Based on the CTR model, a “clicked” column was created in the dataset. Additionally, the click values of the positive and negative samples were “1” and “0,”, respectively.Figure 2
Distribution of the 11 unique telecommunication packages in the “current_type” categorical feature.


## 2.5. Data Wrangling

(i)
Dealing with missing valuesBecause the raw dataset comprised missing values, we used two methods to determine such values from the existing data.(1)
For the numerical features, the datasets were grouped in terms of the current package by first calculating the mean of nonmissing values in a column and then replacing the missing values within each group separately, independent of the other groups. For these 15 numerical features, i.e., online_time, 1_total_fee, 2_total_fee, 3_total_fee, 4_total_fee, month_traffic, contract_time, pay_times, pay_num, last_month_traffic, local_traffic_month, local_caller_time, service1_caller_time, service2_caller_time, and age, their missing values were replaced with the mean corresponding to the feature column. For example, the mean of the nonmissing values of “online_time” in the group of zero telecommunication packages was 74.184570, which was used to replace the missing values within the group of zero telecommunication packages(ii)
In the case of the categorical features, the datasets were grouped by considering the current package and replacing the missing data with frequently occurring values within each group separately. For these 10 categorical features, i.e., service_type, is_mix_service, many_over_bill, contract_type, is_promise_low_consume, net_service, gender, complaint_level, former_complaint_num, and former_complaint_fee, their missing values were replaced with frequently occurring values corresponding to the feature column. For example, frequently occurring values of gender within the group, 0 telecommunication packages, was one; the number of one was 15,627, and the number of two was 4,723, which was used to replace the missing values within the group of zero telecommunication packages(iii)
In the case of scaling using min-max normalization on the numerical features unaffected by the dimension, 17 features, i.e., online_time, 1_total_fee, 2_total_fee, 3_total_fee, 4_total_fee, month_traffic, contract_time, pay_times, pay_num, last_month_traffic, local_traffic_month, local_caller_time, service1_caller_time, service2_caller_time, age, former_complaint_num, and former_complaint_fee, were scaled into a fixed range between 0 and 1 using normalization or min-max normalization(iv)
In the case of feature engineering on the nine categorical features, i.e., is_mix_service, many_over_bill, is_promise_low_consume, gender, service_type, contract_type, complaint_level, net_service, and current_service, and to ensure their compatibility with the recommender system models, the values of these nine categorical features were transformed into numeric labels. For each categorical feature, Scikit-learn [31] was leveraged to map each category to a numeric value


## 3. Results and Discussion

In this section, we evaluate the performance of our proposed model using the dataset provided by the China Unicom Research Labs. The experiments were conducted using a high-performance computer equipped with dual Intel Xeon E5-2660 v4 CPUs running at 2.00 GHz, with 56 cores, 630 GB memory, a hard disk with a capacity of 1 TB, and three Tesla P40 graphics processing units (GPUs). The system runs on the Ubuntu 18.04.1 long term support (LTS) release, with Python 3.7.11, Scikit-learn Version 1.0.1, and TensorFlow Version 1.15.0..


### 3.1. Comparison of Models and Their Hyperparameters

We compared the performance of the proposed model to that of four other models: the DBN, DCN, DeepFM, and DNN. All the models used the adaptive moment estimation (Adam) [32] optimizer, with the learning rate and minibatch size set to 0.001 and 1,000, respectively. Furthermore, because dropout was used to prevent the neural networks from overfitting [33], the dropout rates of all the models were set to 0.2. The Adam optimizer addressed the sparseness faced by the models, whereas the dropout prevented the neural networks from overfitting.Proposed model: the hyperparameters of the proposed model are listed in Table2. The dimensions of word embedding were set to eight, with three and four cross and hidden layers in the deep network. Additionally, the number of hidden neurons in each hidden layer was set to 32, and the number of RBMs was set to four, with a network structure of [200, 100, 50, 20].Table 2
Hyperparameters of the proposed model.
Parameter nameParameter valueEmbedding dimension8Cross layer size3Deep layer size4Number of neurons in each layer32Dropout rate0.2Number of RBM4Network structure of RBMs[200, 100, 50, 20]Minibatch size1000Learning rate0.001DBN: the hyperparameters of the DBN are listed in Table3. The value of k in CK−k was set to two, and four RBMs were used, with the network structure set to [200, 100, 50, 20].Table 3
Hyperparameters of the DBN.
Parameter nameParameter valueValue ofk used in CD−k2Dropout rate0.2Number of RBM4Network structure of RBMs[200, 100, 50, 20]Minibatch size1000Learning rate0.001DCN: the hyperparameters of the DCN are listed in Table4. Similar to the proposed model, the dimensions of word embedding were set to eight, with three and four cross and hidden layers, respectively, in the deep network and 32 hidden neurons in each hidden layer.Table 4
Hyperparameters of the DCN.
Parameter nameParameter valueEmbedding dimension8Cross layer size3Deep layer size4Number of neurons in each layer32Dropout rate0.2Minibatch size1000Learning rate0.001DeepFM: the hyperparameters of the DeepFM are listed in Table5, and the dimensions of word embedding were set to eight, with four hidden layers in the deep network and 32 hidden neurons in each hidden layer.Table 5
Hyperparameters of the DeepFM.
Parameter nameParameter valueEmbedding dimension8Deep layer size4Number of neurons in each layer32Dropout rate0.2Minibatch size1000Learning rate0.001DNN: the hyperparameters of the DNN are listed in Table6, and the deep network comprised four hidden layers, with the network structure set to [200, 100, 50, 20].Table 6
Hyperparameters of the DNN.
Parameter nameParameter valueDeep layer size4Network structure[200, 100, 50, 20]Dropout rate0.2Minibatch size1000Learning rate0.001


### 3.2. Evaluation Methods

In the experiments, we used the area under the ROC curve (AUC) [23], cross entropy (log loss) [34], and recall as the evaluation metrics. The upper bound of AUC was set to one, whereby a larger value indicated improved performance. The log loss metric was used to measure the distance between two distributions, and a smaller log loss value indicated enhanced performance. The recall metric was used to determine the accuracy of the evaluation model in identifying the relevant data, whose upper bound was set to one, whereby a larger value indicated improved performance. All the experiments were repeated 10 times to determine the average results.Figure3 illustrates the AUC metrics. As shown in the figure, the AUC metrics obtained through the proposed model were larger than those obtained using other models. Furthermore, the values of the AUC metrics increased rapidly as the number of epochs increased, and they subsequently attained the maximum optimal values.Figure 3
Change curve of AUC metric versus epochs.Figure4 illustrates the log loss metrics, and it shows that the log loss metrics obtained through the proposed model were less than those of the other models. Moreover, the values of the log loss metrics decreased rapidly with the increase in the number of epochs, and they attained the minimum optimal values.Figure 4
Change curve of log loss metric versus epochs.Figure5 illustrates the recall metrics, and it shows that the recall metrics obtained through the proposed model were higher than those of the other models. The best values of the AUC, log loss, and recall metrics obtained using all the models are listed in Table 7.Figure 5
Change curve of recall metric versus epochs.Table 7
Best log loss, AUC, and recall metrics of different models after 20 epochs.
Model metricDBNDNNDeepFMDCNProposed modelLog loss0.14320.16020.13480.12020.1061AUC0.94130.92330.96020.97130.9814Recall0.94230.92130.96140.97050.9854Obviously, our proposed model outperformed DCN, DeepFM, DNN, and DBN; DCN outperformed DeepFM, DNN, and DBN; DeepFM outperformed DNN and DBN; DBN outperformed DNN. First, compared to other models that do not consider textual feature information, our proposed model can be used to train textual information, and when it is combined with the Word2Vec model, it can be used to extract richer textual information. This approach resulted in the enhanced performance of our proposed model compared to that of the other models. Second, unlike the DBN and DNN, our proposed model considered the extraction of high-level features and the impact of low-level features, thereby ensuring that our proposed model achieved improved performance. Third, unlike the DeepFM and DCN, our proposed model used the DBN rather than the DNN. This implied that our proposed model was less likely to encounter the local optimal situation. Therefore, we trained our proposed model in the deep network layer while considering the global optimal situation.


## 3.1. Comparison of Models and Their Hyperparameters

We compared the performance of the proposed model to that of four other models: the DBN, DCN, DeepFM, and DNN. All the models used the adaptive moment estimation (Adam) [32] optimizer, with the learning rate and minibatch size set to 0.001 and 1,000, respectively. Furthermore, because dropout was used to prevent the neural networks from overfitting [33], the dropout rates of all the models were set to 0.2. The Adam optimizer addressed the sparseness faced by the models, whereas the dropout prevented the neural networks from overfitting.Proposed model: the hyperparameters of the proposed model are listed in Table2. The dimensions of word embedding were set to eight, with three and four cross and hidden layers in the deep network. Additionally, the number of hidden neurons in each hidden layer was set to 32, and the number of RBMs was set to four, with a network structure of [200, 100, 50, 20].Table 2
Hyperparameters of the proposed model.
Parameter nameParameter valueEmbedding dimension8Cross layer size3Deep layer size4Number of neurons in each layer32Dropout rate0.2Number of RBM4Network structure of RBMs[200, 100, 50, 20]Minibatch size1000Learning rate0.001DBN: the hyperparameters of the DBN are listed in Table3. The value of k in CK−k was set to two, and four RBMs were used, with the network structure set to [200, 100, 50, 20].Table 3
Hyperparameters of the DBN.
Parameter nameParameter valueValue ofk used in CD−k2Dropout rate0.2Number of RBM4Network structure of RBMs[200, 100, 50, 20]Minibatch size1000Learning rate0.001DCN: the hyperparameters of the DCN are listed in Table4. Similar to the proposed model, the dimensions of word embedding were set to eight, with three and four cross and hidden layers, respectively, in the deep network and 32 hidden neurons in each hidden layer.Table 4
Hyperparameters of the DCN.
Parameter nameParameter valueEmbedding dimension8Cross layer size3Deep layer size4Number of neurons in each layer32Dropout rate0.2Minibatch size1000Learning rate0.001DeepFM: the hyperparameters of the DeepFM are listed in Table5, and the dimensions of word embedding were set to eight, with four hidden layers in the deep network and 32 hidden neurons in each hidden layer.Table 5
Hyperparameters of the DeepFM.
Parameter nameParameter valueEmbedding dimension8Deep layer size4Number of neurons in each layer32Dropout rate0.2Minibatch size1000Learning rate0.001DNN: the hyperparameters of the DNN are listed in Table6, and the deep network comprised four hidden layers, with the network structure set to [200, 100, 50, 20].Table 6
Hyperparameters of the DNN.
Parameter nameParameter valueDeep layer size4Network structure[200, 100, 50, 20]Dropout rate0.2Minibatch size1000Learning rate0.001


## 3.2. Evaluation Methods

In the experiments, we used the area under the ROC curve (AUC) [23], cross entropy (log loss) [34], and recall as the evaluation metrics. The upper bound of AUC was set to one, whereby a larger value indicated improved performance. The log loss metric was used to measure the distance between two distributions, and a smaller log loss value indicated enhanced performance. The recall metric was used to determine the accuracy of the evaluation model in identifying the relevant data, whose upper bound was set to one, whereby a larger value indicated improved performance. All the experiments were repeated 10 times to determine the average results.Figure3 illustrates the AUC metrics. As shown in the figure, the AUC metrics obtained through the proposed model were larger than those obtained using other models. Furthermore, the values of the AUC metrics increased rapidly as the number of epochs increased, and they subsequently attained the maximum optimal values.Figure 3
Change curve of AUC metric versus epochs.Figure4 illustrates the log loss metrics, and it shows that the log loss metrics obtained through the proposed model were less than those of the other models. Moreover, the values of the log loss metrics decreased rapidly with the increase in the number of epochs, and they attained the minimum optimal values.Figure 4
Change curve of log loss metric versus epochs.Figure5 illustrates the recall metrics, and it shows that the recall metrics obtained through the proposed model were higher than those of the other models. The best values of the AUC, log loss, and recall metrics obtained using all the models are listed in Table 7.Figure 5
Change curve of recall metric versus epochs.Table 7
Best log loss, AUC, and recall metrics of different models after 20 epochs.
Model metricDBNDNNDeepFMDCNProposed modelLog loss0.14320.16020.13480.12020.1061AUC0.94130.92330.96020.97130.9814Recall0.94230.92130.96140.97050.9854Obviously, our proposed model outperformed DCN, DeepFM, DNN, and DBN; DCN outperformed DeepFM, DNN, and DBN; DeepFM outperformed DNN and DBN; DBN outperformed DNN. First, compared to other models that do not consider textual feature information, our proposed model can be used to train textual information, and when it is combined with the Word2Vec model, it can be used to extract richer textual information. This approach resulted in the enhanced performance of our proposed model compared to that of the other models. Second, unlike the DBN and DNN, our proposed model considered the extraction of high-level features and the impact of low-level features, thereby ensuring that our proposed model achieved improved performance. Third, unlike the DeepFM and DCN, our proposed model used the DBN rather than the DNN. This implied that our proposed model was less likely to encounter the local optimal situation. Therefore, we trained our proposed model in the deep network layer while considering the global optimal situation.


## 4. Conclusion

In this study, to provide a potential solution to the recommendation challenges faced by telecommunication enterprises with high-dimensional textual features, low- and high-order feature interactions, and unlabeled data, we proposed a system for recommending telecommunication packages using the DCN, DBN, embedding, and Word2Vec models. The experiments’ results indicated that our proposed model outperformed the existing models, namely, the DBN, DCN, DeepFM, and DNN, in terms of AUC, log loss, and recall. The aspects contributing to the performance improvement of our proposed model are as follows: (1) the addition of the new embedding features generated by applying Word2Vec on the textual features originating from the decimal features enabled our proposed model to learn additional low- and high-order feature interactions. (2) The replacement of the deep segments of the original DCN with those of the DBN transformed our proposed model into an efficient generative model that considered unlabeled data using powerful predictive classification functions. In our future studies, we will investigate distributed training using multiple GPUs and compare the performance of our proposed model to that of other recommender systems based on conventional machine learning algorithms and deep learning algorithms.



---

*Source: 2100841-2022-04-19.xml*