Mining Telecom Product Recommendations | by Meghana Holemadlu Murthy | Towards Data Science

In the current age of information overload, it is becoming increasingly hard for people to find relevant content. Recommender systems have been introduced to help people in retrieving potentially useful information from a huge set of choices. It is a system that suggests products, services, information to users based on the analysis of data and has become increasingly significant in solving the information explosion problem.
In recent years, significant steps have been taken in the direction of providing personalized recommendation services for a wide variety of web-based applications in e-commerce, e-learning, and e-governance. However, this system is surprisingly not present in the telecom sector as of today, though it is a necessity. Hence, this blog explores the need for recommender systems in telcos to assist its customers in selecting the most appropriate telecom plans/products online. The blog ends with the development and implementation of a personalized recommendation system for telecom products/services by mining association rules from data to find the right telcos packages/plans/products for a new customer.
While artificial intelligence (AI) has unquestionable power to transform the telecom industry, it is still very much a nascent field. Yet I believe telcos are uniquely well-positioned to take advantage of AI technology, largely because they are already used to dealing with huge volumes of data on which AI and machine learning (ML) rely. Since this isn’t a passing fad, it’s important for telcos to play the long game with AI and consider less-explored domains that could unlock new revenue streams.
AI and ML require vast quantities of data to function correctly, but the data must be of sufficient quality, too. If the right amount of good-quality data is not available, decisions recommended by an AI or ML model are unlikely to be accurate or successful.
On a positive note, telcos do have access to large quantities of data, perhaps more than companies in most other industries. However, the struggle lies in capturing the right dataset and storing them with the degree of granularity needed for ML.
It is one thing to have high-quality data collected in a traditional way, but it is another to get this data into a specific way which you might need to solve a use case. Most telcos operator’s web page today, consists of a vast number of choices, which in turn becomes increasingly hard for a customer to find his/her desired choice of product.
Don’t you think it would be substantial if the same web page could automatically give you recommendations on various add-on products/bundles based on choices made by existing customers in the past?
Recommendation systems can be fine-tuned to cater to user profiles/preferences based on the choices of other users. The advantage of this system is to suggest the right items (base plans or add-on products) to particular users, based on their explicit and implicit preferences by applying an information filtering algorithm. The algorithm that I have used for implementation is called Apriori.
Apriori Algorithm — It is simply the study of customer transaction database to determine dependencies between purchases of different plans/products.
In other words, it is the study of “what goes with what”.
The availability of detailed information on customer transactions has led to the development of techniques that automatically look for associations between items that are stored in the database. Example — data collected about customer transactions regarding each plan/base package along with the add-on products are captured by the network operator in their database. Such databases consist of a large number of transaction records where each record lists either the base plan or the add-on product purchased by the customer.
By performing basic EDA on customer transaction data using Python, it is very likely to observe that there are certain base packages/plans that appear more often than the rest. Hence it would be a good idea to display a popular list of base packages as a first step. Given all the data is stored in a pandas dataframe ‘df’, we can get the list of popular base-packages/plans using the code below.
df["BASE_PACKAGE_NAME"].value_counts()
Getting this data into a form where each record lists all the items bought by the user wouldn’t be much of a challenge using Python.
At this stage, we would be interested to know if certain groups of items are consistently purchased together, which can be found with the help of the Apriori algorithm.
#Convert the dataframe to a listrecords = df.values.tolist()#Generate association rules with required #minimum support, confidence and liftfrom apyori import aprioriassociation_rules = apriori(records, min_support=0.3346, min_confidence=0.20, min_lift=3, min_length=3)association_results = list(association_rules)
There are four performance metrics within the Apriori algorithm. These parameters are used to exclude association rules in the result that have support and confidence lower than the minimum support, confidence, lift, or length, respectively.
min_support — A float value between 0 and 1; used to select the items with support value greater than the value specified in the parameter.
min_support — A float value between 0 and 1; used to select the items with support value greater than the value specified in the parameter.
2. min_confidence — A float value between 0 and 1; used to filter those rules that have confidence greater than the confidence threshold specified in the parameter.
3. min_lift — This specifies the minimum lift value for the shortlisted rules.
4. min_length — This specifies the minimum number of items that you want to see in your association rules itemsets.
The optimum values for min_support, min_confidence, and min_lift arguments can be set by trying out different values and checking whether they are producing valid association between the itemsets or not.
Obtaining association rules in a comprehensible format could be a challenge; here’s how I overcame that. Note that ‘Predecessor’ is an item found within the data, whereas ‘Successor’ is an item found in combination with the Predecessor.
#Create a dataframe with all the necessary informationdf = pd.DataFrame(columns=('Items','Predecessor','Successor','Support','Confidence','Lift'))Support =[]Confidence = []Lift = []Items = []Predecessor= []Successor=[]for RelationRecord in association_results:    for ordered_stat in RelationRecord.ordered_statistics:        Support.append(RelationRecord.support)        Items.append(RelationRecord.items)        Predecessor.append(ordered_stat.items_base)        Successor.append(ordered_stat.items_add)        Confidence.append(ordered_stat.confidence)        Lift.append(ordered_stat.lift)df['Items'] = list(map(set, Items))                                   df['Predecessor'] = list(map(set, Predecessor))df['Successor'] = list(map(set, Successor))df['Support'] = Supportdf['Confidence'] = Confidencedf['Lift']= Lift#Sort the dataframe based on confidence df.sort_values(by ='Confidence', ascending = False, inplace = True)
Providing the right recommendations would be by picking the items in the ‘Successor’ column given that the ‘Predecessor’ column value is equal to the base-package name selected by the customer from the popular package list.
#Picking the records for a particular plan/base_package that has been selected by the customer on the GUIpackage_name = "1GB/Day(24 day Validity)"index=[]for key, value in df["Predecessor"].iteritems():     if len(value) == 1 and package_name in value:        print(key,value)        index.append(key)        recommendations = df[df.index.isin(index)]
And voila!
Note that the frequently bought percentage is the measure of confidence parameter, which is computed by the Apriori algorithm as stated earlier.
Being a part of the telecom industry for almost 5 years, I have seen that any AI initiative can have various technical and organizational questions; which is not very surprising as AI is such a pie-in-the-sky thing that you need to give the organization something tangible. Hence, making the leap from concept to live project or, making the move from piloting AI proof-of-concept projects to deploying live is a further challenge. The association-rule mining approach described in this blog proves the benefits of automated affinity analysis and is a fundamental business use-case for telcos in my opinion. It is critically important to have a clear understanding of what would constitute success from such an investment.
Taking a step further, the above implementation is limited to provide recommendations for new customers subscribing to the network via web-GUI/kiosk. The model can be improved by providing recommendations to existing/new customers by adding a clustering algorithm in the initial stage; which would segment customers based on age, salary, geographical data, etc. Additionally, some shortcomings of the Apriori algorithm are the profusion of rules that are generated and the fact that rare combinations of itemsets tend to be ignored, as they do not meet the minimum support requirements. Using a more robust algorithm would help to resolve this issue.
Thank you for reading!
[1] https://www.appliedaicourse.com/
[2] Telcos and AI: What it will take to create an AI-Driven Telecoms Industry — May 2019 by STL Partners, supported by Nokia.
[3] Research Article — Improving Top-NRecommendation Performance Using Missing Data by Beijing Research Center for Information Technology in Agriculture and School of Computer Science and Technology, Beijing.
[4] Research Article — A Recommender System Architecture for Predictive Telecom Network Management by IEEE Communications Magazine.