Python | CAP - Cumulative Accuracy Profile analysis - GeeksforGeeks

09 Sep, 2021
CAP popularly called the ‘Cumulative Accuracy Profile’ is used in the performance evaluation of the classification model. It helps us to understand and conclude about the robustness of the classification model. In order to visualize this, three distinct curves are plotted in our plot: 
A random plotA plot obtained by using a SVM classifier or a random forest classifierA perfect plot( an ideal line)
A random plot
A plot obtained by using a SVM classifier or a random forest classifier
A perfect plot( an ideal line)
We are working the DATA to understand the concept. 
Code : Loading dataset. 
Python3
# importing librariesimport pandas as pdimport seaborn as snsimport matplotlib.pyplot as pltimport numpy as np # loading datasetdata = pd.read_csv('C:\\Users\\DELL\\Desktop\\Social_Network_Ads.csv') print ("Data Head : \n\n", data.head())
Output : 
Data Head : 

     User ID  Gender  Age  EstimatedSalary  Purchased
0  15624510    Male   19            19000          0
1  15810944    Male   35            20000          0
2  15668575  Female   26            43000          0
3  15603246  Female   27            57000          0
4  15804002    Male   19            76000          0
Code : Data Input Output. 
Python3
# Input and Outputx = data.iloc[:, 2:4]y = data.iloc[:, 4] print ("Input : \n", x.iloc[0:10, :])
Output : 
Input : 
    Age  EstimatedSalary
0   19            19000
1   35            20000
2   26            43000
3   27            57000
4   19            76000
5   27            58000
6   27            84000
7   32           150000
8   25            33000
9   35            65000
Code : Splitting dataset for training and testing. 
Python3
# splitting datafrom sklearn.model_selection import train_test_splitx_train, x_test, y_train, y_test = train_test_split(        x, y, test_size = 0.3, random_state = 0)
Code : Random Forest Classifier 
Python3
# classifierfrom sklearn.ensemble import RandomForestClassifierclassifier = RandomForestClassifier(n_estimators = 400) # trainingclassifier.fit(x_train, y_train) # predictingpred = classifier.predict(x_test)
Code : Finding the classifier accuracy. 
Python3
# Model Performancefrom sklearn.metrics import accuracy_scoreprint("Accuracy : ", accuracy_score(y_test, pred) *  100)
Output : 
Accuracy :  91.66666666666666
The random plot is made under the assumption that we have plotted the total number of points ranging from 0 to the total number of data points in the dataset. The y-axis has been kept as the total number of points for which the dependent variable from our dataset has the outcome as 1. The random plot can be understood like a linearly increasing relationship. An example is a model that predicts whether a product is bought (positive outcome) by each individual from a group of people (classifying parameter) based on factors such as their gender, age, income etc. If group members would be contacted at random, the cumulative number of products sold would rise linearly toward a maximum value corresponding to the total number of buyers within the group. This distribution is called the “random” CAP. 
Code : Random Model  
Python3
# code for the random plotimport matplotlib.pyplot as pltimport numpy as np # length of the test datatotal = len(y_test) # Counting '1' labels in test dataone_count = np.sum(y_test) # counting '0' labels in test datazero_count = total - one_count plt.figure(figsize = (10, 6)) # x-axis ranges from 0 to total people contacted# y-axis ranges from 0 to the total positive outcomes. plt.plot([0, total], [0, one_count], c = 'b',         linestyle = '--', label = 'Random Model')plt.legend()
Output : 
Code : Random forest classification algorithm is applied to the dataset for the random classifier line plot.  
Python3
lm = [y for _, y in sorted(zip(pred, y_test), reverse = True)]x = np.arange(0, total + 1)y = np.append([0], np.cumsum(lm))plt.plot(x, y, c = 'b', label = 'Random classifier', linewidth = 2)
Output : 
Explanation: pred is the prediction made by the random classifier. We zip the prediction and test values and sort it in the reverse order so that higher values come first and then the lower values. We extract only the y_test values in an array and store it in lm. np.cumsum() creates an array of values while cumulatively adding all previous values in the array to the present value. The x-values will be ranging from 0 to the total + 1. We add one to the total cause arange() does not include one to the array and we want the x-axis to range from 0 to the total. 
We then plot the perfect plot(or the ideal line). A perfect prediction determines exactly which group members will buy the product, such that the maximum number of products sold will be reached with a minimum number of calls. This produces a steep line on the CAP curve that stays flat once the maximum is reached (contacting all other group members will not lead to more products sold), which is the “perfect” CAP.  
Python3
plt.plot([0, one_count, total], [0, one_count, one_count],         c = 'grey', linewidth = 2, label = 'Perfect Model')
Output : 
Explanation: A perfect model finds positive outcomes in the same number of tries as the number of positive outcomes. We have total of 41 positive outcomes in our dataset and so at exactly 41, the maximum is achieved. 
In any case, our classifier algorithm should not produce a line that lies under the random line. It is considered to be a really bad model in that case. Since the plotted classifier line is close to the ideal line we can say that our model is a really good fit. Take the area under the perfect plot and call it aP. Take the area under the prediction model and call it aR. Then take the ratio as aR/aP. This ratio is called the Accuracy Rate. The closer is the value to 1, the better the model. This is one way to analyze it.
Another way to analyze it would be to project a line from about 50% from the axis on the prediction model and project it on the y-axis. Let us say that we obtain the projection value as X%.  
-> 60%        : it is a really bad model
-> 60%<X<70%  : it is still a bad model but better than the first case obviously
-> 70%<X<80%  : it is a good model
-> 80%<X<90%  : it is a very good model
-> 90%<X<100% : it is extraordinarily good and might be one of the overfitting cases.
So according to this analysis, we can determine how accurate our model is. 
Reference:- wikipedia.org  
Akanksha_Rai
simranarora5sos
Machine Learning
Python
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
ML | Linear Regression
Reinforcement learning
Decision Tree
Activation functions in Neural Networks
Decision Tree Introduction with example
Read JSON file using Python
Adding new column to existing DataFrame in Pandas
Python map() function
How to get column names in Pandas dataframe