Decile Analysis in Python

## 1. Motivation
Using the following Kaggle purchase data to do targeting and decile analysis.
https://www.kaggle.com/vijayuv/onlineretail

## 2. Procedures
1. Read retail_data and perform preprocessing
2. Calculate the sum of `TotalPrice` for each `CustomerID`
3. Sort the dataframe in descending order based on the `TotalPrice` per `CustomerID`.
4. Divide the customers into n `group`s (where `𝑛` is a natural number that does not exceed the number of data and does not have the same value in the quartile).
5. Calculate what percentage of sales the each group's `TotalPrice` accounts for. (decile analysis)
6. Sort the groups by their calculated percentages in descending order.
7. Summarize the above steps as a function.
8. Create a cumulative distribution plot to visualize the contribution of each group to the total TotalPrice.

## 3. Steps
### 3-1. Read retail_data and perform preprocessing

#### Code
```Python
#Load the libraries
import pandas as pd
import numpy as np

# Data loading and preprocessing *Takes a lot of time
file_url = "http://archive.ics.uci.edu/ml/machine-learning-databases/00352/Online%20Retail.xlsx"
Excel_data = pd.ExcelFile(file_url)
retale_data = Excel_data.parse('Online Retail')

# Convert InvoiceNo to string type
retail_data['cancel_flg'] = retail_data.InvoiceNo.map(lambda x:str(x)[0])

# Target invoices whose InvoiceNo starts with 5 and whose ID is not Null
retail_data_1 = retail_data[(retail_data.cancel_flg == '5') & (retail_data.CustomerID.notnull())]

# Prepare main dataframe
df = retail_data_1.assign(TotalPrice=retail_data_1.Quantity * retail_data_1.UnitPrice)

```

#### Result

```Python
df.info()
```

![Screenshot 2024-05-25 at 18.03.51.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/96217/7a9d1dae-1710-e552-3b06-f7badbf32a2f.png)

```Python
df.describe()
```
![Screenshot 2024-05-25 at 18.05.07.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/96217/c9f73ba3-94c9-9fc1-1d6f-d01fabdf3984.png)

### 3-2. Calculate the sum of `TotalPrice` for each `CustomerID`

```Python
sum = df.groupby('CustomerID')[['TotalPrice']].sum()
sum_sorted = sum.sort_values(by='TotalPrice', ascending=False)
```
#### Result

```Python
sum
```
![Screenshot 2024-05-25 at 18.07.57.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/96217/0f522403-a90c-e5ec-703d-f401ffe25134.png)


### 3-3. Sort the dataframe in descending order based on the `TotalPrice` per `CustomerID`.


```Python
sum_sorted = sum.sort_values(by='TotalPrice', ascending=False)
```

#### Result

```Python
sum_sorted
```
![Screenshot 2024-05-25 at 18.18.46.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/96217/bd41b668-7ce7-ef54-f794-2e7a950d020c.png)


### 3-4. Divide the customers into n `group`s .
where `𝑛` is a natural number that does not exceed the number of data and does not have the same value in the quartile

```Python
n = 10
group_labels = []
for i in range(n):
    group_labels.append('{0}th'.format(i+1))
sum_sorted['group'] = pd.qcut(sum_sorted['TotalPrice'], n, labels=group_labels, duplicates='drop')
```
#### Result

```Python
sum_sorted
```
![Screenshot 2024-05-25 at 18.11.19.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/96217/91b40083-04d4-17d3-ce6b-347054b4bb97.png)


### 3-5. Calculate what percentage of sales the each group's `TotalPrice` accounts for. (decile analysis)

```Python
# Find what percentage of the total purchase price each group represents
proportions = sum_sorted.groupby('group')['TotalPrice'].sum() / sum_sorted['TotalPrice'].sum()
```
#### Result

```Python
proportions
```
![Screenshot 2024-05-25 at 18.22.16.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/96217/bfbe25a3-d3b6-a027-2fa3-7a340af3cd2a.png)

### 3-6. Sort the groups by their calculated percentages in descending order.
```Python
# Sort in ascending order by proportion height
result = proportions.sort_values(ascending=False)
```
#### Result
```Python
result
```
![Screenshot 2024-05-25 at 18.22.27.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/96217/4a469a5c-4754-8ed8-144b-ddab0cf6889e.png)

```Python
# Rank groups by percentage height and reassign indexes by rank
result.index = result.sort_values().index.map(lambda x: 'decile'+str(x)[0])
```
#### Result
```Python
result
```
![Screenshot 2024-05-25 at 18.22.36.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/96217/f133f5a7-ebad-db23-6998-9871319d17a6.png)

### 3-7. Summarize the above steps as a function.
```Python
def function(df, n):
    sum = df.groupby('CustomerID')[['TotalPrice']].sum()
    sum_sorted = sum.sort_values(by='TotalPrice', ascending=False)
    group_labels = []
    for i in range(n):
        group_labels.append('{0}th'.format(i+1))
    sum_sorted['group'] = pd.qcut(sum_sorted['TotalPrice'], n, labels=group_labels, duplicates='drop')
    proportions = sum_sorted.groupby('group')['TotalPrice'].sum() / sum_sorted['TotalPrice'].sum()
    result = proportions.sort_values(ascending=False)
    result.index = result.sort_values().index.map(lambda x: str(x)[0])
    return result
```

#### Result
```Python
function(df, 10)
```
![Screenshot 2024-05-25 at 18.28.58.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/96217/05f83e95-a29b-7a84-d983-17e7d9eca472.png)

### 3-8. Create a cumulative distribution plot to visualize the contribution of each group to the total `TotalPrice`.

```Python
# Visualize with Matplotlib
import matplotlib.pyplot as plt
N = 10
data = function(df, N)

fig, ax1 = plt.subplots(figsize=(6,4))

# Number of segments determines the abscissa
data_num = len(data)

# Cumulative sum
cum_per = np.cum(data)

# Bar chart
ax1.bar(range(data_num), data)
ax1.set_xticks(range(data_num))

# line graph
ax2 = ax1.twinx()
ax2.plot(range(data_num), cum_per, c="k", marker="o")
ax2.set_ylim([0, 1])
ax2.grid(True, which='both', axis='y')
```
#### Result

![Screenshot 2024-05-25 at 18.32.34.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/96217/e36f5cb9-4ed4-550a-f382-591cd000634a.png)


#### Result (where `N`=20)

![Screenshot 2024-05-25 at 18.30.53.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/96217/d351cac8-880a-5f9d-66b3-5b9e7771f84c.png)



## Acknowledgement
https://archive.ics.uci.edu/ml/machine-learning-databases/00352/Online%20Retail.xlsx

Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.

Daqing Chen, Sai Liang Sain, and Kun Guo, Data mining for the online retail industry: A case study of RFM model-based customer segmentation using data mining, Journal of Database Marketing and Customer Strategy Management, Vol. 19, No. 3, pp. 197â€“208, 2012 (Published online before print: 27 August 2012. doi: 10.1057/dbm.2012.17).