Q- now i want to get the context from my notebook so far ? 

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns




import pandas as pd


path = "data/data.xlsx"
df = pd.read_excel(path)
df = pd.DataFrame(df)
df.head()


df.shape


df.info()



<class 'pandas.core.frame.DataFrame'>
RangeIndex: 7043 entries, 0 to 7042
Data columns (total 33 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   CustomerID         7043 non-null   object 
 1   Count              7043 non-null   int64  
 2   Country            7043 non-null   object 
 3   State              7043 non-null   object 
 4   City               7043 non-null   object 
 5   Zip Code           7043 non-null   int64  
 6   Lat Long           7043 non-null   object 
 7   Latitude           7043 non-null   float64
 8   Longitude          7043 non-null   float64
 9   Gender             7043 non-null   object 
 10  Senior Citizen     7043 non-null   object 
 11  Partner            7043 non-null   object 
 12  Dependents         7043 non-null   object 
 13  Tenure Months      7043 non-null   int64  
 14  Phone Service      7043 non-null   object 
 15  Multiple Lines     7043 non-null   object 
 16  Internet Service   7043 non-null   object 
 17  Online Security    7043 non-null   object 
 18  Online Backup      7043 non-null   object 
 19  Device Protection  7043 non-null   object 
 20  Tech Support       7043 non-null   object 
 21  Streaming TV       7043 non-null   object 
 22  Streaming Movies   7043 non-null   object 
 23  Contract           7043 non-null   object 
 24  Paperless Billing  7043 non-null   object 
 25  Payment Method     7043 non-null   object 
 26  Monthly Charges    7043 non-null   float64
 27  Total Charges      7043 non-null   object 
 28  Churn Label        7043 non-null   object 
 29  Churn Value        7043 non-null   int64  
 30  Churn Score        7043 non-null   int64  
 31  CLTV               7043 non-null   int64  
 32  Churn Reason       1869 non-null   object 
dtypes: float64(3), int64(6), object(24)
memory usage: 1.8+ MB


"""
Notice : Total Charges column has a object data type but it is numerical data ,
dp let's change data type for this column


"""
df['Total Charges'] = pd.to_numeric(df['Total Charges'], errors='coerce')

df.describe()


from typing import Iterable, List
def calculate_null_percentage(column: Iterable) -> float:
    """
    Compute the percentage of null values in a dataframe column.
   
    Args:
    column (Iterable): A column from a dataframe
   
    Returns:
    float: The percentage of null values in the column
    """
    total_count = sum(1 for _ in column)
    null_count = sum(1 for value in column if pd.isna(value))
    return (null_count / total_count) * 100 if total_count > 0 else 0


# Now we have calculate_null_percentage(column) as a higher-order function
# We can use it in any dataframe column
# Leveraging the higher-order function principle
# We can use it in any dataframe column
# Type hints have been added to it
percentage_list: List[float] = df.apply(lambda x: calculate_null_percentage(x), axis=0)
percentage_list




CustomerID            0.000000
Count                 0.000000
Country               0.000000
State                 0.000000
City                  0.000000
Zip Code              0.000000
Lat Long              0.000000
Latitude              0.000000
Longitude             0.000000
Gender                0.000000
Senior Citizen        0.000000
Partner               0.000000
Dependents            0.000000
Tenure Months         0.000000
Phone Service         0.000000
Multiple Lines        0.000000
Internet Service      0.000000
Online Security       0.000000
Online Backup         0.000000
Device Protection     0.000000
Tech Support          0.000000
Streaming TV          0.000000
Streaming Movies      0.000000
Contract              0.000000
Paperless Billing     0.000000
Payment Method        0.000000
Monthly Charges       0.000000
Total Charges         0.156183
Churn Label           0.000000
Churn Value           0.000000
Churn Score           0.000000
CLTV                  0.000000
Churn Reason         73.463013
dtype: float64

When dealing with missing values in your dataset, especially in a critical numerical column like "Total Charges", it's essential to approach the issue methodically to ensure the integrity and quality of your analysis. Here's a comprehensive guide on how to handle the approximately 15.6% missing values in the "Total Charges" column.
1. Understand the Nature of Missing Values
Before deciding on a strategy to handle missing data, it's crucial to understand why these values are missing. This understanding can guide you toward the most appropriate imputation method.
a. Inspect Missing Data Patterns
# Check the number of missing values in 'Total Charges'
missing_total_charges = df['Total Charges'].isnull().sum()
print(f"Missing 'Total Charges': {missing_total_charges} out of {df.shape[0]} records")


b. Analyze Correlation with Other Features
It's possible that "Total Charges" is related to other columns in your dataset, such as "Tenure Months" or "Monthly Charges". For instance, new customers with a tenure of 0 months might naturally have missing total charges.
# Exploratory analysis to find patterns in missing 'Total Charges'
missing_tc = df[df['Total Charges'].isnull()]
print(missing_tc[['Tenure Months', 'Monthly Charges']].describe())

# Visualize the relationship
sns.boxplot(x=df['Tenure Months'] == 0, y=df['Total Charges'])
plt.title("Total Charges vs Tenure Months")
plt.show()


2. Potential Strategies to Handle Missing Values
Based on the insights gained from the exploratory analysis, you can choose one or a combination of the following strategies:
a. Impute Based on Related Features
If "Total Charges" is a product of "Monthly Charges" and "Tenure Months", you can impute missing values using these two columns.
# Impute missing 'Total Charges' by multiplying 'Monthly Charges' with 'Tenure Months'
df['Total Charges'] = df['Total Charges'].fillna(df['Monthly Charges'] * df['Tenure Months'])


Q- next 

df.isnull().sum()
CustomerID           0
Count                0
Country              0
State                0
City                 0
Zip Code             0
Lat Long             0
Latitude             0
Longitude            0
Gender               0
Senior Citizen       0
Partner              0
Dependents           0
Tenure Months        0
Phone Service        0
Multiple Lines       0
Internet Service     0
Online Security      0
Online Backup        0
Device Protection    0
Tech Support         0
Streaming TV         0
Streaming Movies     0
Contract             0
Paperless Billing    0
Payment Method       0
Monthly Charges      0
Total Charges        0
Churn Label          0
Churn Value          0
Churn Score          0
CLTV                 0
dtype: int64

Q- we have deal with them correctly 


import pandas as pd
from typing import List, Tuple


def calculate_cardinality(df: pd.DataFrame) -> pd.Series:
    """
    Calculate the number of unique values for each column in the DataFrame.


    Parameters:
    df (pd.DataFrame): The input DataFrame.


    Returns:
    pd.Series: A Series with column names as index and unique value counts as values.
    """
    return df.nunique()


def find_constant_columns(cardinality: pd.Series) -> List[str]:
    """
    Identify columns with constant values (only one unique value).


    Parameters:
    cardinality (pd.Series): A Series with unique value counts for each column.


    Returns:
    List[str]: A list of column names that are constant.
    """
    return cardinality[cardinality == 1].index.tolist()
def find_high_cardinality_columns(cardinality: pd.Series, total_rows: int, threshold: float = 0.5) -> List[str]:
    """
    Identify columns with high cardinality based on a threshold.


    Parameters:
    cardinality (pd.Series): A Series with unique value counts for each column.
    total_rows (int): Total number of rows in the DataFrame.
    threshold (float): The fraction of unique values relative to the number of rows.
                       Columns with unique value counts >= (threshold * total_rows) will be considered high cardinality.


    Returns:
    List[str]: A list of column names that have high cardinality.
    """
    return cardinality[cardinality / total_rows >= threshold].index.tolist()


def classify_columns_by_cardinality(df: pd.DataFrame, high_cardinality_threshold: float = 0.5) -> Tuple[List[str], List[str]]:
    """
    Classify columns into low and high cardinality based on their unique value counts.


    Parameters:
    df (pd.DataFrame): The input DataFrame.
    high_cardinality_threshold (float): The fraction of unique values to consider a column as high cardinality.


    Returns:
    Tuple[List[str], List[str]]: A tuple containing two lists:
                                 - List of constant (low cardinality) columns.
                                 - List of high cardinality columns.
    """
    cardinality = calculate_cardinality(df)
    low_cardinality = find_constant_columns(cardinality)
    total_rows = df.shape[0]
    high_cardinality = find_high_cardinality_columns(cardinality, total_rows, high_cardinality_threshold)
    return low_cardinality, high_cardinality




# Calculate cardinality and classify columns
low_cardinality, high_cardinality = classify_columns_by_cardinality(df)


# Print results
print("Constant (Low Cardinality) Columns:")
print(low_cardinality)
print("\nHigh Cardinality Columns:")
print(high_cardinality)


# Optionally, you can create new DataFrames with the classified columns
df_low_cardinality = df[low_cardinality]
df_high_cardinality = df[high_cardinality]


# You can also calculate the percentage of columns in each category
total_columns = len(df.columns)
low_cardinality_percentage = len(low_cardinality) / total_columns * 100
high_cardinality_percentage = len(high_cardinality) / total_columns * 100


print(f"\nPercentage of Constant Columns: {low_cardinality_percentage:.2f}%")
print(f"Percentage of High Cardinality Columns: {high_cardinality_percentage:.2f}%")



Constant (Low Cardinality) Columns:
['Count', 'Country', 'State']

High Cardinality Columns:
['CustomerID', 'Total Charges']

Percentage of Constant Columns: 9.38%
Percentage of High Cardinality Columns: 6.25%

"""
Handling Low and High Cardinality Columns
Low Cardinality (Including Constant Columns)
Issues:


Constant Columns: Provide no variance, offering no predictive power. They can introduce unnecessary computational overhead.
Low but Variable Cardinality: Columns with a small number of unique values (e.g., Gender, Marital Status) are not inherently problematic but may require specific encoding.
"""
df = df.drop(columns=low_cardinality )


from typing import List, Tuple


def classify_columns(df: pd.DataFrame) -> Tuple[List[str], List[str]]:
    """
    Classifies columns of a DataFrame into numerical and categorical.


    Args:
        df (pd.DataFrame): Input DataFrame


    Returns:
        Tuple[List[str], List[str]]: A tuple containing two lists:
            - List of numerical column names
            - List of categorical column names
    """
    is_numeric = lambda col: pd.api.types.is_numeric_dtype(df[col])
   
    numerical_columns = list(filter(is_numeric, df.columns))
    categorical_columns = list(filter(lambda col: not is_numeric(col), df.columns))
   
    return numerical_columns, categorical_columns


"""
now given  this function :
we use the high level order feature of it
and the lamabda function to make it more readable
make a call to it
"""
numerical_columns, categorical_columns = classify_columns(df)
(numerical_columns, categorical_columns)

"""
now get the length of
1-numerical features
2-categorical features
"""
(len(numerical_columns), len(categorical_columns))


(10, 22)
"""
oky as we see we have
10- numerical features
23- categorical features
and the problem is a supervised learning problem
where the "y" is $"Churn Value"
Q- what about to lear about some statique properties of this target


"""
"""
Q- firt idea that come to my mind to plot the density fuction of the target
--> create a python target feature and plot it density function
"""


# Create a target feature
target = df['Churn Value']


# Plot the density function of the target
import matplotlib.pyplot as plt
import seaborn as sns


plt.figure(figsize=(10, 6))
sns.kdeplot(data=target, shade=True)
plt.title('Density Function of Churn Value')
plt.xlabel('Churn Value')
plt.ylabel('Density')
plt.show()




"""
oky as we see we have
9- numerical features
24- categorical features
and the problem is a supervised learning problem
where the "y" is $"Churn Value"
Q- what about to lear about some statique properties of this target


"""


"""
Cumulative Distribution Function (CDF):
A CDF would show the proportion of customers below any given churn value. This graph would:




Clearly illustrate the percentiles of churn risk
Help identify thresholds for categorizing customers (e.g., low, medium, high risk)
Reveal the exact proportion of customers in each peak




"""
# Create a target feature
target = df['Churn Value']
# Print some statisticsFeatur
print(f"Mean Churn Value: {np.mean(target)}")
print(f"Median Churn Value: {np.median(target)}")
print(f"25th Percentile: {np.percentile(target, 25)}")
print(f"75th Percentile: {np.percentile(target, 75)}")



Q- in this part i want to focus in two part 

Feature engieerijng and Features selection  

Q- we will be a pair programming you (Aà) and me we brain some ideas 
Test it , get feedback and iterate intil converge oky