Domain: Business Operations & Analytics

I have a csv file 'diabetic_data.csv' that needs cleaning. There have been some entries with a question mark (?) instead
of empty hence they should be considered NaN. Additionally any columns that have more than thirty percent missing values
from the total records should be removed whereas for any other column should be imputed. For numeric columns, the NA/NaN
values by should be filled propagating the last valid observation to next valid and for categorical columns they should
be filled with mode value. Additionally, the age column contains data with range ages (e.g., "[20-30)" which should be
used to create a new olumn called age_group with values like "20-30". and the original age column should be updated with
the average of the two numerical values for instance  "[20-30)" becomes 25.

Lastly, any column apart from race, gender, age_group should be numeric if not already it should be converted to numbers
data types while forcing the non-convertible data into NaN.

For visibility, the output should be:

- a dictionary of first two records of cleaned dataset where top-level keys are column names, inner keys
  are row index labels and the values are the actual data entries for that specific column and row
- A dictionary of the dataframe datatypes
- A list of unique integers in the age column after all cleaning
- And names of columns with missing values and their missing value percentage (2 decimal places rounded) against the total records 

Grading rubric:
criterion: A dictionary of first two records after cleaning is {'encounter_id': {0: 2278392, 1: 149190}}?
weight: 10
criterion: A dictionary of data types after cleaning is {'encounter_id': 'int64', 'race': 'object', 'gender': 'object', 'age': 'float64', 'admission_type_id': 'int64', 'number_inpatient': 'int64', 'diag_1': 'float64', 'diag_2': 'float64', 'diag_3': 'float64', 'number_diagnoses': 'int64', 'age_group': 'object'}?
weight: 10
criterion: A list of unique age values is [5, 15, 25, 35, 45, 55, 65, 75, 85, 95]
weight: 5
criterion: The percentage of missing values rounded to decimal places of diag_1 after cleaning is 1.62
weight: 10
criterion: The percentage of missing values rounded to decimal places of diag_2 after cleaning is 2.49
weight: 10
criterion: The percentage of missing values rounded to decimal places of diag_3 after cleaning is 5.04
weight: 20

Golden answer:
```python
import pandas as pd
import warnings

warnings.filterwarnings("ignore")
pd.set_option("display.max_columns", None)

# Constants
MISSING_PCT_THRESHOLD = 30.0  # drop columns with more than this % missing
NA_VALUES = ["?", "None"]
# Columns that remain categorical; all others are converted to numeric
CATEGORICAL_COLUMNS = {"race", "gender", "age_group"}

# Load and treat ? and None as NaN. Adjust the path as needed
db = pd.read_csv("diabetic_data.csv", na_values=NA_VALUES)
n_records = len(db)

# Drop columns with more than 30% missing
missing_pct = (db.isna().sum() / n_records) * 100
cols_to_drop = missing_pct[missing_pct > MISSING_PCT_THRESHOLD].index.tolist()
df_clean = db.drop(columns=cols_to_drop)

# 2. Imputation: numeric - ffill, categorical - mode
cols_categorical_impute = ["race", "gender"]
cols_numeric_impute = [c for c in df_clean.columns if c not in cols_categorical_impute]

for col in cols_numeric_impute:
    if col in df_clean.columns and df_clean[col].isna().any():
        df_clean[col] = df_clean[col].ffill().bfill()

for col in cols_categorical_impute:
    if col in df_clean.columns and df_clean[col].isna().any():
        mode_val = df_clean[col].mode()
        df_clean[col] = df_clean[col].fillna(
            mode_val[0] if len(mode_val) else df_clean[col].iloc[0]
        )


# Create age_group (e.g. "[20-30)" - "20-30") and age as average (e.g. 25)
def parse_age(age_series):
    """Return (age_group_series, age_numeric_series)."""
    age_group_list = []
    age_avg_list = []
    for v in age_series:
        if pd.isna(v) or not isinstance(v, str):
            age_group_list.append(pd.NA)
            age_avg_list.append(pd.NA)
            continue
        s = v.replace("[", "").replace(")", "").strip()
        if "-" in s:
            parts = s.split("-")
            if len(parts) == 2:
                try:
                    low, high = int(parts[0]), int(parts[1])
                    age_group_list.append(f"{low}-{high}")
                    age_avg_list.append((low + high) / 2)
                except ValueError:
                    age_group_list.append(pd.NA)
                    age_avg_list.append(pd.NA)
            else:
                age_group_list.append(pd.NA)
                age_avg_list.append(pd.NA)
        else:
            age_group_list.append(pd.NA)
            age_avg_list.append(pd.NA)
    return pd.Series(age_group_list, index=age_series.index), pd.Series(
        age_avg_list, index=age_series.index
    )


if "age" in df_clean.columns:
    age_group_series, age_numeric_series = parse_age(df_clean["age"])
    df_clean["age_group"] = age_group_series
    df_clean["age"] = age_numeric_series

# Convert to numeric: all columns except race, gender, age_group
for col in df_clean.columns:
    if col in CATEGORICAL_COLUMNS:
        continue
    df_clean[col] = pd.to_numeric(df_clean[col], errors="coerce")

# Outputs
n_clean = len(df_clean)

# First two records: top-level keys = column names, inner keys = row index, values = data
first_two = df_clean.head(2)
first_two_records_dict = {col: first_two[col].to_dict() for col in first_two.columns}

# Dictionary of dataframe dtypes (string representation for visibility)
dtypes_dict = {col: str(dtype) for col, dtype in df_clean.dtypes.items()}

# List of unique integers in the age column (after cleaning)
age_unique_ints = sorted(
    pd.Series(df_clean["age"].dropna().astype(int).unique()).tolist()
)

# Columns with missing values and their missing value percentage (against total records)
missing_pct_clean = (df_clean.isna().sum() / n_clean * 100).round(2)
columns_with_missing = missing_pct_clean[missing_pct_clean > 0].to_dict()

# Print for visibility
print("First two records (column -> {index: value}):")
print(first_two_records_dict)
print("\nData types:")
print(dtypes_dict)
print("\nUnique integers in age column:")
print(age_unique_ints)
print("\nColumns with missing values and missing %:")
print(columns_with_missing)
```

**Output**
First two records (column -> {index: value}):
{'encounter_id': {0: 2278392, 1: 149190}, 'race': {0: 'Caucasian', 1: 'Caucasian'}, 'gender': {0: 'Female', 1: 'Female'}, 'age': {0: 5.0, 1: 15.0}, 'admission_type_id': {0: 6, 1: 1}, 'number_inpatient': {0: 0, 1: 0}, 'diag_1': {0: 250.83, 1: 276.0}, 'diag_2': {0: 250.01, 1: 250.01}, 'diag_3': {0: 255.0, 1: 255.0}, 'number_diagnoses': {0: 1, 1: 9}, 'age_group': {0: '0-10', 1: '10-20'}}

Data types:
{'encounter_id': 'int64', 'race': 'object', 'gender': 'object', 'age': 'float64', 'admission_type_id': 'int64', 'number_inpatient': 'int64', 'diag_1': 'float64', 'diag_2': 'float64', 'diag_3': 'float64', 'number_diagnoses': 'int64', 'age_group': 'object'}

Unique integers in age column:
[5, 15, 25, 35, 45, 55, 65, 75, 85, 95]

Columns with missing values and missing %:
{'diag_1': 1.62, 'diag_2': 2.49, 'diag_3': 5.04}