Domain: Healthcare

You work with a university’s Student Success Office and need a fair, outlier-robust way to flag students for proactive coaching. You are given a CSV named student_productivity_distraction_dataset_20000.csv. Load the file, drop any row with missing values or invalid values in any required column. 

Create age_band from age using inclusive bins 17–19, 20–22, 23–25, 26–29, and create digital_distraction_hours by summing phone_usage_hours, social_media_hours, youtube_hours, and gaming_hours. Define cohorts as (age_band, gender). 
For each cohort, compute cohort_attendance_median as the median of attendance_percentage within that cohort. Define the cohort’s baseline subset as students with attendance_percentage no less than cohort_attendance_median. If this baseline subset has fewer than 10 students, use the entire cohort as the baseline subset instead. 
Using the baseline subset only, compute baseline median and MAD for each metric, where MAD is Median Absolute Deviation. Use these 12 metrics: higher-is-better = study_hours_per_day, sleep_hours, exercise_minutes, assignments_completed, attendance_percentage, focus_score, final_grade, productivity_score; higher-is-worse = digital_distraction_hours, breaks_per_day, coffee_intake_mg, stress_level. 
For each student and each metric, compute z using the baseline stats for that student’s cohort: if higher-is-better then z = (x − baseline_median) / baseline_MAD, if higher-is-worse then z = (baseline_median − x) / baseline_MAD. If baseline_MAD == 0, set z = 0 for that metric. Convert each z into a component score s = 1 / (1 + exp(-z)). Define RFPI as the equal-weight mean of the 12 component scores (so RFPI is in [0,1]). Define active students as those with attendance_percentage no less than overall_median_attendance computed on the cleaned dataset. Among active students only, compute rfpi_median_active and rfpi_mad_active and define risk_cutoff = rfpi_median_active − 1.5 * rfpi_mad_active. An active student is at_risk if rfpi less than risk_cutoff.

Return answers to the following 10 questions based on the analysis. Round all reported floats to 4 decimals.
1: What is overall_median_attendance?
2: What is rfpi_median_active?
3: What is rfpi_mad_active?
4: What is risk_cutoff?
5: What is at_risk_rate_active (= n_at_risk_active / n_active)?
6: Which cohort (age_band, gender) has the highest active at_risk_rate? Break ties by higher n_students, then age_band ascending, then gender ascending. Report: age_band, gender, at_risk_rate.
7: For that cohort from Q6, what are baseline_n and rfpi_median (computed over all students in that cohort, not just baseline)?
8: Who is the most at-risk active student (sorted by rfpi asc, then final_grade asc, then student_id asc)? Report: student_id, rfpi, lowest_3_components, where lowest_3_components are the 3 metric names with the smallest component scores for that student (tie-break metric name alphabetically).

Grading rubric:
criterion: overall_median_attendance is equal to 70.0350
weight: 10
criterion: rfpi_median_active is equal to 0.4995
weight: 15
criterion: rfpi_mad_active is equal to 0.0575
weight: 15
criterion: risk_cutoff is equal to 0.4132
weight: 10
criterion: at_risk_rate_active is equal to 0.1564
weight: 10
criterion: the cohort with the highest active at_risk_rate is the (17-19, "Other") cohort
weight: 10
criterion: for the highest active at_risk_rate cohort, the at_risk_rate value is equal to 0.1944
weight: 15
criterion: baseline_n and rfpi_median cohorts from Q8 are equal to 71 and 0.4623
weight: 10
criterion: the student ID and RFPI of the most at-risk active student are equal to 16479 and 0.2163
weight: 15
criterion: the metric names with the smallest component scores for the most at-risk active student are: "productivity_score", "stress_level", "digital_distraction_hours"
weight: 10

Golden answer:
```python
import json
import math
from typing import Dict, List, Tuple

import numpy as np
import pandas as pd


# Config
REQ_COLS = [
    "student_id",
    "age",
    "gender",
    "study_hours_per_day",
    "sleep_hours",
    "phone_usage_hours",
    "social_media_hours",
    "youtube_hours",
    "gaming_hours",
    "breaks_per_day",
    "coffee_intake_mg",
    "exercise_minutes",
    "assignments_completed",
    "attendance_percentage",
    "stress_level",
    "focus_score",
    "final_grade",
    "productivity_score",
]

AGE_LABELS = ["17-19", "20-22", "23-25", "26-29"]

GOOD_METRICS = [
    "study_hours_per_day",
    "sleep_hours",
    "exercise_minutes",
    "assignments_completed",
    "attendance_percentage",
    "focus_score",
    "final_grade",
    "productivity_score",
]
BAD_METRICS = [
    "digital_distraction_hours",
    "breaks_per_day",
    "coffee_intake_mg",
    "stress_level",
]
METRICS = GOOD_METRICS + BAD_METRICS


# Helpers
def mad_unscaled(s: pd.Series) -> float:
    """MAD = median(|x - median(x)|), no scaling."""
    med = s.median()
    return float((s - med).abs().median())


def sigmoid_stable(z: np.ndarray) -> np.ndarray:
    """Numerically stable sigmoid, identical to 1/(1+exp(-z)) for normal ranges."""
    z = z.astype(float)
    out = np.empty_like(z, dtype=float)
    pos = z >= 0
    out[pos] = 1.0 / (1.0 + np.exp(-z[pos]))
    ez = np.exp(z[~pos])
    out[~pos] = ez / (1.0 + ez)
    return out


# Load + clean
df = pd.read_csv("student_productivity_distraction_dataset_20000.csv")

# Clean gender
df["gender"] = df["gender"].astype("string").str.strip()
df.loc[df["gender"] == "", "gender"] = pd.NA

# Coerce numeric columns
numeric_cols = [c for c in REQ_COLS if c not in ("gender",)]
for c in numeric_cols:
    df[c] = pd.to_numeric(df[c], errors="coerce")

# Treat +/-inf as invalid
df.replace([np.inf, -np.inf], np.nan, inplace=True)

# Drop rows with missing/invalid in any required column
df = df.dropna(subset=REQ_COLS).copy()

# Age band 
df["age_band"] = pd.cut(
    df["age"],
    bins=[17, 19, 22, 25, 29],
    labels=AGE_LABELS,
    include_lowest=True,
)
df = df.dropna(subset=["age_band"]).copy()
df["age_band"] = df["age_band"].astype(pd.CategoricalDtype(categories=AGE_LABELS, ordered=True))

# Derived feature
df["digital_distraction_hours"] = (
    df["phone_usage_hours"]
    + df["social_media_hours"]
    + df["youtube_hours"]
    + df["gaming_hours"]
)

# Cohort baseline stats (nested median baseline)
stats_rows: List[Dict] = []

for (age_band, gender), g in df.groupby(["age_band", "gender"], observed=False, sort=False):
    cohort_att_med = float(g["attendance_percentage"].median())
    baseline = g[g["attendance_percentage"] >= cohort_att_med]
    if len(baseline) < 10:
        baseline = g

    row: Dict = {
        "age_band": age_band,
        "gender": gender,
        "baseline_n": int(len(baseline)),
    }

    for m in METRICS:
        med = float(baseline[m].median())
        md = mad_unscaled(baseline[m])
        row[f"{m}__median"] = med
        row[f"{m}__mad"] = md

    stats_rows.append(row)

stats = pd.DataFrame(stats_rows).set_index(["age_band", "gender"])
df = df.join(stats, on=["age_band", "gender"], how="left")


# Compute RFPI and component scores
comp_cols: List[str] = []

for m in METRICS:
    med_col = f"{m}__median"
    mad_col = f"{m}__mad"

    x = df[m].to_numpy(dtype=float)
    medv = df[med_col].to_numpy(dtype=float)
    madv = df[mad_col].to_numpy(dtype=float)

    if m in GOOD_METRICS:
        z = np.where(madv == 0.0, 0.0, (x - medv) / madv)
    else:
        z = np.where(madv == 0.0, 0.0, (medv - x) / madv)

    s = sigmoid_stable(z)
    col = f"comp__{m}"
    df[col] = s
    comp_cols.append(col)

df["rfpi"] = df[comp_cols].mean(axis=1)

# Active and risk cutoff
overall_median_attendance = float(df["attendance_percentage"].median())
active = df[df["attendance_percentage"] >= overall_median_attendance].copy()

n_active = int(len(active))
rfpi_median_active = float(active["rfpi"].median()) if n_active else float("nan")
rfpi_mad_active = float((active["rfpi"] - rfpi_median_active).abs().median()) if n_active else float("nan")
risk_cutoff = float(rfpi_median_active - 1.5 * rfpi_mad_active) if n_active else float("nan")

active["is_at_risk"] = active["rfpi"] < risk_cutoff
at_risk_active = active[active["is_at_risk"]].copy()

n_at_risk_active = int(len(at_risk_active))
at_risk_rate_active = float(n_at_risk_active / n_active) if n_active else float("nan")


# 8: cohort with highest active at-risk rate
cohort_active = (
    active.groupby(["age_band", "gender"], observed=False, as_index=False)
    .agg(
        n_students=("student_id", "size"),
        n_at_risk=("is_at_risk", "sum"),
    )
)
cohort_active["at_risk_rate"] = cohort_active["n_at_risk"] / cohort_active["n_students"]
cohort_active["age_band"] = cohort_active["age_band"].astype(df["age_band"].dtype)

cohort_active_sorted = cohort_active.sort_values(
    by=["at_risk_rate", "n_students", "age_band", "gender"],
    ascending=[False, False, True, True],
    kind="mergesort",
)
top_cohort = cohort_active_sorted.iloc[0]
top_age_band = str(top_cohort["age_band"])
top_gender = str(top_cohort["gender"])
top_at_risk_rate = float(top_cohort["at_risk_rate"])

# 9: baseline_n and cohort rfpi_median (over ALL students in that cohort)
baseline_n = int(stats.loc[(top_age_band, top_gender), "baseline_n"])
cohort_all = df[(df["age_band"].astype(str) == top_age_band) & (df["gender"] == top_gender)]
rfpi_median_cohort = float(cohort_all["rfpi"].median())

# 10: most at-risk active student
worst_row = (
    at_risk_active.sort_values(
        by=["rfpi", "final_grade", "student_id"],
        ascending=[True, True, True],
        kind="mergesort",
    )
    .iloc[0]
)

worst_student_id = int(worst_row["student_id"])
worst_rfpi = float(worst_row["rfpi"])

# lowest_3_components
components: List[Tuple[str, float]] = [(m, float(worst_row[f"comp__{m}"])) for m in METRICS]
components_sorted = sorted(components, key=lambda t: (t[1], t[0]))
lowest_3_components = [m for m, _ in components_sorted[:3]]

# Print answers
print(f"1: {overall_median_attendance:.4f}")
print(f"2: {rfpi_median_active:.4f}")
print(f"3: {rfpi_mad_active:.4f}")
print(f"4: {risk_cutoff:.4f}")
print(f"5: {at_risk_rate_active:.4f}")
print(f"6: {top_age_band}, {top_gender}, {top_at_risk_rate:.4f}")
print(f"7: {baseline_n}, {rfpi_median_cohort:.4f}")
print(f"8: {worst_student_id}, {worst_rfpi:.4f}, {json.dumps(lowest_3_components)}")

```

**Explanation**
Loads the CSV, coerces numeric fields, drops invalid/missing rows, then creates age_band and digital_distraction_hours.
For each (age_band, gender) cohort, builds a “baseline” using students with attendance ≥ cohort median (fallback to full cohort if <10), and computes baseline medians + unscaled MADs.
Uses those stats to compute robust z-scores, applies sigmoid to get 0–1 component scores, and averages them into RFPI per student.
Defines active students by overall median attendance, computes a median/MAD-based risk cutoff, flags at-risk actives, and prints the 8 required answers with tie-break rules and rounding.

**gtfa**
1: 70.0350
2: 0.4995
3: 0.0575
4: 0.4132
5: 0.1564
6: 17-19, Other, 0.1944
7: 71, 0.4623
8: 16479, 0.2163, ["productivity_score", "stress_level", "digital_distraction_hours"]