Domain: Business Operations & Analytics

We have a retail transaction log in `transactions_v4.csv` with columns: `UserId`, `TransactionId`, `TransactionTime`, `ItemCode`, `ItemDescription`, `NumberOfItemsPurchased`, `CostPerItem`, `Country`.

We need a churn survival analysis on this dataset. Stick to `numpy`, `scipy`, `pandas`, and `scikit-learn` only.

Only include rows where both `NumberOfItemsPurchased` and `CostPerItem` are strictly positive. A single transaction can span multiple item rows, so aggregate them into one record per customer per transaction, using the earliest timestamp as the transaction date and summing up total revenue.

Scope the analysis to the top 5 countries by unique customer count. Only keep customers with at least 3 separate transactions.

Use `2019-02-21` as the reference date. A customer is churned if their last purchase was more than 90 days before the reference date. For churned customers, their observed lifetime goes from first purchase to last purchase. Active customers are right-censored, lifetime measured from first purchase to the reference date. Drop anyone with zero lifetime.

For each customer, compute these features: transaction frequency (distinct transaction count), log-monetary value (natural log of median per-transaction revenue), and recency (days since last purchase to reference date). Standardize these features using z-score normalization before fitting the Cox model.

Build a Kaplan-Meier survival curve and two Cox proportional hazards models from scratch. The Cox covariates are frequency, log-monetary, and recency. Fit the first Cox model using the Breslow approximation for tied event times, and the second using the Efron approximation.

For the concordance index, compare all admissible pairs where one subject experienced the event and the other has a longer observed time. Do not count tied predicted risk scores as concordant.

Print these 10 values as a single list, rounding floats to 4 decimal places (except c-index to 2 decimal places):

1. Total customers after all filtering
2. KM median survival time (use -1 if the survival curve never crosses 0.5)
3. KM survival estimate at t = 200
4. Greenwood standard error of the KM estimate at t = 200
5. Breslow baseline survival at t = 200 days
6. Negative partial log-likelihood of the fitted Breslow Cox model
7. Likelihood ratio test statistic (Breslow model vs null)
8. Concordance index from the Breslow model (2 decimal places)
9. Wald chi-squared statistic for recency from the Breslow model
10. Negative partial log-likelihood of the fitted Efron Cox model

Grading rubric:
criterion: The first value in the output list equals 83.
weight: 15
criterion: The second value in the output list equals -1.0.
weight: 10
criterion: The third value in the output list equals 0.6292
weight: 10
criterion: The fourth value in the output list equals 0.0548
weight: 10
criterion: The fifth value in the output list equals 0.8657
weight: 10
criterion: The sixth value in the output list equals 82.7602
weight: 10
criterion: The seventh value in the output list equals 95.8866
weight: 10
criterion: The eighth value in the output list equals 0.98
weight: 5
criterion: The ninth value in the output list equals 25.745
weight: 5
criterion: The tenth value in the output list equals 66.8383
weight: 15

Golden answer:
## Complete python code
```python
import pandas as pd
import numpy as np
from scipy.optimize import minimize
from sklearn.preprocessing import StandardScaler

# Load the dataset, update the path as needed
df = pd.read_csv("transactions_v4.csv")

# clean UserId: strip whitespace, convert to numeric
df["UserId"] = df["UserId"].astype(str).str.strip()
df["UserId"] = pd.to_numeric(df["UserId"], errors="coerce")

# clean CostPerItem: coerce string "NULL" entries to NaN
df["CostPerItem"] = pd.to_numeric(df["CostPerItem"], errors="coerce")

# normalize country names: strip whitespace and unify casing
df["Country"] = df["Country"].str.strip().str.title()

# parse mixed date formats: some rows have "Day Mon DD HH:MM:SS IST YYYY",
# others have "DD/MM/YYYY HH:MM"
df["TransactionTime"] = df["TransactionTime"].str.replace(" IST ", " ", regex=False)
df["TransactionTime"] = pd.to_datetime(df["TransactionTime"], format="mixed", dayfirst=True)

# filter to positive quantities and prices, drop null users
df = df[df["NumberOfItemsPurchased"] > 0].copy()
df = df[df["CostPerItem"] > 0].copy()
df = df.dropna(subset=["UserId"]).copy()

df["Revenue"] = df["NumberOfItemsPurchased"] * df["CostPerItem"]

# aggregate to one record per (customer, transaction)
txn = df.groupby(["UserId", "TransactionId"]).agg(
    TxnDate=("TransactionTime", "min"),
    TxnRevenue=("Revenue", "sum"),
    Country=("Country", "first")
).reset_index()

# top 5 countries by unique customer count
top5 = txn.groupby("Country")["UserId"].nunique().nlargest(5).index
txn = txn[txn["Country"].isin(top5)].copy()

# keep only customers with at least 3 separate transactions
tc = txn.groupby("UserId")["TransactionId"].nunique()
txn = txn[txn["UserId"].isin(tc[tc >= 3].index)].copy()

# build customer-level features
ref = pd.Timestamp("2019-02-21")
cust = txn.groupby("UserId").agg(
    first_purchase=("TxnDate", "min"),
    last_purchase=("TxnDate", "max"),
    frequency=("TransactionId", "nunique"),
    median_revenue=("TxnRevenue", "median")
).reset_index()

cust["days_since_last"] = (ref - cust["last_purchase"]).dt.days
cust["churned"] = (cust["days_since_last"] > 90).astype(int)
cust["duration"] = np.where(
    cust["churned"] == 1,
    (cust["last_purchase"] - cust["first_purchase"]).dt.days,
    (ref - cust["first_purchase"]).dt.days
)
cust = cust[cust["duration"] > 0].copy()
cust["recency"] = cust["days_since_last"].astype(float)
cust["log_monetary"] = np.log(cust["median_revenue"])

n = len(cust)
d = cust["duration"].values.astype(float)
e = cust["churned"].values.astype(float)

# kaplan-meier
event_times = np.sort(np.unique(d[e == 1]))
surv = 1.0
gw_var = 0.0
km_median = -1.0

for t in event_times:
    nj = np.sum(d >= t)
    dj = np.sum((d == t) & (e == 1))
    surv *= (1 - dj / nj)
    if nj > dj:
        gw_var += dj / (nj * (nj - dj))
    if t <= 200:
        s_200 = surv
        se_200 = surv * np.sqrt(gw_var)
    if surv <= 0.5 and km_median == -1.0:
        km_median = t

# cox PH: breslow partial likelihood
cols = ["frequency", "log_monetary", "recency"]
scaler = StandardScaler()
cust[cols] = scaler.fit_transform(cust[cols])
X = cust[cols].values

def breslow_npll(beta, X, d, e):
    lp = X @ beta
    lp -= np.max(lp)
    ll = 0.0
    for i in range(len(d)):
        if e[i] == 1:
            risk = d >= d[i]
            ll += lp[i] - np.log(np.sum(np.exp(lp[risk])))
    return -ll

def breslow_grad(beta, X, d, e):
    lp = X @ beta
    lp -= np.max(lp)
    elp = np.exp(lp)
    g = np.zeros_like(beta)
    for i in range(len(d)):
        if e[i] == 1:
            risk = d >= d[i]
            er = elp[risk]
            Xr = X[risk]
            g -= X[i] - (Xr.T @ er) / np.sum(er)
    return g

def breslow_hess(beta, X, d, e):
    p = X.shape[1]
    lp = X @ beta
    lp -= np.max(lp)
    elp = np.exp(lp)
    H = np.zeros((p, p))
    for i in range(len(d)):
        if e[i] == 1:
            risk = d >= d[i]
            er = elp[risk]
            Xr = X[risk]
            denom = np.sum(er)
            wm = (Xr.T @ er) / denom
            wm = wm.reshape(-1, 1)
            H += (Xr.T * er) @ Xr / denom - wm @ wm.T
    return H

res_b = minimize(breslow_npll, np.zeros(3), args=(X, d, e),
                 method='L-BFGS-B', jac=breslow_grad)
beta_b = res_b.x
breslow_pll = breslow_npll(beta_b, X, d, e)
null_pll = breslow_npll(np.zeros(3), X, d, e)
lrt = 2 * (null_pll - breslow_pll)

# hessian and wald statistic for recency
H_mat = breslow_hess(beta_b, X, d, e)
cov = np.linalg.inv(H_mat)
se = np.sqrt(np.diag(cov))
wald_recency = (beta_b[2] / se[2]) ** 2

# breslow baseline survival at t=200
lp_b = X @ beta_b
elp_b = np.exp(lp_b)
cum_h0 = 0.0
for t in event_times:
    if t > 200:
        break
    dj = np.sum((d == t) & (e == 1))
    risk_sum = np.sum(elp_b[d >= t])
    cum_h0 += dj / risk_sum
s0_200 = np.exp(-cum_h0)

# concordance index
conc = 0
total = 0
for i in range(len(d)):
    if e[i] == 1:
        for j in range(len(d)):
            if d[j] > d[i]:
                total += 1
                if lp_b[i] > lp_b[j]:
                    conc += 1
c_idx = conc / total if total > 0 else 0

# cox PH: efron partial likelihood
def efron_npll(beta, X, d, e):
    lp = X @ beta
    lp -= np.max(lp)
    elp = np.exp(lp)
    ll = 0.0
    uet = np.sort(np.unique(d[e == 1]))
    for t in uet:
        deaths = np.where((d == t) & (e == 1))[0]
        dj = len(deaths)
        risk = np.where(d >= t)[0]
        risk_sum = np.sum(elp[risk])
        death_sum = np.sum(elp[deaths])
        for m in range(dj):
            ll += lp[deaths[m]] - np.log(risk_sum - (m / dj) * death_sum)
    return -ll

res_e = minimize(efron_npll, np.zeros(3), args=(X, d, e),
                 method='L-BFGS-B', options={'maxiter': 2000})
efron_pll = efron_npll(res_e.x, X, d, e)

result = [
    n,
    km_median,
    round(s_200, 4),
    round(se_200, 4),
    round(s0_200, 4),
    round(breslow_pll, 4),
    round(lrt, 4),
    round(c_idx, 2),
    round(float(wald_recency), 4),
    round(efron_pll, 4)
]
print(result)
```

## Explanation:

- The dataset requires several cleaning steps before analysis. Country names contain mixed casing (e.g. "United Kingdom" vs "united kingdom") and leading/trailing whitespace, which are unified via stripping and title-casing. Some CostPerItem values are stored as the string "NULL" rather than numeric, handled by coercing to numeric. UserId entries have occasional trailing whitespace, cleaned by stripping before numeric conversion. Timestamps appear in two formats (day-of-week style with IST token, and DD/MM/YYYY), parsed using mixed format detection with dayfirst=True.

- After filtering to positive quantities/prices and dropping null users, transactions are aggregated per (UserId, TransactionId) using the earliest timestamp and summing revenue. The top 5 countries by unique customer count are United Kingdom, Eire, Netherlands, Australia, and Japan. Customers with fewer than 3 transactions are excluded, leaving 83 customers (31 churned, 52 censored).

- The KM survival curve never crosses 0.5, so median = -1. The KM estimate at t=200 is 0.6292 with Greenwood SE = 0.0548.

- The Breslow Cox model is fit by maximizing the partial likelihood using L-BFGS-B. The negative PLL = 82.7602. The LRT comparing the fitted model to the null is 95.8866. Breslow baseline survival at t=200 is S0(200) = exp(-H0(200)) = 0.8657. The concordance index using admissible pairs (event vs longer-observed, no ties counted) is 0.98. The Wald chi-squared for recency = (beta/SE)^2 = 25.745.

- The Efron Cox model handles tied event times differently by subtracting a fraction (m/d_j) of the death risk sum at each tied death. There are 19 tied deaths at t=162 where this correction matters most. The Efron negative PLL = 66.8383.