Domain: Business Operations & Analytics

You are analyzing a retail order-line dataset that contains **data quality issues** including malformed numeric fields, inconsistent formatting, quoted values, and anomalous text patterns.

**Step 1: Data Cleaning**

Before performing any analysis, you must first **clean and recover the dataset** using the following rules:

1. **Remove leading/trailing whitespace** from all text columns
2. **Remove one layer of surrounding double-quotes** from quoted fields (e.g., `"OK"` becomes `OK`)
3. **Convert year fields** formatted as `01-01-YYYY` back to integer year (e.g., `01-01-2020` → `2020`)
4. **Strip percentage symbols** from fields like `tax_pct` and `discount_pct` (e.g., `0.08%` → `0.08` as a float; do NOT divide by 100)
5. **Remove unit suffixes** from numeric fields:
   - `shipping_distance_km`: remove `km` suffix
   - `weight_kg`: remove `kg` suffix
   - `volume_cm3`: remove `cm3` suffix
6. **Remove currency symbols** (e.g., `₹`, `â‚¹`) from all monetary fields and convert to float
7. **Remove zero-padding** from ID fields and convert to integer (e.g., `000400` → `400`)
8. **Coerce remaining numeric-like fields** (e.g., `order_hour`, `quantity`, `churn_risk_score`) to numeric types where applicable
9. **Malformed rows handling**:
   - If a row cannot be parsed into the expected number of columns, **drop the row entirely**
   - Do not attempt to repair or realign malformed rows
10. **Numeric coercion policy**:
   - When converting to numeric, use coercion (invalid values → NaN)
   - Unless explicitly stated otherwise, NaN values should be **ignored in aggregations** (standard pandas behavior)
11. **Percentile definition**:
   - Compute percentiles using **NumPy’s default method (linear interpolation, equivalent to numpy.percentile)**
12. **Division safety**:
   - Any division resulting in NaN or infinity must be replaced with **0.0**

After cleaning, proceed to Step 2.

---

**Step 2: Risk Group Analysis**

Analyze the cleaned dataset at the **order-line level** and group rows by:

- `customer_state`
- `customer_segment`
- `product_category`

For each group, compute a **composite commercial risk score** defined as the **mean of five components**:

1. **Discount behavior score** = (average of `discount_mismatch_ratio` + 90th percentile of `discount_mismatch_ratio`) / 2
   - Where `discount_mismatch_ratio` = `abs(discount_amount - line_subtotal × discount_pct) / line_subtotal`
   - Replace division-by-zero or invalid values with `0.0`
   - Clip negative values to `0.0`

2. **Return rate** = proportion of rows where `return_reason` is non-empty (after trimming whitespace)

3. **Non-final order rate** = proportion of rows where `order_status` is NOT in `{Completed, Delivered}`

4. **Churn risk (normalized)** = `average(churn_risk_score) / 100.0`

5. **Margin weakness** = `1.0 - margin_rate`
   - Where `margin_rate` = `sum(gross_margin_amount) / sum(total_amount)` (handle division-by-zero safely)

**Composite risk score formula:**

```
composite_risk_score = (discount_behavior + return_rate + non_final_rate + churn_normalized + margin_weakness) / 5.0
```

---

**Step 3: Filtering and Ranking**

- **Filter out small groups**: Keep only groups with **at least 200 rows**
- **Rank groups deterministically** using the following tie-break order:
  1. `composite_risk_score` descending
  2. `revenue` (sum of `total_amount`) descending
  3. `n_lines` (row count) descending
  4. `customer_state` ascending (alphabetical)
  5. `customer_segment` ascending (alphabetical)
  6. `product_category` ascending (alphabetical)

Use a **stable sort algorithm** (e.g., mergesort) to ensure deterministic output.

---

**Step 4: Rounding Rules**

- **Revenue**: Round to **2 decimal places**
- **All rates and scores** (composite risk score, discount mismatch metrics, return rate, non-final rate, churn, margin rate): Round to **6 decimal places**

---

**Step 5: Output Format**

Return the **top 7 highest-risk groups** as a **Python list of lists**.

Each inner list must contain exactly **13 elements** in this order:

```python
[
  rank,                              # int (1 through 7)
  customer_state,                    # str
  customer_segment,                  # str
  product_category,                  # str
  n_lines,                           # int (row count)
  revenue,                           # float (rounded to 2 decimals)
  composite_risk_score,              # float (rounded to 6 decimals)
  avg_discount_mismatch_ratio,       # float (rounded to 6 decimals)
  p90_discount_mismatch_ratio,       # float (rounded to 6 decimals)
  return_rate,                       # float (rounded to 6 decimals)
  non_final_rate,                    # float (rounded to 6 decimals)
  avg_churn_risk_normalized,         # float (rounded to 6 decimals, i.e., churn/100)
  margin_rate                        # float (rounded to 6 decimals)
]
```

**Do not use graphs or visualizations.** Print only the final list.
```

Grading rubric:
criterion: The final answer is a Python list containing exactly 7 inner lists.
weight: 5
criterion: Each inner list has exactly 13 elements, and the first element of the 7 rows is the rank sequence 1 through 7 in order.
weight: 10
criterion: The ordered group identifiers for the 7 returned rows are exactly [('AR', 'Government', 'Books'), ('DC', 'Government', 'Electronics'), ('VT', 'Enterprise', 'Beauty'), ('OH', 'SMB', 'Apparel'), ('FM', 'SMB', 'Beauty'), ('ND', 'Government', 'Apparel'), ('SD', 'SMB', 'Beauty')].
weight: 15
criterion: The rank-1 row is exactly [1, 'AR', 'Government', 'Books', 350, 870400.3, 0.406445, 0.039131, 0.099865, 0.242857, 0.66, 0.524457, 0.464585].
weight: 15
criterion: The rank-4 row is exactly [4, 'OH', 'SMB', 'Apparel', 312, 746429.6, 0.399424, 0.039269, 0.107982, 0.230769, 0.637821, 0.505224, 0.450319].
weight: 10
criterion: The rank-7 row is exactly [7, 'SD', 'SMB', 'Beauty', 345, 734623.4, 0.397375, 0.038644, 0.108882, 0.197101, 0.631884, 0.505449, 0.421322].
weight: 10
criterion: For every returned row, revenue is rounded to 2 decimal places or fewer and each rate/score field (columns 7 through 13) is rounded to 6 decimal places or fewer.
weight: 10
criterion: The returned composite risk scores in rank order are exactly [0.406445, 0.40346, 0.401051, 0.399424, 0.39913, 0.398535, 0.397375].
weight: 10
criterion: Each returned row satisfies the minimum group-size filter from the prompt, i.e. the n_lines value (column 5) is at least 200.
weight: 5
criterion: The returned rows are consistent with the prompt's specified deterministic ranking order: composite_risk_score descending, then revenue descending, then n_lines descending, then customer_state ascending, then customer_segment ascending, then product_category ascending.
weight: 5
criterion: All numeric fields used in the final output are correctly parsed from the raw data: percentage fields are converted to decimal form, currency symbols and separators are removed, and no values remain as strings before aggregation.
weight: 5

Golden answer:
### Step-by-Step Explanation

**Step 1: Data Cleaning**

The dataset contains multiple formatting anomalies that must be corrected:

- Text fields have leading/trailing whitespace and are wrapped in double-quotes
- Year fields are formatted as `01-01-YYYY` and must be extracted as integers
- Percentage fields contain `%` symbols and must be converted to decimals (without dividing by 100)
- Numeric fields have unit suffixes (`km`, `kg`, `cm3`) that must be stripped
- Monetary fields contain currency symbols (`₹`, `â‚¹`) that must be removed
- ID fields are zero-padded (e.g., `000400`) and must be converted to integers

After applying all cleaning transformations, the dataset is ready for analysis.

---

**Step 2: Compute Per-Row Metrics**

For each order line, calculate:

- **Discount mismatch ratio** = `abs(discount_amount - line_subtotal × discount_pct) / line_subtotal`
  - Handle division-by-zero by replacing invalid values with `0.0`
  - Clip negative values to `0.0`

- **Is return?** = `1` if `return_reason` is non-empty (after trimming), else `0`

- **Is non-final?** = `1` if `order_status` is NOT in `{Completed, Delivered}`, else `0`

---

**Step 3: Group-Level Aggregation**

Group the cleaned data by `(customer_state, customer_segment, product_category)` and compute:

- `n_lines` = count of rows
- `revenue` = sum of `total_amount`
- `avg_mismatch` = mean of `discount_mismatch_ratio`
- `p90_mismatch` = 90th percentile of `discount_mismatch_ratio` (using NumPy's linear interpolation)
- `return_rate` = mean of `is_return`
- `non_final_rate` = mean of `is_non_final`
- `avg_churn` = mean of `churn_risk_score`
- `margin_rate` = `sum(gross_margin_amount) / sum(total_amount)` (handle division-by-zero)

---

**Step 4: Compute Composite Risk Score**

For each group:

```
discount_behavior = (avg_mismatch + p90_mismatch) / 2.0
churn_normalized = avg_churn / 100.0
margin_weakness = 1.0 - margin_rate

composite_risk_score = (discount_behavior + return_rate + non_final_rate + churn_normalized + margin_weakness) / 5.0
```

---

**Step 5: Filter, Sort, and Rank**

- Keep only groups with `n_lines >= 200`
- Sort deterministically by:
  1. `composite_risk_score` descending
  2. `revenue` descending
  3. `n_lines` descending
  4. `customer_state` ascending
  5. `customer_segment` ascending
  6. `product_category` ascending
- Assign `rank = 1, 2, ..., 7` to the top 7 groups

---

**Step 6: Apply Rounding Rules**

- Revenue: 2 decimals
- All scores/rates: 6 decimals

---

### Python Code

```python
import re
import numpy as np
import pandas as pd


IN_PATH = "unclean.csv"


# -----------------------------
# Cleaning helpers
# -----------------------------
_CURRENCY_RE = re.compile(r"[^\d\.\-]+")  # strips ₹, â‚¹, commas, etc.
_PCT_RE = re.compile(r"^\s*([+-]?\d*\.?\d+)\s*%?\s*$")
_UNIT_RE = re.compile(r"^\s*([+-]?\d*\.?\d+)\s*(km|kg|cm3)\s*$", re.IGNORECASE)
_YEAR_TO_DATE_RE = re.compile(r"^\s*01-01-(\d{4})\s*$")
_QUOTED_RE = re.compile(r'^\s*"(.*)"\s*$', re.DOTALL)


def _strip_quotes_and_ws(s: pd.Series) -> pd.Series:
    s = s.astype("string").str.strip()
    m = s.str.match(_QUOTED_RE)
    s = s.where(~m, s.str.replace(r'^\s*"(.*)"\s*$', r"\1", regex=True))
    return s.str.strip()


def _to_numeric_from_money(s: pd.Series) -> pd.Series:
    s = s.astype("string").str.strip()
    s = s.str.replace(",", "", regex=False)
    s = s.str.replace(_CURRENCY_RE, "", regex=True)
    return pd.to_numeric(s, errors="coerce")


def _to_numeric_from_pct_symbol(s: pd.Series) -> pd.Series:
    s = s.astype("string").str.strip()
    s = s.str.replace(",", "", regex=False)
    s = s.str.replace("%", "", regex=False)
    return pd.to_numeric(s, errors="coerce")


def _to_numeric_from_unit_suffix(s: pd.Series) -> pd.Series:
    s = s.astype("string").str.strip()
    s = s.str.replace(",", "", regex=False)
    s = s.str.replace(r"\s*(km|kg|cm3)\s*$", "", regex=True, flags=re.IGNORECASE)
    return pd.to_numeric(s, errors="coerce")


def _year_to_int(s: pd.Series) -> pd.Series:
    s = s.astype("string").str.strip()
    s = s.where(~s.str.match(_YEAR_TO_DATE_RE), s.str.replace(r"^\s*01-01-(\d{4})\s*$", r"\1", regex=True))
    return pd.to_numeric(s, errors="coerce").astype("Int64")


def _id_zero_pad_to_int(s: pd.Series) -> pd.Series:
    s = s.astype("string").str.strip()
    s = s.str.replace(r"^0+(\d+)$", r"\1", regex=True)
    return pd.to_numeric(s, errors="coerce").astype("Int64")


def clean_dataset(df: pd.DataFrame) -> pd.DataFrame:
    df = df.copy()

    obj_cols = [c for c in df.columns if df[c].dtype == "object" or str(df[c].dtype).startswith("string")]
    for c in obj_cols:
        df[c] = _strip_quotes_and_ws(df[c])

    for c in ["order_year", "signup_year"]:
        if c in df.columns:
            df[c] = _year_to_int(df[c])

    for c in ["tax_pct", "discount_pct"]:
        if c in df.columns:
            df[c] = _to_numeric_from_pct_symbol(df[c])

    for c in ["shipping_distance_km", "weight_kg", "volume_cm3"]:
        if c in df.columns:
            df[c] = _to_numeric_from_unit_suffix(df[c])

    money_cols = [
        "total_amount", "income_monthly", "discount_amount", "line_subtotal",
        "shipping_fee", "gross_margin_amount", "tax_amount", "unit_cost",
        "unit_price", "lifetime_spend"
    ]
    for c in money_cols:
        if c in df.columns:
            df[c] = _to_numeric_from_money(df[c])

    id_cols = [
        "store_id", "order_id", "order_line_id", "fulfillment_center_id",
        "product_id", "customer_id"
    ]
    for c in id_cols:
        if c in df.columns:
            df[c] = _id_zero_pad_to_int(df[c])

    numeric_like_cols = [
        "order_hour", "order_minute", "order_day", "order_month",
        "signup_day", "signup_month", "delivery_days", "quantity",
        "churn_risk_score", "product_reviews_count", "lifetime_orders",
        "credit_score", "household_size", "customer_age", "product_rating_avg"
    ]
    for c in numeric_like_cols:
        if c in df.columns:
            df[c] = pd.to_numeric(df[c], errors="coerce")

    for c in ["customer_state", "customer_segment", "product_category", "order_status", "return_reason", "currency"]:
        if c in df.columns:
            df[c] = df[c].astype("string").str.strip()

    return df


def compute_top7_risk_groups(df: pd.DataFrame) -> list:
    required = [
        "customer_state", "customer_segment", "product_category",
        "line_subtotal", "discount_amount", "discount_pct",
        "order_status", "return_reason", "churn_risk_score",
        "gross_margin_amount", "total_amount"
    ]
    missing = [c for c in required if c not in df.columns]
    if missing:
        raise ValueError(f"Missing required columns: {missing}")

    ls = df["line_subtotal"].astype(float)
    da = df["discount_amount"].astype(float)
    dp = df["discount_pct"].astype(float)

    expected_disc = ls * dp
    with np.errstate(divide="ignore", invalid="ignore"):
        mismatch_ratio = np.abs(da - expected_disc) / ls
    mismatch_ratio = mismatch_ratio.replace([np.inf, -np.inf], np.nan).fillna(0.0)
    mismatch_ratio = mismatch_ratio.clip(lower=0.0)

    df = df.copy()
    df["_discount_mismatch_ratio"] = mismatch_ratio

    rr = df["return_reason"].astype("string").fillna("").str.strip()
    df["_is_return"] = (rr != "").astype(int)

    status = df["order_status"].astype("string").fillna("").str.strip()
    df["_is_non_final"] = (~status.isin(["Completed", "Delivered"])).astype(int)

    df["_revenue"] = df["total_amount"].astype(float)

    grp_cols = ["customer_state", "customer_segment", "product_category"]

    def p90(x: pd.Series) -> float:
        return float(np.percentile(x.to_numpy(dtype=float), 90))

    g = df.groupby(grp_cols, dropna=False)

    agg = g.agg(
        n_lines=("total_amount", "size"),
        revenue=("_revenue", "sum"),
        avg_mismatch=("_discount_mismatch_ratio", "mean"),
        p90_mismatch=("_discount_mismatch_ratio", p90),
        return_rate=("_is_return", "mean"),
        non_final_rate=("_is_non_final", "mean"),
        avg_churn=("churn_risk_score", "mean"),
        gm_sum=("gross_margin_amount", "sum"),
        ta_sum=("total_amount", "sum"),
    ).reset_index()

    with np.errstate(divide="ignore", invalid="ignore"):
        agg["margin_rate"] = (agg["gm_sum"].astype(float) / agg["ta_sum"].astype(float))
    agg["margin_rate"] = agg["margin_rate"].replace([np.inf, -np.inf], np.nan).fillna(0.0)

    agg["discount_behavior"] = (agg["avg_mismatch"] + agg["p90_mismatch"]) / 2.0
    agg["churn_norm"] = agg["avg_churn"].astype(float) / 100.0
    agg["margin_bad"] = 1.0 - agg["margin_rate"].astype(float)

    agg["composite_risk_score"] = (
        agg["discount_behavior"]
        + agg["return_rate"].astype(float)
        + agg["non_final_rate"].astype(float)
        + agg["churn_norm"]
        + agg["margin_bad"]
    ) / 5.0

    agg = agg[agg["n_lines"] >= 200].copy()

    agg = agg.sort_values(
        by=[
            "composite_risk_score", "revenue", "n_lines",
            "customer_state", "customer_segment", "product_category"
        ],
        ascending=[False, False, False, True, True, True],
        kind="mergesort",
    ).reset_index(drop=True)

    top = agg.head(7).copy()
    top.insert(0, "rank", np.arange(1, len(top) + 1))

    def r2(x): return float(np.round(x, 2))
    def r6(x): return float(np.round(x, 6))

    out = []
    for _, row in top.iterrows():
        out.append([
            int(row["rank"]),
            str(row["customer_state"]),
            str(row["customer_segment"]),
            str(row["product_category"]),
            int(row["n_lines"]),
            r2(row["revenue"]),
            r6(row["composite_risk_score"]),
            r6(row["avg_mismatch"]),
            r6(row["p90_mismatch"]),
            r6(row["return_rate"]),
            r6(row["non_final_rate"]),
            r6(row["avg_churn"] / 100.0),
            r6(row["margin_rate"]),
        ])

    return out


def main():
    df_unclean = pd.read_csv(IN_PATH, low_memory=False)
    df_clean = clean_dataset(df_unclean)
    result = compute_top7_risk_groups(df_clean)
    print(result)


if __name__ == "__main__":
    main()
```

---

### Ground Truth Final Answer (GTFA)

```python
[[1, 'AR', 'Government', 'Books', 350, 870400.3, 0.406445, 0.039131, 0.099865, 0.242857, 0.66, 0.524457, 0.464585], [2, 'DC', 'Government', 'Electronics', 523, 1153802.34, 0.40346, 0.037918, 0.099863, 0.231358, 0.650096, 0.505373, 0.438415], [3, 'VT', 'Enterprise', 'Beauty', 686, 1556883.59, 0.401051, 0.038461, 0.099861, 0.240525, 0.632653, 0.506006, 0.443089], [4, 'OH', 'SMB', 'Apparel', 312, 746429.6, 0.399424, 0.039269, 0.107982, 0.230769, 0.637821, 0.505224, 0.450319], [5, 'FM', 'SMB', 'Beauty', 331, 711344.82, 0.39913, 0.040385, 0.108886, 0.23565, 0.628399, 0.515438, 0.45847], [6, 'ND', 'Government', 'Apparel', 327, 708252.43, 0.398535, 0.03972, 0.103479, 0.207951, 0.633028, 0.518318, 0.43822], [7, 'SD', 'SMB', 'Beauty', 345, 734623.4, 0.397375, 0.038644, 0.108882, 0.197101, 0.631884, 0.505449, 0.421322]]
```