Domain: Business Operations & Analytics

You are given four CSV files:
- `Hackathon_Mapping_File.csv`
- `Hackathon_Validation_Data.csv`
- `Hackathon_Ideal_Data.csv`
- `Hackathon_Working_Data.csv`

Your task is to perform a multi-stage data quality analysis pipeline and produce a final ranked output.

---

#### **Step 1: Read and Prepare Data**

1. Read all four CSV files as string-typed DataFrames with empty strings preserved (use `dtype=str, keep_default_na=False`).
2. In the mapping file, forward-fill the `File Name` column to propagate section labels.
3. Extract expected column lists from the mapping file for `Hackathon_Ideal_Data` and `Hackathon_Working_Data`.
4. Count **schema violations** as the total number of missing columns plus extra columns across both transactional files compared to mapping expectations.

---

#### **Step 2: Text Normalization and Canonical Keys**

1. Normalize all `GRP` text fields to uppercase and collapse internal whitespace to single spaces, creating a new column `GRP_N`.
2. In the validation table, create a canonical key from `(STORECODE, MONTH, GRP_N)`.
3. Flag **inconsistent ID rows** where either:
   - The same `ID` appears multiple times, OR
   - The same canonical key `(STORECODE, MONTH, GRP_N)` maps to multiple different `ID` values.

---

#### **Step 3: Null Coercion and Bridging**

1. In both transactional files, treat the following as null tokens: empty string, `-`, `\N`, `NONE`, `NULL`, `NAN` (case-insensitive).
2. Attempt to parse numeric fields (`DAY`, `BILL_AMT`, `QTY`, `VALUE`, `PRICE` in working; `QTY`, `VALUE` in ideal) as floats.
3. Flag **null coercion rows** where any numeric parse fails OR any relevant field contains a null token.
4. In the ideal file, bridge `STORECODE` tokens by extracting the numeric suffix from `P<digits>` format and prepending `N` to create `STORECODE_N`. Mark rows with non-matching formats as null bridges.

---

#### **Step 4: Aggregation**

1. Aggregate both transactional files to the canonical `(STORECODE, MONTH, GRP_N)` level, summing `QTY` and `VALUE`.
2. Count contributing rows, null coercion rows, spacing anomaly rows (where whitespace normalization changes the text), and price mismatch rows (where `PRICE * QTY ≠ VALUE` beyond 1e-6 tolerance) per canonical key.
3. For working data, also count the number of distinct bill ID timestamp length classes (after removing alphabetic prefix) per canonical key.

---

#### **Step 5: Join and Derive Metrics**

1. Left-join both aggregated tables onto the validation canonical key grid.
2. Fill missing aggregates with zero.
3. Mark **missing_working_flag** = 1 when no working rows exist for a canonical key.
4. Mark **missing_ideal_flag** = 1 when no ideal rows exist for a canonical key.
5. Compute quantity delta `dq = working_qty - ideal_qty` and value delta `dv = working_value - ideal_value`.
6. Mark **sign_reversal_flag** = 1 when both sources exist and `dq * dv < 0`.
7. Mark **mixed_timestamp_flag** = 1 when more than one distinct timestamp length class exists for a canonical key in working data.
8. Compute **outlier_flag** = 1 when:
   - Both sources exist,
   - Log-ratio = `log1p(working_value) - log1p(ideal_value)`,
   - Absolute deviation from median log-ratio exceeds 3 times the median absolute deviation (MAD).

---

#### **Step 6: Store-Month Aggregation and Stress Score**

1. Aggregate all flags and anomaly counts to `(STORECODE, MONTH)` level.
2. Compute the **stress score** for each store-month using the following exact formula:

```
stress_score = 7 * inconsistent_id_rows
             + 6 * null_coercion_rows
             + 5 * sign_reversal_rows
             + 4 * missing_working_rows
             + 3 * missing_ideal_rows
             + 2 * mixed_timestamp_rows
             + 3 * outlier_rows
             + 1 * anomaly_rows
             + 2 * price_mismatch_rows
             + 1 * schema_violation_rows
```

Where:
- `inconsistent_id_rows` = sum of inconsistent ID flags
- `null_coercion_rows` = sum of null coercion flags from both sources
- `sign_reversal_rows` = sum of sign reversal flags
- `missing_working_rows` = sum of missing working flags
- `missing_ideal_rows` = sum of missing ideal flags
- `mixed_timestamp_rows` = sum of mixed timestamp flags
- `outlier_rows` = sum of outlier flags
- `anomaly_rows` = sum of spacing anomaly flags
- `price_mismatch_rows` = sum of price mismatch flags
- `schema_violation_rows` = schema violation count computed in Step 1

---

#### **Step 7: Rank and Output**

1. Rank store-month pairs by:
   - **Primary**: `stress_score` descending
   - **Secondary**: `missing_working_rows` descending
   - **Tertiary**: `sign_reversal_rows` descending
   - **Quaternary**: numeric part of `STORECODE` ascending
   - **Quinary**: numeric part of `MONTH` ascending

2. Select the top 10 ranked store-month pairs.

3. For each of the top 10, construct a list with 11 elements (all integers):
   ```
   [rank, STORECODE, MONTH, stress_score, missing_working_rows, missing_ideal_rows,
    sign_reversal_rows, mixed_timestamp_rows, outlier_rows, anomaly_rows, price_mismatch_rows]
   ```

4. Compute summary metrics:
   - `total_stress` = sum of stress scores across top 10
   - `total_missing_working` = sum of missing_working_rows across top 10
   - `total_sign_reversal` = sum of sign_reversal_rows across top 10
   - `md5_hash` = MD5 hash of the JSON-serialized top-10 list (using `separators=(',', ':')`)

5. Return a Python list with exactly 2 elements:
   ```
   [
     <list of 10 ranked rows>,
     [total_stress, total_missing_working, total_sign_reversal, md5_hash]
   ]
   ```


Grading rubric:
criterion: Final output is a Python list with 2 elements.
weight: 10
criterion: Element 1 equals [3651, 238, 99, 'b5135bc611b562f829dce798e72ac289'].
weight: 10
criterion: Element 0 row 1 equals [1, 'N9', 'M1', 398, 26, 22, 15, 33, 6, 69, 0].
weight: 10
criterion: Element 0 row 10 equals [10, 'N10', 'M2', 337, 40, 14, 8, 0, 1, 92, 0].
weight: 10
criterion: Stress values at row positions 1 through 10 are nonincreasing.
weight: 10
criterion: Sum of stress values across the ten rows equals 3651.
weight: 10
criterion: Sum of missing_working values across the ten rows equals 238.
weight: 10
criterion: Sum of sign_reversal values across the ten rows equals 99.
weight: 10
criterion: For each row stress equals 7 times inconsistent_id plus 6 times null_coercion plus 5 times sign_reversal plus 4 times missing_working plus 3 times missing_ideal plus 2 times mixed_timestamp plus 3 times outlier plus anomaly plus 2 times price_mismatch plus schema_violation.
weight: 10
criterion: Concatenated store and month sequence across rows 1 to 10 equals N9M1|N1M1|N2M3|N4M3|N5M1|N4M1|N5M3|N5M2|N7M1|N10M2.
weight: 10

Golden answer:
```python
import pandas as pd
import json
import hashlib
import math

def read_csv_exact(name):
    try:
        return pd.read_csv(name, dtype=str, keep_default_na=False)
    except FileNotFoundError:
        return pd.read_csv(f"{name}", dtype=str, keep_default_na=False)

mapping = read_csv_exact("Hackathon_Mapping_File.csv")
validation = read_csv_exact("Hackathon_Validation_Data.csv")
ideal_raw = read_csv_exact("Hackathon_Ideal_Data.csv")
working_raw = read_csv_exact("Hackathon_Working_Data.csv")

mapping["File Name"] = mapping["File Name"].replace("", pd.NA).ffill()
expected_ideal_cols = mapping.loc[mapping["File Name"] == "Hackathon_Ideal_Data", "Column Name"].tolist()
expected_work_cols = mapping.loc[mapping["File Name"] == "Hackathon_Working_Data", "Column Name"].tolist()

schema_violations = (
    len(set(expected_ideal_cols) - set(ideal_raw.columns))
    + len(set(ideal_raw.columns) - set(expected_ideal_cols))
    + len(set(expected_work_cols) - set(working_raw.columns))
    + len(set(working_raw.columns) - set(expected_work_cols))
)

def norm_text(series):
    return series.astype(str).str.upper().str.replace(r"\s+", " ", regex=True).str.strip()

def ws_norm(series):
    return series.astype(str).str.replace(r"\s+", " ", regex=True).str.strip()

val = validation.copy()
val["ID"] = pd.to_numeric(val["ID"], errors="coerce")
val["STORECODE"] = val["STORECODE"].str.strip()
val["MONTH"] = val["MONTH"].str.strip()
val["GRP_N"] = norm_text(val["GRP"])
val["inconsistent_id_flag"] = (
    val.duplicated(subset=["ID"], keep=False)
    | val.duplicated(subset=["STORECODE", "MONTH", "GRP_N"], keep=False)
).astype(int)

w = working_raw.copy()
w["STORECODE"] = w["STORECODE"].str.strip()
w["MONTH"] = w["MONTH"].str.strip()
w["GRP_N"] = norm_text(w["GRP"])
w["spacing_anomaly_row"] = (ws_norm(w["GRP"]) != w["GRP"].str.strip()).astype(int)

for c in ["DAY", "BILL_AMT", "QTY", "VALUE", "PRICE"]:
    w[c + "_NUM"] = pd.to_numeric(w[c], errors="coerce")

null_tokens = {"", "-", "\\N", "NONE", "NULL", "NAN"}
token_null_flag = pd.Series(False, index=w.index)
for c in ["DAY", "BILL_AMT", "QTY", "VALUE", "PRICE", "BILL_ID", "GRP"]:
    token_null_flag = token_null_flag | w[c].astype(str).str.strip().str.upper().isin(null_tokens)

w["null_coercion_row"] = (
    w[["DAY_NUM", "BILL_AMT_NUM", "QTY_NUM", "VALUE_NUM", "PRICE_NUM"]].isna().any(axis=1)
    | token_null_flag
).astype(int)

suffix = w["BILL_ID"].str.extract(r"^[A-Za-z]+(\d+)$", expand=False)
slen = suffix.str.len()
w["stamp_class"] = "OTHER"
w.loc[slen.notna() & (slen <= 2), "stamp_class"] = "L2"
w.loc[slen.notna() & (slen == 3), "stamp_class"] = "L3"
w.loc[slen.notna() & (slen == 4), "stamp_class"] = "L4"
w.loc[slen.notna() & (slen >= 5), "stamp_class"] = "L5"

w["price_mismatch_row"] = (
    w["QTY_NUM"].notna()
    & (w["QTY_NUM"] != 0)
    & w["VALUE_NUM"].notna()
    & w["PRICE_NUM"].notna()
    & ((w["PRICE_NUM"] * w["QTY_NUM"] - w["VALUE_NUM"]).abs() > 1e-6)
).astype(int)

i = ideal_raw.copy()
i["STORECODE_N"] = "N" + i["STORECODE"].str.extract(r"(\d+)", expand=False).fillna("")
i.loc[~i["STORECODE"].str.match(r"^P\d+$"), "STORECODE_N"] = pd.NA
i["MONTH"] = i["MONTH"].str.strip()
i["GRP_N"] = norm_text(i["GRP"])

for c in ["QTY", "VALUE"]:
    i[c + "_NUM"] = pd.to_numeric(i[c], errors="coerce")

i_token_null_flag = pd.Series(False, index=i.index)
for c in ["QTY", "VALUE", "GRP", "STORECODE"]:
    i_token_null_flag = i_token_null_flag | i[c].astype(str).str.strip().str.upper().isin(null_tokens)

i["null_coercion_row"] = (
    i[["QTY_NUM", "VALUE_NUM"]].isna().any(axis=1)
    | i_token_null_flag
    | i["STORECODE_N"].isna()
).astype(int)

w_agg = w.groupby(["STORECODE", "MONTH", "GRP_N"], as_index=False).agg(
    w_qty=("QTY_NUM", "sum"),
    w_value=("VALUE_NUM", "sum"),
    w_rows=("GRP_N", "size"),
    w_null_coercion_rows=("null_coercion_row", "sum"),
    w_spacing_anomaly_rows=("spacing_anomaly_row", "sum"),
    w_price_mismatch_rows=("price_mismatch_row", "sum"),
    stamp_class_n=("stamp_class", "nunique"),
)
w_agg["mixed_timestamp_flag"] = (w_agg["stamp_class_n"] > 1).astype(int)

i_agg = i.groupby(["STORECODE_N", "MONTH", "GRP_N"], as_index=False).agg(
    i_qty=("QTY_NUM", "sum"),
    i_value=("VALUE_NUM", "sum"),
    i_rows=("GRP_N", "size"),
    i_null_coercion_rows=("null_coercion_row", "sum"),
)

base = val[["ID", "STORECODE", "MONTH", "GRP_N", "GRP", "inconsistent_id_flag"]].copy()
base = base.merge(w_agg, on=["STORECODE", "MONTH", "GRP_N"], how="left")
base = base.merge(i_agg, left_on=["STORECODE", "MONTH", "GRP_N"], right_on=["STORECODE_N", "MONTH", "GRP_N"], how="left")

for c in [
    "w_qty", "w_value", "w_rows", "w_null_coercion_rows", "w_spacing_anomaly_rows",
    "w_price_mismatch_rows", "stamp_class_n", "mixed_timestamp_flag",
    "i_qty", "i_value", "i_rows", "i_null_coercion_rows"
]:
    base[c] = base[c].fillna(0)

base["missing_working_flag"] = (base["w_rows"] == 0).astype(int)
base["missing_ideal_flag"] = (base["i_rows"] == 0).astype(int)
base["null_coercion_rows"] = base["w_null_coercion_rows"] + base["i_null_coercion_rows"]

base["dq"] = base["w_qty"] - base["i_qty"]
base["dv"] = base["w_value"] - base["i_value"]
base["sign_reversal_flag"] = (
    (base["w_rows"] > 0) & (base["i_rows"] > 0) & ((base["dq"] * base["dv"]) < 0)
).astype(int)

mask = (base["w_rows"] > 0) & (base["i_rows"] > 0)
base.loc[mask, "log_ratio"] = base.loc[mask, "w_value"].apply(math.log1p) - base.loc[mask, "i_value"].apply(math.log1p)
med = base.loc[mask, "log_ratio"].median()
mad = (base.loc[mask, "log_ratio"] - med).abs().median()

if pd.isna(mad) or mad == 0:
    base["outlier_flag"] = 0
else:
    base["outlier_flag"] = (mask & ((base["log_ratio"] - med).abs() > (3 * mad))).astype(int)

sm = base.groupby(["STORECODE", "MONTH"], as_index=False).agg(
    inconsistent_id_rows=("inconsistent_id_flag", "sum"),
    null_coercion_rows=("null_coercion_rows", "sum"),
    sign_reversal_rows=("sign_reversal_flag", "sum"),
    mixed_timestamp_rows=("mixed_timestamp_flag", "sum"),
    outlier_rows=("outlier_flag", "sum"),
    anomaly_rows=("w_spacing_anomaly_rows", "sum"),
    price_mismatch_rows=("w_price_mismatch_rows", "sum"),
    missing_working_rows=("missing_working_flag", "sum"),
    missing_ideal_rows=("missing_ideal_flag", "sum"),
)

sm["schema_violation_rows"] = schema_violations
sm["stress_score"] = (
    7 * sm["inconsistent_id_rows"]
    + 6 * sm["null_coercion_rows"]
    + 5 * sm["sign_reversal_rows"]
    + 4 * sm["missing_working_rows"]
    + 3 * sm["missing_ideal_rows"]
    + 2 * sm["mixed_timestamp_rows"]
    + 3 * sm["outlier_rows"]
    + sm["anomaly_rows"]
    + 2 * sm["price_mismatch_rows"]
    + sm["schema_violation_rows"]
)

sm["store_num"] = pd.to_numeric(sm["STORECODE"].str.extract(r"(\d+)", expand=False), errors="coerce")
sm["month_num"] = pd.to_numeric(sm["MONTH"].str.extract(r"(\d+)", expand=False), errors="coerce")

top10 = sm.sort_values(
    ["stress_score", "missing_working_rows", "sign_reversal_rows", "store_num", "month_num"],
    ascending=[False, False, False, True, True]
).head(10).reset_index(drop=True)

top10["rank"] = range(1, 11)

rows = []
for _, r in top10.iterrows():
    rows.append([
        int(r["rank"]),
        str(r["STORECODE"]),
        str(r["MONTH"]),
        int(r["stress_score"]),
        int(r["missing_working_rows"]),
        int(r["missing_ideal_rows"]),
        int(r["sign_reversal_rows"]),
        int(r["mixed_timestamp_rows"]),
        int(r["outlier_rows"]),
        int(r["anomaly_rows"]),
        int(r["price_mismatch_rows"]),
    ])

md5 = hashlib.md5(json.dumps(rows, separators=(",", ":")).encode()).hexdigest()
summary = [sum(x[3] for x in rows), sum(x[4] for x in rows), sum(x[6] for x in rows), md5]
final_answer = [rows, summary]
print(final_answer)
```

## GTFA
```
[[[1, 'N9', 'M1', 398, 26, 22, 15, 33, 6, 69, 0], [2, 'N1', 'M1', 398, 19, 15, 12, 50, 5, 102, 0], [3, 'N2', 'M3', 379, 23, 15, 12, 37, 9, 69, 6], [4, 'N4', 'M3', 362, 28, 16, 10, 43, 6, 48, 0], [5, 'N5', 'M1', 358, 25, 30, 7, 37, 4, 47, 0], [6, 'N4', 'M1', 356, 32, 16, 10, 34, 8, 38, 0], [7, 'N5', 'M3', 356, 12, 35, 10, 0, 5, 138, 0], [8, 'N5', 'M2', 354, 16, 32, 10, 0, 4, 132, 0], [9, 'N7', 'M1', 353, 17, 25, 5, 49, 6, 65, 2], [10, 'N10', 'M2', 337, 40, 14, 8, 0, 1, 92, 0]], [3651, 238, 99, 'b5135bc611b562f829dce798e72ac289']]
```