Domain: Business Operations & Analytics

Domain: Data Science & Revenue Analytics.

A revenue-forensics unit must compute a deterministic Structural Divergence Index (SDI) from `VK_012.csv` for calendar year 2023 only under strict reconciliation and branching rules.

## Parsing and Missingness

Treat all fields as strings first.
Trim leading and trailing whitespace.
Treat the literal strings `"nan"`, `"none"`, `"null"`, and `""` as missing, case-insensitive after trimming.

Parse `order_date` strictly as ISO `YYYY-MM-DD`. Invalid parses become missing.

Coerce the following fields to numeric, with invalid parses becoming missing:
- `price`
- `discount_percent`
- `quantity_sold`
- `rating`
- `review_count`
- `discounted_price`
- `total_revenue`

## Deduplication

Deduplicate by `order_id`.

Within each `order_id`, sort rows by:
1. `disc_err = |discounted_price − price × (1 − discount_percent/100)|`, where `disc_err = +∞` if either side is missing
2. smallest original row index

Keep the first row only.

## Restriction and Required Fields

Restrict to calendar year 2023 only.

Then drop rows missing any of:
- `customer_region`
- `product_category`
- `price`
- `discount_percent`
- `quantity_sold`
- `total_revenue`

## Reconciliation Rules

Define:

`computed_discounted = price × (1 − discount_percent/100)`

For `discounted_price`:
- use observed `discounted_price` if present and `|discounted_price − computed_discounted| ≤ 0.02`
- otherwise use `round(computed_discounted, 2)`

For `total_revenue`:
- use observed `total_revenue` if present and within tolerance `0.05` of reconciled unit price × `quantity_sold`
- otherwise use `round(reconciled_discounted_price × quantity_sold, 2)`

## Canonicalization

Canonicalize region and category exactly as follows.

For region:
1. uppercase `customer_region`
2. remove all non `A–Z` characters
3. call the result `region_c`

For category:
1. lowercase `product_category`
2. remove all non `a–z` characters
3. call the result `cat_c`

No further mapping is allowed.

## Cell Aggregation

Aggregate to `region_c × cat_c`.

For each cell compute:
- `total_rev = sum(reconciled total_revenue)`
- `total_qty = sum(quantity_sold)`

Define:

`base_score = total_rev / (total_qty + 1)`

## Weighted Rating Per Cell

Before weighting:
- set negative `review_count` to missing
- clip `rating` into `[0, 5]`

For each valid row define weight:

`w = ln(1 + review_count)`

For each cell, compute weighted rating from rows with valid `rating` and valid positive-or-zero `review_count` after the rule above.

If a cell has no valid weights, set:

`w_rating = 2.5`

Do not drop the cell.

Define:

`adj_score = base_score × (1 + (5 − w_rating)/5)`

## Sorted Score Vector

Sort all `adj_score` values strictly ascending. If ties occur, break ties by lexicographic order of `(region_c, cat_c)`. Take the first 25 values to form vector `S`.

Compute:
- the Type-7 median `M`
- the Type-7 37th percentile `P37`

Use explicit Type-7 linear interpolation only. No other percentile method is allowed.

## Deterministic Branching Rule

If `P37 > M/2`, multiply all `adj_score` values at odd sorted indices `(i = 1, 3, 5, ...)` by `1.03`.

Otherwise, multiply all `adj_score` values at even sorted indices `(i = 2, 4, 6, ...)` by `0.97`.

After this transformation, define `T` as the modified sorted vector.

## Final Metrics

Compute the Gini coefficient:

`G = (n + 1 − 2 × (sum(cumulative(T)) / sum(T))) / n`

using `n = 25` exactly.

Define:
- `K` = number of values in `S` whose original `adj_score` strictly exceeded `P37`
- `Ω = 1 + (K mod 5)/10`

Finally compute:

`SDI = G × (P37 / M) × Ω`

## Report Exactly These Scalars

1. Rows remaining after strict `order_id` deduplication
2. Rows remaining after year-2023 restriction and required-field dropping
3. Number of canonicalized `region_c × cat_c` cells
4. Type-7 37th percentile `P37` of `adj_score` before branching, rounded to 4 decimals
5. Type-7 median `M` of `adj_score` before branching, rounded to 4 decimals
6. Branch condition result (`1` if `P37 > M/2`, else `0`)
7. Sum of `adj_score` values at odd sorted indices `(i = 1,3,5,...)` before branching, rounded to 4 decimals
8. Sum of transformed scores `T` after parity multiplier, rounded to 4 decimals
9. Gini coefficient `G` after branching transformation, rounded to 4 decimals
10. Count `K` of values in `S` with original `adj_score` strictly greater than `P37`
11. Global invariant multiplier `Ω`
12. Final Structural Divergence Index `SDI`, rounded to 4 decimals

## Dataset Information

File: `VK_012.csv`

Columns:
- `order_id`: string; deduplication key
- `order_date`: string date in ISO `YYYY-MM-DD`
- `product_id`: product identifier, not used in aggregation
- `product_category`: noisy category label
- `price`: list price per unit
- `discount_percent`: percentage discount
- `quantity_sold`: units sold
- `customer_region`: noisy region label
- `payment_method`: payment channel, not used
- `rating`: noisy rating
- `review_count`: noisy review count
- `discounted_price`: possibly inconsistent discounted unit price
- `total_revenue`: possibly inconsistent row revenue

Grading rubric:
criterion: Final answer reports rows remaining after strict order_id deduplication as 49989.
weight: 10
criterion: Final answer reports rows remaining after year-2023 restriction and required-field dropping as 24979.
weight: 10
criterion: Final answer reports number of canonicalized region_c × cat_c cells as 45.
weight: 10
criterion: Final answer reports Type-7 37th percentile P37 of adj_score before branching as 155.2457.
weight: 10
criterion: Final answer reports Type-7 median M of adj_score before branching as 233.9104.
weight: 10
criterion: Final answer reports branch condition result as 1.
weight: 10
criterion: Final answer reports sum of adj_score at odd sorted indices before branching as 2251.5086.
weight: 10
criterion: Final answer reports sum of transformed scores T after parity multiplier as 4512.5049.
weight: 10
criterion: Final answer reports Gini coefficient G after branching transformation as 0.4139.
weight: 10
criterion: Final answer reports count K of cells with original adj_score strictly greater than P37 as 16.
weight: 10
criterion: Final answer reports global invariant multiplier Ω as 1.1.
weight: 10
criterion: Final answer reports final Structural Divergence Index SDI as 0.3022.
weight: 10

Golden answer:
# Code:

```python
import pandas as pd
import numpy as np
import re

INPUT_PATH = "VK_012.csv"

MISSING_LITERALS = {"nan", "none", "null", ""}

NUM_COLS = [
    "price",
    "discount_percent",
    "quantity_sold",
    "rating",
    "review_count",
    "discounted_price",
    "total_revenue",
]

ISO_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")

def trim_and_missing(s: pd.Series) -> pd.Series:
    s = s.astype(str).str.strip()
    s = s.mask(s.str.lower().isin(MISSING_LITERALS), pd.NA)
    return s

def to_num(s: pd.Series) -> pd.Series:
    return pd.to_numeric(s, errors="coerce")

def parse_date_iso_strict(s: pd.Series) -> pd.Series:
    # strict: must match YYYY-MM-DD exactly, else missing
    s2 = s.astype(str).str.strip()
    ok = s2.notna() & s2.map(lambda x: bool(ISO_DATE_RE.fullmatch(x)) if x is not None else False)
    out = pd.Series(pd.NaT, index=s.index)
    out.loc[ok] = pd.to_datetime(s2.loc[ok], format="%Y-%m-%d", errors="coerce")
    return out

def type7_quantile(values: np.ndarray, q: float) -> float:
    x = np.asarray(values, dtype=float)
    x = x[~np.isnan(x)]
    n = len(x)
    if n == 0:
        return float("nan")
    x.sort()
    if n == 1:
        return float(x[0])
    h = 1 + (n - 1) * q
    j = int(np.floor(h))
    g = h - j
    if j <= 0:
        return float(x[0])
    if j >= n:
        return float(x[-1])
    return float((1 - g) * x[j - 1] + g * x[j])

def canon_region(s: pd.Series) -> pd.Series:
    s = s.astype(str).str.upper()
    s = s.str.replace(r"[^A-Z]", "", regex=True)
    s = s.mask(s == "", pd.NA)
    return s

def canon_cat(s: pd.Series) -> pd.Series:
    s = s.astype(str).str.lower()
    s = s.str.replace(r"[^a-z]", "", regex=True)
    s = s.mask(s == "", pd.NA)
    return s

# ─────────────────────────────────────────────────────────────
# LOAD (all fields as strings) + TRIM/MISSING
# ─────────────────────────────────────────────────────────────
df = pd.read_csv(INPUT_PATH, dtype=str)

for c in df.columns:
    df[c] = trim_and_missing(df[c])

# strict date parse (invalid -> missing)
df["order_date_parsed"] = parse_date_iso_strict(df["order_date"])

# numeric coercions (invalid -> missing)
for c in NUM_COLS:
    df[c + "_num"] = to_num(df[c])

# keep original row order index for deterministic tie-breaking
df["_orig_idx"] = np.arange(len(df), dtype=int)

# ─────────────────────────────────────────────────────────────
# DEDUP BY order_id WITH disc_err + orig index tie-break
# disc_err = |discounted_price − price × (1 − discount_percent/100)|
# disc_err = +∞ if either side missing
# ─────────────────────────────────────────────────────────────
computed_disc = df["price_num"] * (1 - df["discount_percent_num"] / 100.0)
disc_err = (df["discounted_price_num"] - computed_disc).abs()
disc_err = disc_err.where(~(df["discounted_price_num"].isna() | computed_disc.isna()), np.inf)
df["_disc_err"] = disc_err

df_sorted = df.sort_values(["order_id", "_disc_err", "_orig_idx"], kind="mergesort")
dedup = df_sorted.drop_duplicates(subset=["order_id"], keep="first").copy()
rows_after_dedup = int(len(dedup))

# ─────────────────────────────────────────────────────────────
# RESTRICT TO YEAR 2023 + DROP REQUIRED-FIELD MISSING
# ─────────────────────────────────────────────────────────────
dedup_2023 = dedup[dedup["order_date_parsed"].dt.year == 2023].copy()

req_mask = (
    dedup_2023["customer_region"].notna()
    & dedup_2023["product_category"].notna()
    & dedup_2023["price_num"].notna()
    & dedup_2023["discount_percent_num"].notna()
    & dedup_2023["quantity_sold_num"].notna()
    & dedup_2023["total_revenue_num"].notna()
)
d = dedup_2023[req_mask].copy()
rows_after_2023_req = int(len(d))

# ─────────────────────────────────────────────────────────────
# RECONCILE discounted_price (tol 0.02) AND total_revenue (tol 0.05)
# ─────────────────────────────────────────────────────────────
computed_disc2 = d["price_num"] * (1 - d["discount_percent_num"] / 100.0)
dp = d["discounted_price_num"]

use_dp = dp.notna() & computed_disc2.notna() & ((dp - computed_disc2).abs() <= 0.02)
disc_recon = np.where(use_dp, dp.to_numpy(dtype=float), np.round(computed_disc2.to_numpy(dtype=float), 2))
disc_recon = pd.Series(disc_recon, index=d.index, dtype=float)

computed_rev = disc_recon * d["quantity_sold_num"].to_numpy(dtype=float)
tr = d["total_revenue_num"].to_numpy(dtype=float)

use_tr = (~np.isnan(tr)) & (~np.isnan(computed_rev)) & (np.abs(tr - computed_rev) <= 0.05)
rev_recon = np.where(use_tr, tr, np.round(computed_rev, 2))
rev_recon = pd.Series(rev_recon, index=d.index, dtype=float)

d["disc_recon"] = disc_recon
d["rev_recon"] = rev_recon

# ─────────────────────────────────────────────────────────────
# CANONICALIZE region/category (NO further mapping)
# ─────────────────────────────────────────────────────────────
d["region_c"] = canon_region(d["customer_region"])
d["cat_c"] = canon_cat(d["product_category"])

# ─────────────────────────────────────────────────────────────
# AGGREGATE TO region_c × cat_c
# ─────────────────────────────────────────────────────────────
agg = (
    d.groupby(["region_c", "cat_c"], dropna=False, sort=False)
     .agg(total_rev=("rev_recon", "sum"), total_qty=("quantity_sold_num", "sum"))
     .reset_index()
)

# ─────────────────────────────────────────────────────────────
# WEIGHTED RATING PER CELL
# ─────────────────────────────────────────────────────────────
tmp = d[["region_c", "cat_c"]].copy()
rc = d["review_count_num"].where(d["review_count_num"].ge(0))
rt = d["rating_num"].clip(lower=0, upper=5)

w = np.log1p(rc)
valid = w.notna() & rt.notna()

tmp["w"] = w.where(valid)
tmp["wr"] = (w * rt).where(valid)

wr = (
    tmp.groupby(["region_c", "cat_c"], dropna=False, sort=False)
       .agg(sum_w=("w", "sum"), sum_wr=("wr", "sum"))
       .reset_index()
)

cell = agg.merge(wr, on=["region_c", "cat_c"], how="left")
cell["w_rating"] = cell["sum_wr"] / cell["sum_w"]
cell.loc[cell["sum_w"].isna() | (cell["sum_w"] <= 0) | cell["w_rating"].isna(), "w_rating"] = 2.5

cell["base_score"] = cell["total_rev"] / (cell["total_qty"] + 1.0)
cell["adj_score"] = cell["base_score"] * (1.0 + (5.0 - cell["w_rating"]) / 5.0)

num_cells = int(len(cell))

# ─────────────────────────────────────────────────────────────
# SORT + TAKE 25 + TYPE-7
# ─────────────────────────────────────────────────────────────
cell_sorted = cell.sort_values(["adj_score", "region_c", "cat_c"], kind="mergesort").reset_index(drop=True)
S = cell_sorted.iloc[:25]["adj_score"].to_numpy(dtype=float)

P37 = type7_quantile(S, 0.37)
M = type7_quantile(S, 0.50)

# ─────────────────────────────────────────────────────────────
# BRANCHING + SUMS
# ─────────────────────────────────────────────────────────────
branch = 1 if (P37 > (M / 2.0)) else 0

sum_odd_before = float(S[0::2].sum())  # i=1,3,5,... (1-based)

T = S.copy()
if branch == 1:
    T[0::2] = T[0::2] * 1.03
else:
    T[1::2] = T[1::2] * 0.97

sum_T = float(T.sum())

# ─────────────────────────────────────────────────────────────
# GINI + K + OMEGA + SDI
# ─────────────────────────────────────────────────────────────
n = 25
cum = np.cumsum(T)
G = float((n + 1 - 2.0 * (cum.sum() / T.sum())) / n)

K = int((S > P37).sum())
Omega = 1 + (K % 5) / 10.0

SDI = float(G * (P37 / M) * Omega)

# ─────────────────────────────────────────────────────────────
# REPORT 12 SCALARS
# ─────────────────────────────────────────────────────────────
out = [
    rows_after_dedup,
    rows_after_2023_req,
    num_cells,
    round(P37, 4),
    round(M, 4),
    branch,
    round(sum_odd_before, 4),
    round(sum_T, 4),
    round(G, 4),
    K,
    Omega,
    round(SDI, 4),
]

for i, v in enumerate(out, 1):
    print(f"{i}. {v}")
```

# Explanation:

1. **Load + clean strings**
   All fields are read as strings, whitespace is trimmed, and literal "nan", "none", "null", "" are converted to missing.

2. **Strict date + numeric coercion**
   order_date must match YYYY-MM-DD exactly.
   Numeric columns are coerced; invalid values become missing.

3. **Deterministic dedup by order_id**
   Rows are sorted within each order_id by (disc_err, original_row_index) and the first is kept.
   Rows remaining after strict deduplication: 49989

4. **Filter to 2023 + required fields**
   Keep only calendar year 2023 rows.
   Drop rows missing required fields.
   Rows remaining after filtering: 24979

5. **Reconcile discounted_price and total_revenue**
   Apply tolerance rules (0.02 and 0.05 respectively).
   Reconciled values are used in aggregation.

6. **Canonicalize region and category**
   region_c: uppercase + remove non A–Z
   cat_c: lowercase + remove non a–z
   Number of canonicalized region_c × cat_c cells: 45

7. **Aggregate + base_score**
   base_score = total_rev / (total_qty + 1)

8. **Weighted rating per cell**
   w = ln(1 + review_count)
   Clip rating into [0,5]
   If no valid weights, w_rating = 2.5
   adj_score = base_score × (1 + (5 − w_rating)/5)

9. **Build S (first 25 sorted adj_score values)**
   All 45 cells are sorted ascending by adj_score, with ties broken lexicographically by (region_c, cat_c). The first 25 values form vector S.
   Type-7 37th percentile P37: 155.2457
   Type-7 median M: 233.9104

10. **Branching rule**
    Check if P37 > M/2.
    Since 155.2457 > 116.9552, condition is true.
    Branch result: 1
    Multiply odd-indexed values (i=1,3,5,...) by 1.03.
    Sum of adj_score at odd sorted indices before branching: 2251.5086
    Sum of transformed scores T after parity multiplier: 4512.5049

11. **Gini coefficient**
    G = (n + 1 − 2 × (sum(cumulative(T)) / sum(T))) / n with n = 25.
    Gini coefficient G: 0.4139

12. **Final metrics**
    K = number of values in S with adj_score strictly > P37 = 16
    Ω = 1 + (16 mod 5)/10 = 1.1
    SDI = G × (P37 / M) × Ω = 0.3022

# GTFA:

1. 49989
2. 24979
3. 45
4. 155.2457
5. 233.9104
6. 1
7. 2251.5086
8. 4512.5049
9. 0.4139
10. 16
11. 1.1
12. 0.3022