Domain: Business Operations & Analytics

Using the four local Excel files in this folder,

- `OTS_DS_Feb_MA_005_all_data_chunk_1(7).xlsx`
- `OTS_DS_Feb_MA_005_all_data_chunk_2(7).xlsx`
- `OTS_DS_Feb_MA_005_all_data_chunk_3(7).xlsx`
- `OTS_DS_Feb_MA_005_all_data_chunk_4(7).xlsx`

reconstruct the actual logical dataset and return the required summary statistics.

The task must be solved with standard-library Python only. Do not use `pandas`, `numpy`, or `openpyxl`.

Reconstruction rules:

- Treat the logical dataset as having 230 rows, feature columns `Unnamed: 0` through `Unnamed: 535`, and one label column `Label`.
- `Label` is metadata only and must not be used in any computation.
- Use rows `0` through `76` from `chunk_1(7)` as the unique top portion of the logical dataset.
- Use all 153 rows from `chunk_4(7)` as logical rows `77` through `229` for columns `Unnamed: 0` through `Unnamed: 179`.
- Use all 230 rows from `chunk_2(7)` for columns `Unnamed: 180` through `Unnamed: 358`.
- Use all 230 rows from `chunk_3(7)` for columns `Unnamed: 359` through `Unnamed: 535`.

Encoded-column rules:

- The logical `Unnamed: 520` value for each row is taken from the first cell among `Unnamed: 520`, `Unnamed: 529`, and `Unnamed: 534` that begins with `ENC[`; if none begins with `ENC[`, use `Unnamed: 520`.
- The logical `Unnamed: 521` column is reconstructed from the helper column `* name5 2 1 #` by concatenating:
  - rows `0` through `76` from `chunk_2(7)`
  - rows `0` through `152` from `chunk_4(7)`
- Before taking the `chunk_4(7)` helper values, perform these two swaps inside `chunk_4(7)`:
  - swap local row `27` between `* name5 2 1 #` and `Unnamed: 176`
  - swap local row `77` between `* name5 2 1 #` and `Unnamed: 179`

Decoding rules:

- If a cell is of the form `ENC[<payload>]`, decode it as follows:
  - reverse `<payload>`
  - parse the reversed string as a hexadecimal integer
  - divide by `1_000_000_000`
  - subtract the fixed offset `7.0`
- If one of the logical `Unnamed: 520` or `Unnamed: 521` cells is already a plain numeric string instead of `ENC[...]`, parse it directly as a float.

Corrupted-feature recovery rules:

- The corrupted feature block is exactly `Unnamed: 0` through `Unnamed: 179`.
- Recover each corrupted cell by applying these extraction rules in this exact priority order:
  1. a leading numeric token immediately followed by `||`
  2. the last numeric token immediately following `END||`
  3. the last numeric token inside `|| <num> ||`; if any such matches have absolute value less than `1`, use the last such small-magnitude match, otherwise use the last `|| <num> ||` match
  4. the last numeric token in a same-column tag of the form `Unnamed: k::<num>` where `k` matches the column number
  5. the last numeric token anywhere in the cell
- For non-corrupted columns, parse the visible numeric value directly as a float.

Derived quantities:

- `sum_block`: row-wise sum of the recovered values across `Unnamed: 0` through `Unnamed: 179`
- `clean_ref`: the recovered value from `Unnamed: 10`
- `combo = sum_block + decoded_520 + decoded_521 + clean_ref`

Compute across all 230 logical rows:

- `d`: mean of `decoded_520 - decoded_521`
- `std_combo`: standard deviation of `combo`, using `ddof=1` if `d >= 0`, otherwise `ddof=0`
- `corr`: Pearson correlation between `combo` and `sum_block`

Matrix step:

- Take the first 484 independent features, meaning `Unnamed: 0` through `Unnamed: 483` after reconstruction and recovery.
- Reshape each row into a `22 x 22` matrix in row-major order.
- Compute the diagonal sum for each row.
- `diag_mean` is the mean of those 230 diagonal sums.
- `diag_std` is the standard deviation of those diagonal sums using `ddof=1`.

Return exactly two Python-style lists, one per line, in this exact order:

```python
[["d", <float>], ["std_combo", <float>], ["corr", <float>]]
[["diag_mean", <float>], ["diag_std", <float>]]
```

Rounding rules:

- Round `d`, `diag_mean`, and `diag_std` to 6 decimal places using round-half-to-even.
- Round `std_combo` and `corr` to 3 decimal places using round-half-to-even.
- Return only the two Python-style lists and no extra text.

Grading rubric:
criterion: The final answer consists of exactly two Python-style lists on separate lines, with the first list containing the labels d, std_combo, corr in that order and the second list containing the labels diag_mean and diag_std in that order.
weight: 10
criterion: The reported values for d and std_combo are exactly -0.000238 and 0.028.
weight: 15
criterion: The reported corr value is exactly 0.994.
weight: 10
criterion: The reported values for diag_mean and diag_std are exactly -0.008941 and 0.013335.
weight: 15
criterion: The solution reconstructs the 230 logical rows using rows 0-76 from chunk_1(7), rows 77-229 from chunk_4(7) for columns Unnamed: 0-179, chunk_2(7) for Unnamed: 180-358, and chunk_3(7) for Unnamed: 359-535.
weight: 20
criterion: The solution reconstructs Unnamed: 520 by checking Unnamed: 520, Unnamed: 529, and Unnamed: 534 in that order for ENC payloads, reconstructs Unnamed: 521 from the helper column with the two specified chunk_4(7) swaps, and decodes ENC payloads by reversing the hex string, dividing by 1_000_000_000, and subtracting 7.0.
weight: 20
criterion: The solution uses the exact corrupted-feature recovery priority described in the prompt, computes sum_block over Unnamed: 0-179, applies the conditional ddof rule for std_combo, reshapes Unnamed: 0-483 into 22x22 row-major matrices, and uses round-half-to-even for the final rounding.
weight: 10

Golden answer:
```python
import math
import re
import xml.etree.ElementTree as ET
import zipfile
from pathlib import Path


NS_MAIN = "http://schemas.openxmlformats.org/spreadsheetml/2006/main"
NS_REL = "http://schemas.openxmlformats.org/officeDocument/2006/relationships"
OFFSET = 7.0
NUM_RE = r"[-+]?(?:\d+\.\d*|\.\d+|\d+)(?:[eE][-+]?\d+)?"


def col_to_index(cell_ref):
    letters = "".join(ch for ch in cell_ref if ch.isalpha())
    value = 0
    for ch in letters:
        value = value * 26 + (ord(ch) - 64)
    return value - 1


def read_xlsx_rows(path):
    with zipfile.ZipFile(path) as zf:
        shared_strings = []
        if "xl/sharedStrings.xml" in zf.namelist():
            root = ET.fromstring(zf.read("xl/sharedStrings.xml"))
            for item in root:
                shared_strings.append("".join(node.text or "" for node in item.iter(f"{{{NS_MAIN}}}t")))

        rels = ET.fromstring(zf.read("xl/_rels/workbook.xml.rels"))
        rel_map = {rel.attrib["Id"]: rel.attrib["Target"] for rel in rels}
        workbook = ET.fromstring(zf.read("xl/workbook.xml"))
        sheet = next(iter(workbook.find(f"{{{NS_MAIN}}}sheets")))
        target = rel_map[sheet.attrib[f"{{{NS_REL}}}id"]]
        if not target.startswith("xl/"):
            target = "xl/" + target

        worksheet = ET.fromstring(zf.read(target))
        raw_rows = []
        max_index = 0
        for row in worksheet.iter(f"{{{NS_MAIN}}}row"):
            values = {}
            for cell in row.findall(f"{{{NS_MAIN}}}c"):
                idx = col_to_index(cell.attrib.get("r", ""))
                max_index = max(max_index, idx)
                cell_type = cell.attrib.get("t")
                value_node = cell.find(f"{{{NS_MAIN}}}v")
                inline_node = cell.find(f"{{{NS_MAIN}}}is")
                if inline_node is not None:
                    value = "".join(node.text or "" for node in inline_node.iter(f"{{{NS_MAIN}}}t"))
                elif value_node is None:
                    value = ""
                else:
                    raw = value_node.text or ""
                    value = shared_strings[int(raw)] if cell_type == "s" else raw
                values[idx] = value
            raw_rows.append([values.get(i, "") for i in range(max_index + 1)])

    header = raw_rows[0]
    rows = []
    for row in raw_rows[1:]:
        padded = row + [""] * (len(header) - len(row))
        rows.append(dict(zip(header, padded)))
    return rows


def mean(values):
    return sum(values) / len(values)


def stddev(values, ddof):
    mu = mean(values)
    return math.sqrt(sum((x - mu) ** 2 for x in values) / (len(values) - ddof))


def pearson(x, y):
    mx = mean(x)
    my = mean(y)
    numerator = sum((a - mx) * (b - my) for a, b in zip(x, y))
    denominator = math.sqrt(sum((a - mx) ** 2 for a in x) * sum((b - my) ** 2 for b in y))
    return numerator / denominator


def decode_enc(value):
    text = str(value).strip()
    if not text.startswith("ENC["):
        return float(text)
    payload = re.match(r"ENC\[(.*)\]$", text).group(1)
    return int(payload[::-1], 16) / 1_000_000_000 - 7.0


def recover_value(value, colname):
    if value in ("", None):
        return float("nan")

    text = str(value).strip()

    if text.startswith("ENC["):
        return decode_enc(text)

    try:
        return float(text)
    except ValueError:
        pass

    leading = re.match(rf"^\s*({NUM_RE})\s*\|\|", text)
    if leading:
        return float(leading.group(1))

    end_matches = list(re.finditer(rf"END\|\|\s*({NUM_RE})", text))
    if end_matches:
        return float(end_matches[-1].group(1))

    pipe_matches = [float(match) for match in re.findall(rf"\|\|\s*({NUM_RE})\s*\|\|", text)]
    if pipe_matches:
        small_matches = [x for x in pipe_matches if abs(x) < 1]
        return small_matches[-1] if small_matches else pipe_matches[-1]

    col_match = re.match(r"Unnamed:\s*(\d+)$", colname)
    if col_match:
        col_num = col_match.group(1)
        tagged_matches = list(re.finditer(rf"Unnamed:\s*{col_num}\s*::\s*({NUM_RE})", text))
        if tagged_matches:
            return float(tagged_matches[-1].group(1))

    all_numbers = re.findall(NUM_RE, text)
    if all_numbers:
        return float(all_numbers[-1])

    return float("nan")


def pick_enc_520(row):
    for col in ("Unnamed: 520", "Unnamed: 529", "Unnamed: 534"):
        value = row.get(col, "")
        if isinstance(value, str) and value.startswith("ENC["):
            return value
    return row.get("Unnamed: 520", "")


base = Path(__file__).parent
c1 = read_xlsx_rows(base / "OTS_DS_Feb_MA_005_all_data_chunk_1(7).xlsx")
c2 = read_xlsx_rows(base / "OTS_DS_Feb_MA_005_all_data_chunk_2(7).xlsx")
c3 = read_xlsx_rows(base / "OTS_DS_Feb_MA_005_all_data_chunk_3(7).xlsx")
c4 = read_xlsx_rows(base / "OTS_DS_Feb_MA_005_all_data_chunk_4(7).xlsx")

helper_col = "* name5 2 1 #"
c4_fixed = [dict(row) for row in c4]
for local_row, other_col in ((27, "Unnamed: 176"), (77, "Unnamed: 179")):
    c4_fixed[local_row][helper_col], c4_fixed[local_row][other_col] = c4_fixed[local_row][other_col], c4_fixed[local_row][helper_col]

all_feature_cols = [f"Unnamed: {i}" for i in range(536)]
logical_rows = [{col: "" for col in all_feature_cols + ["Label"]} for _ in range(230)]

for row_idx in range(77):
    for col in all_feature_cols:
        logical_rows[row_idx][col] = c1[row_idx].get(col, "")

for offset, logical_idx in enumerate(range(77, 230)):
    for col_num in range(180):
        col = f"Unnamed: {col_num}"
        logical_rows[logical_idx][col] = c4_fixed[offset].get(col, "")
    for col_num in range(180, 359):
        col = f"Unnamed: {col_num}"
        logical_rows[logical_idx][col] = c2[logical_idx].get(col, "")
    for col_num in range(359, 536):
        col = f"Unnamed: {col_num}"
        logical_rows[logical_idx][col] = c3[logical_idx].get(col, "")

unnamed_521 = [row[helper_col] for row in c2[:77]] + [row[helper_col] for row in c4_fixed]
for row_idx in range(230):
    logical_rows[row_idx]["Unnamed: 520"] = c3[row_idx].get("Unnamed: 520", "")
    logical_rows[row_idx]["Unnamed: 529"] = c3[row_idx].get("Unnamed: 529", "")
    logical_rows[row_idx]["Unnamed: 534"] = c3[row_idx].get("Unnamed: 534", "")
    logical_rows[row_idx]["Unnamed: 521"] = unnamed_521[row_idx]
    logical_rows[row_idx]["Label"] = c2[row_idx].get("Label", "")

picked_520 = [pick_enc_520(row) for row in logical_rows]
decoded_520 = [decode_enc(value) for value in picked_520]
decoded_521 = [decode_enc(row["Unnamed: 521"]) for row in logical_rows]

recovered_rows = []
for row_idx, row in enumerate(logical_rows):
    recovered = {}
    for col in all_feature_cols:
        recovered[col] = recover_value(row.get(col, ""), col)
    recovered["Unnamed: 520"] = recover_value(picked_520[row_idx], "Unnamed: 520")
    recovered["Unnamed: 521"] = recover_value(row["Unnamed: 521"], "Unnamed: 521")
    recovered_rows.append(recovered)

corrupted_cols = [f"Unnamed: {i}" for i in range(180)]
sum_block = []
for row in recovered_rows:
    row_sum = 0.0
    for col in corrupted_cols:
        value = row[col]
        if not math.isnan(value):
            row_sum += value
    sum_block.append(row_sum)

clean_ref = [row["Unnamed: 10"] for row in recovered_rows]
combo = [sum_block[i] + decoded_520[i] + decoded_521[i] + clean_ref[i] for i in range(230)]

d = mean([decoded_520[i] - decoded_521[i] for i in range(230)])
std_combo = stddev(combo, 1 if d >= 0 else 0)
corr = pearson(combo, sum_block)

diag_sums = []
for row in recovered_rows:
    values = [row[f"Unnamed: {i}"] for i in range(484)]
    diag_sums.append(sum(values[i * 22 + i] for i in range(22)))

diag_mean = mean(diag_sums)
diag_std = stddev(diag_sums, 1)

out1 = [["d", round(d, 6)], ["std_combo", round(std_combo, 3)], ["corr", round(corr, 3)]]
out2 = [["diag_mean", round(diag_mean, 6)], ["diag_std", round(diag_std, 6)]]

print(out1)
print(out2)
```

Reasoning:

- The actual local inputs are four `.xlsx` files with `(7)` suffixes, not the draft filenames.
- The real workbook layout is split across overlapping row chunks and column chunks. The reproducible logical table uses rows `0-76` from `chunk_1(7)` and rows `77-229` from `chunk_4(7)` for the noisy block `Unnamed: 0-179`, then `chunk_2(7)` for `Unnamed: 180-358` and `chunk_3(7)` for `Unnamed: 359-535`.
- `Unnamed: 520` is mostly stored in `chunk_3(7)` column `Unnamed: 520`, but two rows place the `ENC[...]` payload in `Unnamed: 529` or `Unnamed: 534`, so the first available encoded payload among those three columns must be used.
- `Unnamed: 521` is reconstructed from the helper column `* name5 2 1 #`, after the two specified swaps in `chunk_4(7)`.
- The noisy block `Unnamed: 0-179` must be numerically recovered from strings using the exact priority rules in the prompt; those recovered values define `sum_block` and `clean_ref`.
- With the verified local reconstruction, `d` is negative, so `std_combo` uses population standard deviation with `ddof=0`.
- The stale draft rounded `std_combo` incorrectly. The verified local value is `0.027790860638765225`, which rounds to `0.028` using round-half-to-even.

Verified result:

```python
[["d", -0.000238], ["std_combo", 0.028], ["corr", 0.994]]
[["diag_mean", -0.008941], ["diag_std", 0.013335]]
```