Skill: chartgen-ai/excel-data-quality (version 1.0.0)

--- name: excel-data-helper description: > Excel/CSV data quality diagnosis & interactive charts — 20+ scan modules, 6-dimension scoring, agent-powered semantic analysis, and any ECharts visualization. Fully local. Activate this skill whenever the user uploads, attaches, or mentions a .csv, .xlsx, .xls, or .tsv file, even without an explicit request. Present the skill menu immediately. user-invocable: true homepage: https://github.com/excel-data-helper/excel-data-helper-skill metadata: openclaw: requires: runtime: - node >= 14 --- # Excel Data Helper — Data Quality Diagnosis & Chart Skill Local Excel/CSV data profiling, quality scanning (20+ modules, 6-dimension scoring, LLM semantic analysis), and chart generation. International locale support: CJK, European, Middle Eastern, Americas. **Supported files**: `.csv` `.tsv` `.xlsx` `.xls` --- ## Trigger & Menu When a supported file is detected, **do not auto-process** — present this menu first (adapt to the user's language): > **📊 Excel Data Helper** — Hi, I noticed you shared a data file: `<filename>` > > | # | Action | Description | > |:---:|--------|-------------| > | **1** | Quality Check | Overview, scan, scoring | > | **2** | Chart | Any ECharts type: bar, line, pie, scatter, radar, heatmap… | > | **3** | Advanced Chart | Dashboard, Gantt, PPT, diagrams | > | **0** | Skip | Do nothing for now | > > Reply 0–3, or describe what you need. --- ## Routing & Context Match user intent and route to the corresponding sub-skill: - IF user replies `1`, or intent is **overview / quality check / diagnose / score / problems** → follow `references/quality-check.md` - IF user replies `2`, or intent is **chart / plot / visualize / graph** → follow `references/chart.md` - IF user replies `3`, or intent is **dashboard / Gantt / PPT / diagram / complex layout** → follow `references/advanced-chart.md` - IF user replies `0`, or intent is **skip / later / not now** → do nothing, reply: "Got it — file noted. Just let me know when you're ready." - IF intent is **ambiguous or unrelated** → ask to clarify, never guess. Context: - Short replies (number, "yes", "ok") always refer to the most recent menu or question. - After Skip (0), context resets — ignore the file unless user re-references it. - Multiple files — each needs its own explicit choice. --- ## Setup Before first use, install dependencies (one-time): ```bash cd <skill_directory> npm install ``` This installs `xlsx` (SheetJS), `echarts`, and `sharp`. All analysis and chart rendering runs locally. --- ## Rules - Respond in the user's language. - Never auto-process a file — wait for explicit choice. - Parse tool JSON output; present results in clear, readable format — never expose raw JSON. - Use absolute file paths from tool output directly for follow-up operations. - Always include "Excel Data Helper" in the menu header.

path: README.md
content: # Excel Data Helper — Data Quality Diagnosis & Chart Skill

> **Fully local** Excel/CSV data overview, quality diagnosis, and charting — no API key needed.

A skill for [OpenClaw](https://openclaw.com) and other AI agents that provides comprehensive data quality analysis and charting capabilities, powered entirely by local Node.js tools.

---

## What Can It Do?

### 1. Quality Check (22 Scan Modules, 6-Dimension Scoring, LLM Semantic Analysis)
Upload a CSV/Excel file and get a complete data profile (column types, null rates, unique values, patterns, statistics) plus a quality scan across six dimensions:
- **Completeness**: null values, empty strings, empty rows, merged cell patterns
- **Accuracy**: outliers (IQR), rare values, value uniformity (zero-variance / dominant value)
- **Consistency**: mixed types, case issues, date formats, full-width/half-width chars, numbers with embedded units, cross-column logical checks
- **Validity**: email/phone format (intl.), special characters, range checks, encoding/mojibake detection, ID card checksum validation
- **Uniqueness**: duplicate rows, primary key violations, near-duplicate (fuzzy) value detection with Unicode normalization
- **Timeliness**: future dates, abnormally old dates

**International locale support**: multilingual field name recognition (EN, ZH, JA, KO, ES, FR, DE, AR), international date formats (ISO, EU DD.MM.YYYY, US MM/DD/YYYY, CJK 年月日, Korean 년월일, Japanese era), global currency symbols (¥ $ € £ ₩ ₹ ₽ etc.), and region-specific quality patterns.

Plus LLM-powered semantic analysis: synonym detection (cross-language), multi-value cells, business key analysis, cross-column relationship insights, locale-specific observations, and data fitness assessment.

Get a weighted quality score (0-100) with grade, breakdown, and actionable recommendations.

### 2. Charts
Generate PNG chart images (server-side ECharts rendering):
- Any ECharts chart type: bar, line, pie, scatter, area, radar, boxplot, heatmap, funnel, treemap, combo, and more
- LLM analyzes data and determines the best 1–5 charts with optimal types and configs
- PNG images sent directly in conversation — no browser needed

### 3. Advanced Charts (via ChartGen)
For dashboards, Gantt charts, diagrams, and PPT — delegates to the ChartGen skill (requires API key).

---

## Installation

### Natural Language Install (OpenClaw)

> Install this skill for me: `https://github.com/excel-data-helper/excel-data-helper-skill.git`

### Manual Installation

```bash
cd ~/.openclaw/workspace/skills
git clone https://github.com/excel-data-helper/excel-data-helper-skill.git
cd excel-data-helper-skill
npm install
```

---

## Usage Examples

### Quality Check
> "What's in this Excel file?"
> "Check the data quality of this file"
> "Are there any problems with my data?"

### Charts
> "Create a bar chart of sales by category"
> "Suggest some charts for this data"

---

## Tools

| Tool | Purpose |
|------|---------|
| `tools/excel_parser.js` | Parse CSV/Excel files, show structure |
| `tools/quality_scanner.js` | Data profiling + 22-module quality scan with scoring |
| `tools/chart_renderer.js` | Data inspection (`--info`) + render any ECharts chart to PNG (`--config`) |

All tools are CLI scripts that output JSON. Run any tool without arguments to see usage help.

---

## Requirements

- **Node.js** >= 14
- **npm** dependencies: `xlsx` (SheetJS), `echarts`, `sharp` — installed via `npm install`
- No API keys, no external services, no internet required

---

## Architecture

```
SKILL.md (main menu)
├── references/quality-check.md      → tools/quality_scanner.js
├── references/chart.md              → tools/chart_renderer.js
└── references/advanced-chart.md     → ChartGen skill (external)
```

Shared libraries in `tools/lib/`: parser, profiler, scanner, scorer, utils.


path: references/advanced-chart.md
content: # Advanced Chart — Sub-Skill

## When to Use

User needs complex visualizations that go beyond basic charts:
- **Dashboards** with multiple charts in a layout
- **Gantt charts** for project timelines
- **Diagrams** (flowchart, sequence, ER, mind map)
- **PPT generation** with embedded visualizations
- **Advanced analysis** (YoY, cross-file joins, trend analysis)
- **Professional themes** and export options

## Implementation

This capability delegates to the **ChartGen** skill, which provides AI-powered visualization via the ChartGen API.

### Step 1 — Check ChartGen Availability

Verify that the `chartgen` skill is installed and configured. Look for the skill at:
- `../chartgen-skill/SKILL.md` (sibling directory)
- Or installed via OpenClaw's skill system

### Step 2 — If ChartGen is Available

Follow the ChartGen skill's workflow:

1. Read and follow the instructions in the ChartGen `SKILL.md`
2. The ChartGen skill handles:
   - Confirming the request with the user
   - Submitting to ChartGen API
   - Polling for results
   - Delivering artifacts (images, PPT, dashboards)

**Important**: ChartGen requires an API key. If not configured, it will provide setup instructions.

### Step 3 — If ChartGen is NOT Available

Inform the user and offer alternatives:

> The advanced chart capability requires **ChartGen AI** which is not currently installed.
>
> **Options:**
> 1. **Install ChartGen**: Tell me "install skill https://github.com/chartgen-ai/chartgen-skill.git"
> 2. **Use basic charts**: I can create bar, line, pie, scatter, and area charts locally (no API needed)
> 3. **Export data**: I can clean and export your data for use in other visualization tools

### Fallback to Basic Charts

If the user's request can be partially fulfilled with basic charts:

> Your request involves a dashboard layout, which needs ChartGen. However, I can create individual charts locally:
> - A line chart for the time-series data
> - A bar chart for the category comparison
>
> Want me to create these basic charts instead?

## ChartGen Capabilities Reference

When ChartGen is available, it supports:

| Type | Output | Description |
|------|--------|-------------|
| Charts | PNG | All ECharts types: bar, line, pie, scatter, heatmap, radar, treemap, etc. |
| Diagrams | PNG | Flowchart, sequence, class, state, ER, mind map, timeline |
| Dashboards | PNG/HTML | Multi-chart interactive layouts |
| Gantt | PNG | Project timelines with dependencies |
| PPT | PPTX | Presentation slides with visualizations |
| Reports | Text + Charts | Analysis reports with insights |


path: references/chart.md
content: # Chart — Sub-Skill

## Tool

### Inspect data (planning only)

```bash
node tools/chart_renderer.js <file_path> --info
```

Returns column names, inferred types, stats (min/max/mean, unique count, sample values, date range). Use this to understand the data before deciding what to chart.

### Render a chart

```bash
node tools/chart_renderer.js <file_path> --config '<json>' [--output <path>]
```

Outputs a **PNG image** file. The tool only renders — all chart-type decisions, column selection, and ECharts customization are made by YOU in the config JSON.

---

## Config JSON Schema

```jsonc
{
  "type": "bar",                   // REQUIRED — see Supported Types below
  "x": "column_name",             // X-axis / group column
  "y": "col" or ["col1","col2"],  // Y-axis / value column(s)
  "title": "Chart Title",         // shown at top of chart
  "xName": "X Label",             // axis label
  "yName": "Y Label",
  "aggregate": "sum",             // for pie/funnel/treemap: sum|avg|count|max|min
  "series": [                     // optional per-series overrides
    {
      "name": "Display Name",
      "type": "line",             // override type for combo charts
      "smooth": true,
      "area": true,               // fill area below line
      "stack": "group1",
      "itemStyle": { "color": "#ee6666" },
      "label": { "show": true },
      "markLine": { "data": [{ "type": "average" }] },
      "markPoint": { "data": [{ "type": "max" }, { "type": "min" }] }
    }
  ],
  "value": "col",                 // for heatmap: the value (intensity) column
  "radius": "60%",                // for pie
  "roseType": "area",             // for nightingale rose pie
  "symbolSize": 10,               // for scatter
  "width": 900,                   // image width in px
  "height": 600,                  // image height in px
  "limit": 500,                   // max rows (default 500)
  "echarts": { }                  // raw ECharts option — deep-merged last, overrides everything
}
```

### Supported Types

| Type | x | y | Notes |
|------|---|---|-------|
| `bar` | category | numeric(s) | grouped / stacked via `series[].stack` |
| `line` | category / date | numeric(s) | `smooth`, `area` |
| `area` | category / date | numeric(s) | shortcut for line + areaStyle |
| `pie` | category | numeric | auto-aggregates by x; `roseType` for nightingale |
| `scatter` | numeric | numeric | `symbolSize` |
| `radar` | — | numeric[] | y = array of metric columns |
| `boxplot` | — | numeric[] | y = array of columns to box-plot |
| `heatmap` | category | category | requires `value` (z) column |
| `funnel` | category | numeric | sorted descending |
| `treemap` | category | numeric | hierarchical area |
| `combo` | category | numeric[] | use `series[].type` to mix bar/line |

For any ECharts type not listed above (gauge, sankey, graph, sunburst…), pass a complete option object via the `echarts` field — it deep-merges over the base.

---

## How to Present Results

Run `--info` first (if you don't already have the data profile), then walk through **Steps 1–3 in order**.

---

### Step 1 — Analyze Data & Plan Charts

**Read**: `--info` output (or existing `fieldStats` / `sampleData` from a prior quality check).

**Do NOT output to the user yet.** This step is internal analysis.

You MUST:

1. **Identify chartable columns** — classify every column as: date/time, numeric, categorical, or unusable.
2. **Determine the best 1–5 charts** that provide the most insight for this dataset. Consider:
   - Time-series trends (date × numeric → line / area)
   - Category comparisons (category × numeric → bar / pie)
   - Distributions (numeric → boxplot / histogram-style bar)
   - Correlations (numeric × numeric → scatter)
   - Composition (category × numeric → pie / treemap / funnel)
   - Multi-metric overview (many numerics → radar)
   - Cross-dimension density (category × category × numeric → heatmap)
3. **For each planned chart, draft the full config JSON** including type, x, y, title, axis names, series overrides, and any `echarts` customization needed.
4. **Avoid redundancy** — each chart should reveal a different insight. Don't make two bar charts of the same relationship.

---

### Step 2 — Render Charts

**Immediately render all planned charts** — do NOT ask for confirmation first.

For each chart, call the tool:

```bash
node tools/chart_renderer.js <file> --config '<json>'
```

Rules:
- Call the tool once per chart. Each call produces one PNG image file.
- Use meaningful `--output` filenames (ending in `.png`) if the auto-generated name would be unclear.
- If a chart fails, report the error and continue with the remaining charts.
- **Send each generated PNG image to the user** using the file path returned by the tool.

---

### Step 3 — Deliver & Next Steps

**Output to user**: display each PNG image inline, then show a follow-up menu.

You MUST:

1. **Send each PNG image to the user** — use the `imageFile` path from tool output. Display them inline so the user sees the charts directly in the conversation.
2. **Add a brief caption** under each image: chart title + what insight it shows.
3. **Show the follow-up menu**:

> | # | Next |
> |:---:|------|
> | **1** | Adjust or add more charts |
> | **2** | Advanced visualization — dashboards, Gantt, PPT (ChartGen) |
> | **0** | Done |
>
> Reply 0–2, or describe what you'd like to change.

Rules:
- IF user replies `1` → ask what to change (type, columns, add more charts), then re-run Steps 1–3 for new/modified charts only.
- IF user replies `2` → follow `references/advanced-chart.md`.
- IF user replies `0` → end.

---

## Design Guidelines for Chart Selection

When choosing chart types and configs, follow these principles:

1. **Time data** → always prefer line or area for primary view.
2. **<=7 categories** → pie is effective. **>7 categories** → use bar (horizontal if labels are long).
3. **Comparison across 2+ series** → grouped or stacked bar, or combo (bar + line).
4. **Distribution** → boxplot for spread, or a bar of value-frequency bins.
5. **High-cardinality category × category** → heatmap with color intensity.
6. **Multi-dimensional overview** → radar (limit to 5–8 metrics).
7. **Titles and axis labels** — always set them in the user's language, descriptive and concise.
8. **Colors** — use ECharts defaults unless the user requests a theme. For emphasis, set specific colors via `series[].itemStyle`.
9. **Large datasets** — keep `limit` at 500 or lower. For aggregated views (pie, funnel), aggregation handles all rows automatically.


path: references/quality-check.md
content: # Quality Check — Sub-Skill

## Tool

```bash
node tools/quality_scanner.js <file_path>
```

Returns data overview + quality scan in one call.

## Key Output Fields

```jsonc
{
  "fileName": "...",
  "parseInfo": { "fileType", "sheetNames", "warnings", "mojibakeDetected" },
  "tableStats": { "totalRows", "totalColumns", "completeRowsPercentage", "totalNullPercentage", "duplicatePercentage", "memorySizeReadable" },
  "fieldStats": [{ "name", "inferredType", "nullPercentage", "uniquePercentage", "sampleValues", "detectedPatterns", "minValue", "maxValue", "meanValue", "topValues" }],
  "sampleData": [ { /* up to 10 most-complete rows */ } ],
  "dataValidity": { "canProceed": true },
  "issues": [{ "category", "severity", "title", "description", "affectedColumns", "affectedPercentage", "suggestion", "deduction" }],
  "score": { "totalScore": 72.3, "grade": "Fair", "severityCounts": {}, "breakdowns": [{ "categoryName", "weight", "rawDeduction", "scaledDeduction", "finalScore" }] }
}
```

## Scoring: 22 Modules × 6 Dimensions

Each dimension starts at 100. Raw deductions are amplified by severity (critical ×1.5, high ×1.2, low ×0.7) and category sensitivity scale, then subtracted. Total score = weighted sum.

| Dimension | Weight | Scale | Modules |
|-----------|--------|-------|---------|
| Completeness | 25% | ×2.5 | Null values, empty strings, empty rows, merged cell patterns |
| Accuracy | 23% | ×5.0 | Outliers, rare values, value uniformity |
| Consistency | 17% | ×2.0 | Mixed types, case, date format, full-width chars, number-with-unit, cross-column logic |
| Validity | 15% | ×2.5 | Format errors, special chars, range, encoding/mojibake, ID checksum |
| Uniqueness | 15% | ×3.0 | Duplicate rows, primary key, near-duplicates (fuzzy) |
| Timeliness | 5% | ×10 | Future dates, old dates |

**Grades**: >= 90 Excellent, >= 80 Good, >= 70 Fair, >= 60 Poor, < 60 Very Poor

---

## How to Present Results

Run the tool, then walk through **Step 1–6 in order**. Every step is **mandatory** — do not skip any. Do NOT dump raw JSON to the user.

---

### Step 1 — Understand the Data

**Read**: `sampleData`, `fieldStats` (all fields), `parseInfo`.
**Do NOT output anything to the user yet.** This step is internal analysis only.

You MUST complete these 4 tasks before proceeding:

1. **Detect locale/region** — examine field names, sample values, currency symbols, date formats, and text language. Determine the primary data locale (e.g., "Chinese business data", "European HR records", "Japanese logistics", "mixed EN/AR"). Keep this context for all subsequent steps.

2. **Infer business domain** — based on field names and sample values, determine what this dataset represents (e.g., e-commerce orders, employee records, IoT sensor readings, financial transactions).

3. **Build a field mapping** — for EVERY field in `fieldStats`, determine:
   - A human-readable **display name** (translated to the user's language)
   - A one-line **business meaning**
   - The `inferredType` from the tool output

4. **Draft a dataset summary** — 1–2 sentences: what the data represents, its granularity (one row = one what?), and approximate time range (if any date field exists).

---

### Step 2 — Present Data Overview

**Read**: `tableStats`, the field mapping from Step 1.
**Output to user**: the dataset summary, a metrics table, and a column details table.

You MUST produce exactly this structure (adapt values):

> **sales.xlsx** — E-commerce order data, 5,000 orders from 2023-01 to 2024-06.
>
> | Metric | Value |
> |--------|-------|
> | Rows | 5,000 |
> | Columns | 12 |
> | Complete rows | 84.0% |
> | Null cells | 1.33% |
> | Duplicates | 1.0% |
> | File size | 1.2 MB |
>
> | Column | Display Name | Type | Nulls | Unique | Key Stats |
> |--------|-------------|------|-------|--------|-----------|
> | order_id | Order ID | string | 0% | 100% | uuid pattern |
> | amount | Amount | float | 2.1% | 85% | Range: 0.50 – 9,999.00 |
> | created_at | Created At | date | 0% | 95% | 2023-01 to 2024-06 |

Rules:
- Show ALL columns, not just a selection.
- Use the display names from Step 1.
- For numeric columns, show range (min–max) or mean in "Key Stats".
- For categorical columns with `topValues`, show top 2–3 values.
- For columns with `detectedPatterns`, note the pattern (e.g., "email", "uuid", "id_card_cn").

---

### Step 3 — Present Quality Score

**Read**: `score.totalScore`, `score.grade`, `score.severityCounts`, `score.breakdowns`.
**Output to user**: score headline + 6-dimension breakdown table.

You MUST produce exactly this structure:

> **Data Quality Score: 72.3 / 100** (Fair ⚠️)
> 15 issues found: 1 critical, 3 high, 5 medium, 4 low, 2 info
>
> | Dimension | Score | Dimension | Score |
> |-----------|-------|-----------|-------|
> | Completeness | 65.0 | Validity | 78.0 |
> | Accuracy | 82.0 | Uniqueness | 85.0 |
> | Consistency | 90.5 | Timeliness | 95.0 |

Rules:
- Use the emoji: >= 90 ✅, >= 80 🟢, >= 70 ⚠️, >= 60 🟠, < 60 🔴.
- Show severity counts on the same line as the headline.

---

### Step 4 — Narrate Issues & Insights

**Read**: `issues` array (all of them), the business domain and locale from Step 1.
**Output to user**: a narrative analysis grouped by dimension.

You MUST follow these rules:
1. **Do NOT list issues one by one.** Group related issues into paragraphs by dimension (Completeness, Consistency, etc.). Only create a paragraph for dimensions that have issues.
2. **For each group, explain the business impact** — connect the issue to the specific business domain. Example: "The `age` field is 85% empty — for a customer segmentation dataset, this makes age-based analysis impossible."
3. **For every critical or high severity issue, provide a specific actionable recommendation.** Not generic advice like "fix this" — state exactly what to do. Example: "Forward-fill the `region` column (merged cell pattern detected) rather than using mean imputation."
4. **For medium/low issues, summarize briefly** — one sentence per group is enough.
5. **Mention the `suggestion` field from issues as supporting context**, but rephrase it in business terms rather than quoting it verbatim.

---

### Step 5 — Semantic Deep Analysis

**Read**: `sampleData`, `fieldStats.topValues`, `issues`, and the locale from Step 1.
**Output to user**: additional findings that the programmatic scan cannot detect.

You MUST check ALL of the following. For each, either report findings or explicitly state "not detected":

1. **Synonym detection** — Scan `topValues` of every categorical column. Identify value groups that represent the same concept across languages or abbreviations.
   - Examples: "M"/"Male"/"男"/"masculin"/"ذكر", "Y"/"Yes"/"是"/"Oui"/"نعم", "CN"/"China"/"中国".
   - Output: list each synonym group found as `[value1, value2, ...] → suggested canonical value`.

2. **Multi-value cells** — Scan `sampleData` for cells containing commas, semicolons, pipes, slashes, or newlines that suggest multiple values packed in one cell.
   - Output: list affected columns and example values.

3. **Business key duplicates** — Based on the field semantics from Step 1, identify the most likely composite business key (e.g., `[customer_id, order_date]` or `[employee_id, department]`). State whether the dataset has entity-level duplicates using this key, even if no single-column PK violation was found.
   - Output: state the inferred business key and whether duplicates exist.

4. **Cross-column relationships** — Identify logical relationships between columns (date ranges, parent-child, calculated totals). Explain any violations found by the `cross_column_logic` issues in business terms. Also note relationships the tool did NOT check but that appear relevant.
   - Output: list each relationship and its status (valid / violated / unchecked).

5. **Locale-specific issues** — Based on the locale detected in Step 1, check for region-specific problems:
   - CJK: full-width/half-width mixing, encoding, era-based dates
   - European: DD/MM vs MM/DD ambiguity, decimal comma vs dot (1.234,56), GDPR-sensitive fields
   - Middle Eastern: RTL text mixing, Arabic numeral variants (٠١٢ vs 012), Hijri dates
   - Multi-region: mixed languages in same column, inconsistent formats
   - Output: list any locale-specific findings, or state "no locale-specific issues detected".

6. **Data fitness assessment** — Based on ALL findings (Steps 4 + 5), produce this table:

> | Use Case | Fitness | Key Blockers |
> |----------|---------|-------------|
> | BI Reporting | ⚠️ Fair | date inconsistency, 2 fields need cleaning |
> | ML Training | ❌ Poor | 85% null in key feature |
> | Aggregation | ✅ Good | core numerics clean |

   Fitness levels: ✅ Good, ⚠️ Fair, ❌ Poor. Include at least 3 common use cases.

---

### Step 6 — Recommendation & Next Steps

**Read**: `score.totalScore`, `dataValidity.canProceed`, your analysis from Steps 4–5, the issues list.
**Output to user**: a closing statement + a numbered follow-up menu.

Rules:
- IF `dataValidity.canProceed` is false → **warn first**: "⚠️ This file has structural issues that prevent reliable analysis: [reason]. Results above may be incomplete."
- THEN give the score-based recommendation:
  - Score >= 90 → "Data quality is excellent. Ready for analysis."
  - Score 70–89 → "A few issues to address. Recommend cleaning before critical use."
  - Score < 70 → "Significant issues detected. Strongly recommend cleaning before use."

**THEN output the follow-up menu.** You MUST:

1. **Build 1–3 exploration suggestions** based on the actual findings from Steps 4–5. Pick the most valuable from:
   - A dimension with critical/high issues → "Deep-dive into [dimension] issues"
   - Synonym groups found → "Review synonym/duplicate value consolidation"
   - Cross-column violations → "Explore cross-column logic issues"
   - Fitness blockers → "Analyze blockers for [use case]"
   - High-null columns → "Investigate missing data in [column(s)]"
   - Use concrete column/dimension names — not generic text.

2. **Output this menu structure** (adapt the exploration items to the actual data):

> | # | Next |
> |:---:|------|
> | **1** | Chart — visualize this data |
> | **2** | Advanced Chart — dashboards, Gantt, PPT (ChartGen) |
> | **3** | *[exploration suggestion 1, e.g., "Deep-dive: Completeness — 3 fields with >50% nulls"]* |
> | **4** | *[exploration suggestion 2, e.g., "Review synonym groups in `status` and `region`"]* |
> | **5** | *[exploration suggestion 3, if applicable]* |
> | **0** | Done |
>
> Reply 0–N, or describe what you need.

Rules for routing:
- IF user replies `1` → follow `references/chart.md`.
- IF user replies `2` → follow `references/advanced-chart.md`.
- IF user replies an exploration number (3/4/5) → provide the detailed analysis for that topic using the data already available from the scan results and your semantic analysis. Do NOT re-run the tool.
- IF user replies `0` → end.


path: tools/chart_renderer.js
content: #!/usr/bin/env node
"use strict";
const path = require("path");
const fs = require("fs");
const { parseFile } = require("./lib/parser");
const U = require("./lib/utils");
const { inferFieldType } = require("./lib/profiler");

// --------------- data helpers ---------------

function getColumnData(rows, col, limit) {
  return rows.slice(0, limit).map(r => r[col] != null ? r[col] : null);
}

function aggregate(rows, groupCol, valueCol, fn) {
  const groups = {};
  for (const row of rows) {
    const k = row[groupCol] != null ? String(row[groupCol]) : "(empty)";
    if (!groups[k]) groups[k] = [];
    const v = U.tryParseNumber(row[valueCol]);
    if (v !== null) groups[k].push(v);
  }
  const result = {};
  for (const [k, arr] of Object.entries(groups)) {
    if (fn === "count") result[k] = arr.length;
    else if (fn === "avg" || fn === "mean") result[k] = arr.length ? U.round(arr.reduce((a, b) => a + b, 0) / arr.length, 2) : 0;
    else if (fn === "max") result[k] = Math.max(...arr);
    else if (fn === "min") result[k] = Math.min(...arr);
    else result[k] = U.round(arr.reduce((a, b) => a + b, 0), 2);
  }
  return result;
}

// --------------- chart option builders ---------------

function buildOption(config, headers, rows) {
  const limit = config.limit || 500;
  const type = config.type || "bar";

  if (["pie", "funnel", "treemap"].includes(type)) {
    const agg = aggregate(rows, config.x, config.y, config.aggregate || "sum");
    const data = Object.entries(agg).map(([name, value]) => ({ name, value }));
    const base = {
      title: { text: config.title || "", left: "center" },
      tooltip: { trigger: "item" },
    };
    if (type === "pie") {
      base.series = [{ type: "pie", radius: config.radius || "60%", data, label: { show: true } }];
      if (config.roseType) base.series[0].roseType = config.roseType;
    } else if (type === "funnel") {
      base.series = [{ type: "funnel", data, sort: "descending" }];
    } else {
      base.series = [{ type: "treemap", data: data.map(d => ({ name: d.name, value: d.value })) }];
    }
    return deepMerge(base, config.echarts || {});
  }

  if (type === "scatter") {
    const pairs = rows.slice(0, limit)
      .map(r => [U.tryParseNumber(r[config.x]), U.tryParseNumber(r[config.y])])
      .filter(d => d[0] !== null && d[1] !== null);
    const base = {
      title: { text: config.title || "", left: "center" },
      tooltip: { trigger: "item" },
      xAxis: { name: config.xName || config.x, type: "value" },
      yAxis: { name: config.yName || config.y, type: "value" },
      series: [{ type: "scatter", data: pairs, symbolSize: config.symbolSize || 8 }],
    };
    return deepMerge(base, config.echarts || {});
  }

  if (type === "boxplot") {
    const yCols = Array.isArray(config.y) ? config.y : [config.y];
    const boxData = yCols.map(col => {
      const nums = rows.map(r => U.tryParseNumber(r[col])).filter(n => n !== null).sort((a, b) => a - b);
      if (!nums.length) return [0, 0, 0, 0, 0];
      return [nums[0], U.quantile(nums, 0.25), U.quantile(nums, 0.5), U.quantile(nums, 0.75), nums[nums.length - 1]];
    });
    const base = {
      title: { text: config.title || "", left: "center" },
      tooltip: { trigger: "item" },
      xAxis: { type: "category", data: yCols },
      yAxis: { type: "value" },
      series: [{ type: "boxplot", data: boxData }],
    };
    return deepMerge(base, config.echarts || {});
  }

  if (type === "radar") {
    const yCols = Array.isArray(config.y) ? config.y : [config.y];
    const indicator = yCols.map(col => {
      const nums = rows.map(r => U.tryParseNumber(r[col])).filter(n => n !== null);
      return { name: col, max: nums.length ? Math.max(...nums) * 1.2 : 100 };
    });
    const values = yCols.map(col => {
      const nums = rows.map(r => U.tryParseNumber(r[col])).filter(n => n !== null);
      return nums.length ? U.round(nums.reduce((a, b) => a + b, 0) / nums.length, 2) : 0;
    });
    const base = {
      title: { text: config.title || "", left: "center" },
      tooltip: {},
      radar: { indicator },
      series: [{ type: "radar", data: [{ value: values, name: config.title || "Average" }] }],
    };
    return deepMerge(base, config.echarts || {});
  }

  if (type === "heatmap") {
    const xData = [...new Set(rows.map(r => String(r[config.x] ?? "")))];
    const yData = [...new Set(rows.map(r => String(r[config.y] ?? "")))];
    const valCol = config.value || config.z;
    const data = [];
    for (const row of rows.slice(0, limit)) {
      const xi = xData.indexOf(String(row[config.x] ?? ""));
      const yi = yData.indexOf(String(row[config.y] ?? ""));
      const v = U.tryParseNumber(row[valCol]);
      if (xi >= 0 && yi >= 0 && v !== null) data.push([xi, yi, v]);
    }
    const base = {
      title: { text: config.title || "", left: "center" },
      tooltip: {},
      xAxis: { type: "category", data: xData },
      yAxis: { type: "category", data: yData },
      visualMap: { min: 0, max: Math.max(...data.map(d => d[2]), 1), calculable: true },
      series: [{ type: "heatmap", data, label: { show: data.length < 200 } }],
    };
    return deepMerge(base, config.echarts || {});
  }

  // Default: bar / line / area / combo
  const xData = getColumnData(rows, config.x, limit).map(v => v != null ? String(v) : "");
  const yCols = Array.isArray(config.y) ? config.y : [config.y];

  const series = yCols.map((col, idx) => {
    const seriesConf = (config.series && config.series[idx]) || {};
    const sType = seriesConf.type || (type === "area" ? "line" : type);
    const s = {
      name: seriesConf.name || col,
      type: sType,
      data: getColumnData(rows, col, limit).map(v => U.tryParseNumber(v)),
    };
    if (sType === "line" && (type === "area" || seriesConf.area)) s.areaStyle = seriesConf.areaStyle || {};
    if (seriesConf.smooth) s.smooth = true;
    if (seriesConf.stack) s.stack = seriesConf.stack;
    if (seriesConf.itemStyle) s.itemStyle = seriesConf.itemStyle;
    if (seriesConf.label) s.label = seriesConf.label;
    if (seriesConf.markLine) s.markLine = seriesConf.markLine;
    if (seriesConf.markPoint) s.markPoint = seriesConf.markPoint;
    return s;
  });

  const base = {
    title: { text: config.title || "", left: "center" },
    tooltip: { trigger: "axis" },
    legend: yCols.length > 1 ? { data: series.map(s => s.name), top: 30 } : undefined,
    grid: { left: "10%", right: "5%", bottom: "15%", top: yCols.length > 1 ? 60 : 40 },
    xAxis: {
      type: "category",
      data: xData,
      name: config.xName || undefined,
      axisLabel: { rotate: xData.length > 20 ? 45 : 0 },
    },
    yAxis: { type: "value", name: config.yName || undefined },
    series,
  };
  return deepMerge(base, config.echarts || {});
}

function deepMerge(target, source) {
  if (!source || typeof source !== "object") return target;
  const result = { ...target };
  for (const key of Object.keys(source)) {
    if (source[key] && typeof source[key] === "object" && !Array.isArray(source[key]) &&
        result[key] && typeof result[key] === "object" && !Array.isArray(result[key])) {
      result[key] = deepMerge(result[key], source[key]);
    } else {
      result[key] = source[key];
    }
  }
  return result;
}

// --------------- PNG rendering (ECharts SSR + sharp) ---------------

async function renderToPng(option, width, height, outputPath) {
  const echarts = require("echarts");
  const sharp = require("sharp");

  const chart = echarts.init(null, null, {
    renderer: "svg",
    ssr: true,
    width: width || 900,
    height: height || 600,
  });
  chart.setOption(option);
  const svgStr = chart.renderToSVGString();
  chart.dispose();

  await sharp(Buffer.from(svgStr))
    .png({ compressionLevel: 6 })
    .toFile(outputPath);
}

// --------------- info mode ---------------

function getInfo(headers, rows) {
  const columns = headers.map(h => {
    const type = inferFieldType(h, rows);
    const vals = rows.map(r => r[h]).filter(v => !U.isNullish(v));
    const info = { name: h, type, nonNullCount: vals.length };
    if (["integer", "float"].includes(type)) {
      const nums = vals.map(U.tryParseNumber).filter(n => n !== null);
      if (nums.length) {
        info.min = Math.min(...nums);
        info.max = Math.max(...nums);
        info.mean = U.round(nums.reduce((a, b) => a + b, 0) / nums.length, 2);
      }
    }
    if (["categorical", "string"].includes(type)) {
      const uniq = [...new Set(vals.map(String))];
      info.uniqueCount = uniq.length;
      info.sampleValues = uniq.slice(0, 8);
    }
    if (["date", "datetime"].includes(type)) {
      const dates = vals.map(v => U.tryParseDate(v)).filter(d => d !== null).sort((a, b) => a - b);
      if (dates.length) {
        info.minDate = dates[0].toISOString().slice(0, 10);
        info.maxDate = dates[dates.length - 1].toISOString().slice(0, 10);
      }
    }
    return info;
  });
  return { rowCount: rows.length, columnCount: headers.length, columns };
}

// --------------- main ---------------

async function main() {
  const args = process.argv.slice(2);
  if (!args.length) {
    process.stderr.write([
      "Usage:",
      "  chart_renderer.js <file> --info",
      "    Returns column types and stats for chart planning.",
      "",
      "  chart_renderer.js <file> --config '<json>'",
      "    Render a chart to PNG. Config JSON fields:",
      '    { "type":"bar", "x":"col", "y":"col"|["c1","c2"],',
      '      "title":"...", "aggregate":"sum|avg|count",',
      '      "series":[{"name":"...","type":"line","smooth":true}],',
      '      "echarts":{...}, "width":900, "height":600 }',
      "",
      "  Supported types: bar, line, area, pie, scatter, radar,",
      "    boxplot, heatmap, funnel, treemap, combo (+ any via echarts override)",
      "",
      "  chart_renderer.js <file> --config '<json>' --output <path>",
      "    Save PNG to custom path instead of auto-generated filename.",
    ].join("\n") + "\n");
    process.exit(1);
  }

  const filePath = path.resolve(args[0]);
  let mode = null, configStr = null, outputPath = null;

  for (let i = 1; i < args.length; i++) {
    if (args[i] === "--info") mode = "info";
    else if (args[i] === "--config" && args[i + 1]) { mode = "render"; configStr = args[++i]; }
    else if (args[i] === "--output" && args[i + 1]) outputPath = args[++i];
  }

  try {
    const { headers, rows } = parseFile(filePath);

    if (mode === "info") {
      process.stdout.write(JSON.stringify(getInfo(headers, rows), null, 2) + "\n");
      return;
    }

    if (mode !== "render" || !configStr) {
      process.stderr.write("Specify --info or --config '<json>'\n");
      process.exit(1);
    }

    const config = JSON.parse(configStr);
    const option = buildOption(config, headers, rows);

    if (!outputPath) {
      const dir = path.dirname(filePath);
      const base = path.basename(filePath, path.extname(filePath));
      const suffix = config.title ? config.title.replace(/[^a-zA-Z0-9\u4e00-\u9fff]/g, "_").slice(0, 30) : config.type || "chart";
      outputPath = path.join(dir, `${base}_${suffix}.png`);
    }

    const absOut = path.resolve(outputPath);
    await renderToPng(option, config.width, config.height, absOut);

    process.stdout.write(JSON.stringify({
      imageFile: absOut,
      chartType: config.type,
      title: config.title || "",
    }, null, 2) + "\n");
  } catch (err) {
    process.stdout.write(JSON.stringify({ error: err.message }) + "\n");
    process.exit(1);
  }
}

main();


path: tools/data_profiler.js
content: #!/usr/bin/env node
"use strict";
const path = require("path");
const { parseFile } = require("./lib/parser");
const { profile } = require("./lib/profiler");

function main() {
  const filePath = process.argv[2];
  if (!filePath) {
    process.stderr.write("Usage: data_profiler.js <file_path>\n");
    process.stderr.write("Returns JSON data profile (table stats, field stats, sample data, validity).\n");
    process.exit(1);
  }

  try {
    const resolved = path.resolve(filePath);
    const { headers, rows, parseInfo } = parseFile(resolved);
    const result = profile(headers, rows);
    result.parseInfo = parseInfo;
    result.fileName = path.basename(resolved);

    process.stdout.write(JSON.stringify(result, null, 2) + "\n");
  } catch (err) {
    process.stdout.write(JSON.stringify({ error: err.message }) + "\n");
    process.exit(1);
  }
}

main();


path: tools/excel_parser.js
content: #!/usr/bin/env node
"use strict";
const path = require("path");
const { parseFile } = require("./lib/parser");

function main() {
  const filePath = process.argv[2];
  if (!filePath) {
    process.stderr.write("Usage: excel_parser.js <file_path>\n");
    process.stderr.write("Supported: .csv, .tsv, .xlsx, .xls\n");
    process.exit(1);
  }

  try {
    const resolved = path.resolve(filePath);
    const { headers, rows, parseInfo } = parseFile(resolved);

    const result = {
      headers,
      rowCount: rows.length,
      columnCount: headers.length,
      parseInfo,
      preview: rows.slice(0, 5),
    };

    process.stdout.write(JSON.stringify(result, null, 2) + "\n");
  } catch (err) {
    process.stdout.write(JSON.stringify({ error: err.message }) + "\n");
    process.exit(1);
  }
}

main();


path: tools/lib/parser.js
content: "use strict";
const fs = require("fs");
const path = require("path");
const XLSX = require("xlsx");

function parseFile(filePath) {
  const ext = path.extname(filePath).toLowerCase();
  if (ext === ".csv" || ext === ".tsv") return parseCsv(filePath, ext);
  if (ext === ".xlsx" || ext === ".xls") return parseExcel(filePath);
  throw new Error(`Unsupported file type: ${ext}. Supported: .csv, .tsv, .xlsx, .xls`);
}

function parseCsv(filePath, ext) {
  const parseInfo = { fileType: "csv", encoding: null, delimiter: null, hasHeader: true, warnings: [] };

  const buf = fs.readFileSync(filePath);
  let encoding = "utf-8";
  if (buf[0] === 0xEF && buf[1] === 0xBB && buf[2] === 0xBF) encoding = "utf-8";
  else if (buf[0] === 0xFF && buf[1] === 0xFE) encoding = "utf-16le";
  else if (buf[0] === 0xFE && buf[1] === 0xFF) encoding = "utf-16be";
  parseInfo.encoding = encoding;

  const wb = XLSX.readFile(filePath, { type: "file", codepage: 65001, raw: true, cellDates: false });
  const sheetName = wb.SheetNames[0];
  const sheet = wb.Sheets[sheetName];
  if (!sheet) throw new Error("Empty file — no data found");

  const jsonData = XLSX.utils.sheet_to_json(sheet, { defval: null, raw: true });
  if (!jsonData.length) {
    const rawData = XLSX.utils.sheet_to_json(sheet, { header: 1, defval: null });
    if (rawData.length > 0) {
      parseInfo.hasHeader = false;
      parseInfo.warnings.push("No header detected, using row indices as column names");
      const headers = rawData[0].map((_, i) => `Column_${i + 1}`);
      const rows = rawData.slice(1).map(r => {
        const obj = {};
        headers.forEach((h, i) => { obj[h] = r[i] != null ? r[i] : null; });
        return obj;
      });
      return buildResult(headers, rows, parseInfo);
    }
    throw new Error("Empty file — no data found");
  }

  const headers = Object.keys(jsonData[0]);
  const unnamedCount = headers.filter(h => /^__EMPTY/.test(h) || /^Unnamed/.test(h)).length;
  if (unnamedCount === headers.length && jsonData.length > 0) {
    parseInfo.hasHeader = false;
    parseInfo.warnings.push("All columns unnamed, treating first row as data");
  }

  parseInfo.delimiter = ext === ".tsv" ? "\t" : ",";
  return buildResult(headers, jsonData, parseInfo);
}

function parseExcel(filePath) {
  const parseInfo = {
    fileType: "excel", sheetNames: [], activeSheet: null,
    hasMergedCells: false, mergedCellsCount: 0, warnings: [],
  };

  const wb = XLSX.readFile(filePath, { cellDates: true, cellNF: true });
  parseInfo.sheetNames = wb.SheetNames;
  const sheetName = wb.SheetNames[0];
  parseInfo.activeSheet = sheetName;
  const sheet = wb.Sheets[sheetName];
  if (!sheet) throw new Error("Empty workbook — no sheets found");

  if (sheet["!merges"] && sheet["!merges"].length > 0) {
    parseInfo.hasMergedCells = true;
    parseInfo.mergedCellsCount = sheet["!merges"].length;
    parseInfo.warnings.push(`Detected ${sheet["!merges"].length} merged cell regions (auto-filled by SheetJS)`);
  }

  if (wb.SheetNames.length > 1) {
    parseInfo.warnings.push(`Workbook has ${wb.SheetNames.length} sheets, analyzing first: ${sheetName}`);
  }

  const jsonData = XLSX.utils.sheet_to_json(sheet, { defval: null, raw: false });
  if (!jsonData.length) throw new Error("Sheet is empty — no data found");

  let headers = Object.keys(jsonData[0]);
  headers = ensureUniqueHeaders(headers);
  const rows = jsonData.map(row => {
    const obj = {};
    headers.forEach((h, i) => { obj[h] = row[Object.keys(row)[i]] != null ? row[Object.keys(row)[i]] : null; });
    return obj;
  });

  return buildResult(headers, rows, parseInfo);
}

function ensureUniqueHeaders(headers) {
  const seen = {};
  return headers.map(h => {
    const key = h == null ? "Column" : String(h);
    if (seen[key] != null) {
      seen[key]++;
      return `${key}_${seen[key]}`;
    }
    seen[key] = 0;
    return key;
  });
}

function buildResult(headers, rows, parseInfo) {
  parseInfo.rowCount = rows.length;
  parseInfo.columnCount = headers.length;
  parseInfo.mojibakeDetected = detectMojibake(headers, rows);
  return { headers, rows, parseInfo };
}

function detectMojibake(headers, rows) {
  const mojibakeRe = /[\ufffd]{2,}|[\u00c0-\u00c3][\u00a0-\u00bf]{1,2}|[\u0080-\u009f]/;
  let hits = 0;
  const sample = rows.slice(0, 100);
  for (const row of sample) {
    for (const h of headers) {
      const v = row[h];
      if (typeof v === "string" && mojibakeRe.test(v)) { hits++; break; }
    }
  }
  return hits > sample.length * 0.1;
}

module.exports = { parseFile };


path: tools/lib/profiler.js
content: "use strict";
const U = require("./utils");

const PATTERNS = {
  email:        /^[\w.\-+]+@[\w.\-]+\.\w{2,}$/,
  phone_cn:     /^1[3-9]\d{9}$/,
  phone_us:     /^(?:\+1)?[\s\-]?\(?\d{3}\)?[\s\-]?\d{3}[\s\-]?\d{4}$/,
  phone_intl:   /^\+?\d{10,15}$/,
  url:          /^https?:\/\/[\w.\-]+(?:\/[\w.\-/?=&#%]*)*$/,
  ip_v4:        /^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/,
  date_iso:     /^\d{4}-\d{2}-\d{2}$/,
  date_eu_dot:  /^\d{1,2}\.\d{1,2}\.\d{4}$/,
  date_eu_slash:/^\d{1,2}\/\d{1,2}\/\d{4}$/,
  date_cn:      /^\d{4}年\d{1,2}月\d{1,2}日$/,
  date_kr:      /^\d{4}년\s?\d{1,2}월\s?\d{1,2}일$/,
  datetime_iso: /^\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}/,
  percentage:   /^\d+(\.\d+)?%$/,
  currency_cny: /^¥[\d,]+(\.\d{2})?$/,
  currency_usd: /^\$[\d,]+(\.\d{2})?$/,
  currency_eur: /^€[\d,.]+$/,
  currency_gbp: /^£[\d,.]+$/,
  currency_jpy: /^¥?[\d,]+円$/,
  currency_krw: /^₩?[\d,]+(?:원)?$/,
  id_card_cn:   /^\d{17}[\dXx]$/,
  uuid:         /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i,
};

const MAX_UNIQUE_FOR_CATEGORICAL = 50;

function profile(headers, rows, sampleSize) {
  sampleSize = sampleSize || 10;
  const start = Date.now();

  const tableStats = profileTable(headers, rows);
  const fieldStats = headers.map(h => profileField(h, rows));
  const sampleData = extractSample(headers, rows, sampleSize);
  const dataValidity = checkDataValidity(rows, tableStats, fieldStats);

  return {
    tableStats,
    fieldStats,
    sampleData,
    sampleSize: sampleData.length,
    profilingDurationMs: Date.now() - start,
    dataValidity,
  };
}

function profileTable(headers, rows) {
  const totalRows = rows.length;
  const totalCols = headers.length;
  let totalNulls = 0;
  let completeRows = 0;

  for (const row of rows) {
    let allPresent = true;
    for (const h of headers) {
      if (U.isNullish(row[h])) { totalNulls++; allPresent = false; }
    }
    if (allPresent) completeRows++;
  }

  const totalCells = totalRows * totalCols;
  const duplicateRows = countDuplicateRows(headers, rows);

  const columnTypes = {};
  for (const h of headers) {
    const t = inferFieldType(h, rows);
    columnTypes[t] = (columnTypes[t] || 0) + 1;
  }

  const memBytes = estimateMemory(headers, rows);

  return {
    totalRows,
    totalColumns: totalCols,
    memorySizeBytes: memBytes,
    memorySizeReadable: U.formatBytes(memBytes),
    completeRows,
    completeRowsPercentage: U.round(completeRows / Math.max(totalRows, 1) * 100, 2),
    totalNullCells: totalNulls,
    totalNullPercentage: U.round(totalNulls / Math.max(totalCells, 1) * 100, 2),
    columnTypes,
    duplicateRows,
    duplicatePercentage: U.round(duplicateRows / Math.max(totalRows, 1) * 100, 2),
  };
}

function profileField(name, rows) {
  const values = rows.map(r => r[name]);
  const totalCount = values.length;
  const nonNullValues = values.filter(v => !U.isNullish(v));
  const nonNullCount = nonNullValues.length;
  const nullCount = totalCount - nonNullCount;
  const uniqueSet = new Set(nonNullValues.map(String));
  const uniqueCount = uniqueSet.size;
  const inferredType = inferFieldType(name, rows);

  const stat = {
    name,
    inferredType,
    totalCount,
    nonNullCount,
    nullCount,
    nullPercentage: U.round(nullCount / Math.max(totalCount, 1) * 100, 2),
    uniqueCount,
    uniquePercentage: U.round(uniqueCount / Math.max(nonNullCount, 1) * 100, 2),
    sampleValues: nonNullValues.slice(0, 5).map(U.toJsonSafe),
  };

  if (inferredType === "integer" || inferredType === "float") {
    const nums = nonNullValues.map(U.tryParseNumber).filter(n => n !== null);
    if (nums.length) {
      stat.minValue = Math.min(...nums);
      stat.maxValue = Math.max(...nums);
      stat.meanValue = U.round(nums.reduce((a, b) => a + b, 0) / nums.length, 2);
      stat.medianValue = U.round(U.median(nums), 2);
      stat.stdValue = nums.length > 1 ? U.round(U.std(nums), 2) : 0;
    }
  }

  if (["string", "text", "categorical"].includes(inferredType)) {
    const lens = nonNullValues.map(v => String(v).length);
    if (lens.length) {
      stat.minLength = Math.min(...lens);
      stat.maxLength = Math.max(...lens);
      stat.avgLength = U.round(lens.reduce((a, b) => a + b, 0) / lens.length, 1);
    }
  }

  if (uniqueCount <= MAX_UNIQUE_FOR_CATEGORICAL && nonNullCount > 0) {
    stat.topValues = U.topValues(nonNullValues.map(String), 10);
  }

  if (inferredType === "string" && nonNullCount > 0) {
    const detected = detectPatterns(nonNullValues);
    if (detected.length) stat.detectedPatterns = detected;
  }

  return stat;
}

function inferFieldType(name, rows) {
  const sample = rows.slice(0, 100).map(r => r[name]).filter(v => !U.isNullish(v));
  if (!sample.length) return "unknown";

  const boolVals = new Set(["true", "false", "yes", "no", "是", "否", "1", "0"]);
  if (sample.every(v => boolVals.has(String(v).toLowerCase()))) return "boolean";

  if (sample.every(v => { const n = U.tryParseNumber(v); return n !== null; })) {
    return sample.some(v => String(v).includes(".")) ? "float" : "integer";
  }

  const dateCount = sample.filter(v => U.tryParseDate(v) !== null).length;
  if (dateCount > sample.length * 0.8) {
    return sample.some(v => /[T ]\d{2}:/.test(String(v))) ? "datetime" : "date";
  }

  const strs = sample.map(v => String(v));
  const avgLen = strs.reduce((s, v) => s + v.length, 0) / strs.length;
  if (avgLen > 100) return "text";

  const allValues = rows.map(r => r[name]).filter(v => !U.isNullish(v));
  const uniq = new Set(allValues.map(String));
  const uniqueRatio = uniq.size / Math.max(allValues.length, 1);
  if (uniqueRatio < 0.05 && uniq.size <= MAX_UNIQUE_FOR_CATEGORICAL) return "categorical";

  return "string";
}

function detectPatterns(values) {
  const sample = values.slice(0, 100).map(String);
  const found = [];
  for (const [name, regex] of Object.entries(PATTERNS)) {
    const matched = sample.filter(v => regex.test(v)).length;
    if (matched / sample.length > 0.8) found.push(name);
  }
  return found;
}

function extractSample(headers, rows, size) {
  if (!rows.length) return [];
  const scored = rows.map((row, idx) => {
    const filled = headers.filter(h => !U.isNullish(row[h])).length;
    return { idx, filled };
  });
  scored.sort((a, b) => b.filled - a.filled);
  return scored.slice(0, size).map(s => {
    const obj = {};
    headers.forEach(h => { obj[h] = U.toJsonSafe(rows[s.idx][h]); });
    return obj;
  });
}

function countDuplicateRows(headers, rows) {
  const seen = new Set();
  let dups = 0;
  for (const row of rows) {
    const key = headers.map(h => String(row[h] ?? "")).join("\x00");
    if (seen.has(key)) dups++;
    else seen.add(key);
  }
  return dups;
}

function estimateMemory(headers, rows) {
  let bytes = 0;
  for (const row of rows) {
    for (const h of headers) {
      const v = row[h];
      if (v == null) bytes += 8;
      else if (typeof v === "number") bytes += 8;
      else if (typeof v === "string") bytes += v.length * 2;
      else bytes += 16;
    }
  }
  return bytes;
}

function checkDataValidity(rows, tableStats, fieldStats) {
  const issues = [];
  const warnings = [];
  let canProceed = true;
  const { totalRows, totalColumns: totalCols, totalNullPercentage: nullRate,
    completeRowsPercentage: completeRatio, duplicatePercentage: dupRatio } = tableStats;

  if (totalCols > 100) { issues.push(`Too many columns (${totalCols}), may not be a standard table`); canProceed = false; }
  else if (totalCols > 50) warnings.push(`Many columns (${totalCols}), some checks will use simplified mode`);
  if (totalCols < 2) { issues.push("Too few columns for meaningful analysis"); canProceed = false; }
  if (totalRows < 5) { issues.push(`Too few rows (${totalRows}) for statistical analysis`); canProceed = false; }
  if (nullRate > 90) { issues.push(`Extremely high null rate (${nullRate}%), data is nearly empty`); canProceed = false; }
  else if (nullRate > 70) warnings.push(`High null rate (${nullRate}%), data may be incomplete`);

  const unnamedCols = fieldStats.filter(f => /Unnamed|Column_\d+/.test(f.name)).length;
  const unnamedRatio = unnamedCols / Math.max(totalCols, 1);
  if (unnamedRatio > 0.7) { issues.push(`Many unnamed columns (${unnamedCols}/${totalCols}), header may not be detected`); canProceed = false; }
  else if (unnamedRatio > 0.3) warnings.push(`Several unnamed columns (${unnamedCols}), check header`);

  const objectFields = fieldStats.filter(f => ["string", "text", "unknown"].includes(f.inferredType)).length;
  if (objectFields / Math.max(totalCols, 1) > 0.95 && totalCols > 5) {
    warnings.push("Nearly all columns are text type, may not be structured data");
  }
  if (completeRatio < 5 && totalRows > 100) warnings.push(`Very low complete-row ratio (${completeRatio}%)`);
  if (dupRatio > 80) warnings.push(`Very high duplicate-row ratio (${dupRatio}%)`);

  return {
    canProceed,
    issues,
    warnings,
    blockReason: canProceed ? null : (issues[0] || null),
    summary: { totalRows, totalCols, nullRate, unnamedCols, completeRowsPct: completeRatio },
  };
}

module.exports = { profile, inferFieldType, profileField, profileTable };


path: tools/lib/scanner.js
content: "use strict";
const U = require("./utils");
const { inferFieldType } = require("./profiler");

// ===================== Completeness =====================

function scanNullValues(headers, rows, fieldStats) {
  const issues = [];
  const totalRows = rows.length;
  const totalCols = headers.length;
  const thresholds = { critical: 80, high: 50, medium: 20, low: 5 };
  const severityFields = { critical: [], high: [], medium: [], low: [] };
  const nullStats = {};

  for (const h of headers) {
    let count = 0;
    const indices = [];
    rows.forEach((r, i) => { if (U.isNullish(r[h])) { count++; if (indices.length < 100) indices.push(i); } });
    const pct = count / Math.max(totalRows, 1) * 100;
    nullStats[h] = { count, pct };
    if (pct >= thresholds.critical) severityFields.critical.push({ col: h, pct, count, indices });
    else if (pct >= thresholds.high) severityFields.high.push({ col: h, pct, count, indices });
    else if (pct >= thresholds.medium) severityFields.medium.push({ col: h, pct, count, indices });
    else if (pct >= thresholds.low) severityFields.low.push({ col: h, pct, count, indices });
  }

  const problemCount = Object.values(severityFields).reduce((s, a) => s + a.length, 0);
  const problemRatio = problemCount / Math.max(totalCols, 1);
  const weights = { critical: 4, high: 3, medium: 2, low: 1 };
  const totalWeight = Object.entries(severityFields).reduce((s, [k, v]) => s + v.length * weights[k], 0);
  const baseDeduction = Math.min(problemRatio * 40, 30);
  const severityBonus = totalCols > 0 ? Math.min(totalWeight / totalCols * 20, 20) : 0;
  const totalDeduction = baseDeduction + severityBonus;

  function allocDeduction(sev, list) {
    if (!list.length || !totalWeight) return;
    const perField = (weights[sev] * list.length / totalWeight * totalDeduction) / list.length;
    const maxPer = { critical: 10, high: 6, medium: 4, low: 2 }[sev];
    const sevMap = { critical: "critical", high: "high", medium: "medium", low: "low" };
    const titleMap = { critical: "Extremely High Null Rate", high: "High Null Rate", medium: "Multiple Null Values", low: "Few Null Values" };
    for (const f of list) {
      issues.push({
        id: `null_${sev}_${f.col}`, moduleId: "null_value", category: "completeness",
        severity: sevMap[sev], title: `[${f.col}] ${titleMap[sev]} (Missing Values)`,
        description: `Null values account for ${U.round(f.pct, 1)}% of the field`,
        affectedColumns: [f.col], affectedRows: f.count,
        affectedPercentage: U.round(f.pct, 1), affectedRowIndices: f.indices,
        suggestion: sev === "critical" ? "Consider removing this field or checking data collection" : "Check data source or fill missing values",
        deduction: U.round(Math.min(perField, maxPer), 2),
      });
    }
  }
  allocDeduction("critical", severityFields.critical);
  allocDeduction("high", severityFields.high);
  allocDeduction("medium", severityFields.medium);
  allocDeduction("low", severityFields.low);

  return { moduleId: "null_value", moduleName: "Null Value Check", issues };
}

function scanEmptyStrings(headers, rows) {
  const issues = [];
  const totalRows = rows.length;
  const totalCols = headers.length;
  const textCols = headers.filter(h => rows.some(r => typeof r[h] === "string"));
  const problems = [];

  for (const col of textCols) {
    let count = 0;
    const indices = [];
    const emptyVals = new Set(["", "nan", "none", "null", "NULL", "NaN"]);
    rows.forEach((r, i) => {
      if (r[col] != null && emptyVals.has(String(r[col]).trim())) { count++; if (indices.length < 100) indices.push(i); }
    });
    const pct = count / Math.max(totalRows, 1) * 100;
    if (pct >= 10) problems.push({ col, count, pct, indices });
  }

  if (problems.length) {
    const ratio = problems.length / Math.max(totalCols, 1);
    const totalDed = Math.min(ratio * 15 + 2, 10);
    const perField = totalDed / problems.length;
    for (const p of problems) {
      issues.push({
        id: `empty_str_${p.col}`, moduleId: "empty_string", category: "completeness",
        severity: "medium", title: `[${p.col}] Empty Strings Present`,
        description: `Found ${p.count} empty strings or pseudo-null values (${U.round(p.pct, 1)}%)`,
        affectedColumns: [p.col], affectedRows: p.count, affectedPercentage: U.round(p.pct, 1),
        affectedRowIndices: p.indices,
        suggestion: "Convert empty strings to true null values for unified processing",
        deduction: U.round(Math.min(perField, 3), 2),
      });
    }
  }
  return { moduleId: "empty_string", moduleName: "Empty String Check", issues };
}

function scanRowCompleteness(headers, rows) {
  const issues = [];
  const totalRows = rows.length;
  const totalCols = headers.length;

  let emptyRowCount = 0;
  const emptyIndices = [];
  let highNullCount = 0;
  const highNullIndices = [];

  rows.forEach((row, i) => {
    const nulls = headers.filter(h => U.isNullish(row[h])).length;
    if (nulls === totalCols) { emptyRowCount++; if (emptyIndices.length < 100) emptyIndices.push(i); }
    else if (nulls / totalCols > 0.5) { highNullCount++; if (highNullIndices.length < 100) highNullIndices.push(i); }
  });

  if (emptyRowCount > 0) {
    const pct = emptyRowCount / totalRows * 100;
    issues.push({
      id: "empty_rows", moduleId: "row_completeness", category: "completeness",
      severity: "high", title: "Completely Empty Rows Present",
      description: `Found ${emptyRowCount} completely empty rows (${U.round(pct, 1)}%)`,
      affectedRows: emptyRowCount, affectedPercentage: U.round(pct, 1),
      affectedRowIndices: emptyIndices,
      suggestion: "Consider removing completely empty rows",
      deduction: U.round(Math.min(pct / 100 * 20, 10), 2),
    });
  }
  if (highNullCount > 0) {
    issues.push({
      id: "high_null_rows", moduleId: "row_completeness", category: "completeness",
      severity: "info", title: "High Missing Rate Rows Present",
      description: `Found ${highNullCount} rows with >50% fields missing`,
      affectedRows: highNullCount, affectedPercentage: U.round(highNullCount / totalRows * 100, 1),
      affectedRowIndices: highNullIndices,
      suggestion: "Check data source for these rows", deduction: 0,
    });
  }
  return { moduleId: "row_completeness", moduleName: "Row Completeness Check", issues };
}

// ===================== Uniqueness =====================

function scanDuplicateRows(headers, rows) {
  const issues = [];
  const totalRows = rows.length;
  const seen = new Map();

  rows.forEach((row, i) => {
    const key = headers.map(h => String(row[h] ?? "")).join("\x00");
    if (!seen.has(key)) seen.set(key, []);
    seen.get(key).push(i);
  });

  let dupCount = 0;
  let dupGroups = 0;
  const dupIndices = [];
  for (const [, idxs] of seen) {
    if (idxs.length > 1) {
      dupGroups++;
      dupCount += idxs.length;
      for (const i of idxs) { if (dupIndices.length < 100) dupIndices.push(i); }
    }
  }

  if (dupCount > 0) {
    const pct = dupCount / totalRows * 100;
    const sev = pct > 10 ? "critical" : pct > 5 ? "high" : "medium";
    issues.push({
      id: "duplicate_rows", moduleId: "duplicate_row", category: "uniqueness",
      severity: sev, title: "Duplicate Rows Present",
      description: `Found ${dupCount} duplicate rows (${dupGroups} groups)`,
      affectedRows: dupCount, affectedPercentage: U.round(pct, 1),
      affectedRowIndices: dupIndices,
      suggestion: "Consider removing duplicate rows or checking data import",
      deduction: U.round(Math.min(pct * 0.5, 10), 2),
    });
  }
  return { moduleId: "duplicate_row", moduleName: "Duplicate Row Check", issues };
}

function scanPrimaryKey(headers, rows, fieldStats) {
  const issues = [];
  const totalRows = rows.length;
  const pkKeywords = [
    "id", "key", "code", "no", "pk", "uid", "uuid", "index",
    "编号", "编码", "序号",              // Chinese
    "番号", "コード",                   // Japanese
    "번호", "코드",                    // Korean
    "número", "código", "numéro",     // ES/PT/FR
    "nummer", "schlüssel",            // German
    "رقم",                            // Arabic
  ];

  const candidates = fieldStats.filter(f =>
    f.uniquePercentage > 95 && f.nullPercentage < 5 &&
    !["date", "datetime", "boolean"].includes(f.inferredType)
  ).sort((a, b) => {
    const aMatch = pkKeywords.some(k => a.name.toLowerCase().includes(k)) ? 1 : 0;
    const bMatch = pkKeywords.some(k => b.name.toLowerCase().includes(k)) ? 1 : 0;
    return (bMatch - aMatch) || (b.uniquePercentage - a.uniquePercentage);
  }).slice(0, 3);

  for (const fs of candidates) {
    const col = fs.name;
    const vals = rows.map(r => r[col]).filter(v => !U.isNullish(v));
    const dupSet = new Set();
    const seen = new Set();
    const dupIndices = [];
    vals.forEach((v, i) => {
      const s = String(v);
      if (seen.has(s)) { dupSet.add(s); if (dupIndices.length < 100) dupIndices.push(i); }
      else seen.add(s);
    });

    if (dupSet.size > 0) {
      const dupCount = dupIndices.length;
      issues.push({
        id: `pk_duplicate_${col}`, moduleId: "primary_key", category: "uniqueness",
        severity: "high", title: `[${col}] Suspected Primary Key Has Duplicates`,
        description: `Potential primary key has ${dupCount} duplicate values`,
        affectedColumns: [col], affectedRows: dupCount, affectedRowIndices: dupIndices,
        examples: [...dupSet].slice(0, 5),
        suggestion: "Check source of duplicate data if this field should be unique",
        deduction: 5,
      });
    }
  }
  return { moduleId: "primary_key", moduleName: "Primary Key Uniqueness Check", issues };
}

// ===================== Consistency =====================

function scanDataTypeConsistency(headers, rows, fieldStats) {
  const issues = [];
  for (const fs of fieldStats) {
    if (fs.inferredType !== "string") continue;
    const vals = rows.map(r => r[fs.name]).filter(v => !U.isNullish(v));
    if (!vals.length) continue;
    const numCount = vals.filter(v => U.tryParseNumber(v) !== null).length;
    const numPct = numCount / vals.length * 100;
    if (numPct > 10 && numPct < 90) {
      issues.push({
        id: `mixed_type_${fs.name}`, moduleId: "dtype_consistency", category: "consistency",
        severity: "medium", title: `[${fs.name}] Mixed Data Types`,
        description: `Contains ${U.round(numPct, 1)}% numeric and ${U.round(100 - numPct, 1)}% text data`,
        affectedColumns: [fs.name], examples: vals.slice(0, 5),
        suggestion: "Consider unifying data types or splitting the field",
        deduction: 3,
      });
    }
  }
  return { moduleId: "dtype_consistency", moduleName: "Data Type Consistency", issues };
}

function scanCaseConsistency(headers, rows, fieldStats) {
  const issues = [];
  const catFields = fieldStats.filter(f => f.inferredType === "categorical");
  for (const fs of catFields) {
    const vals = rows.map(r => r[fs.name]).filter(v => !U.isNullish(v)).map(String);
    const uniq = [...new Set(vals)];
    const lowerMap = {};
    for (const v of uniq) {
      const lv = v.toLowerCase();
      if (!lowerMap[lv]) lowerMap[lv] = [];
      lowerMap[lv].push(v);
    }
    const caseIssues = Object.values(lowerMap).filter(a => a.length > 1);
    if (caseIssues.length) {
      issues.push({
        id: `case_inconsistent_${fs.name}`, moduleId: "case_consistency", category: "consistency",
        severity: "low", title: `[${fs.name}] Case Inconsistency`,
        description: `Found ${caseIssues.length} groups of values with inconsistent case`,
        affectedColumns: [fs.name], examples: caseIssues.slice(0, 3).map(String),
        suggestion: "Consider unifying case format", deduction: 1,
      });
    }
  }
  return { moduleId: "case_consistency", moduleName: "Case Consistency", issues };
}

function scanDateFormat(headers, rows, fieldStats) {
  const issues = [];
  const dateKeywords = [
    "date", "time", "period", "year", "month", "day", "week",
    "created", "updated", "timestamp", "start", "end",
    "日期", "时间", "年份", "月份",                 // Chinese
    "日付", "時間", "年", "月",                    // Japanese
    "날짜", "시간", "연도", "월",                   // Korean
    "fecha", "hora", "período",                   // Spanish
    "datum", "zeit", "monat",                     // German
    "تاريخ", "وقت",                               // Arabic
    "date", "heure", "mois",                      // French
    "data", "ora",                                // Italian
  ];
  const stdPatterns = [
    [/^\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}/, "ISO DateTime"],
    [/^\d{4}-\d{2}-\d{2}/, "YYYY-MM-DD"],
    [/^\d{4}\/\d{2}\/\d{2}/, "YYYY/MM/DD"],
    [/^\d{4}\.\d{1,2}\.\d{1,2}$/, "YYYY.MM.DD"],
    [/^\d{1,2}\.\d{1,2}\.\d{4}$/, "DD.MM.YYYY"],
    [/^\d{1,2}\/\d{1,2}\/\d{4}/, "DD/MM/YYYY or MM/DD/YYYY"],
    [/^\d{1,2}-\d{1,2}-\d{4}$/, "DD-MM-YYYY"],
    [/^\d{8}$/, "YYYYMMDD"],
    [/^\d{4}年\d{1,2}月\d{1,2}日/, "YYYY年MM月DD日"],
    [/^\d{4}년\s?\d{1,2}월\s?\d{1,2}일/, "YYYY년MM월DD일"],
    [/^令和\d{1,2}年/, "JP Era (Reiwa)"],
    [/^\d{1,2}[-\s][A-Za-z]{3,}[-\s]\d{4}$/, "DD-Mon-YYYY"],
    [/^[A-Za-z]{3,}\s+\d{1,2},?\s+\d{4}$/, "Mon DD, YYYY"],
    [/^\d{10}$/, "Unix Epoch (s)"],
    [/^\d{13}$/, "Unix Epoch (ms)"],
  ];

  const dateFields = fieldStats.filter(f => ["date", "datetime"].includes(f.inferredType));
  const dateNames = new Set(dateFields.map(f => f.name));
  const potentialDateFields = fieldStats.filter(f =>
    !dateNames.has(f.name) && f.inferredType === "string" &&
    dateKeywords.some(k => f.name.toLowerCase().includes(k))
  );

  for (const fs of [...dateFields, ...potentialDateFields]) {
    const vals = rows.map(r => r[fs.name]).filter(v => !U.isNullish(v)).map(String).slice(0, 500);
    if (!vals.length) continue;
    const formats = {};
    let nonStd = 0;
    for (const v of vals) {
      let found = false;
      for (const [re, name] of stdPatterns) {
        if (re.test(v)) { formats[name] = (formats[name] || 0) + 1; found = true; break; }
      }
      if (!found) { formats["Non-standard"] = (formats["Non-standard"] || 0) + 1; nonStd++; }
    }
    const uniqueFormats = Object.keys(formats).length;
    if (!dateNames.has(fs.name) && nonStd / vals.length > 0.1) {
      const sev = nonStd / vals.length > 0.5 ? "high" : "medium";
      issues.push({
        id: `date_format_nonstandard_${fs.name}`, moduleId: "date_format", category: "consistency",
        severity: sev, title: `[${fs.name}] Suspected Date Field Has Non-Standard Format`,
        description: `Field name suggests date/time but contains ${uniqueFormats} formats`,
        affectedColumns: [fs.name], examples: vals.filter(v => !stdPatterns.some(([re]) => re.test(v))).slice(0, 5),
        suggestion: "Convert to standard date format (YYYY-MM-DD)", deduction: sev === "high" ? 4 : 2,
      });
    } else if (dateNames.has(fs.name) && uniqueFormats > 1) {
      issues.push({
        id: `date_format_${fs.name}`, moduleId: "date_format", category: "consistency",
        severity: "medium", title: `[${fs.name}] Inconsistent Date Format`,
        description: `Detected ${uniqueFormats} date formats: ${Object.keys(formats).join(", ")}`,
        affectedColumns: [fs.name], examples: Object.keys(formats),
        suggestion: "Unify to ISO 8601 format (YYYY-MM-DD)", deduction: 2,
      });
    }
  }
  return { moduleId: "date_format", moduleName: "Date Format Consistency", issues };
}

// ===================== Validity =====================

function scanFormatValidity(headers, rows, fieldStats) {
  const issues = [];
  for (const fs of fieldStats) {
    const col = fs.name;
    const colLower = col.toLowerCase();
    const vals = rows.map(r => r[col]).filter(v => !U.isNullish(v)).map(String);
    if (!vals.length) continue;

    const emailKw = [
      "email", "mail", "e-mail", "邮箱", "邮件",
      "メール", "이메일", "correo", "courriel", "بريد",
    ];
    if (emailKw.some(k => colLower.includes(k))) {
      const re = /^[\w.\-+]+@[\w.\-]+\.\w{2,}$/;
      const invalid = vals.filter(v => !re.test(v));
      if (invalid.length) {
        issues.push({
          id: `email_invalid_${col}`, moduleId: "format_validity", category: "validity",
          severity: "medium", title: `[${col}] Invalid Email Format`,
          description: `Found ${invalid.length} invalid email addresses`,
          affectedColumns: [col], affectedRows: invalid.length,
          affectedPercentage: U.round(invalid.length / vals.length * 100, 1),
          examples: invalid.slice(0, 5), suggestion: "Check email format", deduction: 2,
        });
      }
    }
    const phoneKw = [
      "phone", "mobile", "tel", "cell", "fax",
      "电话", "手机", "座机",                     // Chinese
      "電話", "携帯",                            // Japanese
      "전화", "휴대폰",                           // Korean
      "teléfono", "móvil", "celular",           // Spanish
      "téléphone", "portable",                  // French
      "telefon", "handy",                       // German
      "هاتف", "جوال", "موبايل",                   // Arabic
    ];
    if (phoneKw.some(k => colLower.includes(k))) {
      const phonePatterns = [
        /^1[3-9]\d{9}$/,                        // China mobile
        /^\d{3,4}-?\d{7,8}$/,                   // China landline
        /^0\d{1,4}-?\d{6,8}$/,                  // Japan / regional
        /^0[1-9]0-?\d{4}-?\d{4}$/,              // Japan mobile
        /^01[016789]-?\d{3,4}-?\d{4}$/,         // Korea mobile
        /^\+?\d{1,4}[\s\-]?\(?\d{1,5}\)?[\s\-]?\d{3,4}[\s\-]?\d{3,5}$/, // Intl generic
        /^\(\d{3}\)\s?\d{3}-\d{4}$/,            // US (xxx) xxx-xxxx
        /^\d{3}-\d{3}-\d{4}$/,                  // US xxx-xxx-xxxx
        /^0\d{2,4}\s?\d{6,8}$/,                 // EU landline
        /^05\d{8}$/,                            // Middle East mobile (SA, AE)
      ];
      const invalid = vals.filter(v => {
        const cleaned = v.replace(/\s/g, "");
        return !phonePatterns.some(re => re.test(cleaned) || re.test(v));
      });
      if (invalid.length > vals.length * 0.1) {
        issues.push({
          id: `phone_invalid_${col}`, moduleId: "format_validity", category: "validity",
          severity: "medium", title: `[${col}] Suspicious Phone Format`,
          description: `Found ${invalid.length} possibly invalid phone numbers`,
          affectedColumns: [col], affectedRows: invalid.length, examples: invalid.slice(0, 5),
          suggestion: "Check phone number format and country code",
          deduction: 2,
        });
      }
    }
  }
  return { moduleId: "format_validity", moduleName: "Format Validity", issues };
}

function scanSpecialChars(headers, rows) {
  const issues = [];
  const totalRows = rows.length;
  const textCols = headers.filter(h => rows.some(r => typeof r[h] === "string"));
  const invisibleRe = /[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]/;

  for (const col of textCols) {
    const vals = rows.map(r => r[col]).filter(v => !U.isNullish(v)).map(String);
    if (!vals.length) continue;

    const invisibleCount = vals.filter(v => invisibleRe.test(v)).length;
    if (invisibleCount > 0) {
      issues.push({
        id: `invisible_char_${col}`, moduleId: "special_char", category: "validity",
        severity: "medium", title: `[${col}] Contains Invisible Characters`,
        description: `Found ${invisibleCount} records containing invisible control characters`,
        affectedColumns: [col], affectedRows: invisibleCount,
        suggestion: "Clean invisible characters", deduction: 2,
      });
    }

    const leadTrail = vals.filter(v => /^\s/.test(v) || /\s$/.test(v)).length;
    if (leadTrail > vals.length * 0.05) {
      issues.push({
        id: `whitespace_${col}`, moduleId: "special_char", category: "validity",
        severity: "low", title: `[${col}] Extra Whitespace Present`,
        description: `Found ${leadTrail} records with leading/trailing whitespace`,
        affectedColumns: [col], affectedRows: leadTrail,
        suggestion: "Use TRIM to clean whitespace", deduction: 0.5,
      });
    }
  }
  return { moduleId: "special_char", moduleName: "Special Character Check", issues };
}

function scanRangeValidity(headers, rows, fieldStats) {
  const issues = [];
  const numFields = fieldStats.filter(f => ["integer", "float"].includes(f.inferredType));
  const posKw = [
    "count", "amount", "price", "quantity", "age", "salary", "weight", "height", "area", "size",
    "数量", "金额", "价格", "面积", "人数", "年龄",                     // Chinese
    "数量", "金額", "価格", "年齢",                                   // Japanese
    "수량", "금액", "가격", "나이",                                    // Korean
    "cantidad", "precio", "edad", "monto",                          // Spanish
    "betrag", "preis", "alter", "menge",                            // German
    "quantité", "prix", "âge", "montant",                           // French
    "كمية", "مبلغ", "سعر", "عمر",                                    // Arabic
  ];
  const pctKw = [
    "rate", "percent", "%", "ratio", "pct", "proportion",
    "率", "比例", "百分比",                                           // Chinese
    "率", "割合",                                                    // Japanese
    "율", "비율",                                                    // Korean
    "tasa", "porcentaje",                                           // Spanish
    "taux", "pourcentage",                                          // French
    "rate", "prozent", "anteil",                                    // German
    "نسبة", "معدل",                                                  // Arabic
  ];

  for (const fs of numFields) {
    const col = fs.name;
    const colLower = col.toLowerCase();
    const nums = rows.map(r => U.tryParseNumber(r[col])).filter(n => n !== null);
    if (nums.length < 10) continue;

    if (posKw.some(k => colLower.includes(k))) {
      const negCount = nums.filter(n => n < 0).length;
      if (negCount > 0) {
        issues.push({
          id: `negative_${col}`, moduleId: "range_validity", category: "validity",
          severity: "medium", title: `[${col}] Negative Values Present`,
          description: `Field should not be negative but has ${negCount} negative values`,
          affectedColumns: [col], affectedRows: negCount,
          examples: nums.filter(n => n < 0).slice(0, 5),
          suggestion: "Verify negative value data", deduction: 2,
        });
      }
    }
    if (pctKw.some(k => colLower.includes(k))) {
      const oor = nums.filter(n => n < 0 || n > 100).length;
      if (oor > 0) {
        issues.push({
          id: `percent_range_${col}`, moduleId: "range_validity", category: "validity",
          severity: "high", title: `[${col}] Percentage Out of Range`,
          description: `Found ${oor} values outside 0-100 range`,
          affectedColumns: [col], affectedRows: oor,
          suggestion: "Percentage should be between 0-100", deduction: 3,
        });
      }
    }
  }
  return { moduleId: "range_validity", moduleName: "Numeric Range Validity", issues };
}

// ===================== Accuracy =====================

function scanOutliers(headers, rows, fieldStats) {
  const issues = [];
  const numFields = fieldStats.filter(f => ["integer", "float"].includes(f.inferredType));
  const idKw = ["id", "code", "编号", "编码", "no"];
  const iqrMultiplier = 2.0;

  for (const fs of numFields) {
    const col = fs.name;
    if (idKw.some(k => col.toLowerCase().endsWith(k))) continue;
    const nums = rows.map(r => U.tryParseNumber(r[col])).filter(n => n !== null);
    if (nums.length < 20) continue;

    const Q1 = U.quantile(nums, 0.25);
    const Q3 = U.quantile(nums, 0.75);
    const IQR = Q3 - Q1;
    if (IQR === 0) continue;

    const lower = Q1 - iqrMultiplier * IQR;
    const upper = Q3 + iqrMultiplier * IQR;
    const outliers = nums.filter(n => n < lower || n > upper);
    const pct = outliers.length / nums.length * 100;

    if (outliers.length > 0 && pct > 1) {
      const high = nums.filter(n => n > upper).length;
      const low = nums.filter(n => n < lower).length;
      let dir = "";
      if (high > 0 && low > 0) dir = ` (high: ${high}, low: ${low})`;
      else if (high > 0) dir = ` (all high, >${U.round(upper, 2)})`;
      else dir = ` (all low, <${U.round(lower, 2)})`;

      const sev = pct > 10 ? "high" : pct > 5 ? "medium" : "low";
      issues.push({
        id: `outlier_${col}`, moduleId: "outlier", category: "accuracy",
        severity: sev, title: `[${col}] Outliers Present`,
        description: `IQR×${iqrMultiplier} detected ${outliers.length} outliers (${U.round(pct, 1)}%)${dir}`,
        affectedColumns: [col], affectedRows: outliers.length,
        affectedPercentage: U.round(pct, 1),
        examples: outliers.slice(0, 5).map(v => U.round(v, 2)),
        suggestion: "Judge whether outliers need handling based on business context",
        deduction: U.round(Math.min(pct * 0.15, 4), 2),
      });
    }
  }
  return { moduleId: "outlier", moduleName: "Outlier Detection", issues };
}

function scanRareValues(headers, rows, fieldStats) {
  const issues = [];
  const catFields = fieldStats.filter(f => f.inferredType === "categorical" && f.uniqueCount < 50);
  const rarePct = 0.5;

  for (const fs of catFields) {
    const vals = rows.map(r => r[fs.name]).filter(v => !U.isNullish(v)).map(String);
    if (!vals.length) continue;
    const freq = U.countBy(vals);
    const threshold = vals.length * (rarePct / 100);
    const rares = Object.entries(freq).filter(([, c]) => c < threshold);
    if (rares.length > 0 && rares.length < 10) {
      const totalAffected = rares.reduce((s, [, c]) => s + c, 0);
      issues.push({
        id: `rare_value_${fs.name}`, moduleId: "rare_value", category: "accuracy",
        severity: "info", title: `[${fs.name}] Rare Values Present`,
        description: `Found ${rares.length} values with frequency <${rarePct}%`,
        affectedColumns: [fs.name], affectedRows: totalAffected,
        examples: rares.map(([v]) => v),
        suggestion: "May be typos or abnormal data", deduction: 0.5,
      });
    }
  }
  return { moduleId: "rare_value", moduleName: "Rare Value Detection", issues };
}

// ===================== Timeliness =====================

function scanTimeFreshness(headers, rows, fieldStats) {
  const issues = [];
  const dateFields = fieldStats.filter(f => ["date", "datetime"].includes(f.inferredType));
  const now = new Date();

  for (const fs of dateFields) {
    const col = fs.name;
    const dates = rows.map(r => U.tryParseDate(r[col])).filter(d => d !== null);
    if (!dates.length) continue;

    const futureCount = dates.filter(d => d > now).length;
    if (futureCount > 0 && futureCount < dates.length * 0.5) {
      issues.push({
        id: `future_date_${col}`, moduleId: "time_freshness", category: "timeliness",
        severity: "low", title: `[${col}] Contains Future Dates`,
        description: `Found ${futureCount} future date records`,
        affectedColumns: [col], affectedRows: futureCount,
        suggestion: "Confirm if future dates are correct", deduction: 1,
      });
    }

    const oldThreshold = new Date("1990-01-01");
    const oldCount = dates.filter(d => d < oldThreshold).length;
    if (oldCount > 0) {
      issues.push({
        id: `old_date_${col}`, moduleId: "time_freshness", category: "timeliness",
        severity: "low", title: `[${col}] Contains Abnormal Historical Dates`,
        description: `Found ${oldCount} records dated before 1990`,
        affectedColumns: [col], affectedRows: oldCount,
        suggestion: "Verify historical date data", deduction: 0.5,
      });
    }
  }
  return { moduleId: "time_freshness", moduleName: "Data Timeliness", issues };
}

// ===================== Consistency (new) =====================

function scanFullwidthChars(headers, rows) {
  const issues = [];
  const fwDigit = /[\uff10-\uff19]/;
  const fwLetter = /[\uff21-\uff3a\uff41-\uff5a]/;
  const fwPunct = /[\uff01\uff0c\uff0e\uff1a\uff1b\uff1f\u3001\u3002]/;
  const fwAny = /[\uff01-\uff5e\u3000-\u303f]/;

  const textCols = headers.filter(h => rows.some(r => typeof r[h] === "string"));
  for (const col of textCols) {
    const vals = rows.map(r => r[col]).filter(v => !U.isNullish(v)).map(String);
    if (!vals.length) continue;
    let digitCount = 0, letterCount = 0, punctCount = 0, totalAffected = 0;
    const indices = [];
    vals.forEach((v, i) => {
      if (fwAny.test(v)) {
        totalAffected++;
        if (indices.length < 100) indices.push(i);
        if (fwDigit.test(v)) digitCount++;
        if (fwLetter.test(v)) letterCount++;
        if (fwPunct.test(v)) punctCount++;
      }
    });
    if (totalAffected === 0) continue;
    const pct = totalAffected / vals.length * 100;
    const parts = [];
    if (digitCount) parts.push(`digits: ${digitCount}`);
    if (letterCount) parts.push(`letters: ${letterCount}`);
    if (punctCount) parts.push(`punctuation: ${punctCount}`);
    const sev = pct > 30 ? "high" : pct > 5 ? "medium" : "low";
    issues.push({
      id: `fullwidth_${col}`, moduleId: "fullwidth_char", category: "consistency",
      severity: sev, title: `[${col}] Full-Width Characters Detected`,
      description: `${totalAffected} values (${U.round(pct, 1)}%) contain full-width characters (${parts.join(", ")})`,
      affectedColumns: [col], affectedRows: totalAffected,
      affectedPercentage: U.round(pct, 1), affectedRowIndices: indices,
      examples: vals.filter(v => fwAny.test(v)).slice(0, 5),
      suggestion: "Convert full-width characters to half-width for consistent processing",
      deduction: U.round(Math.min(pct * 0.1, 4), 2),
    });
  }
  return { moduleId: "fullwidth_char", moduleName: "Full-Width Character Detection", issues };
}

// ===================== Validity (new) =====================

function scanEncodingIssues(headers, rows, parseInfo) {
  const issues = [];
  const mojibakePatterns = [
    /\ufffd{2,}/,                             // Unicode replacement chars (any encoding)
    /[\u0080-\u009f]/,                        // C1 control chars (should never appear)
    // GBK/GB2312 read as UTF-8 (Chinese)
    /[\u00c0-\u00c3][\u00a0-\u00bf]{1,2}/,
    /\u00e4\u00b8[\u00a0-\u00bf]/,
    /\u00e6[\u00a0-\u00bf]{2}/,
    /\u00c2[\u00a0-\u00bf]/,
    // Shift_JIS read as UTF-8 (Japanese)
    /\u0082[\u00a0-\u00ff]/,
    /\u0083[\u00a0-\u00ff]/,
    // EUC-KR read as UTF-8 (Korean)
    /[\u00b0-\u00c8][\u00a1-\u00fe]/,
    // ISO-8859-1 / Windows-1252 read as UTF-8 (European accented text)
    /Ã[\u00a0-\u00bf]/,
    /Ã©|Ã¨|Ã¤|Ã¶|Ã¼|Ã±|Ã§|Ã¡|Ã³/,
    // Windows-1256 read as UTF-8 (Arabic)
    /[\u00c7\u00c8][\u00a0-\u00bf]{2,}/,
  ];

  if (parseInfo && parseInfo.mojibakeDetected) {
    issues.push({
      id: "encoding_global", moduleId: "encoding_issue", category: "validity",
      severity: "critical", title: "Suspected Encoding / Mojibake Issue",
      description: "File may have been read with incorrect encoding. Garbled character patterns detected during parsing.",
      suggestion: "Re-save the file as UTF-8 or specify the correct encoding",
      deduction: 10,
    });
    return { moduleId: "encoding_issue", moduleName: "Encoding / Mojibake Detection", issues };
  }

  const textCols = headers.filter(h => rows.some(r => typeof r[h] === "string"));
  for (const col of textCols) {
    const vals = rows.map(r => r[col]).filter(v => !U.isNullish(v)).map(String).slice(0, 500);
    if (!vals.length) continue;
    let hitCount = 0;
    const indices = [];
    vals.forEach((v, i) => {
      if (mojibakePatterns.some(re => re.test(v))) {
        hitCount++;
        if (indices.length < 100) indices.push(i);
      }
    });
    if (hitCount === 0) continue;
    const pct = hitCount / vals.length * 100;
    if (pct < 2) continue;
    const sev = pct > 20 ? "critical" : pct > 5 ? "high" : "medium";
    issues.push({
      id: `encoding_${col}`, moduleId: "encoding_issue", category: "validity",
      severity: sev, title: `[${col}] Suspected Garbled Text (Mojibake)`,
      description: `${hitCount} values (${U.round(pct, 1)}%) contain character patterns typical of encoding errors`,
      affectedColumns: [col], affectedRows: hitCount,
      affectedPercentage: U.round(pct, 1), affectedRowIndices: indices,
      examples: vals.filter(v => mojibakePatterns.some(re => re.test(v))).slice(0, 5),
      suggestion: "Re-open the source file with correct encoding (e.g., GBK for Chinese, Shift_JIS for Japanese, EUC-KR for Korean, ISO-8859-1 for European, Windows-1256 for Arabic)",
      deduction: U.round(Math.min(pct * 0.3, 8), 2),
    });
  }
  return { moduleId: "encoding_issue", moduleName: "Encoding / Mojibake Detection", issues };
}

// ===================== Completeness (new) =====================

function scanMergedCellPattern(headers, rows, parseInfo) {
  const issues = [];
  if (!parseInfo || !parseInfo.hasMergedCells) {
    return { moduleId: "merged_cell_pattern", moduleName: "Merged Cell Pattern Detection", issues };
  }

  for (const col of headers) {
    const vals = rows.map(r => r[col]);
    let streakCount = 0, longestStreak = 0, currentStreak = 0, lastNonNull = -1;
    let mergePatternHits = 0;

    for (let i = 0; i < vals.length; i++) {
      if (U.isNullish(vals[i])) {
        currentStreak++;
      } else {
        if (currentStreak >= 2 && lastNonNull >= 0) {
          mergePatternHits++;
          longestStreak = Math.max(longestStreak, currentStreak);
        }
        if (currentStreak > 0) streakCount++;
        currentStreak = 0;
        lastNonNull = i;
      }
    }
    if (currentStreak >= 2 && lastNonNull >= 0) {
      mergePatternHits++;
      longestStreak = Math.max(longestStreak, currentStreak);
    }

    if (mergePatternHits < 2) continue;

    const nullCount = vals.filter(v => U.isNullish(v)).length;
    const pct = nullCount / vals.length * 100;
    issues.push({
      id: `merged_pattern_${col}`, moduleId: "merged_cell_pattern", category: "completeness",
      severity: "high",
      title: `[${col}] Null Pattern Suggests Merged Cells`,
      description: `${mergePatternHits} contiguous null blocks detected (longest: ${longestStreak} rows). These nulls likely originate from merged cells and should be forward-filled, not treated as missing data.`,
      affectedColumns: [col], affectedRows: nullCount,
      affectedPercentage: U.round(pct, 1),
      suggestion: "Forward-fill (propagate the value above) instead of mean/median imputation",
      deduction: U.round(Math.min(mergePatternHits * 0.5, 5), 2),
    });
  }
  return { moduleId: "merged_cell_pattern", moduleName: "Merged Cell Pattern Detection", issues };
}

// ===================== Uniqueness (new) =====================

function scanFuzzyDuplicates(headers, rows, fieldStats) {
  const issues = [];
  const catFields = fieldStats.filter(f =>
    (f.inferredType === "categorical" || f.inferredType === "string") &&
    f.uniqueCount > 1 && f.uniqueCount <= 200
  );

  function normalize(s) {
    return String(s).normalize("NFC").toLowerCase().trim()
      .replace(/[\s\u3000]+/g, " ")
      .replace(/[.\-_'"()（）【】\[\]«»""'']/g, "")
      .replace(/[\u0610-\u061a\u064b-\u065f\u0670]/g, "")  // Arabic diacritics (tashkeel)
      .replace(/[\u0300-\u036f]/g, "");                     // Latin combining diacritics (é→e)
  }

  for (const fs of catFields) {
    const vals = rows.map(r => r[fs.name]).filter(v => !U.isNullish(v)).map(String);
    const uniq = [...new Set(vals)];
    const normMap = {};
    for (const v of uniq) {
      const n = normalize(v);
      if (!normMap[n]) normMap[n] = [];
      normMap[n].push(v);
    }
    const fuzzyGroups = Object.values(normMap).filter(a => a.length > 1);
    const caseGroups = fuzzyGroups.filter(group => {
      const lower = new Set(group.map(v => v.toLowerCase()));
      return lower.size === 1;
    });
    const nonCaseGroups = fuzzyGroups.filter(group => {
      const lower = new Set(group.map(v => v.toLowerCase()));
      return lower.size > 1 || group.some(v => v !== v.trim() || /[\s\u3000]{2,}/.test(v));
    });

    if (nonCaseGroups.length === 0 && caseGroups.length === 0) continue;
    const extraGroups = nonCaseGroups.length;
    if (extraGroups === 0) continue;

    issues.push({
      id: `fuzzy_dup_${fs.name}`, moduleId: "fuzzy_duplicate", category: "uniqueness",
      severity: extraGroups > 5 ? "high" : "medium",
      title: `[${fs.name}] Near-Duplicate Values Detected`,
      description: `${extraGroups} groups of values differ only by whitespace, punctuation, or formatting`,
      affectedColumns: [fs.name],
      examples: nonCaseGroups.slice(0, 3).map(g => g.join(" / ")),
      suggestion: "Normalize values to a consistent format",
      deduction: U.round(Math.min(extraGroups * 0.5, 4), 2),
    });
  }
  return { moduleId: "fuzzy_duplicate", moduleName: "Near-Duplicate Value Detection", issues };
}

// ===================== Consistency (new) =====================

function scanNumberWithUnit(headers, rows, fieldStats) {
  const issues = [];
  const unitPatterns = [
    // Currency prefix: ¥ $ € £ ₩ ₹ ₽ ₺ R$ ﷼ CHF kr
    { re: /^[¥$€£₩₹₽₺﷼]\s*[\d,.]+$/, label: "currency prefix" },
    { re: /^(?:R\$|CHF|kr|SEK|NOK|DKK|AED|SAR|KRW|JPY|INR|RUB|TRY|BRL|MXN|PLN|CZK|HUF|ILS)\s*[\d,.]+$/i, label: "currency code prefix" },
    // Currency suffix
    { re: /^[\d,.]+\s*[¥$€£₩₹₽₺﷼]$/, label: "currency suffix" },
    { re: /^[\d,.]+\s*(?:元|块|角|分|圆)$/, label: "CNY unit" },
    { re: /^[\d,.]+\s*(?:円|万円)$/, label: "JPY unit" },
    { re: /^[\d,.]+\s*(?:원|만원)$/, label: "KRW unit" },
    { re: /^[\d,.]+\s*(?:ريال|درهم|دينار|جنيه|ليرة)$/, label: "MENA currency" },
    { re: /^[\d,.]+\s*(?:EUR|USD|GBP|CHF|kr|руб|₽|zł|Kč|Ft)$/i, label: "currency code suffix" },
    // Magnitude
    { re: /^[\d,.]+\s*[万亿千百kKmMbBtT]$/, label: "magnitude suffix" },
    { re: /^[\d,.]+\s*(?:万|億|兆)$/, label: "CJK magnitude" },
    { re: /^[\d,.]+\s*(?:만|억)$/, label: "Korean magnitude" },
    { re: /^[\d,.]+\s*(?:mil|mln|mrd|bn|tn)$/i, label: "EU magnitude" },
    // Percentage
    { re: /^[\d,.]+\s*%$/, label: "percentage" },
    // Measurement units (international)
    { re: /^[\d,.]+\s*(?:个|件|台|套|箱|吨|只|条|把|张|份)$/, label: "CJK counter" },
    { re: /^[\d,.]+\s*(?:kg|lb|oz|g|mg|ton|tonnes?)$/i, label: "weight" },
    { re: /^[\d,.]+\s*(?:ml|l|L|gal|fl\.?\s?oz)$/i, label: "volume" },
    { re: /^[\d,.]+\s*(?:mm|cm|m|km|in|ft|yd|mi|miles?)$/i, label: "length" },
    { re: /^[\d,.]+\s*(?:m²|m2|km²|km2|sqft|sq\.?\s?m|ha|acres?)$/i, label: "area" },
    { re: /^[\d,.]+\s*(?:°[CF]|℃|℉)$/, label: "temperature" },
  ];

  const strFields = fieldStats.filter(f => f.inferredType === "string" && f.uniqueCount > 5);
  for (const fs of strFields) {
    const vals = rows.map(r => r[fs.name]).filter(v => !U.isNullish(v)).map(String);
    if (vals.length < 10) continue;

    const labelCounts = {};
    let totalHits = 0;
    for (const v of vals) {
      const trimmed = v.trim();
      for (const { re, label } of unitPatterns) {
        if (re.test(trimmed)) {
          labelCounts[label] = (labelCounts[label] || 0) + 1;
          totalHits++;
          break;
        }
      }
    }
    const pct = totalHits / vals.length * 100;
    if (pct < 10) continue;

    const labels = Object.entries(labelCounts)
      .sort((a, b) => b[1] - a[1])
      .map(([l]) => l).join(", ");

    issues.push({
      id: `num_unit_${fs.name}`, moduleId: "number_unit", category: "consistency",
      severity: pct > 50 ? "high" : "medium",
      title: `[${fs.name}] Numbers with Embedded Units`,
      description: `${totalHits} values (${U.round(pct, 1)}%) are numeric but include units (${labels}), preventing numeric analysis`,
      affectedColumns: [fs.name], affectedRows: totalHits,
      affectedPercentage: U.round(pct, 1),
      examples: vals.filter(v => unitPatterns.some(({ re }) => re.test(v.trim()))).slice(0, 5),
      suggestion: "Separate numeric values from units for proper analysis",
      deduction: U.round(Math.min(pct * 0.08, 5), 2),
    });
  }
  return { moduleId: "number_unit", moduleName: "Number-with-Unit Detection", issues };
}

function scanCrossColumnLogic(headers, rows, fieldStats) {
  const issues = [];
  const colLower = {};
  for (const h of headers) colLower[h] = h.toLowerCase();

  // Date pair checks — multilingual start/end keywords
  const startKw = [
    "start", "begin", "from", "since",
    "开始", "起始",                   // Chinese
    "開始",                          // Japanese
    "시작",                          // Korean
    "inicio", "desde",               // Spanish
    "début", "depuis",               // French
    "anfang", "beginn", "von",       // German
    "بداية", "من",                    // Arabic
  ];
  const endKw = [
    "end", "finish", "to", "until", "expir",
    "结束", "截止", "终止",            // Chinese
    "終了", "終わり",                  // Japanese
    "종료", "끝",                     // Korean
    "fin", "hasta",                  // Spanish
    "fin", "jusqu",                  // French
    "ende", "bis",                   // German
    "نهاية", "إلى",                   // Arabic
  ];

  const datePairs = [];
  const dateFields = fieldStats.filter(f => ["date", "datetime"].includes(f.inferredType));
  const dateNames = dateFields.map(f => f.name);
  for (let i = 0; i < dateNames.length; i++) {
    for (let j = i + 1; j < dateNames.length; j++) {
      const a = colLower[dateNames[i]], b = colLower[dateNames[j]];
      const aIsStart = startKw.some(k => a.includes(k));
      const aIsEnd = endKw.some(k => a.includes(k));
      const bIsStart = startKw.some(k => b.includes(k));
      const bIsEnd = endKw.some(k => b.includes(k));
      if (aIsStart && bIsEnd) datePairs.push([dateNames[i], dateNames[j]]);
      else if (bIsStart && aIsEnd) datePairs.push([dateNames[j], dateNames[i]]);
    }
  }

  for (const [startCol, endCol] of datePairs) {
    let violations = 0;
    const indices = [];
    rows.forEach((r, i) => {
      const s = U.tryParseDate(r[startCol]);
      const e = U.tryParseDate(r[endCol]);
      if (s && e && e < s) { violations++; if (indices.length < 100) indices.push(i); }
    });
    if (violations > 0) {
      issues.push({
        id: `date_order_${startCol}_${endCol}`, moduleId: "cross_column_logic", category: "consistency",
        severity: violations > rows.length * 0.05 ? "high" : "medium",
        title: `[${startCol}] > [${endCol}] — Date Order Violation`,
        description: `${violations} rows have start date after end date`,
        affectedColumns: [startCol, endCol], affectedRows: violations,
        affectedRowIndices: indices,
        suggestion: "Check if start/end dates are swapped",
        deduction: U.round(Math.min(violations / rows.length * 30, 5), 2),
      });
    }
  }

  // Numeric total checks
  const numFields = fieldStats.filter(f => ["integer", "float"].includes(f.inferredType));
  const numNames = numFields.map(f => f.name);
  const totalKw = [
    "total", "sum", "grand",
    "合计", "总计", "总额", "总和",             // Chinese
    "合計", "総計", "総額",                    // Japanese
    "합계", "총액", "총합",                     // Korean
    "total", "suma",                          // Spanish
    "total", "somme",                         // French
    "gesamt", "summe",                        // German
    "المجموع", "الإجمالي",                      // Arabic
  ];

  for (const tName of numNames) {
    const tLower = colLower[tName];
    if (!totalKw.some(k => tLower.includes(k))) continue;

    const candidates = numNames.filter(n => n !== tName && !totalKw.some(k => colLower[n].includes(k)));
    if (candidates.length < 2 || candidates.length > 5) continue;

    let matchCount = 0, mismatchCount = 0;
    const indices = [];
    rows.forEach((r, i) => {
      const totalVal = U.tryParseNumber(r[tName]);
      if (totalVal === null) return;
      const partSum = candidates.reduce((s, c) => {
        const v = U.tryParseNumber(r[c]);
        return v !== null ? s + v : s;
      }, 0);
      if (Math.abs(totalVal - partSum) < 0.01) matchCount++;
      else { mismatchCount++; if (indices.length < 100) indices.push(i); }
    });

    if (matchCount > 0 && mismatchCount > 0 && mismatchCount / (matchCount + mismatchCount) > 0.05) {
      issues.push({
        id: `sum_mismatch_${tName}`, moduleId: "cross_column_logic", category: "consistency",
        severity: "medium",
        title: `[${tName}] Sum Mismatch with Component Columns`,
        description: `${mismatchCount} rows where total does not equal sum of parts (${candidates.join(" + ")})`,
        affectedColumns: [tName, ...candidates], affectedRows: mismatchCount,
        affectedRowIndices: indices,
        suggestion: "Verify the calculation formula for the total column",
        deduction: U.round(Math.min(mismatchCount / rows.length * 20, 4), 2),
      });
    }
  }

  return { moduleId: "cross_column_logic", moduleName: "Cross-Column Logical Consistency", issues };
}

// ===================== Validity (new) =====================

function scanIdChecksum(headers, rows, fieldStats) {
  const issues = [];
  const idCardRe = /^\d{17}[\dXx]$/;
  const weights = [7, 9, 10, 5, 8, 4, 2, 1, 6, 3, 7, 9, 10, 5, 8, 4, 2];
  const checkChars = "10X98765432";

  function validateIdCard(id) {
    const s = String(id);
    if (!idCardRe.test(s)) return false;
    let sum = 0;
    for (let i = 0; i < 17; i++) sum += parseInt(s[i], 10) * weights[i];
    return checkChars[sum % 11] === s[17].toUpperCase();
  }

  for (const fs of fieldStats) {
    if (!fs.detectedPatterns || !fs.detectedPatterns.includes("id_card_cn")) continue;
    const col = fs.name;
    const vals = rows.map(r => r[col]).filter(v => !U.isNullish(v)).map(String);
    if (!vals.length) continue;

    const idVals = vals.filter(v => idCardRe.test(v));
    if (idVals.length < 5) continue;

    let invalidCount = 0;
    const indices = [];
    idVals.forEach((v, i) => {
      if (!validateIdCard(v)) { invalidCount++; if (indices.length < 100) indices.push(i); }
    });

    if (invalidCount === 0) continue;
    const pct = invalidCount / idVals.length * 100;
    issues.push({
      id: `id_checksum_${col}`, moduleId: "id_checksum", category: "validity",
      severity: pct > 20 ? "high" : "medium",
      title: `[${col}] Invalid ID Card Checksums`,
      description: `${invalidCount} of ${idVals.length} ID card numbers fail checksum validation (${U.round(pct, 1)}%)`,
      affectedColumns: [col], affectedRows: invalidCount,
      affectedPercentage: U.round(pct, 1), affectedRowIndices: indices,
      suggestion: "Verify ID card numbers — these may be fabricated or mistyped",
      deduction: U.round(Math.min(pct * 0.1, 5), 2),
    });
  }
  return { moduleId: "id_checksum", moduleName: "ID Card Checksum Validation", issues };
}

// ===================== Accuracy (new) =====================

function scanValueUniformity(headers, rows, fieldStats) {
  const issues = [];
  const numFields = fieldStats.filter(f => ["integer", "float"].includes(f.inferredType));

  for (const fs of numFields) {
    const col = fs.name;
    const nums = rows.map(r => U.tryParseNumber(r[col])).filter(n => n !== null);
    if (nums.length < 20) continue;

    // Zero-variance: all values identical
    const uniq = new Set(nums);
    if (uniq.size === 1) {
      issues.push({
        id: `uniform_constant_${col}`, moduleId: "value_uniformity", category: "accuracy",
        severity: "medium",
        title: `[${col}] Constant Value — Zero Variance`,
        description: `All ${nums.length} values are identical (${[...uniq][0]}). This column carries no analytical information.`,
        affectedColumns: [col], affectedRows: nums.length,
        suggestion: "Consider excluding this column from analysis",
        deduction: 2,
      });
      continue;
    }

    // Dominant single value
    const freq = {};
    for (const n of nums) freq[n] = (freq[n] || 0) + 1;
    const sorted = Object.entries(freq).sort((a, b) => b[1] - a[1]);
    const [topVal, topCount] = sorted[0];
    const topPct = topCount / nums.length * 100;

    if (topPct > 80 && nums.length > 50) {
      issues.push({
        id: `uniform_dominant_${col}`, moduleId: "value_uniformity", category: "accuracy",
        severity: "low",
        title: `[${col}] Dominant Repeated Value`,
        description: `${U.round(topPct, 1)}% of values are ${topVal} (${topCount}/${nums.length}). Low variance may indicate data collection issues.`,
        affectedColumns: [col], affectedRows: topCount,
        affectedPercentage: U.round(topPct, 1),
        suggestion: "Verify if the data source is recording values correctly",
        deduction: U.round(Math.min((topPct - 80) * 0.05, 2), 2),
      });
    }
  }
  return { moduleId: "value_uniformity", moduleName: "Value Uniformity Detection", issues };
}

// ===================== Main Scanner =====================

function scan(headers, rows, fieldStats, parseInfo) {
  const modules = [
    scanNullValues(headers, rows, fieldStats),
    scanEmptyStrings(headers, rows),
    scanRowCompleteness(headers, rows),
    scanMergedCellPattern(headers, rows, parseInfo || {}),
    scanDuplicateRows(headers, rows),
    scanPrimaryKey(headers, rows, fieldStats),
    scanFuzzyDuplicates(headers, rows, fieldStats),
    scanDataTypeConsistency(headers, rows, fieldStats),
    scanCaseConsistency(headers, rows, fieldStats),
    scanFullwidthChars(headers, rows),
    scanDateFormat(headers, rows, fieldStats),
    scanNumberWithUnit(headers, rows, fieldStats),
    scanCrossColumnLogic(headers, rows, fieldStats),
    scanFormatValidity(headers, rows, fieldStats),
    scanSpecialChars(headers, rows),
    scanEncodingIssues(headers, rows, parseInfo || {}),
    scanIdChecksum(headers, rows, fieldStats),
    scanRangeValidity(headers, rows, fieldStats),
    scanOutliers(headers, rows, fieldStats),
    scanRareValues(headers, rows, fieldStats),
    scanValueUniformity(headers, rows, fieldStats),
    scanTimeFreshness(headers, rows, fieldStats),
  ];

  const allIssues = [];
  for (const m of modules) {
    for (const issue of m.issues) allIssues.push(issue);
  }

  return { modules, issues: allIssues };
}

module.exports = { scan };


path: tools/lib/scorer.js
content: "use strict";
const U = require("./utils");

const CATEGORY_WEIGHTS = {
  completeness: 0.25,
  accuracy:     0.23,
  consistency:  0.17,
  validity:     0.15,
  uniqueness:   0.15,
  timeliness:   0.05,
};

const CATEGORY_NAMES = {
  completeness: "Completeness",
  accuracy:     "Accuracy",
  consistency:  "Consistency",
  validity:     "Validity",
  uniqueness:   "Uniqueness",
  timeliness:   "Timeliness",
};

// Severity multiplier — makes critical/high issues hit harder in the score.
// Modules already encode severity into deduction values, but this adds a
// scorer-level amplification to ensure severity is consistently reflected.
const SEVERITY_MULT = { critical: 1.5, high: 1.2, medium: 1.0, low: 0.7, info: 0 };

// Category sensitivity scale — compensates for categories with fewer modules
// or more conservative per-module deduction caps. Higher scale = each raw
// deduction point costs more category-score points.
//
// Rationale:
//   completeness  2.5 — well-quantified by modules (nulls can produce large deductions)
//   accuracy      5.0 — few modules, small deductions; each finding is significant
//   consistency   2.0 — many modules, deductions accumulate naturally
//   validity      2.5 — moderate module count, balanced amplification
//   uniqueness    3.0 — few modules, each duplicate finding is impactful
//   timeliness   10.0 — single module, any date anomaly should be noticeable
const CATEGORY_SCALE = {
  completeness: 2.5,
  accuracy:     5.0,
  consistency:  2.0,
  validity:     2.5,
  uniqueness:   3.0,
  timeliness:  10.0,
};

const MAX_CATEGORY_DEDUCTION = 80;

function calculate(issues) {
  const categoryIssues = {};
  for (const cat of Object.keys(CATEGORY_WEIGHTS)) categoryIssues[cat] = [];
  for (const issue of issues) {
    const cat = issue.category;
    if (categoryIssues[cat]) categoryIssues[cat].push(issue);
  }

  const breakdowns = [];
  let weightedScore = 0;

  for (const [cat, weight] of Object.entries(CATEGORY_WEIGHTS)) {
    const catIssues = categoryIssues[cat] || [];

    // Sum deductions with severity multipliers
    let rawDeduction = 0;
    for (const issue of catIssues) {
      const ded = issue.deduction || 0;
      const mult = SEVERITY_MULT[issue.severity] || 1.0;
      rawDeduction += ded * mult;
    }

    // Apply category sensitivity scale and cap
    const scale = CATEGORY_SCALE[cat] || 1.0;
    const scaledDeduction = Math.min(U.round(rawDeduction * scale, 2), MAX_CATEGORY_DEDUCTION);
    const catScore = Math.max(0, U.round(100 - scaledDeduction, 2));

    breakdowns.push({
      category: cat,
      categoryName: CATEGORY_NAMES[cat],
      weight,
      rawDeduction: U.round(rawDeduction, 2),
      scaledDeduction,
      finalScore: catScore,
      issueCount: catIssues.length,
    });

    weightedScore += catScore * weight;
  }

  const totalScore = U.round(weightedScore, 1);
  const [grade, gradeColor] = getGrade(totalScore);

  const severityCounts = { critical: 0, high: 0, medium: 0, low: 0, info: 0 };
  for (const issue of issues) {
    const sev = issue.severity || "medium";
    severityCounts[sev] = (severityCounts[sev] || 0) + 1;
  }

  return {
    totalScore,
    grade,
    gradeColor,
    issueCount: issues.length,
    severityCounts,
    breakdowns,
    summary: generateSummary(totalScore, issues, breakdowns),
  };
}

function getGrade(score) {
  if (score >= 90) return ["Excellent", "green"];
  if (score >= 80) return ["Good", "blue"];
  if (score >= 70) return ["Fair", "yellow"];
  if (score >= 60) return ["Poor", "orange"];
  return ["Very Poor", "red"];
}

function generateSummary(totalScore, issues, breakdowns) {
  let summary;
  if (totalScore >= 90) summary = "Data quality is excellent, safe to use directly.";
  else if (totalScore >= 80) summary = "Data quality is good with minor issues worth noting.";
  else if (totalScore >= 70) summary = "Data quality is fair; recommend addressing issues before use.";
  else if (totalScore >= 60) summary = "Data quality is poor; multiple issues require attention.";
  else summary = "Data quality is very poor; strongly recommend comprehensive cleaning.";

  const worst = breakdowns.reduce((a, b) => a.finalScore < b.finalScore ? a : b);
  if (worst.finalScore < 80) {
    summary += ` Weakest dimension: ${worst.categoryName} (${worst.finalScore}).`;
  }

  const crit = issues.filter(i => i.severity === "critical").length;
  const high = issues.filter(i => i.severity === "high").length;
  if (crit > 0) summary += ` ${crit} critical issue(s) require immediate attention.`;
  if (high > 0) summary += ` ${high} high-priority issue(s) to address.`;

  return summary;
}

module.exports = { calculate };


path: tools/lib/utils.js
content: "use strict";

function formatBytes(size) {
  const units = ["B", "KB", "MB", "GB", "TB"];
  let i = 0;
  while (size >= 1024 && i < units.length - 1) { size /= 1024; i++; }
  return `${size.toFixed(1)} ${units[i]}`;
}

function round(val, decimals) {
  if (val == null || isNaN(val)) return null;
  const f = Math.pow(10, decimals);
  return Math.round(val * f) / f;
}

function isNullish(v) {
  return v === null || v === undefined || v === "" ||
    (typeof v === "number" && isNaN(v)) ||
    (typeof v === "string" && ["nan", "none", "null", "undefined"].includes(v.toLowerCase().trim()));
}

function median(arr) {
  if (!arr.length) return null;
  const s = arr.slice().sort((a, b) => a - b);
  const mid = Math.floor(s.length / 2);
  return s.length % 2 ? s[mid] : (s[mid - 1] + s[mid]) / 2;
}

function quantile(arr, q) {
  if (!arr.length) return null;
  const s = arr.slice().sort((a, b) => a - b);
  const pos = (s.length - 1) * q;
  const lo = Math.floor(pos);
  const hi = Math.ceil(pos);
  if (lo === hi) return s[lo];
  return s[lo] + (s[hi] - s[lo]) * (pos - lo);
}

function std(arr) {
  if (arr.length < 2) return 0;
  const m = arr.reduce((a, b) => a + b, 0) / arr.length;
  const variance = arr.reduce((sum, v) => sum + (v - m) * (v - m), 0) / (arr.length - 1);
  return Math.sqrt(variance);
}

function mode(arr) {
  if (!arr.length) return null;
  const freq = {};
  for (const v of arr) { freq[v] = (freq[v] || 0) + 1; }
  let best = null, bestCount = 0;
  for (const [k, c] of Object.entries(freq)) {
    if (c > bestCount) { best = k; bestCount = c; }
  }
  return best;
}

function countBy(arr) {
  const m = {};
  for (const v of arr) { m[v] = (m[v] || 0) + 1; }
  return m;
}

function topValues(arr, n) {
  const freq = countBy(arr);
  return Object.entries(freq)
    .sort((a, b) => b[1] - a[1])
    .slice(0, n)
    .map(([value, count]) => ({ value, count, percentage: round(count / arr.length * 100, 2) }));
}

function toJsonSafe(v) {
  if (v === null || v === undefined) return null;
  if (typeof v === "number" && isNaN(v)) return null;
  if (typeof v === "number" && !isFinite(v)) return null;
  if (v instanceof Date) return v.toISOString();
  return v;
}

const MONTH_ABBR = {
  jan: 0, feb: 1, mar: 2, apr: 3, may: 4, jun: 5,
  jul: 6, aug: 7, sep: 8, oct: 9, nov: 10, dec: 11,
  // French
  janv: 0, févr: 1, mars: 2, avr: 3, mai: 4, juin: 5,
  juil: 6, août: 7, sept: 8, oct: 9, nov: 10, déc: 11,
  // German
  jän: 0, mär: 2, mai: 4, okt: 9, dez: 11,
  // Spanish
  ene: 0, abr: 3, ago: 7, dic: 11,
};

function tryParseDate(v) {
  if (v == null || v === "") return null;
  if (v instanceof Date) return isNaN(v.getTime()) ? null : v;
  const s = String(v).trim();

  // ISO / YYYY-MM-DD / YYYY/MM/DD (with optional time)
  if (/^\d{4}[-/]\d{2}[-/]\d{2}/.test(s)) {
    const d = new Date(s);
    return isNaN(d.getTime()) ? null : d;
  }
  // YYYY.MM.DD (EU/CJK dot-separated)
  const dotYmd = s.match(/^(\d{4})\.(\d{1,2})\.(\d{1,2})$/);
  if (dotYmd) {
    const d = new Date(+dotYmd[1], +dotYmd[2] - 1, +dotYmd[3]);
    return isNaN(d.getTime()) ? null : d;
  }
  // DD.MM.YYYY (German, Swiss, Russian, Turkish, etc.)
  const dotDmy = s.match(/^(\d{1,2})\.(\d{1,2})\.(\d{4})$/);
  if (dotDmy) {
    const day = +dotDmy[1], m = +dotDmy[2], y = +dotDmy[3];
    if (m >= 1 && m <= 12 && day >= 1 && day <= 31) {
      const d = new Date(y, m - 1, day);
      return isNaN(d.getTime()) ? null : d;
    }
  }
  // YYYYMMDD compact
  if (/^\d{8}$/.test(s)) {
    const y = +s.slice(0, 4), m = +s.slice(4, 6), day = +s.slice(6, 8);
    if (m >= 1 && m <= 12 && day >= 1 && day <= 31) {
      const d = new Date(y, m - 1, day);
      return isNaN(d.getTime()) ? null : d;
    }
  }
  // DD/MM/YYYY (UK, Europe, Middle East, LatAm) — also catches MM/DD/YYYY (US)
  const slashMatch = s.match(/^(\d{1,2})\/(\d{1,2})\/(\d{4})$/);
  if (slashMatch) {
    const a = +slashMatch[1], b = +slashMatch[2], y = +slashMatch[3];
    // If first > 12, must be day (DD/MM/YYYY)
    if (a > 12 && b <= 12) return new Date(y, b - 1, a);
    // If second > 12, must be day (MM/DD/YYYY)
    if (b > 12 && a <= 12) return new Date(y, a - 1, b);
    // Ambiguous — default to MM/DD/YYYY for compatibility
    const d = new Date(y, a - 1, b);
    return isNaN(d.getTime()) ? null : d;
  }
  // DD-MM-YYYY with dashes (common in India, parts of EU)
  const dashDmy = s.match(/^(\d{1,2})-(\d{1,2})-(\d{4})$/);
  if (dashDmy) {
    const a = +dashDmy[1], b = +dashDmy[2], y = +dashDmy[3];
    if (a > 12 && b <= 12) return new Date(y, b - 1, a);
    if (b > 12 && a <= 12) return new Date(y, a - 1, b);
    const d = new Date(y, b - 1, a); // default DD-MM-YYYY
    return isNaN(d.getTime()) ? null : d;
  }
  // DD-MMM-YYYY / DD MMM YYYY (e.g., 15-Mar-2024, 15 Mar 2024)
  const dMyMatch = s.match(/^(\d{1,2})[-\s]([a-záéíóúàèìòùäöüâêîôûñç]+)[-\s](\d{4})$/i);
  if (dMyMatch) {
    const mon = MONTH_ABBR[dMyMatch[2].toLowerCase()];
    if (mon !== undefined) {
      const d = new Date(+dMyMatch[3], mon, +dMyMatch[1]);
      return isNaN(d.getTime()) ? null : d;
    }
  }
  // MMM DD, YYYY (e.g., Mar 15, 2024)
  const mdyStrMatch = s.match(/^([a-záéíóúàèìòùäöüâêîôûñç]+)\s+(\d{1,2}),?\s+(\d{4})$/i);
  if (mdyStrMatch) {
    const mon = MONTH_ABBR[mdyStrMatch[1].toLowerCase()];
    if (mon !== undefined) {
      const d = new Date(+mdyStrMatch[3], mon, +mdyStrMatch[2]);
      return isNaN(d.getTime()) ? null : d;
    }
  }
  // Chinese/Japanese: YYYY年MM月DD日
  const cjkFull = s.match(/^(\d{4})年(\d{1,2})月(\d{1,2})日$/);
  if (cjkFull) {
    const d = new Date(+cjkFull[1], +cjkFull[2] - 1, +cjkFull[3]);
    return isNaN(d.getTime()) ? null : d;
  }
  // Chinese/Japanese: YYYY年MM月 (month-only)
  const cjkMonth = s.match(/^(\d{4})年(\d{1,2})月$/);
  if (cjkMonth) {
    const d = new Date(+cjkMonth[1], +cjkMonth[2] - 1, 1);
    return isNaN(d.getTime()) ? null : d;
  }
  // Korean: YYYY년 MM월 DD일 or YYYY년MM월DD일
  const krMatch = s.match(/^(\d{4})년\s*(\d{1,2})월\s*(\d{1,2})일$/);
  if (krMatch) {
    const d = new Date(+krMatch[1], +krMatch[2] - 1, +krMatch[3]);
    return isNaN(d.getTime()) ? null : d;
  }
  // Korean month-only: YYYY년 MM월
  const krMonth = s.match(/^(\d{4})년\s*(\d{1,2})월$/);
  if (krMonth) {
    const d = new Date(+krMonth[1], +krMonth[2] - 1, 1);
    return isNaN(d.getTime()) ? null : d;
  }
  // Japanese era (Reiwa): 令和N年M月D日
  const jpEra = s.match(/^令和(\d{1,2})年(\d{1,2})月(\d{1,2})日$/);
  if (jpEra) {
    const d = new Date(2018 + (+jpEra[1]), +jpEra[2] - 1, +jpEra[3]);
    return isNaN(d.getTime()) ? null : d;
  }
  // Unix epoch: 10-digit (seconds) or 13-digit (milliseconds)
  if (/^\d{10}$/.test(s)) {
    const d = new Date(+s * 1000);
    if (d.getFullYear() >= 2000 && d.getFullYear() <= 2100) return d;
  }
  if (/^\d{13}$/.test(s)) {
    const d = new Date(+s);
    if (d.getFullYear() >= 2000 && d.getFullYear() <= 2100) return d;
  }
  return null;
}

function tryParseNumber(v) {
  if (v == null || v === "") return null;
  if (typeof v === "number") return isNaN(v) ? null : v;
  const s = String(v).trim().replace(/,/g, "");
  if (s === "") return null;
  const n = Number(s);
  return isNaN(n) ? null : n;
}

module.exports = {
  formatBytes, round, isNullish, median, quantile, std, mode,
  countBy, topValues, toJsonSafe, tryParseDate, tryParseNumber,
};


path: tools/quality_scanner.js
content: #!/usr/bin/env node
"use strict";
const path = require("path");
const { parseFile } = require("./lib/parser");
const { profile } = require("./lib/profiler");
const { scan } = require("./lib/scanner");
const { calculate } = require("./lib/scorer");

function main() {
  const filePath = process.argv[2];
  if (!filePath) {
    process.stderr.write("Usage: quality_scanner.js <file_path>\n");
    process.stderr.write("Returns JSON quality scan (modules, issues, score).\n");
    process.exit(1);
  }

  try {
    const resolved = path.resolve(filePath);
    const { headers, rows, parseInfo } = parseFile(resolved);
    const prof = profile(headers, rows);

    if (!prof.dataValidity.canProceed) {
      process.stdout.write(JSON.stringify({
        error: "data_not_suitable",
        message: prof.dataValidity.blockReason,
        dataValidity: prof.dataValidity,
      }, null, 2) + "\n");
      process.exit(0);
    }

    const { modules, issues } = scan(headers, rows, prof.fieldStats, parseInfo);
    const score = calculate(issues);

    process.stdout.write(JSON.stringify({
      fileName: path.basename(resolved),
      parseInfo,
      tableStats: prof.tableStats,
      fieldStats: prof.fieldStats,
      sampleData: prof.sampleData,
      dataValidity: prof.dataValidity,
      modules: modules.map(m => ({
        moduleId: m.moduleId,
        moduleName: m.moduleName,
        issueCount: m.issues.length,
      })),
      issues,
      score,
    }, null, 2) + "\n");
  } catch (err) {
    process.stdout.write(JSON.stringify({ error: err.message }) + "\n");
    process.exit(1);
  }
}

main();


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)