Skill: kaiyuelv/data-validator-pro (version 1.0.0)

--- name: data-quality-validator description: Data quality validation and profiling toolkit for tabular data. Use when checking data completeness, detecting anomalies, validating schemas, profiling datasets, or assessing data cleanliness. Triggers on phrases like "data quality", "data validation", "schema validation", "data profiling", "missing data", "anomaly detection", "data completeness", "dirty data". --- # Data Quality Validator Toolkit for validating and profiling tabular data quality. ## Features - **Schema validation** - Check column types, constraints, and rules - **Completeness analysis** - Missing value detection and reporting - **Anomaly detection** - Statistical outlier detection - **Profiling** - Summary statistics and distribution analysis - **Constraint checking** - Range checks, uniqueness, regex patterns ## Quick Start ```python from scripts.data_profiler import DataProfiler from scripts.schema_validator import SchemaValidator # Profile a dataset profiler = DataProfiler() report = profiler.profile(df) # pandas DataFrame print(report["missing"]) print(report["outliers"]) # Validate against schema schema = { "age": {"type": "int", "min": 0, "max": 150}, "email": {"type": "str", "regex": r"^\S+@\S+\.\S+$"}, "id": {"type": "int", "unique": True} } validator = SchemaValidator(schema) errors = validator.validate(df) for err in errors: print(err) ``` ## Scripts - `scripts/data_profiler.py` - Dataset profiling and summary stats - `scripts/schema_validator.py` - Schema-based validation engine - `scripts/anomaly_detector.py` - Statistical anomaly detection ## References - `references/validation_rules.md` - Common validation patterns

path: README.md
content: # Data Quality Validator

Data quality validation and profiling toolkit for tabular data.

## 功能 | Features

- 数据画像 | Data profiling (statistics, distributions)
- 模式验证 | Schema validation (types, ranges, regex)
- 缺失值分析 | Missing value analysis
- 异常检测 | Anomaly detection (IQR, Z-score)
- 约束检查 | Constraint checking (uniqueness, enums)

## 安装 | Installation

```bash
pip install -r requirements.txt
```

## 快速开始 | Quick Start

```python
from scripts.data_profiler import DataProfiler
from scripts.schema_validator import SchemaValidator

profiler = DataProfiler()
report = profiler.profile(df)

schema = {"age": {"type": "int", "min": 0, "max": 120}}
validator = SchemaValidator(schema)
errors = validator.validate(df)
```

## 目录结构 | Structure

```
data-quality-validator/
├── SKILL.md
├── README.md
├── requirements.txt
├── scripts/
│   ├── data_profiler.py
│   ├── schema_validator.py
│   └── anomaly_detector.py
├── examples/
│   └── basic_usage.py
└── tests/
    └── test_validator.py
```


path: examples/basic_usage.py
content: #!/usr/bin/env python3
"""
Basic usage examples for data-quality-validator.
"""
import sys
sys.path.insert(0, "../scripts")

import pandas as pd
from data_profiler import DataProfiler
from schema_validator import SchemaValidator
from anomaly_detector import AnomalyDetector

def demo_profiler():
    df = pd.DataFrame({
        "age": [25, 30, None, 200, 45],
        "name": ["Alice", "Bob", "Charlie", "Dave", "Eve"],
        "salary": [50000, 60000, 55000, None, 70000],
    })
    profiler = DataProfiler()
    report = profiler.profile(df)
    print("Profile report:", report)

def demo_validator():
    df = pd.DataFrame({
        "age": [25, 30, 200, -5],
        "email": ["[REDACTED_SECRET]", "bad", "[REDACTED_SECRET]", "[REDACTED_SECRET]"],
    })
    schema = {
        "age": {"type": "int", "min": 0, "max": 150},
        "email": {"type": "str", "regex": r"^\S+@\S+\.\S+$"},
    }
    validator = SchemaValidator(schema)
    errors = validator.validate(df)
    print("Validation errors:", errors)

def demo_anomaly():
    df = pd.DataFrame({"value": [1, 2, 3, 4, 5, 100]})
    detector = AnomalyDetector(method="iqr")
    report = detector.report(df, columns=["value"])
    print("Anomaly report:", report)

if __name__ == "__main__":
    demo_profiler()
    demo_validator()
    demo_anomaly()


path: references/validation_rules.md
content: # Validation Rules Reference

## Type Checks

- `int` - Integer dtype
- `float` - Float dtype
- `str` - String dtype
- `bool` - Boolean dtype
- `datetime` - Datetime64 dtype

## Constraints

- `min` / `max` - Numeric range (inclusive)
- `regex` - String pattern match
- `unique` - No duplicate values
- `enum` - Value must be in allowed list
- `nullable` - Whether nulls are allowed (default True)

## Example Schema

```python
schema = {
    "user_id": {"type": "int", "unique": True, "nullable": False},
    "age": {"type": "int", "min": 0, "max": 150},
    "email": {"type": "str", "regex": r"^\S+@\S+\.\S+$"},
    "country": {"type": "str", "enum": ["US", "UK", "CN", "JP"]},
    "score": {"type": "float", "min": 0.0, "max": 100.0},
}
```

## Anomaly Detection Methods

- **IQR** - Interquartile range (1.5 * IQR rule)
- **Z-Score** - Standard deviations from mean
- **MAD** - Median absolute deviation (robust to outliers)


path: requirements.txt
content: pandas>=2.0.0
numpy>=1.24.0


path: scripts/anomaly_detector.py
content: """
Statistical anomaly detection methods.
"""
from typing import Dict, Any, List, Optional
import pandas as pd
import numpy as np

class AnomalyDetector:
    def __init__(self, method: str = "iqr", threshold: Optional[float] = None):
        self.method = method
        self.threshold = threshold

    def detect(self, series: pd.Series) -> pd.Series:
        """Return boolean mask of anomalies."""
        if self.method == "iqr":
            return self._iqr(series)
        elif self.method == "zscore":
            return self._zscore(series, self.threshold or 3.0)
        elif self.method == "mad":
            return self._mad(series, self.threshold or 3.5)
        else:
            raise ValueError(f"Unknown method: {self.method}")

    def _iqr(self, series: pd.Series) -> pd.Series:
        q1 = series.quantile(0.25)
        q3 = series.quantile(0.75)
        iqr = q3 - q1
        lower = q1 - 1.5 * iqr
        upper = q3 + 1.5 * iqr
        return (series < lower) | (series > upper)

    def _zscore(self, series: pd.Series, threshold: float = 3.0) -> pd.Series:
        mean = series.mean()
        std = series.std()
        if std == 0:
            return pd.Series([False] * len(series), index=series.index)
        z = (series - mean).abs() / std
        return z > threshold

    def _mad(self, series: pd.Series, threshold: float = 3.5) -> pd.Series:
        median = series.median()
        mad = (series - median).abs().median()
        if mad == 0:
            return pd.Series([False] * len(series), index=series.index)
        modified_z = 0.6745 * (series - median).abs() / mad
        return modified_z > threshold

    def report(self, df: pd.DataFrame, columns: Optional[List[str]] = None) -> Dict[str, Any]:
        target = columns or df.select_dtypes(include=[np.number]).columns.tolist()
        report = {}
        for col in target:
            mask = self.detect(df[col])
            report[col] = {
                "anomaly_count": mask.sum(),
                "anomaly_pct": round(mask.mean() * 100, 2),
                "threshold_method": self.method,
            }
        return report


path: scripts/data_profiler.py
content: """
Data profiler for summary statistics and quality metrics.
"""
from typing import Dict, Any, Optional
import pandas as pd
import numpy as np

class DataProfiler:
    def profile(self, df: pd.DataFrame) -> Dict[str, Any]:
        report = {
            "row_count": len(df),
            "column_count": len(df.columns),
            "columns": {},
            "missing": {},
            "outliers": {},
            "duplicates": df.duplicated().sum(),
        }
        for col in df.columns:
            series = df[col]
            col_report = {
                "dtype": str(series.dtype),
                "non_null": series.notna().sum(),
                "null_count": series.isna().sum(),
                "null_pct": round(series.isna().mean() * 100, 2),
            }
            if pd.api.types.is_numeric_dtype(series):
                col_report.update({
                    "min": series.min(),
                    "max": series.max(),
                    "mean": series.mean(),
                    "median": series.median(),
                    "std": series.std(),
                })
                # IQR outliers
                q1 = series.quantile(0.25)
                q3 = series.quantile(0.75)
                iqr = q3 - q1
                lower = q1 - 1.5 * iqr
                upper = q3 + 1.5 * iqr
                outliers = series[(series < lower) | (series > upper)]
                col_report["outlier_count"] = len(outliers)
                report["outliers"][col] = {
                    "count": len(outliers),
                    "lower_bound": lower,
                    "upper_bound": upper,
                }
            else:
                col_report["unique_count"] = series.nunique()
                col_report["top_value"] = series.mode().iloc[0] if not series.mode().empty else None
            report["columns"][col] = col_report
            report["missing"][col] = col_report["null_count"]
        return report

    def compare(self, df1: pd.DataFrame, df2: pd.DataFrame) -> Dict[str, Any]:
        """Compare two datasets for schema drift."""
        drift = {
            "columns_added": list(set(df2.columns) - set(df1.columns)),
            "columns_removed": list(set(df1.columns) - set(df2.columns)),
            "type_changes": {},
        }
        for col in set(df1.columns) & set(df2.columns):
            if df1[col].dtype != df2[col].dtype:
                drift["type_changes"][col] = {"before": str(df1[col].dtype), "after": str(df2[col].dtype)}
        return drift


path: scripts/schema_validator.py
content: """
Schema validator for constraint checking.
"""
from typing import Dict, Any, List, Optional
import pandas as pd
import numpy as np
import re

class SchemaValidator:
    def __init__(self, schema: Dict[str, Dict[str, Any]]):
        self.schema = schema

    def validate(self, df: pd.DataFrame) -> List[Dict[str, Any]]:
        errors = []
        for col, rules in self.schema.items():
            if col not in df.columns:
                errors.append({"column": col, "error": "missing_column", "message": f"Column '{col}' not found"})
                continue
            series = df[col]
            # Type check
            expected_type = rules.get("type")
            if expected_type:
                valid = self._check_type(series, expected_type)
                if not valid["ok"]:
                    errors.append({"column": col, "error": "type_mismatch", "message": valid["message"]})
            # Range check
            if "min" in rules:
                invalid = series[series < rules["min"]]
                if not invalid.empty:
                    errors.append({"column": col, "error": "min_violation", "count": len(invalid), "min": rules["min"]})
            if "max" in rules:
                invalid = series[series > rules["max"]]
                if not invalid.empty:
                    errors.append({"column": col, "error": "max_violation", "count": len(invalid), "max": rules["max"]})
            # Regex
            if "regex" in rules and (series.dtype == object or pd.api.types.is_string_dtype(series)):
                pattern = re.compile(rules["regex"])
                invalid = series[~series.astype(str).apply(lambda x: bool(pattern.match(x)) if pd.notna(x) else True)]
                if not invalid.empty:
                    errors.append({"column": col, "error": "regex_mismatch", "count": len(invalid), "regex": rules["regex"]})
            # Uniqueness
            if rules.get("unique"):
                dups = series[series.duplicated(keep=False)]
                if not dups.empty:
                    errors.append({"column": col, "error": "duplicate_values", "count": len(dups)})
            # Enum
            if "enum" in rules:
                invalid = series[~series.isin(rules["enum"])]
                if not invalid.empty:
                    errors.append({"column": col, "error": "enum_violation", "count": len(invalid), "allowed": rules["enum"]})
        return errors

    def _check_type(self, series: pd.Series, expected: str) -> Dict[str, Any]:
        mapping = {
            "int": pd.api.types.is_integer_dtype,
            "float": pd.api.types.is_float_dtype,
            "str": pd.api.types.is_string_dtype,
            "bool": pd.api.types.is_bool_dtype,
            "datetime": pd.api.types.is_datetime64_any_dtype,
        }
        checker = mapping.get(expected)
        if checker is None:
            return {"ok": True, "message": f"Unknown type '{expected}'"}
        ok = checker(series)
        return {"ok": ok, "message": f"Expected {expected}, got {series.dtype}"}


path: tests/test_validator.py
content: """
Unit tests for data-quality-validator.
"""
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "../scripts"))

import unittest
import pandas as pd
import numpy as np
from data_profiler import DataProfiler
from schema_validator import SchemaValidator
from anomaly_detector import AnomalyDetector

class TestDataProfiler(unittest.TestCase):
    def test_profile_basic(self):
        df = pd.DataFrame({"a": [1, 2, 3], "b": ["x", "y", "z"]})
        p = DataProfiler()
        r = p.profile(df)
        self.assertEqual(r["row_count"], 3)
        self.assertEqual(r["column_count"], 2)
        self.assertEqual(r["missing"]["a"], 0)

    def test_profile_outliers(self):
        df = pd.DataFrame({"a": [1, 2, 3, 100]})
        p = DataProfiler()
        r = p.profile(df)
        self.assertEqual(r["outliers"]["a"]["count"], 1)

class TestSchemaValidator(unittest.TestCase):
    def test_type_check(self):
        df = pd.DataFrame({"age": [1, 2, 3]})
        v = SchemaValidator({"age": {"type": "int"}})
        self.assertEqual(len(v.validate(df)), 0)

    def test_range_violation(self):
        df = pd.DataFrame({"age": [200]})
        v = SchemaValidator({"age": {"type": "int", "max": 150}})
        errs = v.validate(df)
        self.assertTrue(any(e["error"] == "max_violation" for e in errs))

    def test_regex_mismatch(self):
        df = pd.DataFrame({"email": ["bad"]})
        v = SchemaValidator({"email": {"type": "str", "regex": r"^\S+@\S+\.\S+$"}})
        errs = v.validate(df)
        self.assertTrue(any(e["error"] == "regex_mismatch" for e in errs))

    def test_unique_violation(self):
        df = pd.DataFrame({"id": [1, 1, 2]})
        v = SchemaValidator({"id": {"type": "int", "unique": True}})
        errs = v.validate(df)
        self.assertTrue(any(e["error"] == "duplicate_values" for e in errs))

class TestAnomalyDetector(unittest.TestCase):
    def test_iqr(self):
        s = pd.Series([1, 2, 3, 4, 5, 100])
        d = AnomalyDetector("iqr")
        mask = d.detect(s)
        self.assertTrue(mask.iloc[-1])

    def test_zscore(self):
        s = pd.Series([1, 2, 3, 4, 5, 500])
        d = AnomalyDetector("zscore", threshold=2.0)
        mask = d.detect(s)
        self.assertTrue(mask.iloc[-1])

    def test_report(self):
        df = pd.DataFrame({"a": [1, 2, 3, 100]})
        d = AnomalyDetector("iqr")
        r = d.report(df, columns=["a"])
        self.assertEqual(r["a"]["anomaly_count"], 1)

if __name__ == "__main__":
    unittest.main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)