Skill: mtoby8326/openclaw-security-mtoby8326 (version 1.0.0)

--- name: openclaw-security description: "Multi-region async PII detection for OpenClaw sessions. Scans user input, prompts, context, and knowledge base content for sensitive personal data across CN, US, AU, UK, DE, FR, SG, MY, TH, ID regions. Detects phone numbers, emails, names, addresses, passports, bank cards, national IDs, social accounts. Use when: (1) user asks to audit or scan for PII / sensitive data, (2) 'security scan', (3) 'check for personal information', (4) 'PII detection', (5) background audit on session content, (6) 'sensitive data check', (7) 'privacy audit'." --- # OpenClaw Security - PII Audit Skill Multi-region async PII detection engine for OpenClaw sessions. Detects 8 categories of sensitive personal data across 10 country/region jurisdictions and logs audit events locally as NDJSON. ## 中文速览（PII 审计） ### 基本信息 - 技能名称：`openclaw-security` - 能力：多区域异步 PII 检测，支持后台审计与本地合规留痕 ### 检测范围 - 8 类标签：`PHONE` / `EMAIL` / `PERSON_NAME` / `ADDRESS` / `PASSPORT` / `BANK_CARD` / `NATIONAL_ID` / `SOCIAL_ACCOUNT` - 10 区域：CN / US / AU / SG / MY / TH / ID / DE / UK / FR（支持 `+CC` 国际手机号） - 来源类型：`input` / `prompt` / `context` / `knowledge_base` ### 关键规则 - 风险分级：`high`（证件/银行卡或组合信息），`low`（单一弱标识） - 智能采样：`input` 100%（5m），`prompt` 20%（24h），`context` 20%（1h），`knowledge_base` 100%（24h） - 调用方无需自行判断是否跳过扫描；如需强制扫描，使用 `--no-cache` - 后台扫描禁止 `--text`，请使用 `--file` + `--delete-after-read` - 输入上限 32,768 字符，超限截断并记录 `truncated: true` - 审计结果本地 NDJSON 落盘，默认保留 7 天，可 `cleanup.py --dry-run` 先演练 ## Quick Start Scan via file (recommended for background / automated scans): ```powershell python scripts/audit_worker.py --session-id SESSION_001 --source-type input --file content.txt ``` Scan via file + auto-delete (secure temp-file workflow): ```powershell python scripts/audit_worker.py --session-id SESSION_001 --source-type input --file tmp_scan.txt --delete-after-read ``` Scan via stdin: ```powershell echo "张三的手机号是13812345678" | python scripts/audit_worker.py --session-id SESSION_001 --source-type input ``` Quick manual test (WARNING: content visible in process list): ```powershell python scripts/audit_worker.py --session-id S001 --source-type input --text "short test" --json ``` ## Source Types - `input` — User input text - `prompt` — System or user prompts - `context` — Conversation context - `knowledge_base` — Knowledge base content ## Detection Labels PHONE, EMAIL, PERSON_NAME, ADDRESS, PASSPORT, BANK_CARD, NATIONAL_ID, SOCIAL_ACCOUNT ## Supported Regions CN, US, AU, SG, MY, TH, ID, DE, UK, FR (+ INTL via +CC phone prefix) ## Risk Levels - **high**: NATIONAL_ID / PASSPORT / BANK_CARD detected, or combination of PERSON_NAME + contact info + ADDRESS - **low**: Single weak identifier (EMAIL, SOCIAL_ACCOUNT, PHONE alone) ## Smart Sampling The audit worker includes built-in smart sampling to efficiently handle large contexts: - **User input** (`input`): 100% scan rate, 5-min cache TTL — every user message is scanned, but identical repeats within 5 minutes are skipped. - **System prompts** (`prompt`): 20% scan rate, 24-hour cache TTL — prompts rarely change; first scan is cached for 24 hours. - **Conversation context** (`context`): 20% scan rate, 1-hour cache TTL — context overlaps heavily; only sample 1 in 5 submissions. - **Knowledge base** (`knowledge_base`): 100% first-scan rate, 24-hour cache TTL — static content is fully scanned once, then deduped for 24 hours. Bypass sampling for manual / forced scans: ```powershell python scripts/audit_worker.py --session-id S001 --source-type context --text "text" --no-cache ``` ## Async Audit Workflow When auditing session content as a background task: 1. **Respond to user first** — never block the main response for audit. 2. **Feed all content types** — the script internally decides whether to actually scan based on sampling config and cache. The Agent does not need to decide when to skip. 3. **Use temp-file + `--delete-after-read`** — NEVER pass content via `--text` in background scans. Write content to a temp file, pass `--file`, and let the script auto-delete it. 4. Run audit in background: ```powershell # Step 1: Write content to temp file (no PII in command-line args) $tmpFile = [System.IO.Path]::GetTempFileName() [System.IO.File]::WriteAllText($tmpFile, $userInput, [System.Text.Encoding]::UTF8) # Step 2: Background scan — script reads and deletes the temp file Start-Process -NoNewWindow -FilePath python -ArgumentList "scripts/audit_worker.py --session-id $sid --source-type input --file $tmpFile --delete-after-read" # Same pattern for other source types: $tmpPrompt = [System.IO.Path]::GetTempFileName() [System.IO.File]::WriteAllText($tmpPrompt, $systemPrompt, [System.Text.Encoding]::UTF8) Start-Process -NoNewWindow -FilePath python -ArgumentList "scripts/audit_worker.py --session-id $sid --source-type prompt --file $tmpPrompt --delete-after-read" ``` 5. Review results: `openclaw-security-audit/YYYY-MM-DD/events.ndjson` 6. All outcomes (detected, clean, skipped) are logged for complete audit trail. ## Retention Default: 7 days. Cleanup: ```powershell python scripts/cleanup.py --days 7 ``` Dry run first: ```powershell python scripts/cleanup.py --days 7 --dry-run ``` ## Input Size Limit Maximum input: **32,768 characters** (32K). Content exceeding this limit is truncated to the first 32K characters. The audit record carries `truncated: true` and original `input_chars` count. ## Audit Record Schema Every scan invocation writes an NDJSON record — including `clean` and `skipped` outcomes. Each NDJSON line contains: - `event_id` — UUID - `session_id` — Caller-provided session ID (required) - `source_type` — One of: input, prompt, context, knowledge_base - `status` — `detected`, `clean`, or `skipped` - `labels` — Array of detected PII types (detected only) - `regions` — Array of matched regions/country codes (detected only) - `risk_level` — high or low (detected only) - `matched_count` — Number of PII matches - `matches` — Array of {label, confidence, masked_preview, region} (detected only) - `content_hash` — SHA256 prefix for dedup (no raw content stored) - `input_chars` — Original input size in characters - `truncated` — Whether input was truncated to 32K - `created_at` — ISO 8601 UTC timestamp ## Safety Rules - NEVER store raw sensitive values — only masked previews + content hash - NEVER pass content via `--text` in background scans — use `--file` + `--delete-after-read` - Audit logs are local-only, never transmitted externally - All file I/O uses UTF-8 encoding explicitly, with file locking for concurrent safety - No external dependencies — stdlib only - Input capped at 32K characters to prevent resource exhaustion ## Configuration Environment variable override for audit output directory: ```powershell $env:OPENCLAW_AUDIT_DIR = "C:\path\to\custom\audit\dir" ``` See `references/patterns.md` for detection pattern details.

path: README.md
content: # OpenClaw Security Skill 🔒

A multi-region async PII (Personally Identifiable Information) detection engine that runs as an OpenClaw Skill. Scans session content for sensitive data across **10 country/region jurisdictions** and logs audit events locally in NDJSON format.
## 🛡️ 中文速览：PII 审计 Skill 信息

### 基本信息

**技能名称**: `openclaw-security`  
**描述**: 多区域异步 PII（敏感个人信息）检测引擎

### 核心功能

检测 **8 类敏感个人信息**，覆盖 **10 个国家/地区**：

- `PHONE`（手机号）
- `EMAIL`（电子邮箱）
- `PERSON_NAME`（真实姓名）
- `ADDRESS`（物理地址）
- `PASSPORT`（护照号码）
- `BANK_CARD`（银行卡号）
- `NATIONAL_ID`（身份证/国民ID）
- `SOCIAL_ACCOUNT`（社交账号）

支持区域：CN / US / AU / SG / MY / TH / ID / DE / UK / FR（并支持 `+CC` 国际手机号前缀）

### 检测来源类型

- `input` — 用户输入文本
- `prompt` — 系统或用户提示词
- `context` — 对话上下文
- `knowledge_base` — 知识库内容

### 风险等级

- **high**：检测到 `NATIONAL_ID` / `PASSPORT` / `BANK_CARD`，或组合信息（姓名 + 联系方式 + 地址）
- **low**：单一弱标识符（单独邮箱 / 社交账号 / 手机号）

### 智能采样策略

| 来源类型 | 扫描率 | 缓存 TTL | 说明 |
|---|---:|---|---|
| `input` | 100% | 5分钟 | 每条消息都扫描，重复内容跳过 |
| `prompt` | 20% | 24小时 | 提示词变化少 |
| `context` | 20% | 1小时 | 上下文重叠度高 |
| `knowledge_base` | 100% | 24小时 | 首次全扫，后续去重 |

采样与缓存决策由脚本内部处理，调用方只需送入内容。需要强制扫描时使用 `--no-cache`。

### 使用方式

```powershell
# 1) 文件扫描（推荐）
python scripts/audit_worker.py --session-id SESSION_001 --source-type input --file content.txt

# 2) 文件 + 自动删除（安全工作流）
python scripts/audit_worker.py --session-id SESSION_001 --source-type input --file tmp_scan.txt --delete-after-read

# 3) stdin
echo "张三的手机号是13812345678" | python scripts/audit_worker.py --session-id SESSION_001 --source-type input

# 4) 强制扫描（绕过缓存/采样）
python scripts/audit_worker.py --session-id SESSION_001 --source-type context --file context.txt --no-cache
```

> `--text` 仅建议手工快速测试；后台审计不要使用 `--text`，请使用 `--file` + `--delete-after-read`。

### 异步审计工作流（后台）

```powershell
# 步骤1：写入临时文件（命令行参数中不包含 PII）
$tmpFile = [System.IO.Path]::GetTempFileName()
[System.IO.File]::WriteAllText($tmpFile, $userInput, [System.Text.Encoding]::UTF8)

# 步骤2：后台扫描，脚本读取后自动删除临时文件
Start-Process -NoNewWindow -FilePath python -ArgumentList "scripts/audit_worker.py --session-id $sid --source-type input --file $tmpFile --delete-after-read"
```

### 审计记录格式（NDJSON）

每次扫描都会写一条记录（包括 `detected` / `clean` / `skipped`）：

- `event_id` — UUID
- `session_id` — 会话 ID
- `source_type` — 来源类型
- `status` — `detected` / `clean` / `skipped`
- `labels` — 检测到的 PII 类型数组
- `regions` — 命中区域数组
- `risk_level` — 风险等级
- `matched_count` — 匹配数量
- `matches` — 匹配详情（`label` / `confidence` / `masked_preview` / `region`）
- `content_hash` — 内容哈希（用于去重）
- `input_chars` — 输入字符数
- `truncated` — 是否被截断
- `created_at` — 时间戳

### 配置项

- 输入限制：最大 32,768 字符（超过则截断）
- 日志保留：默认 7 天
- 输出目录：`openclaw-security-audit/YYYY-MM-DD/events.ndjson`
- 缓存文件：`.scan-cache.json`

```powershell
# 环境变量覆盖输出目录
$env:OPENCLAW_AUDIT_DIR = "C:\path\to\custom\audit\dir"

# 清理旧日志
python scripts/cleanup.py --days 7
python scripts/cleanup.py --days 7 --dry-run
```

### 安全规则

- 永不存储原始敏感值（仅存脱敏预览 + 内容哈希）
- 后台扫描不使用 `--text`，必须使用 `--file` + `--delete-after-read`
- 审计日志仅本地保存，不外传
- UTF-8 编码 + 文件锁，保障并发安全
- 无外部依赖（仅 Python 标准库）

### 相关文件

- `scripts/audit_worker.py` — 主审计脚本
- `scripts/cleanup.py` — 清理脚本
- `references/patterns.md` — 检测模式详解

## ✨ Features

- **8 PII Categories** — Phone, Email, National ID, Passport, Bank Card, Person Name, Address, Social Account
- **10 Regions** — CN, US, AU, SG, MY, TH, ID, DE, UK, FR (+ any country via +CC phone prefix)
- **Zero Dependencies** — Pure Python stdlib, works out of the box
- **Async & Non-blocking** — Audit decoupled from main workflow
- **Smart False Positive Control** — Checksums (CN ID, AU TFN, TH ID, FR NIR), Luhn (bank card), keyword gating (SSN, passport, name, address)
- **Region Classification** — Each match tagged with ISO country code
- **Overlap Dedup** — Same text range: highest confidence wins
- **Risk Scoring** — Two-level (high/low) with single-label and combo rules
- **Smart Sampling** — Per-source-type sampling rates + content-hash cache dedup
- **Complete Audit Trail** — All outcomes (detected, clean, skipped) logged for compliance
- **Concurrent-Safe** — File locking on NDJSON + cache writes for parallel background scans
- **32K Input Cap** — Truncates oversized content; records original size + truncation flag
- **Secure Input Channel** — `--file` + `--delete-after-read` for background scans (no PII in process args)
- **Local NDJSON Storage** — Partitioned by date, grep/SIEM-friendly
- **Auto Cleanup** — Configurable retention (default: 7 days) + scan cache pruning

## 🌍 Supported Regions

| Region | National ID | Phone (local) | Address | Name |
|--------|------------|---------------|---------|------|
| CN | ID Card (18-digit, checksum) | 1[3-9]X mobile, landline | Province/City structural | Keyword-gated |
| US | SSN (keyword-gated) | (XXX) XXX-XXXX | State + ZIP | Mr./Mrs. + First Last |
| AU | TFN (9-digit, checksum) | 04XX mobile | State + postcode | Keyword-gated |
| SG | NRIC/FIN (keyword-gated) | +65 | — | — |
| MY | MyKad (12-digit, date check) | +60 | — | — |
| TH | National ID (13-digit, checksum) | +66 | — | — |
| ID | NIK/KTP (16-digit, date check) | +62 | — | — |
| DE | Steuer-ID (keyword-gated) | +49 | Street + PLZ | Keyword-gated |
| UK | NIN (keyword-gated) | 07XXX mobile | Postcode | — |
| FR | NIR/INSEE (mod-97 check) | +33 | Rue + code postal | Keyword-gated |

## 🚀 Quick Start

```bash
git clone https://github.com/mtoby8326/openclaw-security-skill.git
cd openclaw-security-skill
```

### Scan Text

```bash
# Scan from file (recommended)
python scripts/audit_worker.py --session-id S001 --source-type knowledge_base \
  --file path/to/content.txt

# Scan from file + auto-delete (secure temp-file workflow)
python scripts/audit_worker.py --session-id S001 --source-type input \
  --file tmp_scan.txt --delete-after-read

# Quick manual test (WARNING: content visible in process list)
python scripts/audit_worker.py --session-id S001 --source-type input \
  --text "Name: Zhang San, Phone: 13812345678" --json
```

### Sample Output

```json
{
  "status": "detected",
  "risk_level": "high",
  "labels": ["NATIONAL_ID", "PERSON_NAME", "PHONE"],
  "regions": ["CN", "INTL", "UK", "US"],
  "matched_count": 8,
  "audit_file": "openclaw-security-audit/2026-03-06/events.ndjson"
}
```

## 🏷️ Detection Labels

| Label | Description | Confidence | Validation |
|-------|-------------|------------|------------|
| `PHONE` | Mobile, landline, international | 0.85-0.92 | Format + country code mapping |
| `EMAIL` | Email address | 0.95 | RFC format match |
| `NATIONAL_ID` | CN/US/AU/SG/MY/TH/ID/DE/UK/FR IDs | 0.85-0.98 | Checksum / keyword-gated |
| `PASSPORT` | Passport number (multi-region) | 0.85 | Keyword-gated |
| `BANK_CARD` | Bank card (13-19 digits) | 0.92 | Luhn algorithm |
| `PERSON_NAME` | CN/Western/DE/FR names | 0.70-0.75 | Keyword / title gated |
| `ADDRESS` | CN/US/AU/UK/DE/FR addresses | 0.75-0.80 | Structural + keyword |
| `SOCIAL_ACCOUNT` | WeChat/QQ/Twitter etc. | 0.80 | Keyword-gated |

## 🎯 Smart Sampling

The audit worker includes built-in smart sampling to avoid redundant scans on large or repetitive context:

| Source Type | Scan Rate | Cache TTL | Rationale |
|---|---|---|---|
| `input` | 100% | 5 min | Every user message scanned; identical repeats within 5 min skipped |
| `prompt` | 20% | 24 hours | System prompts rarely change; scan once, cache long |
| `context` | 20% | 1 hour | Conversation context overlaps heavily; sample 1 in 5 |
| `knowledge_base` | 100% | 24 hours | Static content fully scanned once, then deduped |

**How it works**: The script computes a SHA256 content hash, checks a file-backed cache (`.scan-cache.json`), and applies the sampling rate. The caller (Agent) does not need to decide when to skip — just feed all content through, and the script handles the rest.

```bash
# Force scan (bypass cache + sampling)
python scripts/audit_worker.py --session-id S001 --source-type context --text "..." --no-cache
```

## ⚠️ Risk Level Rules

**HIGH**: `NATIONAL_ID`, `PASSPORT`, or `BANK_CARD` detected, or combo of `PERSON_NAME` + contact + `ADDRESS`

**LOW**: Single weak identifier (email, phone, or social account alone)

## 📋 Audit Record Schema

Every scan invocation writes an NDJSON record — including `clean` and `skipped` outcomes.

```json
{
  "event_id": "uuid",
  "session_id": "caller-provided session ID (required)",
  "source_type": "input | prompt | context | knowledge_base",
  "status": "detected | clean | skipped",
  "labels": ["PHONE", "NATIONAL_ID"],
  "regions": ["CN", "US"],
  "risk_level": "high",
  "matched_count": 3,
  "matches": [
    {"label": "PHONE", "confidence": 0.90, "masked_preview": "13*******00", "region": "CN"},
    {"label": "NATIONAL_ID", "confidence": 0.90, "masked_preview": "***-**-**20", "region": "US"}
  ],
  "content_hash": "sha256[:16]",
  "input_chars": 256,
  "truncated": false,
  "created_at": "ISO 8601 UTC"
}
```

> **Security Principle**: Raw sensitive values are never stored — only minimally masked previews and content hashes.

## 📁 Project Structure

```
openclaw-security/
├── SKILL.md                      # OpenClaw Skill definition
├── README.md
├── .gitignore
├── scripts/
│   ├── audit_worker.py           # Main entry: detect → risk score → NDJSON sink
│   ├── cleanup.py                # Audit log + cache cleanup (UTC-aware)
│   ├── file_lock.py              # Cross-platform file lock (O_CREAT|O_EXCL)
│   └── detectors/                # PII detector modules (multi-region)
│       ├── __init__.py           # Detector registry
│       ├── base.py               # Base class + Match dataclass (with region)
│       ├── phone.py              # Phone (CN/US/AU/UK local + INTL +CC)
│       ├── email_detector.py     # Email (universal)
│       ├── national_id.py        # National IDs (10 countries)
│       ├── passport.py           # Passport (multi-language keywords)
│       ├── bank_card.py          # Bank card (Luhn, universal)
│       ├── person_name.py        # Names (CN/Western/DE/FR)
│       ├── address.py            # Addresses (CN/US/AU/UK/DE/FR)
│       └── social_account.py     # Social accounts
├── references/
│   └── patterns.md               # Detection pattern reference
└── openclaw-security-audit/      # Audit log output (excluded by .gitignore)
    └── YYYY-MM-DD/
        └── events.ndjson
```

## 🧹 Log Cleanup

```bash
python scripts/cleanup.py              # default 7-day retention
python scripts/cleanup.py --days 30    # custom retention
python scripts/cleanup.py --dry-run    # preview only
```

## ⚙️ Configuration

```bash
# Override audit output directory
export OPENCLAW_AUDIT_DIR="/path/to/custom/audit/dir"          # Linux/macOS
$env:OPENCLAW_AUDIT_DIR = "C:\path\to\custom\audit\dir"        # PowerShell
```

## 🛡️ Security Design

- **Minimal Masking** — Only 2-3 characters exposed per sensitive value; raw values never stored
- **Secure Input Channel** — `--file` + `--delete-after-read` prevents PII exposure in process args
- **Concurrent-Safe** — FileLock on all shared files (NDJSON, cache) prevents corruption
- **32K Input Cap** — Truncation prevents ReDoS and memory exhaustion attacks
- **Local Only** — Audit logs never transmitted externally
- **Keyword Gating** — Weak signals require context keywords to fire
- **Algorithm Validation** — CN ID / AU TFN / TH ID / FR NIR checksums; Bank Card Luhn; US SSN range validation
- **Overlap Dedup** — Highest confidence result kept per character range
- **Complete Audit Trail** — All outcomes (detected, clean, skipped) logged for compliance proof

## 🗺️ Roadmap

- [x] Smart sampling with content-hash dedup (v0.2.0)
- [x] Security hardening: file locking, 32K cap, secure input channel, audit-all, tighter masking (v0.3.0)
- [ ] Batch scan mode (`--batch`)
- [ ] `tool_output` source type
- [ ] NER model enhancement for name/address
- [ ] HTML audit report generation
- [ ] PIPL / GDPR / CCPA compliance label mapping
- [ ] Scheduled audit (cron / Task Scheduler)
- [ ] More SEA regions (VN, PH)

## 📄 License

Apache 2.0

## 🤝 Contributing

Issues and PRs welcome! To add a new detector:
1. Create a module in `scripts/detectors/`, extending `BaseDetector`
2. Implement `detect(text)` returning `Match` objects with `region` set
3. Register in `__init__.py`

---

**Made with ❤️ for the OpenClaw community**


path: references/patterns.md
content: # Detection Patterns Reference

## Label: PHONE
- **INTL**: `+CC` prefix → region resolved from country code map (+86=CN, +1=US, +61=AU, +65=SG, +60=MY, +66=TH, +62=ID, +49=DE, +44=UK, +33=FR)
- **CN mobile**: `1[3-9]X XXXX XXXX` (11 digits)
- **CN landline**: `0XX(X)-XXXXXXX(X)` (requires separator)
- **US**: `(XXX) XXX-XXXX` or `XXX-XXX-XXXX` (requires parentheses or dashes)
- **AU mobile**: `04XX XXX XXX`
- **UK mobile**: `07XXX XXXXXX`
- **Confidence**: 0.85–0.92

## Label: EMAIL
- **Pattern**: `[REDACTED_SECRET]` (RFC-like, ASCII)
- **Confidence**: 0.95
- **Region**: Not assigned (universal)

## Label: NATIONAL_ID
### CN — Chinese ID Card
- 18 digits: 6 region + 8 birthday + 3 seq + 1 check digit
- **Validation**: ISO 7064 Mod 11-2 checksum
- **Confidence**: 0.98 | **Gate**: Checksum (no keyword required)

### US — Social Security Number (SSN)
- 9 digits: `XXX-XX-XXXX`
- **Validation**: Area ≠ 000/666/9xx, group ≠ 00, serial ≠ 0000
- **Confidence**: 0.90 | **Gate**: Keyword (SSN, Social Security)

### AU — Tax File Number (TFN)
- 9 digits: `XXX XXX XXX`
- **Validation**: Weighted checksum (1,4,3,7,5,8,6,9,10), sum mod 11 = 0
- **Confidence**: 0.92 | **Gate**: Keyword (TFN, Tax File Number)

### SG — NRIC/FIN
- Format: `[STFGM]` + 7 digits + check letter
- **Confidence**: 0.88 | **Gate**: Keyword (NRIC, FIN, IC)

### MY — MyKad
- 12 digits: `YYMMDD-SS-GGGG`
- **Validation**: Date validity on first 6 digits
- **Confidence**: 0.88 | **Gate**: Keyword (MyKad, IC, NRIC)

### TH — Thai National ID
- 13 digits: `X-XXXX-XXXXX-XX-X`
- **Validation**: Mod 11 check digit
- **Confidence**: 0.90 | **Gate**: Keyword (Thai ID, national ID)

### ID — NIK/KTP
- 16 digits: 6 region + 6 date (DD+40 for female) + 4 seq
- **Validation**: Date validity
- **Confidence**: 0.85 | **Gate**: Keyword (NIK, KTP)

### DE — Steuer-ID
- 11 digits, first digit ≠ 0
- **Confidence**: 0.85 | **Gate**: Keyword (Steuer-ID, IdNr, Tax ID)

### UK — National Insurance Number (NIN)
- Format: 2 letters + 6 digits + 1 letter (e.g. AB123456C)
- Letter restrictions: specific character sets for each position
- **Confidence**: 0.90 | **Gate**: Keyword (NIN, National Insurance)

### FR — NIR/INSEE
- 15 digits: sex + year + month + dept + commune + seq + check
- **Validation**: `check = 97 - (first_13 mod 97)`
- **Confidence**: 0.92 | **Gate**: Keyword (NIR, INSEE, social security)

## Label: PASSPORT
- **CN**: `[EGDSPH]` + optional letter + 7-8 digits
- **DE**: `C` + 8 alphanumeric
- **FR**: 2 digits + 2 letters + 5 digits
- **Generic**: 1-2 uppercase letters + 6-9 digits
- **Confidence**: 0.85 | **Gate**: Keyword (passport, visa, immigration — multi-language)

## Label: BANK_CARD
- **Formatted**: `XXXX XXXX XXXX XXXX` (space/dash groups of 4)
- **Continuous**: 13-19 consecutive digits
- **Validation**: Luhn algorithm (universal)
- **Confidence**: 0.92 | **Region**: Not assigned (universal)

## Label: PERSON_NAME
- **CN**: Keyword-gated (2-4 Chinese characters after context keywords or before honorific titles)
- **Western**: Keyword-gated (`name:`, `Mr./Mrs./Dr.` + First Last)
- **DE**: Keyword-gated (Name/Vorname/Nachname + name)
- **FR**: Keyword-gated (nom/prénom + name)
- **Confidence**: 0.70–0.75

## Label: ADDRESS
- **CN**: Structural (province + city + district) or keyword-gated
- **US**: Number + street + state abbreviation + ZIP (5 or 5+4)
- **AU**: Number + street + state abbreviation + 4-digit postcode
- **UK**: Postcode pattern (e.g. SW1A 1AA), keyword-boosted
- **DE**: Street name (straße/str./weg) + house number + PLZ
- **FR**: Number + street type (rue/avenue/boulevard) + code postal
- **Confidence**: 0.75–0.80

## Label: SOCIAL_ACCOUNT
- **WeChat**: keyword + letter-start ID (6-20 chars)
- **QQ**: keyword + 5-12 digit number
- **Twitter/X**: keyword + @handle
- **Generic**: keyword + account value
- **Confidence**: 0.80 | **Region**: Not assigned

## Risk Level Rules
- **HIGH**: NATIONAL_ID / PASSPORT / BANK_CARD present, or combo (PERSON_NAME + PHONE/EMAIL + ADDRESS)
- **LOW**: All other detections

## False Positive Control
1. Regex → checksum/format validation → context keyword confirmation
2. National IDs: CN/AU/TH/FR use algorithmic checksums; US/SG/MY/DE/UK use keyword gating
3. Passport: Always keyword-gated (multi-language)
4. Bank card: Luhn validation (universal)
5. Person name / address: Keyword or structural gating
6. Overlapping matches: Higher confidence wins


path: scripts/audit_worker.py
content: #!/usr/bin/env python3
"""OpenClaw Security Audit Worker — PII detection and local NDJSON logging.

Usage:
    echo "text" | python audit_worker.py --session-id S001 --source-type input
    python audit_worker.py --session-id S001 --source-type input --text "short text"
    python audit_worker.py --session-id S001 --source-type knowledge_base --file path.txt
    python audit_worker.py --session-id S001 --source-type input --file tmp.txt --delete-after-read
    python audit_worker.py --session-id S001 --source-type input --text "text" --json
    python audit_worker.py --session-id S001 --source-type input --text "text" --no-cache

Security note:
    --text passes content via command-line args, which are visible in the
    process list.  For background / automated scans, prefer --file with
    --delete-after-read to avoid exposing PII in process metadata.
"""

import argparse
import hashlib
import json
import os
import random
import stat
import sys
import time
import uuid
from datetime import datetime, timezone
from pathlib import Path

# Ensure detectors package is importable
SCRIPT_DIR = Path(__file__).resolve().parent
sys.path.insert(0, str(SCRIPT_DIR))

from detectors import ALL_DETECTORS
from detectors.base import Match
from file_lock import FileLock

# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
VERSION = "0.3.0"

DEFAULT_AUDIT_DIR = os.environ.get(
    'OPENCLAW_AUDIT_DIR',
    str(SCRIPT_DIR.parent / 'openclaw-security-audit')
)

HIGH_RISK_LABELS = {'NATIONAL_ID', 'PASSPORT', 'BANK_CARD'}

# Maximum input size in characters.  Content exceeding this limit is truncated
# to the first MAX_INPUT_CHARS characters; the audit record will carry
# ``truncated: true`` and the original ``input_chars`` count.
MAX_INPUT_CHARS = 32768

# Smart sampling configuration per source_type.
# rate:      probability [0.0, 1.0] of scanning content that is NOT in cache.
# cache_ttl: seconds before a cached content_hash expires and becomes eligible
#            for re-scanning.  Set to 0 to disable caching for a source type.
SAMPLE_CONFIG = {
    'input':          {'rate': 1.00, 'cache_ttl': 300},     # 100%, 5-min cache
    'prompt':         {'rate': 0.20, 'cache_ttl': 86400},   # 20%,  24-hour cache
    'context':        {'rate': 0.20, 'cache_ttl': 3600},    # 20%,  1-hour cache
    'knowledge_base': {'rate': 1.00, 'cache_ttl': 86400},   # 100%, 24-hour cache (dedup)
}

CACHE_MAX_ENTRIES = 5000   # Hard cap; prune to 3000 when exceeded
CACHE_FILE_NAME = '.scan-cache.json'


# ---------------------------------------------------------------------------
# Scan Cache — file-backed dedup + sampling gate
# ---------------------------------------------------------------------------
class ScanCache:
    """Lightweight file-backed cache tracking recently scanned content hashes.

    The cache is stored as a JSON object ``{content_hash: epoch_seconds}`` in
    the audit directory.  All file I/O is protected by a FileLock to support
    concurrent background scans.
    """

    def __init__(self, audit_dir: str):
        self.path = Path(audit_dir) / CACHE_FILE_NAME
        self._lock_path = str(self.path) + '.lock'
        self.data: dict = self._load()

    # -- persistence --------------------------------------------------------
    def _load(self) -> dict:
        if self.path.exists():
            try:
                with FileLock(self._lock_path):
                    return json.loads(self.path.read_text(encoding='utf-8'))
            except (json.JSONDecodeError, OSError):
                return {}
        return {}

    def save(self) -> None:
        self.path.parent.mkdir(parents=True, exist_ok=True)
        with FileLock(self._lock_path):
            self.path.write_text(
                json.dumps(self.data, ensure_ascii=False),
                encoding='utf-8',
            )

    # -- query / update -----------------------------------------------------
    def is_fresh(self, content_hash: str, ttl: int) -> bool:
        """Return True if *content_hash* was scanned within the last *ttl* seconds."""
        ts = self.data.get(content_hash)
        if ts is None:
            return False
        return (time.time() - ts) < ttl

    def record(self, content_hash: str) -> None:
        """Mark *content_hash* as just-scanned and prune if over capacity."""
        self.data[content_hash] = time.time()
        if len(self.data) > CACHE_MAX_ENTRIES:
            # Keep the most recent 3000 entries
            ranked = sorted(self.data.items(), key=lambda kv: kv[1], reverse=True)
            self.data = dict(ranked[:3000])

    # -- decision -----------------------------------------------------------
    def should_scan(self, content_hash: str, source_type: str) -> bool:
        """Decide whether to scan based on cache freshness + sampling rate.

        Returns ``False`` (skip) when either:
        1. The content was scanned within the TTL window, OR
        2. The random draw exceeds the sampling rate for this source_type.
        """
        cfg = SAMPLE_CONFIG.get(source_type, SAMPLE_CONFIG['input'])
        if cfg['cache_ttl'] > 0 and self.is_fresh(content_hash, cfg['cache_ttl']):
            return False
        return random.random() < cfg['rate']


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def dedupe_overlapping(matches):
    """Remove lower-confidence matches whose character ranges overlap with
    higher-confidence matches.  Higher confidence wins."""
    sorted_matches = sorted(matches, key=lambda m: -m.confidence)
    result = []
    used_ranges = []
    for m in sorted_matches:
        overlaps = False
        for s, e in used_ranges:
            if m.start < e and m.end > s:
                overlaps = True
                break
        if not overlaps:
            result.append(m)
            used_ranges.append((m.start, m.end))
    return result


def compute_risk(labels):
    """Determine risk level based on detected label set.

    HIGH: any single high-risk label, or the combo PERSON_NAME + contact + ADDRESS.
    LOW:  everything else.
    """
    label_set = set(labels)
    if label_set & HIGH_RISK_LABELS:
        return 'high'
    has_name = 'PERSON_NAME' in label_set
    has_contact = bool(label_set & {'PHONE', 'EMAIL'})
    has_address = 'ADDRESS' in label_set
    if has_name and has_contact and has_address:
        return 'high'
    return 'low'


def _write_audit_record(record: dict, audit_dir: str) -> str:
    """Persist an audit record to the day-partitioned NDJSON file.

    File writes are protected by a FileLock to support concurrent processes.
    Returns the path of the output file.
    """
    today = datetime.now(timezone.utc).strftime('%Y-%m-%d')
    day_dir = Path(audit_dir) / today
    day_dir.mkdir(parents=True, exist_ok=True)

    # Best-effort restrictive permissions (owner-only) — limited on Windows
    try:
        day_dir.chmod(stat.S_IRWXU)
    except OSError:
        pass

    out_file = day_dir / 'events.ndjson'
    line = json.dumps(record, ensure_ascii=False) + '\n'

    with FileLock(str(out_file) + '.lock'):
        with open(out_file, 'a', encoding='utf-8') as f:
            f.write(line)

    return str(out_file)


# ---------------------------------------------------------------------------
# Core
# ---------------------------------------------------------------------------
def scan(text, session_id, source_type, audit_dir, *,
         use_cache=True, input_chars=0, truncated=False):
    """Run all detectors, dedupe, compute risk, write NDJSON record.

    When *use_cache* is True (default), the scan cache and per-source-type
    sampling rates are respected — identical content is not re-scanned within
    the configured TTL, and low-priority sources may be randomly skipped.

    Every invocation writes an audit record (detected, clean, OR skipped)
    to ensure a complete audit trail.

    Returns a summary dict (never raises on detection failure).
    """
    content_hash = hashlib.sha256(text.encode('utf-8')).hexdigest()[:16]
    now = datetime.now(timezone.utc).isoformat()
    base_fields = {
        "event_id": str(uuid.uuid4()),
        "session_id": session_id,
        "source_type": source_type,
        "detector_version": VERSION,
        "content_hash": content_hash,
        "input_chars": input_chars,
        "truncated": truncated,
        "created_at": now,
    }

    def _safe_write(record):
        """Write audit record; log warning instead of crashing on I/O errors."""
        try:
            return _write_audit_record(record, audit_dir)
        except Exception as exc:
            print(f'[WARN] Failed to write audit record: {exc}', file=sys.stderr)
            return None

    # --- Smart sampling gate -----------------------------------------------
    cache = None
    if use_cache:
        try:
            cache = ScanCache(audit_dir)
        except Exception as exc:
            print(f'[WARN] Cache load failed, scanning without cache: {exc}',
                  file=sys.stderr)
            cache = None
        if cache is not None and not cache.should_scan(content_hash, source_type):
            skip_record = {
                **base_fields,
                "status": "skipped",
                "reason": "cached_or_sampled_out",
                "matched_count": 0,
            }
            _safe_write(skip_record)
            return {
                "status": "skipped",
                "reason": "cached_or_sampled_out",
                "content_hash": content_hash,
                "matched_count": 0,
            }

    # --- Detection ---------------------------------------------------------
    all_matches = []
    for detector in ALL_DETECTORS:
        try:
            hits = detector.detect(text)
            all_matches.extend(hits)
        except Exception as exc:
            print(f'[WARN] Detector {detector.label} error: {exc}', file=sys.stderr)

    all_matches = dedupe_overlapping(all_matches)

    # Update cache regardless of detection result
    if cache is not None:
        try:
            cache.record(content_hash)
            cache.save()
        except Exception as exc:
            print(f'[WARN] Cache save failed: {exc}', file=sys.stderr)

    # --- No PII found → clean ---------------------------------------------
    if not all_matches:
        clean_record = {
            **base_fields,
            "status": "clean",
            "matched_count": 0,
        }
        _safe_write(clean_record)
        return {"status": "clean", "matched_count": 0}

    # --- PII detected → full record ----------------------------------------
    labels = sorted(set(m.label for m in all_matches))
    risk = compute_risk(labels)
    regions = sorted(set(m.region for m in all_matches if m.region))

    record = {
        **base_fields,
        "status": "detected",
        "labels": labels,
        "regions": regions,
        "risk_level": risk,
        "matched_count": len(all_matches),
        "matches": [
            {
                "label": m.label,
                "confidence": m.confidence,
                "masked_preview": m.masked_preview,
                "region": m.region,
            }
            for m in all_matches
        ],
    }

    audit_file = _safe_write(record)

    result = {
        "status": "detected",
        "risk_level": risk,
        "labels": labels,
        "regions": regions,
        "matched_count": len(all_matches),
    }
    if audit_file:
        result["audit_file"] = audit_file
    return result


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main():
    parser = argparse.ArgumentParser(
        description='OpenClaw PII Audit Worker — detect and log sensitive data'
    )
    parser.add_argument('--session-id', default='unknown',
                        help='Session identifier (default: unknown, warns if not set)')
    parser.add_argument('--source-type',
                        choices=['input', 'prompt', 'context', 'knowledge_base'],
                        default='input',
                        help='Content source type (default: input)')
    parser.add_argument('--file',
                        help='Read content from file instead of stdin')
    parser.add_argument('--text',
                        help='Inline text to scan (WARNING: visible in process list)')
    parser.add_argument('--delete-after-read', action='store_true',
                        help='Delete the --file after reading (for temp-file workflow)')
    parser.add_argument('--audit-dir', default=DEFAULT_AUDIT_DIR,
                        help='Audit output directory')
    parser.add_argument('--json', action='store_true',
                        help='Output result as JSON')
    parser.add_argument('--no-cache', action='store_true',
                        help='Bypass scan cache and sampling (force full scan)')
    args = parser.parse_args()

    # --- Validate flags ---
    if args.session_id == 'unknown':
        print('[WARN] --session-id not set, using "unknown". '
              'Provide a session ID for proper audit correlation.',
              file=sys.stderr)

    if args.text and args.file:
        print('[WARN] Both --text and --file provided; --file is ignored.',
              file=sys.stderr)

    if args.delete_after_read and not args.file:
        print('[WARN] --delete-after-read has no effect without --file.',
              file=sys.stderr)

    # --- Read input (with size cap applied during read) ---
    read_limit = MAX_INPUT_CHARS + 1  # read one extra char to detect truncation

    if args.text:
        text = args.text
    elif args.file:
        try:
            with open(args.file, 'r', encoding='utf-8') as f:
                text = f.read(read_limit)
        except OSError as exc:
            print(f'[ERROR] Cannot read file {args.file}: {exc}', file=sys.stderr)
            sys.exit(1)
        if args.delete_after_read:
            try:
                os.remove(args.file)
            except OSError as exc:
                print(f'[WARN] Could not delete temp file {args.file}: {exc}',
                      file=sys.stderr)
    else:
        text = sys.stdin.read(read_limit)

    if not text.strip():
        print('No input provided.', file=sys.stderr)
        sys.exit(1)

    # --- Size cap ---
    original_chars = len(text)
    truncated = False
    if original_chars > MAX_INPUT_CHARS:
        text = text[:MAX_INPUT_CHARS]
        truncated = True
        # original_chars is approximate when reading from file (could be larger)
        print(f'[INFO] Input truncated to {MAX_INPUT_CHARS} chars',
              file=sys.stderr)

    # --- Scan ---
    result = scan(
        text, args.session_id, args.source_type, args.audit_dir,
        use_cache=not args.no_cache,
        input_chars=original_chars,
        truncated=truncated,
    )

    # --- Output ---
    if args.json:
        print(json.dumps(result, ensure_ascii=False, indent=2))
    else:
        if result['status'] == 'skipped':
            print(f'[SKIP] Content already scanned or sampled out '
                  f'(hash={result["content_hash"]})')
        elif result['status'] == 'clean':
            print('[CLEAN] No PII detected.')
        else:
            print(f'[{result["risk_level"].upper()}] '
                  f'Detected {result["matched_count"]} PII match(es)')
            print(f'  Labels:  {", ".join(result["labels"])}')
            print(f'  Regions: {", ".join(result["regions"])}')
            print(f'  Audit:   {result["audit_file"]}')


if __name__ == '__main__':
    main()


path: scripts/cleanup.py
content: #!/usr/bin/env python3
"""Cleanup audit logs older than retention period.

Usage:
    python cleanup.py                  # default 7-day retention
    python cleanup.py --days 30        # 30-day retention
    python cleanup.py --dry-run        # preview only
"""

import argparse
import json
import re
import shutil
import time
from datetime import datetime, timedelta, timezone
from pathlib import Path

DEFAULT_AUDIT_DIR = str(Path(__file__).resolve().parent.parent / 'openclaw-security-audit')
DATE_PATTERN = re.compile(r'^\d{4}-\d{2}-\d{2}$')


def prune_scan_cache(audit_path, max_age_seconds, dry_run=False):
    """Remove expired entries from .scan-cache.json.

    Uses FileLock for consistency with audit_worker.py's ScanCache.
    """
    import sys
    sys.path.insert(0, str(Path(__file__).resolve().parent))
    from file_lock import FileLock

    cache_file = audit_path / '.scan-cache.json'
    if not cache_file.exists():
        return

    lock_path = str(cache_file) + '.lock'
    try:
        with FileLock(lock_path):
            data = json.loads(cache_file.read_text(encoding='utf-8'))
    except (json.JSONDecodeError, OSError):
        return

    now = time.time()
    before = len(data)
    data = {k: v for k, v in data.items() if (now - v) < max_age_seconds}
    pruned = before - len(data)

    if pruned == 0:
        print(f'Scan cache: {before} entries, 0 expired.')
        return

    if dry_run:
        print(f'[DRY-RUN] Scan cache: would prune {pruned}/{before} expired entries.')
    else:
        try:
            with FileLock(lock_path):
                cache_file.write_text(
                    json.dumps(data, ensure_ascii=False), encoding='utf-8')
        except OSError as exc:
            print(f'[WARN] Could not write scan cache: {exc}')
            return
        print(f'Scan cache: pruned {pruned}/{before} expired entries.')


def cleanup(audit_dir, days, dry_run=False):
    audit_path = Path(audit_dir)
    if not audit_path.exists():
        print(f'Audit directory not found: {audit_dir}')
        return

    cutoff = datetime.now(timezone.utc) - timedelta(days=days)
    removed = 0

    for entry in sorted(audit_path.iterdir()):
        if not entry.is_dir() or not DATE_PATTERN.match(entry.name):
            continue
        try:
            dir_date = datetime.strptime(entry.name, '%Y-%m-%d').replace(
                tzinfo=timezone.utc)
        except ValueError:
            continue

        if dir_date < cutoff:
            if dry_run:
                print(f'[DRY-RUN] Would remove: {entry}')
            else:
                shutil.rmtree(entry)
                print(f'[REMOVED] {entry}')
            removed += 1

    unit = 'directory' if removed == 1 else 'directories'
    prefix = 'would be ' if dry_run else ''
    print(f'Total: {removed} {unit} {prefix}removed.')

    # Also prune stale scan cache entries
    prune_scan_cache(audit_path, days * 86400, dry_run)


def main():
    parser = argparse.ArgumentParser(description='Cleanup old audit logs')
    parser.add_argument('--days', type=int, default=7,
                        help='Retention period in days (default: 7)')
    parser.add_argument('--audit-dir', default=DEFAULT_AUDIT_DIR,
                        help='Audit directory path')
    parser.add_argument('--dry-run', action='store_true',
                        help='Show what would be removed without deleting')
    args = parser.parse_args()

    cleanup(args.audit_dir, args.days, args.dry_run)


if __name__ == '__main__':
    main()


path: scripts/detectors/address.py
content: """Address detector — multi-region support.

CN: Structural province/city/district patterns + keyword-gated.
US: State abbreviation + ZIP code anchor.
AU: State abbreviation + 4-digit postcode.
UK: Distinctive postcode format (e.g. SW1A 1AA).
DE/FR: Keyword-gated with postal code patterns.
SEA: Keyword-gated only.
"""

import re
from .base import BaseDetector, Match

# US state abbreviations
_US_STATES = (
    'AL|AK|AZ|AR|CA|CO|CT|DE|FL|GA|HI|ID|IL|IN|IA|KS|KY|LA|ME|MD|MA|MI'
    '|MN|MS|MO|MT|NE|NV|NH|NJ|NM|NY|NC|ND|OH|OK|OR|PA|RI|SC|SD|TN|TX'
    '|UT|VT|VA|WA|WV|WI|WY|DC'
)

# AU state abbreviations
_AU_STATES = 'NSW|VIC|QLD|WA|SA|TAS|ACT|NT'


class AddressDetector(BaseDetector):
    label = "ADDRESS"

    # ---- Multi-language address keyword ----
    ADDR_KW = re.compile(
        r'address|地址|住址|收货地址|寄送地址|通讯地址'
        r'|Adresse|adresse|alamat|ที่อยู่|住所',
        re.IGNORECASE
    )

    # ---- CN: keyword-gated ----
    CN_KEYWORD = re.compile(
        r'(?:地址|住址|收货地址|寄送地址|通讯地址|address)'
        r'[：:\s]*'
        r'([\u4e00-\u9fff\w\d\-\u2014#\u53f7\u697c\u680b\u5355\u5143'
        r'\u5ba4\u5c42\u5e62\u5f04\u5df7\u8def\u8857\u9053\u533a\u53bf'
        r'\u5e02\u7701]{8,60})',
        re.IGNORECASE
    )
    # CN: Province + City structural
    CN_PROVINCIAL = re.compile(
        r'([\u4e00-\u9fff]{2,6}(?:\u7701|\u81ea\u6cbb\u533a)'
        r'[\u4e00-\u9fff]{2,10}(?:\u5e02|\u5dde|\u76df)'
        r'[\u4e00-\u9fff\w\d\-\u2014#\u53f7\u697c\u680b\u5355\u5143'
        r'\u5ba4\u5c42\u5e62\u5f04\u5df7\u8def\u8857\u9053\u533a\u53bf]{4,40})'
    )
    # CN: Direct-controlled municipalities
    CN_MUNICIPAL = re.compile(
        r'((?:\u5317\u4eac|\u4e0a\u6d77|\u5929\u6d25|\u91cd\u5e86)\u5e02?'
        r'[\u4e00-\u9fff]{2,6}(?:\u533a|\u53bf)'
        r'[\u4e00-\u9fff\w\d\-\u2014#\u53f7\u697c\u680b\u5355\u5143'
        r'\u5ba4\u5c42\u5e62\u5f04\u5df7\u8def\u8857\u9053]{4,30})'
    )

    # ---- US: street + state abbreviation + ZIP ----
    US_ADDR = re.compile(
        r'(\d{1,6}\s[\w\s.]{3,40},?\s*(?:' + _US_STATES + r')\s+\d{5}(?:\-\d{4})?)',
        re.IGNORECASE
    )

    # ---- AU: street + state + 4-digit postcode ----
    AU_ADDR = re.compile(
        r'(\d{1,5}\s[\w\s.]{3,40},?\s*(?:' + _AU_STATES + r')\s+\d{4})',
        re.IGNORECASE
    )

    # ---- UK: postcode pattern (very distinctive) ----
    UK_POSTCODE = re.compile(
        r'\b([A-Z]{1,2}\d[A-Z\d]?\s?\d[A-Z]{2})\b'
    )

    # ---- DE: street + PLZ (5-digit postal code) ----
    DE_ADDR = re.compile(
        r'([\w\s\-\.]+(?:stra[sß]e|str\.|weg|platz|gasse|allee)'
        r'\s*\d{1,5}[\w\s,]*\d{5}\s+[\w\s\-]+)',
        re.IGNORECASE
    )

    # ---- FR: number + street type + code postal ----
    FR_ADDR = re.compile(
        r'(\d{1,5}[\s,]+(?:rue|avenue|boulevard|place|chemin|'
        r'all[eé]e|impasse)[\w\s\-\u00e0-\u00ff]+\d{5}\s+[\w\s\-\u00e0-\u00ff]+)',
        re.IGNORECASE
    )

    def _mask_addr(self, addr):
        """Mask address keeping first 4 and last 2 characters."""
        if len(addr) > 6:
            return addr[:4] + '****' + addr[-2:]
        return '****'

    def detect(self, text):
        matches = []
        seen = set()

        def _add(addr, start, end, region, confidence):
            if addr in seen or len(addr) < 8:
                return
            seen.add(addr)
            matches.append(Match(
                label=self.label, confidence=confidence,
                masked_preview=self._mask_addr(addr),
                start=start, end=end, region=region,
            ))

        # CN patterns (no global keyword required — structural is distinctive)
        for m in self.CN_KEYWORD.finditer(text):
            _add(m.group(1), m.start(1), m.end(1), 'CN', 0.78)
        for m in self.CN_PROVINCIAL.finditer(text):
            _add(m.group(1), m.start(1), m.end(1), 'CN', 0.75)
        for m in self.CN_MUNICIPAL.finditer(text):
            _add(m.group(1), m.start(1), m.end(1), 'CN', 0.75)

        # US
        for m in self.US_ADDR.finditer(text):
            _add(m.group(1).strip(), m.start(1), m.end(1), 'US', 0.80)

        # AU
        for m in self.AU_ADDR.finditer(text):
            _add(m.group(1).strip(), m.start(1), m.end(1), 'AU', 0.78)

        # UK postcode (requires address keyword nearby for full confidence)
        has_addr_kw = bool(self.ADDR_KW.search(text))
        for m in self.UK_POSTCODE.finditer(text):
            conf = 0.80 if has_addr_kw else 0.60
            if conf < 0.65:
                continue  # Skip low confidence without keyword
            _add(m.group(1), m.start(1), m.end(1), 'UK', conf)

        # DE (keyword-gated)
        if self.ADDR_KW.search(text):
            for m in self.DE_ADDR.finditer(text):
                _add(m.group(1).strip(), m.start(1), m.end(1), 'DE', 0.75)
            for m in self.FR_ADDR.finditer(text):
                _add(m.group(1).strip(), m.start(1), m.end(1), 'FR', 0.75)

        return matches


path: scripts/detectors/bank_card.py
content: """Bank card number detector with Luhn algorithm validation."""

import re
from .base import BaseDetector, Match


class BankCardDetector(BaseDetector):
    label = "BANK_CARD"

    # Formatted: groups of 4 digits separated by space or dash
    FORMATTED = re.compile(r'(?<!\d)(\d{4}[\s\-]\d{4}[\s\-]\d{4}[\s\-]?\d{1,7})(?!\d)')
    # Continuous: 13-19 digit sequence
    CONTINUOUS = re.compile(r'(?<!\d)(\d{13,19})(?!\d)')

    @staticmethod
    def _luhn_check(number):
        """Validate card number using Luhn algorithm."""
        digits = [int(d) for d in number]
        odd_digits = digits[-1::-2]
        even_digits = digits[-2::-2]
        total = sum(odd_digits)
        for d in even_digits:
            d *= 2
            total += d - 9 if d > 9 else d
        return total % 10 == 0

    def detect(self, text):
        matches = []
        seen = set()

        for pattern in [self.FORMATTED, self.CONTINUOUS]:
            for m in pattern.finditer(text):
                raw = m.group()
                clean = re.sub(r'[\s\-]', '', raw)
                if not clean.isdigit():
                    continue
                if len(clean) < 13 or len(clean) > 19:
                    continue
                if clean in seen:
                    continue
                seen.add(clean)

                if self._luhn_check(clean):
                    matches.append(Match(
                        label=self.label,
                        confidence=0.92,
                        masked_preview=clean[:2] + ' **** **** **** ' + clean[-2:],
                        start=m.start(),
                        end=m.end(),
                    ))
        return matches


path: scripts/detectors/base.py
content: """Base detector interface and Match data structure."""

from dataclasses import dataclass


@dataclass
class Match:
    """A single PII match result."""
    label: str
    confidence: float
    masked_preview: str
    start: int
    end: int
    region: str = ""  # ISO country code: CN, US, AU, SG, MY, TH, ID, DE, UK, FR


class BaseDetector:
    """Base class for all PII detectors."""
    label = ""

    def detect(self, text):
        """Scan text and return list of Match objects.

        Args:
            text: Input string to scan.

        Returns:
            list[Match]: Detected PII matches.
        """
        raise NotImplementedError

    def _mask(self, value, keep_start=3, keep_end=4, mask_char="*"):
        """Mask a string, keeping only start/end characters visible.

        Args:
            value: Original string to mask.
            keep_start: Number of characters to keep at the start.
            keep_end: Number of characters to keep at the end.
            mask_char: Character used for masking.

        Returns:
            Masked string.
        """
        if len(value) <= keep_start + keep_end:
            return mask_char * len(value)
        mid_len = len(value) - keep_start - keep_end
        return value[:keep_start] + mask_char * mid_len + value[-keep_end:] if keep_end else value[:keep_start] + mask_char * mid_len


path: scripts/detectors/email_detector.py
content: """Email address detector."""

import re
from .base import BaseDetector, Match


class EmailDetector(BaseDetector):
    label = "EMAIL"

    PATTERN = re.compile(
        r'[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}',
        re.ASCII
    )

    def detect(self, text):
        matches = []
        for m in self.PATTERN.finditer(text):
            raw = m.group()
            local, domain = raw.rsplit('@', 1)
            masked = self._mask(local, 2, 0) + '@' + domain
            matches.append(Match(
                label=self.label,
                confidence=0.95,
                masked_preview=masked,
                start=m.start(),
                end=m.end(),
            ))
        return matches


path: scripts/detectors/national_id.py
content: """National ID detector — multi-region support.

Supported:
  CN  - Chinese ID card (18-digit, ISO 7064 checksum)
  US  - Social Security Number (SSN, context-keyword gated)
  AU  - Tax File Number (TFN, 9-digit with check algorithm)
  SG  - NRIC/FIN (letter + 7 digits + check letter)
  MY  - MyKad (12 digits, date + state + seq)
  TH  - Thai National ID (13 digits with check digit)
  ID  - NIK/KTP (16 digits, region + date + seq)
  DE  - Steuer-ID (11 digits, context-keyword gated)
  UK  - National Insurance Number (NIN, context-keyword gated)
  FR  - NIR/INSEE (15 digits with mod-97 check)
"""

import re
from .base import BaseDetector, Match


class NationalIdDetector(BaseDetector):
    label = "NATIONAL_ID"

    # ---- Context keywords per region ----
    CN_KW = re.compile(
        r'身份证|ID\s*card|identity\s*card|居民身份', re.IGNORECASE)
    US_KW = re.compile(
        r'SSN|Social\s*Security|social\s*security\s*number', re.IGNORECASE)
    AU_KW = re.compile(
        r'TFN|Tax\s*File\s*Number|tax\s*file', re.IGNORECASE)
    SG_KW = re.compile(
        r'NRIC|FIN|identity\s*card|IC\s*number', re.IGNORECASE)
    MY_KW = re.compile(
        r'MyKad|IC\s*number|NRIC|kad\s*pengenalan', re.IGNORECASE)
    TH_KW = re.compile(
        r'Thai\s*ID|national\s*ID|บัตรประชาชน|เลขบัตร', re.IGNORECASE)
    ID_KW = re.compile(
        r'NIK|KTP|kartu\s*tanda\s*penduduk', re.IGNORECASE)
    DE_KW = re.compile(
        r'Steuer[\-\s]?ID|Steueridentifikationsnummer|IdNr|Tax\s*ID', re.IGNORECASE)
    UK_KW = re.compile(
        r'NIN|National\s*Insurance|NI\s*number|NINO', re.IGNORECASE)
    FR_KW = re.compile(
        r'NIR|INSEE|num[eé]ro\s*de\s*s[eé]curit[eé]\s*sociale|social\s*security',
        re.IGNORECASE)

    # ---- Patterns ----
    # CN: 6 region + 8 birthday + 3 seq + 1 check
    CN_PAT = re.compile(
        r'(?<!\d)(\d{6})((?:19|20)\d{2})((?:0[1-9]|1[0-2]))'
        r'((?:0[1-9]|[12]\d|3[01]))(\d{3})(\d|X|x)(?!\d)')
    # US SSN: XXX-XX-XXXX (9 digits, with or without separators)
    US_SSN_PAT = re.compile(
        r'(?<!\d)(\d{3})[\s\-]?(\d{2})[\s\-]?(\d{4})(?!\d)')
    # AU TFN: XXX XXX XXX (9 digits)
    AU_TFN_PAT = re.compile(
        r'(?<!\d)(\d{3})[\s\-]?(\d{3})[\s\-]?(\d{3})(?!\d)')
    # SG NRIC/FIN: [STFGM] + 7 digits + check letter
    SG_NRIC_PAT = re.compile(
        r'(?<![A-Za-z])([STFGM]\d{7}[A-Z])(?![A-Za-z])')
    # MY MyKad: XXXXXX-XX-XXXX (12 digits)
    MY_MYKAD_PAT = re.compile(
        r'(?<!\d)(\d{6})[\s\-]?(\d{2})[\s\-]?(\d{4})(?!\d)')
    # TH: 13 digits (optionally formatted X-XXXX-XXXXX-XX-X)
    TH_PAT = re.compile(
        r'(?<!\d)(\d[\s\-]?\d{4}[\s\-]?\d{5}[\s\-]?\d{2}[\s\-]?\d)(?!\d)')
    # ID NIK: 16 consecutive digits
    ID_NIK_PAT = re.compile(r'(?<!\d)(\d{16})(?!\d)')
    # DE Steuer-ID: 11 digits (optionally spaced XX XXX XXX XXX)
    DE_STID_PAT = re.compile(
        r'(?<!\d)(\d{2}[\s]?\d{3}[\s]?\d{3}[\s]?\d{3})(?!\d)')
    # UK NIN: 2 letters + 6 digits + 1 letter (spaces optional)
    UK_NIN_PAT = re.compile(
        r'(?<![A-Za-z])([A-CEGHJ-PR-TW-Z][A-CEGHJ-NPR-TW-Z]'
        r'[\s]?\d{2}[\s]?\d{2}[\s]?\d{2}[\s]?[A-D])(?![A-Za-z])')
    # FR NIR: 15 digits (1/2 sex + year + month + dept + commune + seq + check)
    FR_NIR_PAT = re.compile(
        r'(?<!\d)([12][\s]?\d{2}[\s]?\d{2}[\s]?\d{2}[\s]?\d{3}[\s]?\d{3}[\s]?\d{2})(?!\d)')

    # ---- CN checksum (ISO 7064 Mod 11-2) ----
    _CN_W = [7, 9, 10, 5, 8, 4, 2, 1, 6, 3, 7, 9, 10, 5, 8, 4, 2]
    _CN_CHK = '10X98765432'

    def _cn_check(self, s):
        if len(s) != 18 or not s[:17].isdigit():
            return False
        total = sum(int(d) * w for d, w in zip(s[:17], self._CN_W))
        return s[17].upper() == self._CN_CHK[total % 11]

    # ---- AU TFN check (weights 1,4,3,7,5,8,6,9,10) ----
    _AU_W = [1, 4, 3, 7, 5, 8, 6, 9, 10]

    def _au_tfn_check(self, digits):
        if len(digits) != 9 or not digits.isdigit():
            return False
        return sum(int(d) * w for d, w in zip(digits, self._AU_W)) % 11 == 0

    # ---- TH check digit (mod 11) ----
    def _th_check(self, digits):
        if len(digits) != 13 or not digits.isdigit():
            return False
        total = sum(int(digits[i]) * (13 - i) for i in range(12))
        return int(digits[12]) == (11 - total % 11) % 10

    # ---- FR NIR check (mod 97) ----
    def _fr_nir_check(self, digits):
        if len(digits) != 15 or not digits.isdigit():
            return False
        return int(digits[13:15]) == 97 - (int(digits[:13]) % 97)

    # ---- Main detect ----
    def detect(self, text):
        matches = []

        # CN (checksum-gated, no keyword required)
        for m in self.CN_PAT.finditer(text):
            raw = m.group()
            if self._cn_check(raw):
                matches.append(Match(
                    label=self.label, confidence=0.98,
                    masked_preview=raw[:3] + '**************' + raw[-1:],
                    start=m.start(), end=m.end(), region='CN'))

        # US SSN (keyword-gated)
        if self.US_KW.search(text):
            for m in self.US_SSN_PAT.finditer(text):
                area, grp, serial = m.group(1), m.group(2), m.group(3)
                if area in ('000', '666') or area[0] == '9':
                    continue
                if grp == '00' or serial == '0000':
                    continue
                matches.append(Match(
                    label=self.label, confidence=0.90,
                    masked_preview=f'***-**-**{serial[-2:]}',
                    start=m.start(), end=m.end(), region='US'))

        # AU TFN (keyword-gated + checksum)
        if self.AU_KW.search(text):
            for m in self.AU_TFN_PAT.finditer(text):
                digits = m.group(1) + m.group(2) + m.group(3)
                if self._au_tfn_check(digits):
                    matches.append(Match(
                        label=self.label, confidence=0.92,
                        masked_preview=digits[:2] + ' ***** ' + digits[-1:],
                        start=m.start(), end=m.end(), region='AU'))

        # SG NRIC (keyword-gated)
        if self.SG_KW.search(text):
            for m in self.SG_NRIC_PAT.finditer(text):
                raw = m.group(1)
                matches.append(Match(
                    label=self.label, confidence=0.88,
                    masked_preview=raw[0] + '*******' + raw[-1:],
                    start=m.start(), end=m.end(), region='SG'))

        # MY MyKad (keyword-gated + date validation)
        if self.MY_KW.search(text):
            for m in self.MY_MYKAD_PAT.finditer(text):
                digits = m.group(1) + m.group(2) + m.group(3)
                if len(digits) != 12:
                    continue
                mm, dd = int(digits[2:4]), int(digits[4:6])
                if mm < 1 or mm > 12 or dd < 1 or dd > 31:
                    continue
                matches.append(Match(
                    label=self.label, confidence=0.88,
                    masked_preview=digits[:2] + '********' + digits[-1:],
                    start=m.start(), end=m.end(), region='MY'))

        # TH National ID (keyword-gated + checksum)
        if self.TH_KW.search(text):
            for m in self.TH_PAT.finditer(text):
                digits = re.sub(r'[\s\-]', '', m.group())
                if self._th_check(digits):
                    matches.append(Match(
                        label=self.label, confidence=0.90,
                        masked_preview=digits[:2] + '-***********-' + digits[-1],
                        start=m.start(), end=m.end(), region='TH'))

        # ID NIK (keyword-gated + date validation)
        if self.ID_KW.search(text):
            for m in self.ID_NIK_PAT.finditer(text):
                d = m.group(1)
                dd, mm = int(d[6:8]), int(d[8:10])
                if dd > 40:
                    dd -= 40  # female offset
                if mm < 1 or mm > 12 or dd < 1 or dd > 31:
                    continue
                matches.append(Match(
                    label=self.label, confidence=0.85,
                    masked_preview=d[:2] + '************' + d[-1:],
                    start=m.start(), end=m.end(), region='ID'))

        # DE Steuer-ID (keyword-gated)
        if self.DE_KW.search(text):
            for m in self.DE_STID_PAT.finditer(text):
                digits = re.sub(r'\s', '', m.group(1))
                if len(digits) != 11 or not digits.isdigit():
                    continue
                if digits[0] == '0':
                    continue
                matches.append(Match(
                    label=self.label, confidence=0.85,
                    masked_preview=digits[:2] + ' ******* ' + digits[-1:],
                    start=m.start(), end=m.end(), region='DE'))

        # UK NIN (keyword-gated)
        if self.UK_KW.search(text):
            for m in self.UK_NIN_PAT.finditer(text):
                raw = re.sub(r'\s', '', m.group(1))
                matches.append(Match(
                    label=self.label, confidence=0.90,
                    masked_preview=raw[:2] + ' ****** ' + raw[-1],
                    start=m.start(), end=m.end(), region='UK'))

        # FR NIR (keyword-gated + mod-97 check)
        if self.FR_KW.search(text):
            for m in self.FR_NIR_PAT.finditer(text):
                digits = re.sub(r'\s', '', m.group(1))
                if self._fr_nir_check(digits):
                    matches.append(Match(
                        label=self.label, confidence=0.92,
                        masked_preview=digits[0] + ' ************ ' + digits[-1:],
                        start=m.start(), end=m.end(), region='FR'))

        return matches


path: scripts/detectors/passport.py
content: """Passport number detector — multi-region, context-keyword gated.

All passport detection requires a context keyword (passport, visa, etc.)
to avoid false positives on generic alphanumeric strings.
"""

import re
from .base import BaseDetector, Match


class PassportDetector(BaseDetector):
    label = "PASSPORT"

    # Multi-language context keywords
    CONTEXT_KW = re.compile(
        r'passport|护照|签证|visa|出入境|immigration'
        r'|Reisepass|passeport|パスポート|여권|paspor',
        re.IGNORECASE
    )

    # Region-specific patterns: (compiled_regex, default_region)
    PATTERNS = [
        # CN: E/G/D/S/P/H + optional letter + 7-8 digits
        (re.compile(r'(?<![A-Za-z])([EGDSPHegdsph][A-Za-z]?\d{7,8})(?![A-Za-z\d])'), 'CN'),
        # DE: C + 8 alphanumeric
        (re.compile(r'(?<![A-Za-z])(C[A-Z0-9]{8})(?![A-Za-z\d])'), 'DE'),
        # FR: 2 digits + 2 letters + 5 digits
        (re.compile(r'(?<![A-Za-z\d])(\d{2}[A-Z]{2}\d{5})(?![A-Za-z\d])'), 'FR'),
        # Generic: 1-2 uppercase letters + 6-9 digits (covers US, AU, UK, SEA, etc.)
        (re.compile(r'(?<![A-Za-z])([A-Z]{1,2}\d{6,9})(?![A-Za-z\d])'), 'INTL'),
    ]

    def detect(self, text):
        if not self.CONTEXT_KW.search(text):
            return []

        matches = []
        seen = set()

        for pattern, default_region in self.PATTERNS:
            for m in pattern.finditer(text):
                raw = m.group(1)
                if raw in seen:
                    continue
                digits = re.sub(r'[^0-9]', '', raw)
                if len(digits) < 6:
                    continue
                seen.add(raw)
                matches.append(Match(
                    label=self.label,
                    confidence=0.85,
                    masked_preview=raw[:2] + '****' + raw[-2:],
                    start=m.start(1), end=m.end(1),
                    region=default_region,
                ))
        return matches


path: scripts/detectors/person_name.py
content: """Person name detector — multi-region, context-keyword gated.

CN: Chinese name (2-4 characters) after/before context keywords.
INTL: Western name (First Last) after context keywords.
All detections require keyword context to reduce false positives.
"""

import re
from .base import BaseDetector, Match


class PersonNameDetector(BaseDetector):
    label = "PERSON_NAME"

    # ---- CN: keyword before name ----
    CN_BEFORE = re.compile(
        r'(?:姓名|名字|收件人|联系人|持卡人|户名|开户人|患者|被告|原告|当事人)'
        r'[：:\s]*'
        r'([\u4e00-\u9fff]{2,4})',
    )
    # CN: name followed by honorific title
    CN_AFTER_TITLE = re.compile(
        r'([\u4e00-\u9fff]{2,4})'
        r'(?:先生|女士|同志|老师|教授|医生|律师)',
    )

    # ---- Multi-language keyword + name ----
    # Structured: "Name:" / "姓名：" + value (CN or Western)
    STRUCTURED_CN = re.compile(
        r'(?:name|姓名|收件人|联系人|持卡人|户名)'
        r'[\s：:=]*'
        r'([\u4e00-\u9fff]{2,4})',
        re.IGNORECASE
    )
    STRUCTURED_WESTERN = re.compile(
        r'(?:name|full\s*name|recipient|account\s*holder|customer|passenger|patient)'
        r'[\s：:=]*'
        r'([A-Z][a-z]{1,15}\s[A-Z][a-z]{1,15}(?:\s[A-Z][a-z]{1,15})?)',
        re.IGNORECASE
    )

    # ---- Western name after title (Mr./Mrs./Ms./Dr./Prof.) ----
    WESTERN_TITLE = re.compile(
        r'(?:Mr\.?|Mrs\.?|Ms\.?|Miss|Dr\.?|Prof\.?)\s+'
        r'([A-Z][a-z]{1,15}\s[A-Z][a-z]{1,15}(?:\s[A-Z][a-z]{1,15})?)'
    )

    # ---- DE/FR name keywords ----
    DE_NAME = re.compile(
        r'(?:Name|Vorname|Nachname|Inhaber|Empf[aä]nger)'
        r'[\s：:=]*'
        r'([A-Z\u00c0-\u00ff][a-z\u00e0-\u00ff]{1,15}\s'
        r'[A-Z\u00c0-\u00ff][a-z\u00e0-\u00ff]{1,15})',
        re.IGNORECASE
    )
    FR_NAME = re.compile(
        r'(?:nom|pr[eé]nom|destinataire|titulaire)'
        r'[\s：:=]*'
        r'([A-Z\u00c0-\u00ff][a-z\u00e0-\u00ff]{1,15}\s'
        r'[A-Z\u00c0-\u00ff][a-z\u00e0-\u00ff]{1,15})',
        re.IGNORECASE
    )

    def _mask_name(self, name):
        """Mask name: keep first char, mask the rest."""
        parts = name.split()
        if len(parts) >= 2:
            return parts[0][0] + '***' + ' ' + parts[-1][0] + '***'
        return name[0] + '*' * (len(name) - 1)

    def detect(self, text):
        matches = []
        seen = set()

        def _add(name, start, end, region, confidence):
            if name in seen:
                return
            seen.add(name)
            if len(name) <= 1:
                return
            masked = self._mask_name(name)
            matches.append(Match(
                label=self.label, confidence=confidence,
                masked_preview=masked, start=start, end=end,
                region=region,
            ))

        # CN patterns
        for pat in [self.CN_BEFORE, self.CN_AFTER_TITLE, self.STRUCTURED_CN]:
            for m in pat.finditer(text):
                _add(m.group(1), m.start(1), m.end(1), 'CN', 0.70)

        # Western structured (English keyword + First Last)
        for m in self.STRUCTURED_WESTERN.finditer(text):
            _add(m.group(1), m.start(1), m.end(1), 'INTL', 0.72)

        # Western title prefix (Mr./Mrs./Dr. + name)
        for m in self.WESTERN_TITLE.finditer(text):
            _add(m.group(1), m.start(1), m.end(1), 'INTL', 0.75)

        # DE name patterns
        for m in self.DE_NAME.finditer(text):
            _add(m.group(1), m.start(1), m.end(1), 'DE', 0.70)

        # FR name patterns
        for m in self.FR_NAME.finditer(text):
            _add(m.group(1), m.start(1), m.end(1), 'FR', 0.70)

        return matches


path: scripts/detectors/phone.py
content: """Phone number detector — multi-region support.

Regions: CN, US, AU, UK (local patterns) + any country via +CC prefix.
Country code mapping: +86=CN, +1=US, +61=AU, +65=SG, +60=MY,
+66=TH, +62=ID, +49=DE, +44=UK, +33=FR.
"""

import re
from .base import BaseDetector, Match

# Map country calling codes to ISO region codes
COUNTRY_CODES = {
    '86': 'CN', '1': 'US', '61': 'AU', '65': 'SG', '60': 'MY',
    '66': 'TH', '62': 'ID', '49': 'DE', '44': 'UK', '33': 'FR',
}


class PhoneDetector(BaseDetector):
    label = "PHONE"

    # --- International: +CC followed by digits (highest priority) ---
    INTL = re.compile(r'\+(\d{1,3})[\s\-.]?(\d[\d\s\-.]{5,14}\d)')

    # --- China: mobile 1[3-9]X XXXX XXXX ---
    CN_MOBILE = re.compile(r'(?<!\d)1[3-9]\d[\s\-]?\d{4}[\s\-]?\d{4}(?!\d)')
    # China: landline 0XX(X)-XXXXXXX(X), require separator to reduce FP
    CN_LANDLINE = re.compile(r'(?<!\d)0\d{2,3}[\s\-]\d{7,8}(?!\d)')

    # --- US: (XXX) XXX-XXXX ---
    US_PAREN = re.compile(r'(?<!\d)\(\d{3}\)[\s\-.]?\d{3}[\s\-.]?\d{4}(?!\d)')
    # US: XXX-XXX-XXXX (require dashes to be distinctive)
    US_DASH = re.compile(r'(?<!\d)\d{3}\-\d{3}\-\d{4}(?!\d)')

    # --- Australia: mobile 04XX XXX XXX ---
    AU_MOBILE = re.compile(r'(?<!\d)04\d{2}[\s\-]?\d{3}[\s\-]?\d{3}(?!\d)')

    # --- UK: mobile 07XXX XXXXXX ---
    UK_MOBILE = re.compile(r'(?<!\d)07\d{3}[\s\-]?\d{6}(?!\d)')

    def _resolve_region(self, cc):
        """Resolve country calling code to region code."""
        return COUNTRY_CODES.get(cc, 'INTL')

    def detect(self, text):
        matches = []
        seen = set()  # (start, end) dedup

        def _add(start, end, raw, region, confidence):
            if (start, end) in seen:
                return
            seen.add((start, end))
            clean = re.sub(r'[\s\-.()+]', '', raw)
            matches.append(Match(
                label=self.label,
                confidence=confidence,
                masked_preview=self._mask(clean, keep_start=2, keep_end=2),
                start=start, end=end,
                region=region,
            ))

        # International format (highest priority, any country)
        for m in self.INTL.finditer(text):
            cc = m.group(1)
            region = self._resolve_region(cc)
            _add(m.start(), m.end(), m.group(), region, 0.92)

        # CN mobile
        for m in self.CN_MOBILE.finditer(text):
            _add(m.start(), m.end(), m.group(), 'CN', 0.90)

        # CN landline
        for m in self.CN_LANDLINE.finditer(text):
            _add(m.start(), m.end(), m.group(), 'CN', 0.85)

        # US with parentheses
        for m in self.US_PAREN.finditer(text):
            _add(m.start(), m.end(), m.group(), 'US', 0.88)

        # US with dashes
        for m in self.US_DASH.finditer(text):
            _add(m.start(), m.end(), m.group(), 'US', 0.85)

        # AU mobile
        for m in self.AU_MOBILE.finditer(text):
            _add(m.start(), m.end(), m.group(), 'AU', 0.88)

        # UK mobile
        for m in self.UK_MOBILE.finditer(text):
            _add(m.start(), m.end(), m.group(), 'UK', 0.88)

        return matches


path: scripts/detectors/social_account.py
content: """Social account detector — WeChat, QQ, Twitter/X, and generic account patterns."""

import re
from .base import BaseDetector, Match


class SocialAccountDetector(BaseDetector):
    label = "SOCIAL_ACCOUNT"

    # WeChat ID after keyword: letter-start, 6-20 chars
    WECHAT = re.compile(
        r'(?:\u5fae\u4fe1|wechat|WeChat|wx)[\u53f7ID\uff1a:\s]*([a-zA-Z][\w\-]{5,19})',
        re.IGNORECASE
    )

    # QQ number after keyword: 5-12 digits
    QQ = re.compile(
        r'(?:QQ|qq)[\u53f7\uff1a:\s]*(\d{5,12})'
    )

    # Twitter / X handle
    TWITTER = re.compile(
        r'(?:twitter|\u63a8\u7279|X)[\uff1a:\s]*@?([a-zA-Z_][\w]{1,15})',
        re.IGNORECASE
    )

    # Generic: after "账号/帐号/account" keyword
    GENERIC = re.compile(
        r'(?:\u8d26\u53f7|\u5e10\u53f7|account)[\uff1a:\s]*([\w@.\-]{4,30})',
        re.IGNORECASE
    )

    def detect(self, text):
        matches = []
        seen = set()

        for pattern in [self.WECHAT, self.QQ, self.TWITTER, self.GENERIC]:
            for m in pattern.finditer(text):
                account = m.group(1)
                if account in seen:
                    continue
                seen.add(account)
                if len(account) > 4:
                    masked = account[:2] + '****' + account[-2:]
                else:
                    masked = account[:1] + '***'
                matches.append(Match(
                    label=self.label,
                    confidence=0.80,
                    masked_preview=masked,
                    start=m.start(1),
                    end=m.end(1),
                ))
        return matches


path: scripts/detectors/__init__.py
content: """PII detector registry."""

from .phone import PhoneDetector
from .email_detector import EmailDetector
from .national_id import NationalIdDetector
from .passport import PassportDetector
from .bank_card import BankCardDetector
from .person_name import PersonNameDetector
from .address import AddressDetector
from .social_account import SocialAccountDetector

ALL_DETECTORS = [
    PhoneDetector(),
    EmailDetector(),
    NationalIdDetector(),
    PassportDetector(),
    BankCardDetector(),
    PersonNameDetector(),
    AddressDetector(),
    SocialAccountDetector(),
]


path: scripts/file_lock.py
content: """Cross-platform file lock using atomic exclusive-create.

Usage:
    from file_lock import FileLock

    with FileLock('/path/to/data.json.lock'):
        # read / write data.json safely
        ...

The lock is acquired by atomically creating a `.lock` file (O_CREAT | O_EXCL).
If another process holds the lock, we spin-wait with 50ms intervals up to a
configurable timeout.  On timeout, the stale lock file is force-removed and
acquisition is retried once (guards against dead processes leaving orphan locks).

Stdlib-only.  Works on Windows and POSIX.
"""

import os
import time


class FileLockTimeout(Exception):
    """Raised when the lock cannot be acquired within the timeout."""


class FileLock:
    """Context manager for file-based mutual exclusion.

    Parameters:
        lock_path:      Full path to the lock file (e.g. ``data.json.lock``).
        timeout:        Maximum seconds to wait for the lock (default 5).
        poll_interval:  Seconds between acquisition attempts (default 0.05).
    """

    def __init__(self, lock_path: str, *, timeout: float = 5.0,
                 poll_interval: float = 0.05):
        self.lock_path = lock_path
        self.timeout = timeout
        self.poll_interval = poll_interval
        self._fd = None

    # ------------------------------------------------------------------
    def acquire(self) -> None:
        deadline = time.monotonic() + self.timeout
        while True:
            try:
                self._fd = os.open(
                    self.lock_path,
                    os.O_CREAT | os.O_EXCL | os.O_WRONLY,
                )
                return  # lock acquired
            except (FileExistsError, OSError):
                if time.monotonic() >= deadline:
                    # Timeout — attempt stale-lock recovery once
                    self._remove_stale()
                    try:
                        self._fd = os.open(
                            self.lock_path,
                            os.O_CREAT | os.O_EXCL | os.O_WRONLY,
                        )
                        return
                    except (FileExistsError, OSError):
                        raise FileLockTimeout(
                            f'Could not acquire lock: {self.lock_path}'
                        )
                time.sleep(self.poll_interval)

    def release(self) -> None:
        if self._fd is not None:
            try:
                os.close(self._fd)
            except OSError:
                pass
            self._fd = None
        self._remove_stale()

    # ------------------------------------------------------------------
    def _remove_stale(self) -> None:
        try:
            os.remove(self.lock_path)
        except OSError:
            pass

    # ------------------------------------------------------------------
    def __enter__(self):
        self.acquire()
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.release()
        return False


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)