Skill: li2092/ontology-engineer (version 1.1.1)

--- name: ontology-engineer version: 1.1.1 description: >- Extract candidate ontology models from enterprise business systems AND build/maintain personal knowledge graphs from any file system. Use when: ontology extraction, 本体提取, schema analysis, entity extraction, data dictionary (数据字典), 表结构分析, knowledge graph (知识图谱), 全量扫描, file scan, personal knowledge (个人知识), or analyzing business system data models. Three operating modes: (A) Database/schema extraction — SCAN→EXTRACT→MERGE from SQL DDL, Word/Excel data dictionaries. Outputs ontology.json + review.md. (B) Filesystem scanning — index→analyze pipeline for personal knowledge graph. Reads Office/PDF/text, extracts entities and domain structures. Outputs graph.jsonl + schema.yaml. (C) External data scanning — same as B for others' data spaces (clients, partners). Handles .docx .doc .wps .pdf .xlsx .xls .et .pptx .ppt .dps .md .txt .sql .yaml .json .csv. Uses python-docx, PyMuPDF, openpyxl, python-pptx. Supports multimodal image analysis. No external API keys or network access required — the LLM running this skill IS the semantic analysis engine. All processing is local. File scanning is user-scoped via mandatory Step 1.5 confirmation before any analysis begins. metadata.openclaw: homepage: "https://github.com/li2092/ontology-engineer" requires: bins: - python3 anyBins: - libreoffice - word install: - type: uv packages: - python-docx - PyMuPDF - openpyxl - xlrd - python-pptx - Pillow - pyyaml --- # Ontology Engineer Extract candidate ontology models from existing data. Build and maintain personal knowledge graphs. **Core principle**: Make implicit business models in existing data explicit. Don't create from scratch. **Division of labor**: Scripts handle mechanical extraction (file scanning, format conversion, table parsing). LLM handles semantic judgment (entity identification, property selection, relationship discovery, naming, cross-source merging). **Security model**: - **No external API calls.** The LLM running this skill (Claude, OpenClaw, etc.) IS the semantic engine. No credentials, no network endpoints, no data exfiltration paths. - **User-scoped scanning.** Step 1.5 is a MANDATORY interactive checkpoint — the user reviews and approves every folder before any content is read. Nothing is analyzed without explicit confirmation. - **Local-only output.** All artifacts (graph.jsonl, schema.yaml, review.md) are written to a user-specified local directory. No data leaves the machine. - **Append-only writes.** Scripts only create/append files. No deletion, no modification of existing user files. ## When This Skill Adds Value (and When It Doesn't) Knowledge graphs and ontology extraction are not universally useful. Before starting, assess fit: | Scenario | Value | Why | |----------|-------|-----| | **3+ heterogeneous systems** with inconsistent naming for the same concepts | High (Mode A) | Cross-system concept alignment is the core use case | | **Agent product** needs factual grounding to reduce hallucination | High (Mode B/C) | Graph becomes Agent's fact base — auto-query before every response | | **1000+ entities** with dense relationships across long time spans | High | Pattern discovery humans can't do manually (churn, cross-sell, capability mapping) | | **Client consulting engagement** analyzing their data landscape | High (Mode A) | Core consulting deliverable: "here's what your data assets look like" | | Small org, <200 entities, info fits in one person's head + Excel | Low (Mode B) | Graph just re-stores what user already knows — use as PoC/capability validation only | | Single system, no cross-system integration need | Low (Mode A) | Read the schema directly; ontology layer adds overhead without value | **Rule of thumb**: If the user's reaction to the output is "I already knew all this", the graph isn't producing incremental value. Redirect to Mode A (client projects) or Agent integration. **Detailed value scenarios**: [references/value-scenarios.md](references/value-scenarios.md) ## Operating Modes | Mode | Input | Output | Use When | |------|-------|--------|----------| | **A: Database Extraction** | SQL DDL, data dictionaries, Word/Excel schemas | ontology.json + review.md | Analyzing enterprise business systems | | **B: Filesystem Scanning** | Local/cloud directories | graph.jsonl + schema.yaml | Building personal knowledge graph | | **C: External Data** | Others' data spaces, shared drives | graph.jsonl (source=external) | Acquiring others' business models | --- ## Mode A: Database Extraction Three-phase workflow for extracting ontology from structured data sources. ### Phase 1: SCAN Run `scripts/scan_directory.py` to discover and classify files by priority (P1-P7). ```bash python scripts/scan_directory.py "<dir>" --output scan_result.json --report ``` Review scan report. Process P1-P2 files first, expand as needed. ### Phase 2: EXTRACT 1. Convert .doc files if needed: `scripts/convert_doc.py` 2. Extract tables from Word/Excel: `scripts/extract_tables.py` 3. Read extracted data, apply Rules 1-7 (see [analysis-rules.md](references/analysis-rules.md)) 4. For text formats (.sql, .json, .yaml, .md, .csv): read directly ### Phase 3: MERGE 1. Cross-source entity deduplication (Rule 5) 2. Relationship consolidation 3. Output: `ontology.json` + `review.md` **Detailed rules**: [references/analysis-rules.md](references/analysis-rules.md) (Rules 1-7) **Quality checks**: [references/quality-checks.md](references/quality-checks.md) **Script details**: [references/script-operations.md](references/script-operations.md) **Modeling decisions**: [references/modeling-decisions.md](references/modeling-decisions.md) --- ## Mode B/C: Knowledge Graph Two-step pipeline for building personal knowledge graphs from file systems. ### Step 1: File Indexing (script, no LLM) ```bash python scripts/scan_filesystem.py --root /path --config namespace_rules.yaml --extract-metadata ``` Creates Document + Project entities in graph.jsonl. Pure mechanical operation. **Key features**: Auto namespace inference, duplicate detection, .docx/.pdf metadata extraction, universal noise filtering. ### Step 1.5: User Scope Confirmation (MANDATORY interactive step) After Step 1 completes, **present the scan summary to the user and ask for scope confirmation** before proceeding to Step 2. The user knows which folders matter most. Display a table of all discovered projects/namespaces with document counts, then ask: ``` 扫描完成，发现 {N} 个项目，共 {M} 篇文档。请标记每个文件夹的优先级： - 🔴 重点（高采样率，优先分析） - ⚪ 普通（默认采样率） - ⚫ 忽略（跳过，不分析） - 或输入"全部"跳过选择，按默认策略处理所有文件夹 | # | 项目 | 文档数 | 格式分布 | 默认优先级 | |---|------|--------|----------|-----------| | 1 | work/myfiles | 15,617 | .doc .docx .pdf .xlsx | 🔴 重点 | | 2 | work/classified | 1,578 | .doc .pdf .xlsx | ⚪ 普通 | | ... | ... | ... | ... | ... | 请输入调整（如 "2=忽略, 5=重点"）或 "全部" 或 "确认"： ``` **Rules**: - User can mark any folder as 重点/普通/忽略 - User can type "全部" to skip selection and use defaults - 重点 folders get 2-3x sampling rate, 忽略 folders are skipped entirely - Default priority is auto-inferred: human work folders=重点, AI-generated=普通, downloads/cache=忽略 - **Never skip this step.** Even if obvious, let the user confirm. ### Step 2: Semantic Analysis (LLM, core step) Five phases: Sampling → Document Reading → Aggregation → Cross-project Alignment → Output. **Key decisions** (details in [knowledge-graph-workflow.md](references/knowledge-graph-workflow.md)): - Minimum 10% coverage, 重点 folders 2-3x, 忽略 folders skip - **Structured lists (Rule 13)**: Files named 列表/台账/名单/登记表/清单 etc. → full extraction (every row = one entity), NOT sampling. See [analysis-rules.md](references/analysis-rules.md) Rule 13. - Dual-track extraction: Track A (named entities) + Track B (domain terms) - Subagents must be `general-purpose` type (Bash access). Never use Explore type. - Format tools: see [formats-and-deps.md](references/formats-and-deps.md) - **Relation semantics**: Use enriched relation format with direction, cardinality, temporal range. See [relation-ontology.md](references/relation-ontology.md). ### Step 3: Runtime Evolution Agent enriches the knowledge graph during daily conversations. `source.type = "runtime"`. **When to trigger** (passive, no user action needed): - User mentions a person by name + role/org → check graph, append if new - User discusses a project/event with dates → append Event - User makes a strategic decision or key insight → append Note - User mentions a new organization/client → append Organization **How to append**: ```bash python query_graph.py search "张三" # Check if entity exists # If not found, append to graph.jsonl: echo '{"op":"create","ts":"...","entity":{"id":"per-NNNNN","type":"Person","graph":"core/persons","source":{"type":"runtime","conversation_id":"..."},...}}' >> graph.jsonl ``` **Rules**: - Only append entities with concrete evidence from the conversation - Never overwrite existing entities — only add new ones or note conflicts - Use `source.type = "runtime"` to distinguish from scan-derived entities - Keep it lightweight: 1-3 entities per conversation, not a full re-scan **Full workflow details**: [references/knowledge-graph-workflow.md](references/knowledge-graph-workflow.md) **Analysis rules (8-12)**: [references/analysis-rules.md](references/analysis-rules.md) **Format support & deps**: [references/formats-and-deps.md](references/formats-and-deps.md) --- ## Key Principles - **Model business concepts, not database tables.** Table names ≠ object names. - **Extract then express.** Make implicit models explicit, don't create from nothing. - **Experts judge.** Produce candidates; final decisions belong to humans. When in doubt, flag it. - **Invest in invariants.** Stable entities and relationships, not technical details. - **Handle what exists.** Real projects use Word and Excel. Adapt to the data. - **Scripts extract, LLM analyzes.** Mechanical extraction via Python. Semantic judgment via LLM. - **Coverage over perfection.** 60% of files at moderate depth beats 3% at maximum depth. - **Generic skeleton + domain discovery.** 8 core types (BFO-aligned). Domain types discovered by scanning. - **Single source of truth.** All data in one graph.jsonl. Soft partition via graph/labels/source. - **Relations carry semantics.** Direction, cardinality, temporal range, evidence. Not just type + target. - **Append-only evolution.** Never delete entities. Deprecate, reclassify, version. ## Ontology Theory References | Reference | When to Read | |-----------|-------------| | [modeling-decisions.md](references/modeling-decisions.md) | Core type boundaries, entity vs enum, promotion judgment | | [relation-ontology.md](references/relation-ontology.md) | Relation format, core relation catalog, ternary relations | | [ontology-evolution.md](references/ontology-evolution.md) | Schema versioning, entity reclassification, conflict resolution | | [constraints-and-inference.md](references/constraints-and-inference.md) | Type/relation constraints, inference rules, inconsistency detection | | [value-scenarios.md](references/value-scenarios.md) | When this skill adds value and when it doesn't | --- ## Output Formats ### graph.jsonl (Mode B/C) ```jsonl {"op":"create","ts":"2026-01-15T10:00:00Z","entity":{"id":"per-00001","type":"Person","graph":"core/persons","labels":["employee"],"source":{"type":"scan","scan_id":"step2-r1"},"properties":{"name":"张三","roles":["项目经理"],"organizations":["某科技公司"]},"relations":[{"type":"works_at","target_id":"org-00002","direction":"forward","cardinality":"N:1","temporal":{"start":"2019-01","end":null},"confidence":"high"}],"created_at":"2026-01-15T10:00:00Z"}} ``` Required: `id`, `type`, `graph`, `source`, `created_at`. Optional: `labels`, `properties`, `relations`. Relation fields: `type` + `target_id` required. Optional: `direction` (forward/reverse/bidirectional), `cardinality` (1:1/1:N/N:1/N:M), `temporal` ({start, end}), `evidence` (source entity ID), `confidence` (high/medium/low). See [relation-ontology.md](references/relation-ontology.md). ### schema.yaml (Mode B/C) ```yaml meta: version: "2.0" core_types: # 8 fixed (BFO-aligned): Person, Organization, Project, Task, Document, Event, Note, Goal domain_types: # Discovered by Step 2 Track B, grouped by domain namespaces: # core/, work/*, personal/*, external/*, uncategorized/* source_types: # scan | runtime | manual | email | cloud | chat relation_schema: # Relation fields: type, target_id, direction, cardinality, temporal, evidence, confidence relation_types: # Core relation catalog grouped by source type pair constraints: # type_constraints (required props, enums), relation_constraints, id_pattern inference_rules: # Transitive subsidiary, symmetric partner, inverse works_at, etc. schema_evolution: # Version format, backward compatibility rules ``` ### ontology.json (Mode A) ```json { "meta": {"generated_by": "ontology-engineer", "source_files": [], "domain": "..."}, "object_types": [{"name": "...", "english": "...", "core_properties": [], "confidence": "high|medium|low"}], "link_types": [{"from": "A", "relation": "verb", "to": "B", "cardinality": "1:N", "evidence": "..."}], "review_flags": [{"type": "promotion|merge|ambiguity|missing", "item": "...", "question": "..."}] } ``` ### review.md 1. Scan summary 2. Model overview 3. Object catalog 4. Relationship map 5. Review items 6. Cross-source merges 7. Data quality notes 8. Decision log --- ## Scripts | Script | Mode | Purpose | |--------|------|---------| | `scripts/scan_filesystem.py` | B/C | File indexing, namespace inference, metadata extraction | | `scripts/scan_directory.py` | A | File discovery with P1-P7 priority classification | | `scripts/convert_doc.py` | A | .doc → .docx conversion | | `scripts/extract_tables.py` | A | Table extraction from Word/Excel | **Details**: [references/script-operations.md](references/script-operations.md) --- ## Agent Integration | Component | Purpose | Status | |-----------|---------|--------| | `query_graph.py` | Search entities by type/name/graph/labels, traverse relations | Done | | Runtime write | Agent appends new entities during conversation (Step 3) | Done | | MCP Server | Expose graph as tools: `search_entities`, `get_relations` | Planned | | Prompt injection | Agent auto-queries graph for context before handling tasks | Planned | **Query tool usage**: ```bash python query_graph.py stats # Overview python query_graph.py search "关键词" # Search python query_graph.py type Person --limit 20 # By type python query_graph.py get per-00001 # Details python query_graph.py relations per-00001 # Relations python query_graph.py domain --limit 30 # Domain terms python query_graph.py export Person --format csv # Export ```

path: references/analysis-rules.md
content: # Analysis Rules

## Rules 1-7: Database/Schema Extraction (Mode A)

### Rule 1: Entity Identification (W3C Direct Mapping)

| Table Pattern | Maps To |
|--------------|---------|
| Independent business table | Object Type |
| Pure association table (only FKs + few attrs) | Link Type |
| Enum/dictionary table | Attribute value domain (don't model independently) |
| Technical table (logs, audit, temp) | Exclude |

Three-question test:
1. Do business people mention this concept daily?
2. Does it have an independent lifecycle (create → modify → archive)?
3. Can it exist independently (has own PK and business meaning)?

### Rule 2: Property Selection

- **Keep**: Fields with business decision value (max 15 per object type)
- **Exclude**: create_time, update_by, is_deleted, version, sort_order
- **Naming**: Business language ("客户名称" not "cust_nm"), keep original as annotation

### Rule 3: Promotion Judgment (Stanford Ontology 101)

Promote field to Object Type when:
- It has multiple dimensions of information (fare_class → pricing rules + refund policy + mileage)
- It needs its own properties (not just a value, but a "concept with substance")
- Multiple objects reference it (shared business entity)

Mark as `review_flag` for expert confirmation.

### Rule 4: Relationship Discovery

Evidence sources:
- **Foreign keys**: A has B's FK → A relates to B
- **Naming**: Fields ending in `_id`, `_code`, `_ref`
- **Business logic**: Semantics imply relationships even without FK
- **Association tables**: `_rel`, `_map`, `_link` or only two FKs → M:N

Each relationship needs: direction (A → verb → B), cardinality (1:1, 1:N, N:M), evidence.

### Rule 5: Cross-Source Entity Merging

- Identify same concept across systems (CRM.customer vs billing.party)
- Mark merge evidence: shared FK, naming similarity, semantic alignment
- Mark as `review_flag` for expert confirmation

### Rule 6: 30-Second Rule

- Every concept explainable to business person in 30 seconds
- If you can't explain it, adjust granularity
- Simple > complex; ontologies enrich incrementally

### Rule 7: Evidence Grading

| Source Type | Grade |
|------------|-------|
| SQL DDL with column + type + constraint | Definitive |
| Excel data dictionary with descriptions | Strong |
| Word document with table structure | Moderate |
| Requirements/architecture description | Inferred |
| Business logic or naming convention only | Weak |

---

## Rules 8-12: Knowledge Graph (Mode B/C)

### Rule 8: File Source Classification

| Level | Extensions | Step 2 Priority |
|-------|-----------|-----------------|
| **Core work docs** | .docx .doc .wps .pdf .xlsx .xls .et .pptx .ppt .dps | **Highest** — most users' primary output |
| Text docs | .md .txt .csv .tsv .rtf .odt .ods .odp | Medium — developer/note-taker format |
| Structured defs | .sql .ddl .yaml .yml .json .xml .graphql .proto | By content |
| Code files | .py .js .ts .go .rs .java etc. | No content analysis |

Office documents are the primary analysis target, not an edge case.

Must skip: node_modules, .git, __pycache__, .venv, venv, dist, build, site-packages, $RECYCLE.BIN.

### Rule 9: Project Structure Inference

- First-level directory = project grouping
- Second-level directory = specific project
- Project markers (README.md, package.json, .git) confirm boundaries
- Namespace convention: `core/`, `work/*`, `personal/*`, `external/*`, `uncategorized/*`

### Rule 10: Dual-Track Entity Discovery

Step 2 runs two tracks in parallel:

**Track A — Core Type Instantiation**: Extract instances of 8 core types (Person, Organization, Project, Task, Document, Event, Note, Goal) from document content.

**Track B — Domain Structure Discovery**: Cluster analysis to find naturally emerging domain knowledge structures. Not "leftovers from core types" — a parallel classification of how the user's professional knowledge is organized.

Domain type criteria (need 2+):
1. Appears in ≥3 files within the domain
2. Has describable attributes
3. Has nameable relationships with other domain concepts
4. Domain practitioners use this term daily (30-second rule)

Core types answer "what universal things exist in your world." Domain types answer "how is your professional knowledge organized."

### Rule 11: Smart Sampling

- Input is Step 1 index, not hardcoded filenames
- **Minimum 10% coverage** of total indexed documents
- Per-project scaling: ≤20 all, 21-100→30-50, 101-500→50-100, 501-2000→100-200, >2000→200-500
- Every subdirectory covered (2-3 files each)
- All file formats represented (don't skip .xlsx/.pptx)
- No time-based sorting; old docs can be more valuable
- Second round: high-value areas up to 30-50% coverage

### Rule 12: Cross-Project Concept Alignment

- Detection: similar attributes, similar relationship structure, overlapping descriptions
- Handling: mark as review_flag (type: merge), list candidates and evidence
- Never auto-merge: only flag, let user confirm

### Rule 13: Structured List Full Extraction

**Problem**: Sampling works for narrative documents (reports, memos) where information is distributed across paragraphs. But structured registries (client lists, asset inventories, contact directories) have zero redundancy — every row is a unique entity. Sampling these files loses data permanently.

**Detection** (two paths):

1. **Filename-based** (Phase 2.1, before reading): Match against patterns:
   - Chinese: 列表, 台账, 名单, 登记表, 花名册, 清单, 通讯录, 客户列表, 资产清册, 人员名册, 统计表
   - English: list, roster, directory, registry, inventory, catalog, manifest
   - Mark these for **mandatory full extraction** regardless of sampling quota

2. **Content-based** (Phase 2.2, during reading): If an opened file reveals:
   - Excel: Sequential numbered rows, each row = one entity with consistent columns
   - Word: Tables where each row has a name/company/person in the first column
   - Any format: Repeating structure where rows are independent entities, not parts of a narrative
   - → **Switch to full extraction mode mid-analysis**. Do not summarize. Extract every row.

**Extraction rules**:
- Read ALL sheets (Excel), ALL tables (Word)
- One entity per data row
- Map column headers to entity properties
- Preserve relationships encoded in the structure (e.g., "使用的产品" column → product usage relations)
- If >500 rows, batch into groups of 100 for processing

**Why this matters**: A real example — a client list Excel with 139 companies was sampled at 10%, yielding only 13 entities in the knowledge graph. The other 126 companies were invisible. Full extraction recovers 10x more entities from a single file at minimal additional cost.


path: references/constraints-and-inference.md
content: # Constraints and Inference

约束防止脏数据进入图谱，推理从已有事实中推导新知识。两者配合让知识图谱从"数据仓库"升级为"推理引擎"。

## 1. 类型约束

### 1.1 必需属性

每个 core type 的最低数据要求。缺少必需属性的实体不应写入 graph.jsonl。

| Type | 必需属性 | 说明 |
|------|----------|------|
| Person | `name` | 至少有姓名。无名人物用"某星辰集团员工"等占位，标 confidence=low |
| Organization | `name` | 至少有组织名 |
| Project | `name`, `status` | 项目必须有名称和状态（planning/active/paused/completed/archived） |
| Task | `title`, `status` | 任务必须有标题和状态（open/in_progress/blocked/done/cancelled） |
| Document | `title` | 至少有标题。path 可选（口述提到的文档可能无路径） |
| Event | `title`, `date` | 事件必须有日期，没有日期的归 Note 而非 Event |
| Note | `content` | 笔记的本体就是内容 |
| Goal | `description`, `status` | 目标必须有描述和状态（proposed/active/achieved/abandoned） |

### 1.2 属性值域约束

| Type.属性 | 允许值 | 违规处理 |
|-----------|--------|----------|
| Organization.type | company, department, team, institution, government, ngo, startup, other | 不在列表中 → 用 other + labels 补充 |
| Project.status | planning, active, paused, completed, archived | 不合法值 → review_flag |
| Task.status | open, in_progress, blocked, done, cancelled | 同上 |
| Task.priority | low, medium, high, critical | 同上 |
| Goal.status | proposed, active, achieved, abandoned | 同上 |
| Document.format | md, docx, pdf, xlsx, pptx, html, txt, sql, yaml, json, csv, doc, xls, ppt, other | 未知格式 → other |
| *.confidence | high, medium, low | 只有这三档 |

### 1.3 ID 格式约束

正则：`^[a-z][a-z0-9_]+-[0-9]{3,}$`

| Type | 前缀 | 示例 |
|------|------|------|
| Person | per- | per-00001（张三） |
| Organization | org- | org-00002（蓝天信息） |
| Project | project- | project-001 |
| Task | task- | task-00005 |
| Document | doc- | doc-00033 |
| Event | evt- | evt-00001 |
| Note | note- | note-00002 |
| Goal | goal- | goal-00006 |

ID 一旦分配不可回收。deprecated 实体的 ID 永久保留。

## 2. 关系约束

### 2.1 合法关系矩阵

标记哪些 source_type → target_type 的关系是合法的。未在矩阵中的关系写入时触发 review_flag。

| Source \ Target | Person | Org | Project | Task | Doc | Event | Note | Goal |
|-----------------|--------|-----|---------|------|-----|-------|------|------|
| **Person** | knows | works_at, leads, member_of | participates_in, manages | assigned_to(reverse) | authored(reverse) | has_participant(reverse) | authored_by(reverse) | owned_by(reverse) |
| **Org** | employs | subsidiary_of, partner_of, competitor_of, customer_of | contracted_by(reverse) | - | - | related_to(reverse) | about(reverse) | set_by(reverse) |
| **Project** | has_owner | contracted_by, delivered_to | - | has_task | has_document | - | - | - |
| **Task** | assigned_to | - | belongs_to | depends_on, blocks | produces | - | - | measured_by(reverse) |
| **Doc** | authored_by | - | belongs_to | - | references | - | extracted_from(reverse) | - |
| **Event** | has_participant | related_to | belongs_to | produces | - | - | - | - |
| **Note** | authored_by | about | about | about | extracted_from | about | - | - |
| **Goal** | owned_by | set_by | belongs_to | measured_by | - | driven_by | driven_by | - |

`-` 表示这对组合没有常见关系。如果扫描中发现了，生成 review_flag 让用户判断是否需要新增关系类型。

### 2.2 基数约束

| 关系 | 硬约束 | 理由 |
|------|--------|------|
| Person.works_at → Organization | 最多 3 个（end=null 的） | 兼职上限。超过 3 个当前有效的雇佣关系，大概率有脏数据 |
| Task.belongs_to → Project | 恰好 1 个 | 每个 Task 必须归属且只归属一个 Project |
| Document.authored_by → Person | 最多 5 个 | 联合作者上限。超过 5 个更可能是组织署名 → 建 Organization |
| Organization.subsidiary_of → Organization | 最多 1 个 | 一个组织只有一个直接上级。间接上级用推理得出 |
| Goal.measured_by → Task | 至少 1 个 | 没有 Task 衡量的 Goal 是空谈。标 review_flag |

### 2.3 自引用与循环约束

| 约束 | 检查方式 | 示例 |
|------|----------|------|
| 禁止自引用 | target_id != source_id | Organization 不能 subsidiary_of 自己 |
| subsidiary_of 无环 | 沿 subsidiary_of 追溯不能回到起点 | 蓝天信息 → 星辰集团国际 → 星辰集团 → ... 不能回到蓝天信息 |
| depends_on 无环 | Task 依赖链不能成环 | A depends_on B, B depends_on A → 检测到循环，review_flag |
| blocks 无环 | 同上 | 阻塞关系也不能成环 |
| references 允许环 | Document 可以互相引用 | 07号文档 references 技术底座方案，反过来也可以 |

## 3. 推理规则

从已有事实推导新知识。推理产出的关系标记 `source.type = "inferred"`，与扫描/手动数据区分。

### 规则 1: 传递性 — subsidiary_of

```
A subsidiary_of B, B subsidiary_of C
→ A indirectly_affiliated_with C
```

示例：蓝天信息 subsidiary_of 星辰集团国际, 星辰集团国际 subsidiary_of 星辰集团 → 蓝天信息 indirectly_affiliated_with 星辰集团

### 规则 2: 对称性 — partner_of / competitor_of / knows

```
A partner_of B → B partner_of A
A competitor_of B → B competitor_of A
A knows B → B knows A
```

写入时只存一条（direction=bidirectional），查询时双向返回。

### 规则 3: 逆关系 — works_at / employs

```
A works_at B → B employs A
A authored_by B → B authored A
A assigned_to B → B has_assignee A
```

不需要重复存储。query_graph.py 在查询 Organization 的关系时，自动反向查 Person.works_at。

### 规则 4: 继承推理 — 组织归属传递

```
A works_at B, B subsidiary_of C
→ A affiliated_with C
```

示例：张三 works_at 蓝天信息, 蓝天信息 subsidiary_of 星辰集团 → 张三 affiliated_with 星辰集团

注意：affiliated_with 比 works_at 弱——张三不直接为星辰集团工作，但属于星辰集团体系。

### 规则 5: 冲突检测 — 角色不一致

```
A works_at B (role=X) in doc-001
A works_at B (role=Y) in doc-002, where X != Y
→ review_flag: role_conflict
```

示例：某文档说李四是"总经理"，另一文档说是"法定代表人"。两个都可能对（一人多职），但需要人工确认，不自动合并。

### 规则 6: 时间推理 — 跳槽检测

```
A works_at B (end: 2023-06)
A works_at C (start: 2023-09)
→ infer: A 从 B 跳槽到 C (gap: 3个月)
```

如果 gap < 6 个月，高置信度。gap > 12 个月，可能中间还有其他经历，标 confidence=low。

### 规则 7: 项目归属推理

```
A authored doc-X, doc-X belongs_to Project-Y
→ A participates_in Project-Y (confidence=medium)
```

写了项目文档的人大概率参与了该项目。但不绝对——可能是外部审稿人。

### 规则 8: 孤立实体预警

```
Entity E has zero relations (no incoming, no outgoing)
→ review_flag: isolated_entity
```

孤立实体要么是数据不完整（缺少关系提取），要么是垃圾数据。定期检查清理。

## 4. 不一致检测清单

query_graph.py 应支持 `--check-consistency` 模式，检查以下项目：

| 检查项 | 检测方法 | 严重程度 |
|--------|----------|----------|
| 同名不同 ID | 按 name 分组，找 count > 1 的 | 高 — 可能是重复实体 |
| 同 ID 不同属性值 | 同一 ID 出现在多条 create/update 记录中，属性值矛盾 | 中 — 可能是正常更新 |
| 循环依赖 | 沿 subsidiary_of / depends_on / blocks 做 DFS 检测环 | 高 — 违反约束 |
| 孤立实体 | relations 为空且不被任何其他实体引用 | 低 — 可能只是数据不全 |
| 过期关系未标记 | temporal.end 日期已过但未标记 deprecated | 中 — 关系可能已失效 |
| ID 格式违规 | 不匹配 `^[a-z][a-z0-9_]+-[0-9]{3,}$` | 高 — 违反格式约束 |
| 基数超限 | Person.works_at(end=null) > 3 个 | 中 — 可能有脏数据 |
| 必需属性缺失 | Event 没有 date, Task 没有 status | 高 — 违反类型约束 |

**执行方式**：

```bash
python query_graph.py --check-consistency
# 输出: consistency_report.md，按严重程度排序
```

建议在每次 Step 2 完成后运行一次，在 graph.jsonl 追加 100+ 实体后再运行一次。


path: references/formats-and-deps.md
content: # Supported Formats and Dependencies

## File Format Support

| Format | Extensions | Processing | Dependencies |
|--------|-----------|------------|--------------|
| Word (new) | .docx | python-docx: text + tables + images | python-docx, Pillow |
| Word (legacy) | .doc | temp-copy + COM read (Win) / libreoffice (Linux) | pywin32 (Win) / libreoffice |
| WPS Office | .wps .et .dps | LibreOffice convert → standard format | libreoffice |
| Excel (new) | .xlsx | openpyxl: all sheets, headers, data | openpyxl |
| Excel (legacy) | .xls | xlrd | xlrd |
| PDF | .pdf | PyMuPDF: text + images (Read tool also works) | PyMuPDF (fitz) |
| Presentations | .pptx .ppt | python-pptx: slides + notes | python-pptx |
| Markdown/Text | .md .txt | Direct read | — |
| SQL/DDL | .sql .ddl | Direct read | — |
| JSON/YAML | .json .yaml .yml | Direct read | — |
| CSV/TSV | .csv .tsv | Direct read | — |
| XML | .xml | Direct read | — |
| RTF | .rtf | striprtf | striprtf (optional) |
| OpenDocument | .odt .ods .odp | odfpy or LibreOffice | odfpy (optional) |
| macOS Office | .pages .numbers .key | macOS iWork export only | macOS only |

### .xlsx/.pptx Special Notes

- **.xlsx**: Not just "data." May contain client lists, budgets, project tracking, KPI dashboards, personnel rosters. Iterate ALL sheets. Sheet names are domain signals.
- **.pptx**: Strategy summaries, client pitches, competitive analysis. Info density often higher than corresponding Word docs. Read ALL slides + speaker notes.
- Both formats have priority equal to .docx. Never deprioritize.

### .doc Extraction: Temp-Copy COM Pattern (Windows)

Direct COM opening fails when file paths contain CJK characters (garbled encoding in COM API). Solution:

```python
import shutil, tempfile, win32com.client, pythoncom

def extract_doc(filepath):
    tmp_dir = tempfile.mkdtemp(prefix='doc_')
    tmp_path = os.path.join(tmp_dir, f'doc_{seq}.doc')  # unique name per file
    shutil.copy2(filepath, tmp_path)

    pythoncom.CoInitialize()
    word = win32com.client.DispatchEx("Word.Application")
    word.Visible = False
    word.DisplayAlerts = 0

    doc = word.Documents.Open(str(Path(tmp_path).resolve()), ReadOnly=True)
    text = doc.Content.Text
    doc.Close(False)
    word.Quit()
    pythoncom.CoUninitialize()

    shutil.rmtree(tmp_dir, ignore_errors=True)
    return text
```

**Key points**:
- Use **unique temp file names** per document (avoid lock collisions — Word may not release the file immediately after `Close`)
- Keep ONE Word instance for batch processing (creating Word is slow, ~2s each)
- If a file causes Word to hang, restart the Word instance and continue
- `._` prefix files are macOS resource forks — skip them
- Tested: **99% success rate** on 500 CJK-path .doc files

### WPS Format Notes

`.wps` (doc), `.et` (spreadsheet), `.dps` (presentation) — WPS Office formats, common in China. Convert via LibreOffice `--convert-to` to standard formats, then extract normally.

### Image Processing

.docx/.pdf embedded images (architecture diagrams, ER diagrams, flowcharts):
- .docx: python-docx `document.inline_shapes` and `document.part.rels`
- .pdf: PyMuPDF per-page or per-image extraction
- Feed to multimodal LLM for visual analysis
- If LLM doesn't support images: note in review.md for manual review

## Python Dependencies

### Core (required)

```bash
pip install python-docx openpyxl xlrd PyMuPDF Pillow python-pptx pyyaml
```

| Package | Purpose |
|---------|---------|
| python-docx | .docx text/tables/images |
| openpyxl | .xlsx reading |
| xlrd | .xls (legacy) reading |
| PyMuPDF (fitz) | .pdf text/images |
| Pillow | Image processing |
| python-pptx | .pptx slides/notes |
| pyyaml | schema.yaml read/write |

### Platform-specific

```bash
# Windows .doc conversion (requires Microsoft Word installed)
pip install pywin32

# Linux/macOS .doc/.wps conversion
sudo apt install libreoffice  # or: brew install libreoffice
```

### Optional

```bash
pip install striprtf odfpy olefile
```

| Package | Purpose |
|---------|---------|
| striprtf | .rtf text extraction |
| odfpy | OpenDocument (.odt/.ods/.odp) |
| olefile | .doc OLE2 structure parsing (fallback when COM unavailable) |

### LLM Requirements

| Capability | Purpose | Minimum |
|-----------|---------|---------|
| Text understanding | Phase 2.2 semantic reading | Any long-context LLM |
| Multimodal image input | Phase 2.2 image analysis | Claude Opus/Sonnet, GPT-4o |
| Tool use | Phase 2.2 structured output | Recommended, not required |

Step 1 (scan_filesystem.py) uses only Python stdlib — no extra deps.


path: references/knowledge-graph-workflow.md
content: # Knowledge Graph Workflow (Mode B/C)

## Infrastructure

```
{ontology_home}/
├── schema.yaml      ← 8 core types + discovered domain types
├── graph.jsonl      ← All entity data (append-only, soft-partitioned)
└── scan_filesystem.py (copy from skill scripts/)
```

## Step 1 — File Indexing (automated script, no LLM)

Run `scripts/scan_filesystem.py`:

```bash
python scripts/scan_filesystem.py --root /path/to/scan
python scripts/scan_filesystem.py --root /path --config namespace_rules.yaml
python scripts/scan_filesystem.py --root /path --extract-metadata
python scripts/scan_filesystem.py --root /path --dry-run
```

Creates Document + Project entities. Pure mechanical operation. See references/script-operations.md.

## Step 1.5 — User Scope Confirmation (MANDATORY)

After Step 1, **ask the user to review and confirm the analysis scope** before proceeding.

### Why this step exists

The user knows their own files best. Blindly sampling wastes compute on low-value folders (downloads, reference archives, AI-generated output). A 30-second user confirmation saves hours of misguided analysis.

### Procedure

1. Summarize Step 1 results: total projects, total documents, format distribution
2. Present a table of all projects/namespaces sorted by document count
3. Auto-suggest priorities based on heuristics:
   - Human work folders (contracts, meeting notes, client correspondence) → 重点
   - AI-generated folders (generate-docx.py, batch output) → 普通
   - Downloads, caches, reference archives → 忽略
4. Ask the user to confirm or adjust:
   - **重点** (priority): 2-3x normal sampling rate
   - **普通** (normal): default sampling rate
   - **忽略** (skip): excluded from Step 2 entirely
   - **全部** (all): accept defaults, analyze everything — user explicitly chooses to skip prioritization
5. User can also flag specific subfolders (e.g., "DM6 in Downloads is email, treat as 重点")

### Rules

- **Never skip this step.** Even experienced users benefit from reviewing the scan before committing compute.
- Respect "全部" — if the user wants everything analyzed, don't argue. Just apply default sampling rates.
- Store user choices in a `scope_config.json` alongside graph.jsonl for reproducibility.
- If the scan is small (<500 docs), suggest "全部" as default but still ask.

## Step 2 — Semantic Analysis (LLM, core step)

### Phase 2.1: Sampling

Minimum **10% coverage** of non-ignored documents (per Step 1.5 scope). Two rounds: wide → deep.

**Wide sampling**: Cover all non-ignored projects, all subdirectories, all formats. Per-project amounts scale with size (see Rule 11 in analysis-rules.md). 重点 projects get 2-3x sampling.

**Coverage validation**: After sampling, verify every non-ignored Project and every major subdirectory has representation. Fill gaps.

**Deep sampling**: After Phase 2.2, increase to 30-50% for high-value areas (≥10 named entities found, multiple formats coexist, new domain term clusters discovered, client/project names in path).

Stop when 5 consecutive docs yield no new entities or terms.

**Structured List Detection (Rule 13 — MANDATORY)**:

Before sampling, scan Step 1 index for files whose names match structured list patterns (列表/台账/名单/登记表/花名册/清单/通讯录/客户列表/list/roster/registry/inventory/catalog). These files are **exempt from sampling** — they go directly to Phase 2.2 for full extraction (every row, every sheet). Rationale: structured registries have zero information redundancy; sampling them loses entities permanently.

### Phase 2.2: LLM Document Reading

#### Pre-check: Project Environment Validation (MANDATORY)

Before reading any project's documents:

1. Read project context files (CLAUDE.md, README.md, etc.) if they exist
2. Determine document origin:
   - References to `generate-docx.py`, `python-docx`, batch generation → **AI-generated, deprioritize**
   - Timestamps clustered on same day, numbered filenames, similar sizes → batch generation signals
   - Contracts, meeting notes, handmade Excel, client correspondence → **human-written, highest priority**
3. Spend analysis budget on human-written work documents, not AI output

#### Pre-check: Directory Structure Analysis (free information)

File paths reveal domain structure without reading content:
- `工作记录/某部委项目/` → client project exists
- `Desktop/学习资料/` → personal learning area

#### Reading Office/PDF Files (MANDATORY, not optional)

Most users' core work is in Office formats. These are the PRIMARY target.

| Format | Method |
|--------|--------|
| .docx | python-docx: full text + all tables + embedded images |
| .doc/.wps | Convert via pywin32/LibreOffice → .docx → extract |
| .pdf | Read tool (direct) or PyMuPDF for text + images |
| .xlsx/.xls/.et | openpyxl/xlrd: **all sheets**, headers, first N rows. Sheet names are domain signals. Contains client lists, budgets, KPI tracking. |
| .pptx/.ppt/.dps | python-pptx: **all slides** + notes. Strategy summaries, client pitches. Info density often higher than Word. |
| .md/.txt | Read tool directly |

**Subagents must be `general-purpose` type** (with Bash). Never use `Explore` type for Phase 2.2.

**Never skip a file because it's binary.** Extract text with tools, then analyze.

#### Per-Document LLM Analysis

For each document, first determine its **document class**:

| Class | Examples | Extraction Mode |
|-------|----------|----------------|
| **Structured list** | Client lists, asset registers, contact directories, inventories, ledgers | **Full extraction**: every row = one entity, all columns = properties |
| **Narrative document** | Reports, memos, proposals, meeting notes, emails | **Semantic extraction**: summary + Track A + Track B (below) |

**If structured list** (Rule 13): Do NOT summarize. Read all sheets/tables, extract every data row as an independent entity. Map column headers to properties. This is non-negotiable — a 139-row client list must produce ~139 entities, not a 3-sentence summary.

**If narrative document**, extract:

**A. Understanding**: 1-2 sentence summary, domain, document type, knowledge density rating. High density → flag for deep sampling.

**B. Track A (Core Instances)**: Named persons (with role), organizations (with type), events (with dates), goals (with metrics), tasks, notes.

**C. Track B (Domain Terms)**: Professional vocabulary (no quantity limit), term relationships, domain classification (let it emerge, don't predefine).

**Content-based reclassification**: If you open a file expecting narrative content but discover it's actually a structured list (e.g., a .docx that's entirely tables of names), switch to full extraction mode immediately. Don't force narrative analysis on tabular data.

#### Image Analysis

.docx/.pdf embedded images (architecture diagrams, ER diagrams, flowcharts) → multimodal LLM visual analysis. Extract nodes and relationships. If LLM doesn't support images, note in review.md.

#### Large Document Chunking

>50KB text: split by sections, analyze each, merge in Phase 2.3.

#### Parallel Processing

>50 docs: split by Project, one general-purpose subagent per Project. Main agent aggregates in Phase 2.3.

### Phase 2.3: Aggregation

**Track A**: Deduplicate persons/organizations across documents. Same name + consistent context → merge, add labels for all projects. Unclear → review_flag.

**Track B**: Aggregate all term annotations → cluster by co-occurrence and semantic similarity → identify domain centers → name domains → identify core types within each domain → map inter-type relationships.

**Domains emerge from data, not predefined.**

### Phase 2.4: Cross-Project Alignment (Rule 12)

### Phase 2.5: Output

| Output | Target |
|--------|--------|
| Core type instances | graph.jsonl |
| Domain definitions | schema.yaml → domain_types |
| Domain type definitions | schema.yaml → domain_types → types |
| Domain instances | graph.jsonl |
| Review report | review.md |

## Step 3 — Runtime Evolution

Agent enriches the knowledge graph during daily conversations. `source.type = "runtime"`.

### Trigger Conditions

The agent should check and potentially update the graph when the user:
- Mentions a person with role/org context (e.g., "跟张三开了个会，他是XX的产品经理")
- Discusses events with dates (e.g., "上周五某IT系统项目通过验收了")
- Makes strategic decisions or key observations worth preserving
- Mentions new organizations, clients, or partners

### Procedure

1. **Check**: Query graph for existing entity (`python query_graph.py search "name"`)
2. **Compare**: If found, verify info is consistent. If new info, note but don't overwrite.
3. **Append**: If not found, create new entity with `source.type = "runtime"` and conversation context.

### Rules

- **Lightweight**: 1-3 entities per conversation max. This is passive enrichment, not a re-scan.
- **Evidence-based**: Only append what the user explicitly stated. Don't infer or guess.
- **Append-only**: Never modify or delete scan-derived entities. Runtime entities have lower authority than scan-derived ones.
- **Conflict handling**: If runtime info contradicts scan-derived data, note the conflict in a Note entity rather than overwriting.

## Mode C: External Data Scanning

Same as Mode B, but: source.target = "external", namespace = `external/{id}`, no prior knowledge assumed, can combine with Mode A for database schemas.

## Future Data Sources

| Source | Adapter | source.type |
|--------|---------|-------------|
| Email (Outlook/Gmail) | .ost/.pst or IMAP/API | email |
| Cloud drives | Sync directory or API | cloud |
| Chat records (WeChat/Slack) | Export or API | chat |
| Database | DDL/data dictionary (Mode A) | database |

## Backup and Migration

Everything is files. Copy directory, git commit, or cloud sync. graph.jsonl is append-only plain text. Filter by `graph` field to export subsets.


path: references/modeling-decisions.md
content: # Ontology Modeling Decision Guide

## Is this table a business entity?

```
Q1: Do business people mention this concept daily?
  NO  → EXCLUDE (technical: logs, audit, config, sequences)
  YES → Q2

Q2: Does it have an independent lifecycle (create → use → archive)?
  NO  → Likely ENUM (dictionary/lookup) or LINK (association)
  YES → Q3

Q3: Can it exist independently (own PK and business meaning)?
  NO  → Likely sub-table or LINK
  YES → ENTITY
```

## ENUM vs ENTITY

```
>5 business-meaningful attributes?
  NO  → ENUM (e.g., gender table with 3 fields)
  YES → Consider ENTITY (e.g., country table with visa rules, 17 fields)

Referenced by >3 other entities?
  YES → Consider promoting to ENTITY
  NO  → Keep as ENUM
```

## Promote property to ENTITY? (Stanford Ontology 101)

```
Does this field have its own attributes to describe?
  NO  → Keep as property
  YES → continue

Do multiple entities reference this concept?
  NO  → Keep as property (local concern)
  YES → PROMOTE, create review_flag
```

## Cross-system merge

```
Same table name in different systems?
  YES → Auto-merge, keep version with more complete attributes

Similar Chinese name (>80%) AND field overlap (>60%)?
  YES → review_flag type=merge, do NOT auto-merge

Different name but same business concept?
  → review_flag, note shared FKs / similar fields
```

## Subagent Architecture (large projects)

For >100 files, parallelize with Claude Code Task tool:

- **Scanner subagent** (Explore): Run scan_directory.py, recommend batches
- **Extractor subagent(s)** (general-purpose): Process file batches, run extraction scripts, apply rules
- **Merger subagent** (general-purpose): Combine fragments, deduplicate, generate outputs

## Incremental Mode

If extraction interrupted or adding files:
1. Check existing extracted_tables.json
2. Process only new/unprocessed files
3. Merge with existing
4. Re-run Phase 3 on combined data

## Core Types 设计理由与边界定义

8 个 core types 对齐 BFO (Basic Formal Ontology) 的两大范畴：

- **Continuant（持续体）**：在时间中持续存在、可被反复引用的实体
- **Occurrent（发生体）**：占据时间区间、有起止的过程或事件

| Type | BFO 范畴 | 理由 |
|------|----------|------|
| Person | Continuant | 人在时间中持续存在，属性可变但身份不变 |
| Organization | Continuant | 组织持续存在，成员和结构可变 |
| Project | Continuant | 项目是长期存在的工作容器，状态可变 |
| Document | Continuant | 文档是持久化的信息载体，有文件路径 |
| Goal | Continuant | 目标是持续存在的期望状态描述 |
| Note | Continuant | 知识片段一旦创建即持续存在 |
| Task | Occurrent | 任务是有起止的工作过程 |
| Event | Occurrent | 事件是有时间定位的发生 |

### Person
人物实体。同事、客户联系人、行业人物、私人关系人。
- **算**：有姓名的自然人，如 `[per-00001] 张三`、`[per-00009] 李四`
- **不算**：岗位角色（"客户经理"是 Person 的 role 属性，不是独立实体）；团队（归 Organization）
- **与 Organization 的边界**：Person 通过 works_at 关系挂在 Organization 下，不内嵌

### Organization
组织实体。公司、部门、团队、政府机构、行业协会。
- **算**：有独立名称和运作结构的组织体，如 `[org-00002] 蓝天信息技术有限公司`、`[org-00013] 某部委`
- **不算**：临时工作小组（归 Project 的参与者）；一个人的工作室（除非对外有独立品牌）
- **层级处理**：用 subsidiary_of / parent_of 关系表达层级，如星辰集团 → 星辰集团国际 → 蓝天信息

### Project
长期工作容器。有明确目标、多个参与者、跨越一段时间。
- **算**：某战略项目文档体系、某IT系统项目、某AI产品开发，如 `[task-00005]` 对应的整体项目
- **不算**：一次性的具体工作（归 Task）；一场会议（归 Event）
- **与 Task 的边界**：Project 是"装 Task 的容器"。Project 没有单一 assignee，Task 有。Project 的完成取决于其下 Task 的完成。如"某IT系统项目"是 Project，"外派周八到某地"是其下的 Task

### Task
具体可交付的工作项。有明确的执行者、截止时间、完成标准。
- **算**：`[task-00001] 某IT系统项目实施`、`[task-00004] 某系统升级方案`
- **不算**：长期的战略方向（归 Goal）；已经结束的事情的记录（归 Event）
- **与 Event 的边界**：Task 面向未来或进行中，有 status（open/in_progress/done）。Event 面向过去，有 date。"编写升级方案"是 Task，"方案评审会（2025-06-01）"是 Event。Task 完成后不变成 Event——它的 status 变为 done

### Document
有文件路径的信息载体。实际存在于文件系统中的文件。
- **算**：`[doc-00001]` 以上所有有 path 属性的实体，Word/PDF/Excel/MD 文件
- **不算**：从文档中提取的知识片段（归 Note）；没有对应文件的口头信息
- **与 Note 的边界**：Document 有 `path` 属性指向文件系统中的真实文件。Note 是从 Document（或对话、会议）中提取的语义片段，没有独立文件路径。一个 Document 可以产出多个 Note。如 `[doc-00033] 项目需求规格书V2` 是 Document，从中提取的"系统需支持多语言"是 Note

### Event
已发生或已确定时间的事件。时间是必需属性。
- **算**：`[evt-00001] 2020年9月蓝天信息并入星辰集团国际`、`[evt-00013] 赴某地某IT系统项目现场实施`
- **不算**：正在进行且无确定结束时间的工作（归 Task）；周期性例行会议（除非某次会议产生了重要决定）
- **与 Task 的边界**：Event 回答"发生了什么"（what happened），Task 回答"要做什么"（what to do）。Event 有确切 date，Task 有 deadline（可能未定）。Event 是点或短区间，Task 是过程

### Note
非结构化知识片段。洞察、事实、观察、总结。
- **算**：`[note-00002] 业务占比:客户A占45%，客户B占35%，客户C占20%`、`[note-00004] 核心劣势:...`
- **不算**：有可度量结果的目标（归 Goal）；有时间和地点的事件（归 Event）
- **与 Goal 的边界**：Note 是对现状的描述或洞察，Goal 是对未来的期望。"蓝天信息市场占有率本地100%、国内5%"是 Note（现状事实），"将国内份额提升到15%"是 Goal（期望状态）

### Goal
可度量的目标状态。有期望达成的结果和判断标准。
- **算**：`[goal-00001] 巩固提升核心业务`、`[goal-00006] 立足本地市场、辐射周边地区`
- **不算**：模糊的愿景（"成为行业领先"太虚，除非配有量化指标）；具体的执行步骤（归 Task）
- **与 Task 的边界**：Goal 回答"要达到什么状态"，Task 回答"具体怎么做"。Goal 通过 measured_by 关系关联 Task，用 Task 的完成来衡量 Goal 的达成


path: references/ontology-evolution.md
content: # Ontology Evolution

知识图谱不是一次性建好的静态模型。实体会被重新分类，schema 会增加字段，多次扫描会产生冲突。本文定义 schema 版本控制、实体重分类、冲突处理和向后兼容的机制。

## 1. Schema 版本控制

schema.yaml 顶部已有 `meta.version`。版本控制规则：

```yaml
meta:
  version: "1.1"          # 语义化版本：major.minor
  last_updated: "2026-03-17"
  changelog:
    - version: "1.1"
      date: "2026-03-17"
      changes: "新增关系语义属性(temporal/confidence/evidence)"
    - version: "1.0"
      date: "2026-03-16"
      changes: "初始版本，8 core types + 10 domain types"
```

**版本递增规则**：
- **minor**（1.0 → 1.1）：新增可选字段、新增 domain type、新增关系类型。旧数据不需迁移
- **major**（1.x → 2.0）：修改 core type 定义、删除字段、修改 ID 格式。旧数据需要迁移脚本

**graph.jsonl 中的版本标记**：

每个实体的 source 中记录写入时的 schema 版本：

```json
{
  "id": "per-00001",
  "source": {
    "type": "scan",
    "scan_id": "scan-20260316-step2",
    "schema_version": "1.0"
  }
}
```

查询时可按 schema_version 过滤，定位需要迁移的旧实体。

## 2. 实体重分类

场景：扫描时把"2020年9月蓝天信息并入星辰集团国际"记为 Note，复查后发现应该是 Event。

**操作流程**：

```jsonl
// 1. 创建新 Event 实体
{"op":"create","ts":"2026-03-17T10:00:00Z","entity":{"id":"evt-00164","type":"Event","properties":{"title":"蓝天信息以研发中心形式并入星辰集团国际","date":"2020-09"},"source":{"type":"manual","reason":"reclassify from note-00007","schema_version":"1.1"}}}

// 2. 标记旧 Note 为 deprecated
{"op":"update","ts":"2026-03-17T10:00:01Z","entity_id":"note-00007","changes":{"deprecated_by":"evt-00164","deprecated_at":"2026-03-17","deprecated_reason":"reclassified as Event"}}
```

**核心原则**：
- **append-only**：不删除旧实体，只追加 deprecated 标记
- **双向链接**：新实体的 source.reason 引用旧 ID，旧实体的 deprecated_by 引用新 ID
- **query_graph.py 默认行为**：排除带 deprecated_by 字段的实体。加 `--include-deprecated` 参数可查看全量

**批量重分类**：

Step 2 完成后发现一批 Note 应该是 Event（都有明确日期），用脚本批量处理：

```python
# 伪代码：筛选有 date 属性的 Note，逐条重分类
for note in notes_with_date_property:
    create_event(from_note=note)
    mark_deprecated(note, new_id=event.id)
```

## 3. Schema 合并冲突

两次扫描可能对同一概念给出不同定义。

**场景举例**：
- 第一次扫描从邮件中提取 `org-00002 蓝天信息`，type 记为 "IT企业"
- 第二次扫描从合同文档中提取同名实体，type 记为 "研发中心"（并入星辰集团国际后的角色）

**处理策略**：

| 冲突类型 | 策略 | 示例 |
|----------|------|------|
| 属性值矛盾 | 保留两个版本，标记 `review_flag` | 蓝天信息的 type 到底是"IT企业"还是"研发中心"？两个都对——不同时期不同角色 |
| 同名不同实体 | 分别保留，用 ID 区分 | 两个"赵六"——per-00006 是蓝天信息的，per-00XXX 是星辰总部的 |
| 不同名同一实体 | 合并为一条，保留别名 | "蓝天信息" = "蓝天信息" = "蓝天信息"，aliases 数组保存所有名称变体 |

**合并规则**：
- **属性取并集**：A 有 email 但没 phone，B 有 phone 但没 email → 合并后两个都有
- **约束取交集**：A 说 Organization.type 可选 5 种，B 说可选 3 种 → 合并后只保留 A、B 都认可的
- **冲突属性不自动覆盖**：值矛盾时两个都保留，生成 review_flag 等用户裁决

**review_flag 格式**：

```json
{
  "type": "merge_conflict",
  "entity_id": "org-00002",
  "field": "type",
  "values": ["IT企业", "研发中心"],
  "sources": ["scan-20260316-step2-email", "scan-20260317-step2-contracts"],
  "suggestion": "两个都对，建议用 labels 同时标记"
}
```

## 4. 向后兼容

新版本 schema 不能让旧数据失效。

**规则**：

| 变更类型 | 兼容性 | 处理方式 |
|----------|--------|----------|
| 新增可选字段 | 向后兼容 | 旧实体中该字段为 null，query 时视为"未知" |
| 修改字段名 | 不兼容 | 必须递增 major 版本，提供迁移脚本 |
| 新增关系属性（temporal/confidence/evidence） | 向后兼容 | 旧关系只有 type + target_id 继续有效 |
| 新增 core type | 不兼容 | 8 个 core types 是固定的，不应新增 |
| 新增 domain type | 向后兼容 | 直接追加到 schema.yaml 的 domain_types |
| 收紧属性值域（如 type_enum 减少选项） | 不兼容 | 需要检查旧数据中是否有被删除的选项 |

**实际操作**：

关系新增 temporal/confidence/evidence 字段（1.0 → 1.1）：
- 旧关系 `{"type":"works_at","target_id":"org-00002"}` 继续合法
- 读取时：temporal=null（时间未知）、confidence=medium（默认中等）、evidence=null（来源未知）
- 不需要回填旧数据，除非后续扫描发现了时间或来源信息

**迁移脚本约定**：

如果发生 major 版本变更，在 `scripts/` 下创建 `migrate_vX_to_vY.py`：
- 读取 graph.jsonl 全量
- 按新 schema 转换每条记录
- 输出新的 graph.jsonl（旧文件重命名为 graph.jsonl.v1.bak）
- 更新所有实体的 source.schema_version


path: references/quality-checks.md
content: # Quality Self-Check and Error Detection

After each phase, perform checks before proceeding. Fix errors immediately.

## Phase 1 (SCAN) Checks

| Check | How | Common Error | Fix |
|-------|-----|-------------|-----|
| macOS resource forks excluded | No `._*` in scan_result.json | Double-counted files | Add `._` filter |
| Priority classification accuracy | Sample 10 P7 files | System-named Excel files misclassified | Promote to P2 manually |
| Extension coverage | Check by_extension stats | .xls missed (case) | `.lower()` on comparison |
| File count sanity | Compare against `find . -type f` | Path encoding on Windows CJK | Use `pathlib.Path` |

## Phase 2 (EXTRACT) Checks

| Check | How | Common Error | Fix |
|-------|-----|-------------|-----|
| Column role accuracy | Print 5 sample tables' column_roles | "字段描述" → field_name instead of field_name_cn | Longest-match-wins strategy |
| Table name extraction | Count `table_name: null` | Name in merged first row | Check merged rows |
| Schema detection rate | schema_tables / total_tables | Below 30% | Tighten `is_schema_header()` |
| Large file handling | Check `truncated: true` flags | Data lost | Re-extract with higher limits |
| .doc conversion | Check for FAILED entries | COM failure | Skip, try LibreOffice fallback |

### Column Role Detection Pitfalls

```
"字段描述" matches both "字段" (field_name) and "描述" (description)
→ Use longest-keyword-match-wins

"数据类型" matches "数据" and "类型"
→ Add "数据类型" as specific keyword for field_type

"是否主键" matches "是否" (nullable) and "主键"
→ Add compound keyword to primary_key
```

Always spot-check 5 random tables after extraction.

## Phase 3 (MERGE) Checks

| Check | How | Common Error | Fix |
|-------|-----|-------------|-----|
| Entity count plausibility | Compare against schema tables | Too few = aggressive exclusion | Review EXCLUDE decisions |
| Cross-system duplicates | Search similar Chinese names | Same concept counted separately | Create merge review_flag |
| Relationship completeness | Each entity has ≥1 relationship | Orphan entities | Check for missed FKs |
| Property deduplication | Same property twice in one entity | Multi-source extraction | Keep stronger evidence_grade |
| JSON validity | `python -c "import json; json.load(...)"` | Syntax errors | Validate before claiming done |

## Error Correction Workflow

```
1. IDENTIFY: What exactly is wrong? (specific example)
2. DIAGNOSE: Why? (root cause in script or analysis)
3. FIX: Modify script/approach
4. VERIFY: Re-run and confirm
5. PROPAGATE: Same error type elsewhere?
6. DOCUMENT: Note in review.md "Data Quality Notes"
```

Do NOT silently work around errors or wait until the end.

## Common Error Patterns

| Error | Root Cause | Fix |
|-------|-----------|-----|
| Column role mismatch | Ambiguous keywords | Longest-match + compound keywords |
| macOS resource forks | `._*` not filtered | Add prefix filter |
| Excel data dict misclassified | Filename is system name | Promote P7 Excel manually |
| Console encoding crash (Win) | GBK codec | `sys.stdout = TextIOWrapper(buffer, encoding='utf-8')` |
| .doc conversion timeout | Large/corrupted file | Set timeout, skip, log |
| Merged cell table names | Title in merged row | Check identical-cell rows |


path: references/relation-ontology.md
content: # Relation Ontology

关系是知识图谱的骨架。当前关系只有 `type` + `target_id`，语义太薄。本文定义关系的完整语义结构、核心关系类型目录、以及三元关系的处理策略。

## 1. 关系的语义属性

扩展后的关系结构：

```json
{
  "type": "works_at",
  "target_id": "org-00002",
  "direction": "forward",
  "cardinality": "N:1",
  "temporal": {"start": "2019-01", "end": null},
  "evidence": "doc-00123",
  "confidence": "high"
}
```

各属性语义：

| 属性 | 值域 | 语义 | 默认值 |
|------|------|------|--------|
| `type` | 关系类型名 | 关系的语义标签，英文动词或动词短语 | (必需) |
| `target_id` | 实体 ID | 关系指向的目标实体 | (必需) |
| `direction` | forward / reverse / bidirectional | forward: 主语→宾语（张三 works_at 蓝天信息）；reverse: 宾语→主语（从 target 看回来）；bidirectional: 双向对等（partner_of） | forward |
| `cardinality` | 1:1 / 1:N / N:1 / N:M | 从源实体到目标实体的基数约束。N:1 表示多个源可指向同一目标（多人可 works_at 同一公司） | N:M |
| `temporal` | `{start, end}` | 关系有效期。start/end 为 ISO 日期或 null。null start = 起始不详，null end = 当前有效 | null（无时间信息） |
| `evidence` | 实体 ID | 支撑该关系的来源实体（通常是 Document 或 Event），用于溯源 | null |
| `confidence` | high / medium / low | 关系可信度。high = 文档明确记载；medium = 从上下文推断；low = 间接证据或过时信息 | medium |

**向后兼容**：所有新增字段均可选（null = 未知）。旧数据中只有 type + target_id 的关系继续有效，视为 direction=forward, confidence=medium, 其余 null。

**示例**（真实数据）：

```json
// per-00001 张三 在蓝天信息工作
{
  "type": "works_at",
  "target_id": "org-00002",
  "direction": "forward",
  "cardinality": "N:1",
  "temporal": {"start": null, "end": null},
  "confidence": "high"
}

// org-00002 蓝天信息 是星辰集团国际的子公司
{
  "type": "subsidiary_of",
  "target_id": "org-00001",
  "direction": "forward",
  "cardinality": "N:1",
  "temporal": {"start": "2020-09", "end": null},
  "evidence": "evt-00001",
  "confidence": "high"
}
```

## 2. 核心关系类型目录

按源实体类型分组，列出最常见的关系。

### Person 相关

| 关系 | 方向 | 目标 | 基数 | 时间性 | 示例 |
|------|------|------|------|--------|------|
| `works_at` | Person → Org | Organization | N:1 | 有（在职期间） | 张三 works_at 蓝天信息 |
| `leads` | Person → Org | Organization | N:1 | 有 | 李四 leads 蓝天信息 |
| `member_of` | Person → Org | Organization | N:M | 有 | 孙九 member_of 市场业务部 |
| `participates_in` | Person → Project | Project | N:M | 有 | 陈七 participates_in 某IT系统项目 |
| `manages` | Person → Project | Project | 1:N | 有 | 王五 manages 某升级项目 |
| `authored` | Person → Document | Document | 1:N | 无 | 张三 authored 某战略项目战略报告 |
| `knows` | Person → Person | Person | N:M | 无（双向） | 同事/合作关系 |

### Organization 相关

| 关系 | 方向 | 目标 | 基数 | 时间性 | 示例 |
|------|------|------|------|--------|------|
| `subsidiary_of` | Org → Org | Organization | N:1 | 有 | 蓝天信息 subsidiary_of 星辰集团 |
| `partner_of` | Org ↔ Org | Organization | N:M | 有（双向） | 蓝天信息 partner_of 某机场 |
| `competitor_of` | Org ↔ Org | Organization | N:M | 无（双向） | 蓝天信息 competitor_of CompetitorA |
| `customer_of` | Org → Org | Organization | N:M | 有 | 某航空公司 customer_of 蓝天信息 |
| `regulates` | Org → Org | Organization | 1:N | 无 | IATA regulates 星辰集团 |

### Project 相关

| 关系 | 方向 | 目标 | 基数 | 时间性 | 示例 |
|------|------|------|------|--------|------|
| `contracted_by` | Project → Org | Organization | N:1 | 有 | 某IT系统项目 contracted_by 某国际机场 |
| `delivered_to` | Project → Org | Organization | N:M | 有 | 某AI产品 delivered_to 中小企业客户群 |
| `uses_methodology` | Project → Methodology | domain type | N:M | 无 | 某战略项目 uses_methodology BLM |

### Task 相关

| 关系 | 方向 | 目标 | 基数 | 时间性 | 示例 |
|------|------|------|------|--------|------|
| `belongs_to` | Task → Project | Project | N:1 | 无 | 升级方案编制 belongs_to 某升级项目 |
| `assigned_to` | Task → Person | Person | N:1 | 有 | 某IT系统实施 assigned_to 周八 |
| `depends_on` | Task → Task | Task | N:M | 无 | 需求文档 depends_on 调研完成 |
| `blocks` | Task → Task | Task | N:M | 无 | 审批未过 blocks 开发启动 |

### Event 相关

| 关系 | 方向 | 目标 | 基数 | 时间性 | 示例 |
|------|------|------|------|--------|------|
| `related_to` | Event → Project/Org | any | N:M | 无 | 蓝天信息并入星辰集团国际 related_to 星辰集团国际 |
| `has_participant` | Event → Person | Person | N:M | 无 | 某行业大会 has_participant 张三 |
| `produces` | Event → Task/Doc/Note | any | 1:N | 无 | 评审会 produces 修改任务 |

### Document 相关

| 关系 | 方向 | 目标 | 基数 | 时间性 | 示例 |
|------|------|------|------|--------|------|
| `belongs_to` | Doc → Project | Project | N:1 | 无 | 战略报告 belongs_to 某战略项目 |
| `authored_by` | Doc → Person | Person | N:1 | 无 | 战略报告 authored_by 张三 |
| `references` | Doc → Doc | Document | N:M | 无 | 07号文档 references 技术底座方案 |
| `describes` | Doc → any | any | N:M | 无 | 公司介绍 describes 蓝天信息 |

### Goal 相关

| 关系 | 方向 | 目标 | 基数 | 时间性 | 示例 |
|------|------|------|------|--------|------|
| `set_by` | Goal → Org/Project | Org or Project | N:1 | 有 | 国际化目标 set_by 蓝天信息 |
| `owned_by` | Goal → Person | Person | N:1 | 有 | AI内训目标 owned_by 张三 |
| `measured_by` | Goal → Task | Task | 1:N | 无 | 份额提升 measured_by 客户拓展任务 |
| `driven_by` | Goal → Note/Event | Note or Event | N:M | 无 | 转型目标 driven_by 市场现状分析 |

### Note 相关

| 关系 | 方向 | 目标 | 基数 | 时间性 | 示例 |
|------|------|------|------|--------|------|
| `about` | Note → any | any | N:M | 无 | 业务占比数据 about 蓝天信息 |
| `extracted_from` | Note → Document | Document | N:1 | 无 | 核心劣势 extracted_from 公司介绍文档 |
| `authored_by` | Note → Person | Person | N:1 | 无 | 洞察 authored_by 张三 |

## 3. 三元关系处理

有些关系涉及第三方。例如：蓝天信息通过某IT系统合同向某机场提供IT服务。

**策略一：拆成二元关系 + 共享实体**

将三元关系 (A, 通过C, 服务B) 拆为：
- A → C: `signs`（蓝天信息签订 某IT系统合同）
- C → B: `serves`（合同服务于某机场）
- A → B: `delivered_to`（蓝天信息交付给某机场）

中间实体 C（合同/协议）可建模为 Document 或 Event。

**策略二：via 属性**

关系上加 `via` 字段，适合不想为中间实体单独建模的场景：

```json
{
  "type": "serves",
  "target_id": "org-00005",
  "via": "evt-00007",
  "confidence": "high"
}
```

含义：蓝天信息 serves 某国际机场，via 某IT系统合同签订事件。

**选择标准**：
- 中间实体有独立属性需要记录（金额、日期、条款）→ 策略一，建独立实体
- 中间实体只是关系的背景信息 → 策略二，用 via 属性
- 拿不准 → 先用策略二，后续需要时再升级为策略一

**示例**（真实场景）：

```json
// 策略二：张三通过某战略项目为蓝天信息制定AI战略
// per-00001 的 relations 中：
{
  "type": "advises",
  "target_id": "org-00002",
  "via": "task-00005",
  "confidence": "high"
}

// 策略一：蓝天信息通过外派协议为星辰集团国际提供某IT系统实施
// 外派协议建为 doc-NNNNN：
// org-00002 → doc-NNNNN: signs
// doc-NNNNN → org-00024: serves
// org-00002 → org-00024: delivered_to
```


path: references/script-operations.md
content: # Script Operations

## scan_filesystem.py (Mode B/C)

**Input**: Directory path + optional config
**Output**: graph.jsonl (Document + Project entities)

```bash
python scripts/scan_filesystem.py --root <dir> [options]

Options:
  --config <yaml>        Custom namespace rules
  --extract-metadata     Extract author/title from .docx/.pdf
  --dry-run              Stats only, no write
  --output <path>        Custom output path
  --no-dedup             Disable duplicate detection
```

Features:
- Auto-infers namespaces from directory structure (or reads config YAML)
- Skips dependencies, caches, system files via universal patterns
- Lightweight dedup (same filename + same size)
- Optional .docx/.pdf metadata extraction (author/title/subject)
- Cross-platform (Python stdlib only for core; python-docx/PyMuPDF for metadata)

Config format (namespace_rules.yaml, optional):
```yaml
namespace_map:
  MyProjects: work
  Downloads: external/downloads
expand_children:
  - MyProjects
```

Without config: namespaces auto-inferred from project markers (README, package.json, .git).

## scan_directory.py (Mode A)

**Input**: Project root directory
**Output**: scan_result.json (file manifest with P1-P7 priority)

```bash
python scripts/scan_directory.py "<dir>" --output scan_result.json --report
```

Priority classification patterns:
- P1: 数据库设计, 表结构, DB Design, 数据模型, ER图
- P2: 数据字典, 字段定义, 系统清单, 代码表
- P3: 接口说明, API Spec, 报文, 消息格式
- P4-P7: Requirements, architecture, data flow, other

## convert_doc.py (Mode A)

**Input**: Directory or single .doc file
**Output**: .docx files in output directory

```bash
python scripts/convert_doc.py "<input>" --output-dir "<output>" --max-files 100 --scan-result scan_result.json
```

- Windows: COM automation (requires Word)
- Linux: `libreoffice --headless --convert-to docx`
- Sorts by priority if scan_result provided
- Skips already-converted files

## extract_tables.py (Mode A)

**Input**: File or directory + optional scan_result.json
**Output**: extracted_tables.json

```bash
python scripts/extract_tables.py "<input>" --output extracted_tables.json --scan-result scan_result.json
```

Handles .docx (python-docx), .xlsx (openpyxl), .xls (xlrd). Column role detection with longest-keyword-match. Size strategies: full (<5MB), selective (5-20MB), table_priority (20-50MB), sampling (>50MB).

## Word Table Column Role Keywords

```
field_name:    字段名, 字段, 列名, column, field, 英文名
field_name_cn: 中文名, 字段中文, 中文描述, 业务名称
field_type:    类型, 数据类型, type, data type
field_length:  长度, length, size, 精度
nullable:      是否为空, nullable, 是否必填
primary_key:   主键, pk, primary
foreign_key:   外键, fk, foreign, 关联
default_value: 默认值, default, 缺省值
description:   说明, 描述, 备注, comment, 含义, 业务含义
enum_values:   枚举值, 取值范围, 值域, 可选值
```

Table name extraction: Look in preceding paragraphs for `表名：xxx`, `表 1.2 — xxx`, `xxx表结构`, `Table: xxx`.


path: references/value-scenarios.md
content: # 知识图谱 / 本体提取的价值场景

## 什么时候知识图谱没用

小规模、单系统、信息在人脑里能管住的场景。

- 200 人公司、150 个客户、十几条产品线 — 人脑 + Excel 完全覆盖
- 数据集中在一两个 Excel 里，Ctrl+F 比任何 query 工具快
- 存进图谱的信息用户本人已经全部掌握，图谱只是换了格式存了一遍
- 没有下游消费者（Agent、BI、数据中台），图谱是死数据

**判断标准**：如果用户的反应是"这些我都知道"，说明图谱没有产生增量价值。

## 什么时候知识图谱有用

### 场景一：跨系统概念对齐（本体的核心战场）

多个系统对同一业务概念使用不同名称、不同字段、不同粒度。

例：某大型集团几万人、几十个子公司、几百个系统。CRM 里叫"蓝天航空"，计费系统里叫"蓝天"，合同系统里叫"蓝天航空有限责任公司"，订座终端里是三字码 "LT"。四个名字指向同一业务实体。

本体提供跨系统的概念对齐层，让"蓝天航空去年给集团贡献了多少收入"这类问题可以被回答 — 因为它知道四个名字是同一个东西，数据分散在五个系统里。

**适用条件**：3+ 个异构系统、实体名称不统一、需要跨系统聚合查询。

### 场景二：Agent 的事实基座（与 LLM 产品直接相关）

Agent 最大的问题是幻觉。客户问"我们用了蓝天信息哪些产品"，Agent 没有事实基础就会编。

知识图谱不是给人查的，是给 Agent 查的：
- 客户来电 → Agent 先查图谱拿到客户画像（产品、在建项目、客户经理）→ 带上下文回答
- 写投标书 → Agent 查"我们在同类机场做过哪些项目" → 生成经验描述
- 售前准备 → Agent 查"这个客户去年有什么工单/投诉" → 风险预判

Agent 对话前自动 query 图谱、注入上下文，比人手动喂 prompt 高效一个量级。

**适用条件**：有 Agent 产品需要事实性回答、客户/产品/项目信息分散在多个文件中。

### 场景三：规模化模式发现

150 个客户没什么好分析的。但几千个客户、几万个合同、十几年的交易记录拉通后，知识图谱能发现：
- 哪些客户在流失（三年没新项目）
- 哪些产品组合最常被一起采购（交叉销售机会）
- 哪些项目团队复用率最高（能力中心在哪）

Excel 做不了这个 — 数据分散在几十个文件、字段名不统一、时间跨度太大。

**适用条件**：实体数 >1000、关系密度高、需要发现非显性模式。

## 个人知识图谱的定位

个人知识图谱（Mode B）的真实价值是：

1. **练手 + 验证 Skill 可用性** — 在自己数据上跑通全流程，才能在客户数据上有信心
2. **Agent 上下文注入的 PoC** — 证明"图谱 + Agent"模式可行，再推广到产品
3. **跨时间的记忆外存** — 三年后你忘了某个项目的关键人物，图谱还记得

不要期望个人图谱本身产生直接业务价值。它的价值在于**作为能力验证和产品原型**。

## 本体提取的核心交付物（咨询视角）

对客户项目（Mode A）做本体提取，交付物是：

| 产出 | 客户价值 |
|------|---------|
| 实体目录（业务语言命名） | 客户第一次看到自己有多少业务概念 |
| 跨系统映射表 | 知道哪些系统在说同一件事 |
| 关系网络图 | 理解实体间的依赖和影响链 |
| review_flag 清单 | 标出需要业务专家确认的模糊判断 |

这才是咨询式营销要卖的东西 — 不是"帮你建知识图谱"，而是"帮你看清自己的数据资产长什么样"。


path: scripts/convert_doc.py
content: #!/usr/bin/env python3
"""
convert_doc.py - Batch convert .doc files to .docx format.

Windows: Uses win32com COM automation (requires pywin32 + Microsoft Word installed).
Linux:   Uses LibreOffice headless mode (requires libreoffice installed).

Usage:
    python convert_doc.py <input_dir_or_file> [--output-dir <dir>] [--max-files 100]

Examples:
    # Convert all .doc files in a directory
    python convert_doc.py /path/to/project --output-dir /path/to/converted

    # Convert a single file
    python convert_doc.py /path/to/file.doc

    # Convert only the first 50 .doc files (sorted by priority)
    python convert_doc.py /path/to/project --max-files 50
"""

import argparse
import json
import shutil
import subprocess
import sys
import time
from pathlib import Path


def convert_doc_to_docx_win32com(doc_path: Path, output_path: Path) -> bool:
    """Convert .doc to .docx using win32com (Windows only)."""
    import win32com.client
    import pythoncom

    pythoncom.CoInitialize()
    word = None
    doc = None
    try:
        word = win32com.client.Dispatch("Word.Application")
        word.Visible = False
        word.DisplayAlerts = False

        doc = word.Documents.Open(str(doc_path.resolve()))
        # wdFormatXMLDocument = 12
        doc.SaveAs2(str(output_path.resolve()), FileFormat=12)
        doc.Close(False)
        return True
    except Exception as e:
        print(f"  ERROR converting {doc_path.name}: {e}", file=sys.stderr)
        return False
    finally:
        if doc:
            try:
                doc.Close(False)
            except Exception:
                pass
        if word:
            try:
                word.Quit()
            except Exception:
                pass
        pythoncom.CoUninitialize()


def convert_doc_to_docx_libreoffice(doc_path: Path, output_dir: Path) -> bool:
    """Convert .doc to .docx using LibreOffice headless (Linux/macOS)."""
    lo_cmd = shutil.which("libreoffice") or shutil.which("soffice")
    if not lo_cmd:
        print("ERROR: LibreOffice not found. Install with: sudo apt install libreoffice", file=sys.stderr)
        return False

    try:
        result = subprocess.run(
            [lo_cmd, "--headless", "--convert-to", "docx", "--outdir", str(output_dir), str(doc_path)],
            capture_output=True,
            text=True,
            timeout=120,
        )
        return result.returncode == 0
    except subprocess.TimeoutExpired:
        print(f"  TIMEOUT converting {doc_path.name}", file=sys.stderr)
        return False
    except Exception as e:
        print(f"  ERROR converting {doc_path.name}: {e}", file=sys.stderr)
        return False


def convert_doc_to_docx(doc_path: Path, output_dir: Path) -> bool:
    """
    Cross-platform .doc → .docx conversion.
    Automatically selects win32com (Windows) or LibreOffice (Linux/macOS).
    """
    output_path = output_dir / (doc_path.stem + ".docx")

    if output_path.exists():
        return True  # Already converted

    if sys.platform == "win32":
        return convert_doc_to_docx_win32com(doc_path, output_path)
    else:
        return convert_doc_to_docx_libreoffice(doc_path, output_dir)


def batch_convert(
    input_path: Path,
    output_dir: Path,
    max_files: int = 0,
    scan_result_path: Path | None = None,
) -> dict:
    """
    Batch convert .doc files to .docx.

    Args:
        input_path: A single .doc file or a directory to scan.
        output_dir: Directory to write .docx output files.
        max_files: Max number of files to convert (0 = all).
        scan_result_path: Optional path to scan_result.json for priority ordering.

    Returns:
        Conversion report dict.
    """
    output_dir.mkdir(parents=True, exist_ok=True)

    # Collect .doc files
    if input_path.is_file() and input_path.suffix.lower() == ".doc":
        doc_files = [input_path]
    elif input_path.is_dir():
        doc_files = sorted(input_path.rglob("*.doc"))
        # Exclude already-.docx files that happen to match (case sensitivity)
        doc_files = [f for f in doc_files if f.suffix.lower() == ".doc"]
    else:
        print(f"Error: '{input_path}' is not a .doc file or directory", file=sys.stderr)
        return {"success": 0, "failed": 0, "skipped": 0, "files": []}

    # If scan result available, sort by priority
    if scan_result_path and scan_result_path.exists():
        with open(scan_result_path, "r", encoding="utf-8") as f:
            scan_data = json.load(f)
        priority_map = {}
        for entry in scan_data.get("files", []):
            priority_map[entry["path"]] = entry["priority"]

        def sort_key(p):
            return priority_map.get(str(p), "P7")

        doc_files.sort(key=sort_key)

    # Apply max_files limit
    if max_files > 0:
        doc_files = doc_files[:max_files]

    total = len(doc_files)
    if total == 0:
        print("No .doc files found.")
        return {"success": 0, "failed": 0, "skipped": 0, "files": []}

    print(f"Converting {total} .doc files to .docx...")
    if sys.platform == "win32":
        print("Backend: win32com (Microsoft Word)")
    else:
        print("Backend: LibreOffice headless")

    success = 0
    failed = 0
    skipped = 0
    results = []
    start_time = time.time()

    for i, doc_path in enumerate(doc_files, 1):
        output_path = output_dir / (doc_path.stem + ".docx")

        if output_path.exists():
            skipped += 1
            results.append({"source": str(doc_path), "output": str(output_path), "status": "skipped"})
            continue

        print(f"  [{i}/{total}] {doc_path.name}...", end=" ", flush=True)

        ok = convert_doc_to_docx(doc_path, output_dir)
        if ok:
            success += 1
            print("OK")
            results.append({"source": str(doc_path), "output": str(output_path), "status": "success"})
        else:
            failed += 1
            print("FAILED")
            results.append({"source": str(doc_path), "output": str(output_path), "status": "failed"})

    elapsed = time.time() - start_time
    report = {
        "success": success,
        "failed": failed,
        "skipped": skipped,
        "total": total,
        "elapsed_seconds": round(elapsed, 1),
        "output_dir": str(output_dir),
        "files": results,
    }

    print(f"\nDone in {elapsed:.1f}s: {success} converted, {failed} failed, {skipped} skipped")
    return report


def main():
    parser = argparse.ArgumentParser(description="Batch convert .doc to .docx")
    parser.add_argument("input", help="Input .doc file or directory")
    parser.add_argument("--output-dir", "-o", default=None, help="Output directory for .docx files")
    parser.add_argument("--max-files", "-n", type=int, default=0, help="Max files to convert (0=all)")
    parser.add_argument("--scan-result", default=None, help="Path to scan_result.json for priority ordering")
    parser.add_argument("--report", default=None, help="Save conversion report to JSON file")
    args = parser.parse_args()

    input_path = Path(args.input)
    if not input_path.exists():
        print(f"Error: '{input_path}' does not exist", file=sys.stderr)
        sys.exit(1)

    # Default output dir: <input>_converted or same dir as input file
    if args.output_dir:
        output_dir = Path(args.output_dir)
    elif input_path.is_dir():
        output_dir = input_path / "_converted_docx"
    else:
        output_dir = input_path.parent

    scan_result = Path(args.scan_result) if args.scan_result else None

    report = batch_convert(input_path, output_dir, args.max_files, scan_result)

    if args.report:
        with open(args.report, "w", encoding="utf-8") as f:
            json.dump(report, f, ensure_ascii=False, indent=2)
        print(f"Report saved to: {args.report}")


if __name__ == "__main__":
    main()


path: scripts/extract_tables.py
content: #!/usr/bin/env python3
"""
extract_tables.py - Extract table structures from Word (.docx) and Excel (.xls/.xlsx) files.

Identifies database schema tables, data dictionary entries, and field definitions.
Outputs structured JSON suitable for ontology extraction by LLM.

Usage:
    python extract_tables.py <file_or_directory> [--output extracted_tables.json]
    python extract_tables.py document.docx --output tables.json
    python extract_tables.py data_dictionary.xlsx --output tables.json
"""

import argparse
import json
import re
import sys
from pathlib import Path

# ---------------------------------------------------------------------------
# Column role recognition keywords
# ---------------------------------------------------------------------------

COLUMN_ROLE_KEYWORDS = {
    "field_name": ["字段名", "字段名称", "列名", "column", "field name", "英文名", "字段英文", "属性名"],
    "field_name_cn": ["中文名", "字段中文", "中文描述", "中文说明", "业务名称", "字段描述", "字段说明"],
    "field_type": ["数据类型", "字段类型", "数据格式", "类型", "data type", "type"],
    "field_length": ["字段长度", "长度", "length", "size", "精度"],
    "nullable": ["是否为空", "nullable", "允许空", "是否必填", "必填", "null"],
    "primary_key": ["是否主键", "主键", "pk", "primary"],
    "foreign_key": ["外键", "fk", "foreign", "关联", "引用"],
    "default_value": ["默认值", "default", "缺省值"],
    "description": ["业务含义", "说明", "描述", "备注", "comment", "注释", "含义", "定义"],
    "enum_values": ["枚举值", "取值范围", "值域", "可选值", "枚举", "代码值"],
}

# Keywords that indicate a table row is a header for database schema
SCHEMA_HEADER_INDICATORS = [
    "字段名", "字段", "列名", "column", "field",
    "类型", "数据类型", "type",
    "说明", "描述", "备注", "comment",
]

# Table name extraction patterns (from paragraphs preceding a table)
TABLE_NAME_PATTERNS = [
    r"表[名称]?[：:]\s*(.+)",
    r"表\s*\d+[\s\-.]+(.+)",
    r"(\w+)\s*表\s*[结构定义]",
    r"Table[\s:]+(\w+)",
    r"(\w+)\s+Table\s+(?:Structure|Definition)",
]

# Size thresholds for processing strategy
SIZE_SMALL = 5 * 1024 * 1024       # < 5 MB
SIZE_MEDIUM = 20 * 1024 * 1024     # 5-20 MB
SIZE_LARGE = 50 * 1024 * 1024      # 20-50 MB
# > 50 MB = huge (sampling mode)

MAX_ROWS_MEDIUM = 500
MAX_ROWS_LARGE = 100
SAMPLE_TABLES = 10


def get_size_strategy(file_size: int) -> str:
    """Determine processing strategy based on file size."""
    if file_size < SIZE_SMALL:
        return "full"
    elif file_size < SIZE_MEDIUM:
        return "selective"
    elif file_size < SIZE_LARGE:
        return "table_priority"
    else:
        return "sampling"


def detect_column_roles(header_cells: list[str]) -> dict[int, str]:
    """
    Map column indices to semantic roles based on header text matching.
    Uses longest-keyword-match-wins strategy to avoid ambiguous matches
    (e.g., "字段描述" should match description, not field_name).

    Returns: {col_index: role_name} e.g. {0: "field_name", 1: "field_type", 3: "description"}
    """
    roles = {}
    for i, cell_text in enumerate(header_cells):
        text_lower = cell_text.strip().lower()
        if not text_lower:
            continue

        best_role = None
        best_kw_len = 0
        for role, keywords in COLUMN_ROLE_KEYWORDS.items():
            for kw in keywords:
                if kw.lower() in text_lower and len(kw) > best_kw_len:
                    best_role = role
                    best_kw_len = len(kw)

        if best_role:
            roles[i] = best_role

    return roles


def is_schema_header(cells: list[str]) -> bool:
    """Check if a row of cells looks like a database schema table header."""
    combined = " ".join(c.strip().lower() for c in cells)
    match_count = sum(1 for indicator in SCHEMA_HEADER_INDICATORS if indicator.lower() in combined)
    return match_count >= 2


def extract_table_name_from_context(preceding_text: str) -> str | None:
    """Try to extract a table name from the text preceding a table."""
    for pattern in TABLE_NAME_PATTERNS:
        m = re.search(pattern, preceding_text, re.IGNORECASE)
        if m:
            return m.group(1).strip()
    return None


# ---------------------------------------------------------------------------
# Word (.docx) extraction
# ---------------------------------------------------------------------------

def extract_tables_from_docx(filepath: Path, strategy: str = "full") -> list[dict]:
    """Extract table structures from a .docx file."""
    from docx import Document

    doc = Document(str(filepath))
    tables_found = []

    # Build paragraph text index for table name context
    paragraphs = [p.text.strip() for p in doc.paragraphs]

    # Track which paragraph index corresponds to which table
    # (Word tables are interleaved with paragraphs in document body)
    para_idx = 0

    for table_idx, table in enumerate(doc.tables):
        if strategy == "sampling" and table_idx >= SAMPLE_TABLES:
            break

        rows = table.rows
        if len(rows) < 2:
            continue  # Need at least header + 1 data row

        # Get header row
        header_cells = [cell.text.strip() for cell in rows[0].cells]

        # Check if this looks like a schema table
        if strategy in ("table_priority", "sampling"):
            if not is_schema_header(header_cells):
                continue

        # Even in "full" or "selective" mode, only extract schema-like tables
        if not is_schema_header(header_cells):
            # Still capture it but mark as non-schema
            pass

        # Detect column roles
        column_roles = detect_column_roles(header_cells)

        # Try to find table name from preceding paragraphs
        table_name = None
        # Heuristic: look at a few paragraphs before, searching backward
        search_paras = paragraphs[max(0, para_idx - 3):para_idx + 1]
        for p_text in reversed(search_paras):
            table_name = extract_table_name_from_context(p_text)
            if table_name:
                break

        # If no name found from context, try the first cell of a potential title row
        if not table_name and header_cells:
            # Sometimes the first row is actually a merged title row
            if len(set(header_cells)) == 1 and header_cells[0]:
                table_name = header_cells[0]
                # Then the real header is row[1]
                if len(rows) >= 3:
                    header_cells = [cell.text.strip() for cell in rows[1].cells]
                    column_roles = detect_column_roles(header_cells)

        # Determine row limit based on strategy
        if strategy == "full":
            max_rows = len(rows)
        elif strategy == "selective":
            max_rows = min(len(rows), MAX_ROWS_MEDIUM)
        else:
            max_rows = min(len(rows), MAX_ROWS_LARGE)

        # Extract data rows
        data_rows = []
        start_row = 1
        # If we consumed an extra header row above, adjust
        for row in rows[start_row:max_rows]:
            cells = [cell.text.strip() for cell in row.cells]
            if all(c == "" for c in cells):
                continue  # Skip empty rows

            row_data = {}
            for col_idx, role in column_roles.items():
                if col_idx < len(cells):
                    row_data[role] = cells[col_idx]

            # Also keep raw cells for LLM context
            row_data["_raw"] = cells
            data_rows.append(row_data)

        is_schema = is_schema_header(header_cells)

        tables_found.append({
            "source_file": str(filepath),
            "table_index": table_idx,
            "table_name": table_name,
            "is_schema_table": is_schema,
            "header": header_cells,
            "column_roles": {str(k): v for k, v in column_roles.items()},
            "row_count": len(data_rows),
            "total_rows_in_source": len(rows) - 1,
            "truncated": len(rows) - 1 > max_rows - 1,
            "fields": data_rows,
        })

    return tables_found


# ---------------------------------------------------------------------------
# Excel (.xlsx) extraction
# ---------------------------------------------------------------------------

def extract_tables_from_xlsx(filepath: Path, strategy: str = "full") -> list[dict]:
    """Extract table structures from a .xlsx file."""
    import openpyxl

    read_only = strategy != "full"
    wb = openpyxl.load_workbook(str(filepath), read_only=read_only, data_only=True)

    tables_found = []
    max_sheets = 3 if strategy in ("table_priority", "sampling") else len(wb.sheetnames)

    for sheet_idx, sheet_name in enumerate(wb.sheetnames[:max_sheets]):
        ws = wb[sheet_name]

        # Determine row limit
        if strategy == "full":
            max_data_rows = None
        elif strategy == "selective":
            max_data_rows = MAX_ROWS_MEDIUM
        else:
            max_data_rows = MAX_ROWS_LARGE

        # Read rows
        all_rows = []
        for i, row in enumerate(ws.iter_rows(values_only=True)):
            if max_data_rows and i > max_data_rows:
                break
            cells = [str(c).strip() if c is not None else "" for c in row]
            if all(c == "" for c in cells):
                continue
            all_rows.append(cells)

        if len(all_rows) < 2:
            continue  # Need header + data

        # Find the header row (first row with schema-like indicators, or row 0)
        header_row_idx = 0
        for idx, row in enumerate(all_rows[:5]):  # Check first 5 rows
            if is_schema_header(row):
                header_row_idx = idx
                break

        header_cells = all_rows[header_row_idx]
        column_roles = detect_column_roles(header_cells)

        # Extract data rows
        data_rows = []
        for row in all_rows[header_row_idx + 1:]:
            row_data = {}
            for col_idx, role in column_roles.items():
                if col_idx < len(row):
                    row_data[role] = row[col_idx]
            row_data["_raw"] = row
            data_rows.append(row_data)

        is_schema = is_schema_header(header_cells) or bool(column_roles)

        tables_found.append({
            "source_file": str(filepath),
            "sheet_name": sheet_name,
            "sheet_index": sheet_idx,
            "table_name": sheet_name,  # Use sheet name as default table name
            "is_schema_table": is_schema,
            "header": header_cells,
            "column_roles": {str(k): v for k, v in column_roles.items()},
            "row_count": len(data_rows),
            "total_rows_in_source": len(all_rows) - header_row_idx - 1,
            "truncated": max_data_rows is not None and len(all_rows) > max_data_rows,
            "fields": data_rows,
        })

    wb.close()
    return tables_found


# ---------------------------------------------------------------------------
# Excel (.xls) extraction
# ---------------------------------------------------------------------------

def extract_tables_from_xls(filepath: Path, strategy: str = "full") -> list[dict]:
    """Extract table structures from a .xls file (legacy Excel format)."""
    import xlrd

    wb = xlrd.open_workbook(str(filepath))
    tables_found = []
    max_sheets = 3 if strategy in ("table_priority", "sampling") else wb.nsheets

    for sheet_idx in range(min(wb.nsheets, max_sheets)):
        ws = wb.sheet_by_index(sheet_idx)

        if ws.nrows < 2:
            continue

        # Determine row limit
        if strategy == "full":
            max_data_rows = ws.nrows
        elif strategy == "selective":
            max_data_rows = min(ws.nrows, MAX_ROWS_MEDIUM)
        else:
            max_data_rows = min(ws.nrows, MAX_ROWS_LARGE)

        # Read rows
        all_rows = []
        for i in range(min(ws.nrows, max_data_rows)):
            cells = [str(ws.cell_value(i, j)).strip() for j in range(ws.ncols)]
            all_rows.append(cells)

        # Find header row
        header_row_idx = 0
        for idx, row in enumerate(all_rows[:5]):
            if is_schema_header(row):
                header_row_idx = idx
                break

        header_cells = all_rows[header_row_idx]
        column_roles = detect_column_roles(header_cells)

        data_rows = []
        for row in all_rows[header_row_idx + 1:]:
            if all(c == "" for c in row):
                continue
            row_data = {}
            for col_idx, role in column_roles.items():
                if col_idx < len(row):
                    row_data[role] = row[col_idx]
            row_data["_raw"] = row
            data_rows.append(row_data)

        is_schema = is_schema_header(header_cells) or bool(column_roles)

        tables_found.append({
            "source_file": str(filepath),
            "sheet_name": ws.name,
            "sheet_index": sheet_idx,
            "table_name": ws.name,
            "is_schema_table": is_schema,
            "header": header_cells,
            "column_roles": {str(k): v for k, v in column_roles.items()},
            "row_count": len(data_rows),
            "total_rows_in_source": ws.nrows - header_row_idx - 1,
            "truncated": ws.nrows > max_data_rows,
            "fields": data_rows,
        })

    return tables_found


# ---------------------------------------------------------------------------
# Unified extraction interface
# ---------------------------------------------------------------------------

def extract_tables(filepath: Path) -> list[dict]:
    """
    Extract tables from a supported file format.
    Automatically selects strategy based on file size.
    """
    if not filepath.exists():
        print(f"Error: File not found: {filepath}", file=sys.stderr)
        return []

    file_size = filepath.stat().st_size
    strategy = get_size_strategy(file_size)
    ext = filepath.suffix.lower()

    if strategy != "full":
        size_mb = round(file_size / (1024 * 1024), 1)
        print(f"  Large file ({size_mb} MB), using '{strategy}' strategy")

    if ext == ".docx":
        return extract_tables_from_docx(filepath, strategy)
    elif ext == ".xlsx":
        return extract_tables_from_xlsx(filepath, strategy)
    elif ext == ".xls":
        return extract_tables_from_xls(filepath, strategy)
    else:
        print(f"  Unsupported format: {ext}", file=sys.stderr)
        return []


def extract_from_directory(root: Path, scan_result_path: Path | None = None) -> dict:
    """
    Extract tables from all supported files in a directory.

    If scan_result_path is provided, processes files in priority order (P1 first).
    """
    supported_exts = {".docx", ".xlsx", ".xls"}

    if scan_result_path and scan_result_path.exists():
        with open(scan_result_path, "r", encoding="utf-8") as f:
            scan_data = json.load(f)
        file_list = [
            entry for entry in scan_data.get("files", [])
            if entry["extension"] in supported_exts
        ]
        # Sort by priority
        file_list.sort(key=lambda x: x["priority"])
        files_to_process = [Path(entry["path"]) for entry in file_list]
    else:
        files_to_process = sorted(
            f for f in root.rglob("*")
            if f.is_file() and f.suffix.lower() in supported_exts
        )

    all_tables = []
    file_results = []
    total = len(files_to_process)

    print(f"Extracting tables from {total} files...")

    for i, filepath in enumerate(files_to_process, 1):
        print(f"[{i}/{total}] {filepath.name}")
        try:
            tables = extract_tables(filepath)
            schema_tables = [t for t in tables if t.get("is_schema_table")]
            all_tables.extend(tables)
            file_results.append({
                "file": str(filepath),
                "tables_found": len(tables),
                "schema_tables": len(schema_tables),
                "status": "success",
            })
            print(f"  Found {len(tables)} tables ({len(schema_tables)} schema-like)")
        except Exception as e:
            print(f"  ERROR: {e}", file=sys.stderr)
            file_results.append({
                "file": str(filepath),
                "tables_found": 0,
                "schema_tables": 0,
                "status": f"error: {e}",
            })

    total_tables = len(all_tables)
    schema_count = sum(1 for t in all_tables if t.get("is_schema_table"))
    total_fields = sum(t.get("row_count", 0) for t in all_tables)

    summary = {
        "total_files_processed": total,
        "total_tables_found": total_tables,
        "total_schema_tables": schema_count,
        "total_fields_extracted": total_fields,
        "files": file_results,
    }

    return {"tables": all_tables, "summary": summary}


def main():
    parser = argparse.ArgumentParser(description="Extract tables from Word/Excel files")
    parser.add_argument("input", help="File or directory to process")
    parser.add_argument("--output", "-o", default="extracted_tables.json", help="Output JSON file")
    parser.add_argument("--scan-result", default=None, help="Path to scan_result.json for priority ordering")
    args = parser.parse_args()

    input_path = Path(args.input)
    if not input_path.exists():
        print(f"Error: '{input_path}' does not exist", file=sys.stderr)
        sys.exit(1)

    if input_path.is_file():
        tables = extract_tables(input_path)
        result = {
            "tables": tables,
            "summary": {
                "total_files_processed": 1,
                "total_tables_found": len(tables),
                "total_schema_tables": sum(1 for t in tables if t.get("is_schema_table")),
                "total_fields_extracted": sum(t.get("row_count", 0) for t in tables),
            },
        }
    else:
        scan_result = Path(args.scan_result) if args.scan_result else None
        result = extract_from_directory(input_path, scan_result)

    # Save output
    output_path = Path(args.output)
    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(result, f, ensure_ascii=False, indent=2)

    s = result["summary"]
    print(f"\nResults saved to: {output_path}")
    print(f"Tables: {s['total_tables_found']} ({s['total_schema_tables']} schema-like)")
    print(f"Fields: {s['total_fields_extracted']}")


if __name__ == "__main__":
    main()


path: scripts/namespace_rules.example.yaml
content: # 命名空间规则配置（可选）
# 无此文件时，脚本按目录结构自动推断命名空间
# 有此文件时，匹配的顶层目录使用自定义映射，其余仍自动推断

namespace_map:
  # 格式: 顶层目录名 → 命名空间
  Projects: work
  GitHub: work/github
  Documents: work/documents
  MyFiles: work/myfiles
  Notes: personal/notes
  AI: personal/ai
  Writing: personal/writing
  Downloads: external/downloads
  Reference: external/reference

# 这些目录的子目录应展开为独立项目（如 Projects/my-app → work/my-app）
expand_children:
  - Projects
  - GitHub


path: scripts/scan_directory.py
content: #!/usr/bin/env python3
"""
scan_directory.py - Directory scanner with priority classification for ontology extraction.

Scans a project directory tree, classifies files by ontology-extraction priority (P1-P7),
and outputs a JSON manifest + statistics report.

Usage:
    python scan_directory.py <directory> [--output scan_result.json]
"""

import argparse
import json
import re
import sys
from collections import Counter, defaultdict
from pathlib import Path


# ---------------------------------------------------------------------------
# Priority classification rules
# ---------------------------------------------------------------------------

PRIORITY_RULES = [
    {
        "priority": "P1",
        "label": "Database Design Documents",
        "patterns": [
            r"数据库设计", r"表结构", r"db.?design", r"database.?design",
            r"table.?structure", r"数据模型", r"data.?model", r"ER图",
            r"实体关系", r"entity.?relation",
        ],
    },
    {
        "priority": "P2",
        "label": "Data Dictionaries",
        "patterns": [
            r"数据字典", r"字段定义", r"data.?dictionary", r"field.?def",
            r"系统清单", r"字段清单", r"代码表", r"code.?table",
            r"枚举", r"enum",
        ],
    },
    {
        "priority": "P3",
        "label": "Interface Specifications",
        "patterns": [
            r"接口说明", r"接口规格", r"接口文档", r"api.?spec",
            r"interface", r"webservice", r"web.?service", r"报文",
            r"消息格式", r"message.?format",
        ],
    },
    {
        "priority": "P4",
        "label": "Requirements Specifications",
        "patterns": [
            r"需求规格", r"需求说明", r"需求分析", r"srs",
            r"requirement", r"业务需求", r"功能需求",
            r"用例", r"use.?case",
        ],
    },
    {
        "priority": "P5",
        "label": "Architecture Documents",
        "patterns": [
            r"总体方案", r"概要设计", r"系统架构", r"架构设计",
            r"总体设计", r"技术方案", r"system.?architecture",
            r"high.?level.?design", r"hld",
        ],
    },
    {
        "priority": "P6",
        "label": "Data Flow Documents",
        "patterns": [
            r"数据流向", r"数据流转", r"data.?flow", r"数据交换",
            r"数据同步", r"etl", r"数据迁移", r"data.?migration",
        ],
    },
]

# Supported file extensions for ontology extraction
SUPPORTED_EXTENSIONS = {
    ".doc", ".docx", ".xls", ".xlsx",
    ".sql", ".ddl",
    ".json", ".jsonschema", ".yaml", ".yml",
    ".md", ".txt", ".csv",
}


def classify_file(filepath: Path) -> str:
    """Classify a file into priority P1-P7 based on filename patterns."""
    name_lower = filepath.stem.lower()
    # Also check parent directory name for context
    parent_lower = filepath.parent.name.lower()
    search_text = f"{name_lower} {parent_lower}"

    for rule in PRIORITY_RULES:
        for pattern in rule["patterns"]:
            if re.search(pattern, search_text, re.IGNORECASE):
                return rule["priority"]

    return "P7"


def scan_directory(root: Path) -> dict:
    """
    Scan directory tree and classify all supported files.

    Returns a dict with:
      - files: list of {path, extension, size_bytes, priority, priority_label}
      - stats: summary statistics
    """
    files = []
    ext_counter = Counter()
    priority_counter = Counter()
    total_size = 0
    skipped_count = 0

    for filepath in sorted(root.rglob("*")):
        if not filepath.is_file():
            continue

        # Skip macOS resource fork files (._*)
        if filepath.name.startswith("._"):
            skipped_count += 1
            continue

        ext = filepath.suffix.lower()
        if ext not in SUPPORTED_EXTENSIONS:
            skipped_count += 1
            continue

        size = filepath.stat().st_size
        priority = classify_file(filepath)

        # Find label for priority
        priority_label = "Other Documents"
        for rule in PRIORITY_RULES:
            if rule["priority"] == priority:
                priority_label = rule["label"]
                break

        files.append({
            "path": str(filepath),
            "relative_path": str(filepath.relative_to(root)),
            "extension": ext,
            "size_bytes": size,
            "size_mb": round(size / (1024 * 1024), 2),
            "priority": priority,
            "priority_label": priority_label,
        })

        ext_counter[ext] += 1
        priority_counter[priority] += 1
        total_size += size

    # Build statistics
    stats = {
        "root_directory": str(root),
        "total_supported_files": len(files),
        "total_skipped_files": skipped_count,
        "total_size_mb": round(total_size / (1024 * 1024), 2),
        "by_extension": dict(sorted(ext_counter.items())),
        "by_priority": dict(sorted(priority_counter.items())),
        "size_distribution": {
            "small_lt_5mb": sum(1 for f in files if f["size_bytes"] < 5 * 1024 * 1024),
            "medium_5_20mb": sum(1 for f in files if 5 * 1024 * 1024 <= f["size_bytes"] < 20 * 1024 * 1024),
            "large_20_50mb": sum(1 for f in files if 20 * 1024 * 1024 <= f["size_bytes"] < 50 * 1024 * 1024),
            "huge_gt_50mb": sum(1 for f in files if f["size_bytes"] >= 50 * 1024 * 1024),
        },
    }

    return {"files": files, "stats": stats}


def format_report(result: dict) -> str:
    """Format a human-readable statistics report."""
    stats = result["stats"]
    files = result["files"]
    lines = []

    lines.append("=" * 60)
    lines.append("ONTOLOGY ENGINEER - Directory Scan Report")
    lines.append("=" * 60)
    lines.append(f"Root: {stats['root_directory']}")
    lines.append(f"Total supported files: {stats['total_supported_files']}")
    lines.append(f"Total skipped files: {stats['total_skipped_files']}")
    lines.append(f"Total size: {stats['total_size_mb']} MB")
    lines.append("")

    lines.append("--- By Extension ---")
    for ext, count in sorted(stats["by_extension"].items()):
        lines.append(f"  {ext:10s} {count:>5d} files")
    lines.append("")

    lines.append("--- By Priority ---")
    priority_labels = {r["priority"]: r["label"] for r in PRIORITY_RULES}
    priority_labels["P7"] = "Other Documents"
    for p in ["P1", "P2", "P3", "P4", "P5", "P6", "P7"]:
        count = stats["by_priority"].get(p, 0)
        label = priority_labels.get(p, "")
        lines.append(f"  {p} ({label:30s}) {count:>5d} files")
    lines.append("")

    lines.append("--- File Size Distribution ---")
    sd = stats["size_distribution"]
    lines.append(f"  < 5 MB:    {sd['small_lt_5mb']:>5d} files")
    lines.append(f"  5-20 MB:   {sd['medium_5_20mb']:>5d} files")
    lines.append(f"  20-50 MB:  {sd['large_20_50mb']:>5d} files")
    lines.append(f"  > 50 MB:   {sd['huge_gt_50mb']:>5d} files")
    lines.append("")

    # Top 10 largest files
    sorted_by_size = sorted(files, key=lambda f: f["size_bytes"], reverse=True)[:10]
    if sorted_by_size:
        lines.append("--- Top 10 Largest Files ---")
        for f in sorted_by_size:
            lines.append(f"  [{f['priority']}] {f['size_mb']:>8.2f} MB  {f['relative_path']}")
        lines.append("")

    # P1-P3 file listing (high value)
    high_value = [f for f in files if f["priority"] in ("P1", "P2", "P3")]
    if high_value:
        lines.append("--- High-Value Files (P1-P3) ---")
        for f in sorted(high_value, key=lambda x: (x["priority"], x["relative_path"])):
            lines.append(f"  [{f['priority']}] {f['relative_path']}")
        lines.append("")

    lines.append("=" * 60)
    return "\n".join(lines)


def main():
    parser = argparse.ArgumentParser(description="Scan directory for ontology extraction")
    parser.add_argument("directory", help="Root directory to scan")
    parser.add_argument("--output", "-o", default="scan_result.json", help="Output JSON file path")
    parser.add_argument("--report", "-r", action="store_true", help="Print human-readable report")
    args = parser.parse_args()

    root = Path(args.directory)
    if not root.is_dir():
        print(f"Error: '{root}' is not a valid directory", file=sys.stderr)
        sys.exit(1)

    result = scan_directory(root)

    # Save JSON
    output_path = Path(args.output)
    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(result, f, ensure_ascii=False, indent=2)
    print(f"Scan result saved to: {output_path}")

    # Print report (force UTF-8 on Windows to avoid GBK encoding errors)
    import io
    sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")
    if args.report:
        print()
        print(format_report(result))
    else:
        # Always print summary
        stats = result["stats"]
        print(f"Found {stats['total_supported_files']} supported files ({stats['total_size_mb']} MB)")
        print(f"Priority distribution: {dict(sorted(stats['by_priority'].items()))}")


if __name__ == "__main__":
    main()


path: scripts/scan_filesystem.py
content: #!/usr/bin/env python3
"""Personal Knowledge Graph - Filesystem Scanner (Step 1)

通用文件系统扫描器。不硬编码任何用户特定的路径或目录名。
适配任何操作系统、任何目录结构。

Usage:
    python scan_filesystem.py --root /path/to/scan
    python scan_filesystem.py --root /path --config namespace_rules.yaml
    python scan_filesystem.py --root /path --dry-run
    python scan_filesystem.py --root /path --extract-metadata
"""

import os
import sys
import json
import hashlib
import argparse
from datetime import datetime, timezone
from pathlib import Path
from collections import defaultdict

# UTF-8 输出（跨平台兼容）
try:
    sys.stdout = open(sys.stdout.fileno(), mode='w', encoding='utf-8', buffering=1)
    sys.stderr = open(sys.stderr.fileno(), mode='w', encoding='utf-8', buffering=1)
except Exception:
    pass  # 某些环境不支持重设 stdout

# ============================================================
# 配置：文件分类（通用，不含用户特定路径）
# ============================================================

ONTOLOGY_DIR = os.path.dirname(os.path.abspath(__file__))
GRAPH_FILE = os.path.join(ONTOLOGY_DIR, "graph.jsonl")

# 知识文档（必定索引）
DOC_EXTS = {
    '.md', '.txt', '.docx', '.doc', '.pdf', '.rtf',
    '.xlsx', '.xls', '.pptx', '.ppt', '.csv', '.tsv',
    '.wps', '.et', '.dps',          # WPS Office 格式（文档/表格/演示）
    '.pages', '.numbers', '.key',   # macOS 办公格式
    '.odt', '.ods', '.odp',         # OpenDocument 格式
}

# 结构化定义文件（选择性索引）
SCHEMA_EXTS = {'.sql', '.ddl', '.yaml', '.yml', '.json', '.xml', '.graphql', '.proto'}

# 代码文件（只统计，不逐个索引）
CODE_EXTS = {
    '.py', '.js', '.ts', '.tsx', '.jsx', '.html', '.htm', '.css', '.scss',
    '.go', '.rs', '.java', '.kt', '.swift', '.c', '.cpp', '.h', '.hpp',
    '.rb', '.php', '.lua', '.r', '.R', '.jl', '.scala',
    '.sh', '.bash', '.zsh', '.fish', '.bat', '.ps1', '.cmd',
    '.toml', '.ini', '.cfg', '.conf', '.env.example',
    '.vue', '.svelte', '.astro',
}

# ============================================================
# 跳过规则（通用模式，不含用户特定目录名）
# ============================================================

# 跳过的目录名（精确匹配，全小写比较）
SKIP_DIR_NAMES = {
    # 包管理器依赖
    'node_modules', '__pycache__', 'site-packages',
    '.venv', 'venv', 'env', '.env',
    '.tox', '.nox', 'vendor', '.bundle',
    # 构建产物
    'dist', 'build', 'out', 'output', '_build',
    '.next', '.nuxt', '.output', '.svelte-kit',
    'target',  # Rust/Maven
    # 缓存
    '.cache', '.npm', '.yarn', '.pnpm-store', 'bun-cache',
    '.gradle', '.m2', '.cargo',
    # IDE 和工具
    '.git', '.svn', '.hg',
    '.idea', '.vscode', '.vs', '.fleet',
    # 系统
    '$recycle.bin', 'system volume information',
    '.trash', '.trashes',
}

# 跳过的路径片段（包含即跳过整个子树，小写比较）
SKIP_PATH_PATTERNS = [
    # 虚拟环境（各种命名）
    '/venv/', '/virtualenv/', '/site-packages/',
    '/.staging',
    # IM 缓存（通用模式，不限定具体 IM 软件）
    '/gamecaches/', '/appcache/', '/miniapp/',
    # 系统缓存（通用）
    '/cache/', '/caches/', '/tmp/', '/temp/',
    # 应用数据（通常不含用户知识）
    '/appdata/', '/application data/',
]

# 跳过的文件名（精确匹配）
SKIP_FILE_NAMES = {
    # 包管理器锁文件
    'package-lock.json', 'yarn.lock', 'pnpm-lock.yaml', 'bun.lockb',
    'cargo.lock', 'poetry.lock', 'pipfile.lock', 'composer.lock',
    'gemfile.lock', 'flake.lock',
    # 系统文件
    '.ds_store', 'thumbs.db', 'desktop.ini', 'ehthumbs.db',
    # 编辑器临时文件
    '.swp', '.swo', '.swn',
}

# JSON 文件中跳过的（开发配置，不是知识内容）
SKIP_JSON_NAMES = {
    'package.json', 'tsconfig.json', 'jsconfig.json',
    '.eslintrc.json', '.prettierrc.json', 'tslint.json',
    'babel.config.json', '.babelrc',
    'launch.json', 'settings.json', 'extensions.json',
    'devcontainer.json', 'turbo.json', 'nx.json',
    'composer.json', 'deno.json',
}

# ============================================================
# 项目标记文件（用于检测项目边界）
# ============================================================

PROJECT_MARKERS = {
    # 文档项目
    'readme.md', 'readme', 'readme.txt', 'readme.rst',
    'claude.md', 'agents.md', 'gemini.md',
    'index.md', 'index.html',
    # 代码项目
    'package.json', 'cargo.toml', 'go.mod', 'pyproject.toml',
    'setup.py', 'setup.cfg', 'pom.xml', 'build.gradle',
    'makefile', 'cmakelists.txt', 'meson.build',
    'gemfile', 'requirements.txt',
    '.git',
}

# ============================================================
# 命名空间推断（通用逻辑，无硬编码映射）
# ============================================================

def load_namespace_rules(config_path):
    """从 YAML 配置文件加载自定义命名空间规则。"""
    if not config_path or not os.path.exists(config_path):
        return {}, set()
    try:
        import yaml
        with open(config_path, 'r', encoding='utf-8') as f:
            cfg = yaml.safe_load(f) or {}
        ns_map = cfg.get('namespace_map', {})
        expand = set(cfg.get('expand_children', []))
        return ns_map, expand
    except ImportError:
        print("[WARN] pyyaml 未安装，跳过配置文件加载。pip install pyyaml")
        return {}, set()


def safe_name(dirname):
    """将目录名转为安全的命名空间段。"""
    return dirname.lower().replace(' ', '-').replace('_', '-')


def detect_project_boundary(dirpath):
    """检测目录是否是一个项目的根目录。"""
    try:
        entries = {e.lower() for e in os.listdir(dirpath)}
    except OSError:
        return False
    return bool(entries & PROJECT_MARKERS)


class NamespaceInferrer:
    """通用命名空间推断器。"""

    def __init__(self, scan_root, ns_map=None, expand_children=None):
        self.scan_root = os.path.abspath(scan_root)
        self.ns_map = ns_map or {}          # 用户自定义映射
        self.expand = expand_children or set()  # 需要展开子目录的顶层目录
        self._project_cache = {}

    def infer(self, filepath):
        """推断文件的命名空间。"""
        rel = os.path.relpath(filepath, self.scan_root).replace('\\', '/')
        parts = rel.split('/')

        if len(parts) < 2:
            return 'uncategorized'

        top_dir = parts[0]

        # 1. 用户自定义映射优先
        if top_dir in self.ns_map:
            ns = self.ns_map[top_dir]
            if top_dir in self.expand and len(parts) >= 3:
                return f"{ns}/{safe_name(parts[1])}"
            return ns

        # 2. 自动推断：检测项目边界
        #    向下查找最近的项目标记，确定项目层级
        best_project_depth = 1  # 默认取第一层为项目
        for depth in range(1, min(len(parts), 4)):  # 最多看3层
            candidate = os.path.join(self.scan_root, *parts[:depth])
            if candidate not in self._project_cache:
                self._project_cache[candidate] = detect_project_boundary(candidate)
            if self._project_cache[candidate]:
                best_project_depth = depth

        # 用检测到的项目层级构建命名空间
        ns_parts = [safe_name(p) for p in parts[:best_project_depth]]
        return '/'.join(ns_parts)

    def infer_project_name(self, dirpath):
        """推断项目名称。"""
        rel = os.path.relpath(dirpath, self.scan_root).replace('\\', '/')
        parts = rel.split('/')
        # 返回最深的有意义的目录名
        for i in range(min(len(parts), 3) - 1, -1, -1):
            candidate = os.path.join(self.scan_root, *parts[:i+1])
            if detect_project_boundary(candidate):
                return parts[i]
        return parts[0]


# ============================================================
# 文件分类
# ============================================================

def classify_file(filepath):
    """分类文件，返回 'doc' | 'schema' | 'code' | None。"""
    name = os.path.basename(filepath).lower()

    if name in SKIP_FILE_NAMES:
        return None

    # 跳过隐藏文件（以 . 开头，但不是 .md 等有效扩展名的文件）
    if name.startswith('.') and not any(name.endswith(ext) for ext in DOC_EXTS | SCHEMA_EXTS):
        return None

    ext = os.path.splitext(name)[1].lower()

    if ext in DOC_EXTS:
        return 'doc'

    if ext in SCHEMA_EXTS:
        if ext == '.json' and name in SKIP_JSON_NAMES:
            return None
        return 'schema'

    if ext in CODE_EXTS:
        return 'code'

    return None


def file_title(filepath):
    """从文件名提取标题。"""
    name = os.path.basename(filepath)
    title, _ = os.path.splitext(name)
    return title


# ============================================================
# 元数据提取（可选功能）
# ============================================================

def extract_docx_metadata(filepath):
    """从 .docx 文件提取作者/标题等元数据。需要 python-docx。"""
    try:
        from docx import Document
        doc = Document(filepath)
        props = doc.core_properties
        meta = {}
        if props.author:
            meta['author'] = props.author
        if props.title and props.title.strip():
            meta['doc_title'] = props.title.strip()
        if props.subject:
            meta['subject'] = props.subject
        if props.keywords:
            meta['keywords'] = props.keywords
        return meta
    except Exception:
        return {}


def extract_pdf_metadata(filepath):
    """从 .pdf 文件提取元数据。需要 PyMuPDF。"""
    try:
        import fitz
        doc = fitz.open(filepath)
        meta = doc.metadata or {}
        result = {}
        if meta.get('author'):
            result['author'] = meta['author']
        if meta.get('title') and meta['title'].strip():
            result['doc_title'] = meta['title'].strip()
        if meta.get('subject'):
            result['subject'] = meta['subject']
        if meta.get('keywords'):
            result['keywords'] = meta['keywords']
        doc.close()
        return result
    except Exception:
        return {}


# ============================================================
# 重复检测
# ============================================================

class DuplicateDetector:
    """基于文件名+大小的轻量级重复检测。"""

    def __init__(self):
        self._seen = {}  # (filename, size) → first_path
        self.duplicates = []

    def check(self, filepath, size_bytes):
        """返回 True 如果文件疑似重复。"""
        key = (os.path.basename(filepath).lower(), size_bytes)
        if key in self._seen:
            self.duplicates.append((filepath, self._seen[key]))
            return True
        self._seen[key] = filepath
        return False


# ============================================================
# 实体构建
# ============================================================

def make_entity(entity_id, entity_type, graph, source, properties, labels=None, relations=None):
    """构建一条 graph.jsonl 记录。"""
    now = datetime.now(timezone.utc).isoformat()
    return {
        "op": "create",
        "ts": now,
        "entity": {
            "id": entity_id,
            "type": entity_type,
            "graph": graph,
            "labels": labels or [],
            "source": source,
            "properties": properties,
            "relations": relations or [],
            "created_at": now,
        }
    }


# ============================================================
# 扫描器
# ============================================================

class FileScanner:
    def __init__(self, scan_root, dry_run=False, extract_metadata=False,
                 ns_map=None, expand_children=None, skip_duplicates=True):
        self.scan_root = os.path.abspath(scan_root)
        self.dry_run = dry_run
        self.extract_metadata = extract_metadata
        self.skip_duplicates = skip_duplicates
        self.scan_id = f"scan-{datetime.now().strftime('%Y%m%d-%H%M%S')}"

        self.ns_inferrer = NamespaceInferrer(self.scan_root, ns_map, expand_children)
        self.dup_detector = DuplicateDetector()

        self.entities = []
        self.projects = {}

        self.stats = {
            'dirs_scanned': 0,
            'dirs_skipped': 0,
            'files_doc': 0,
            'files_schema': 0,
            'files_code': 0,
            'files_skipped': 0,
            'files_duplicate': 0,
            'metadata_extracted': 0,
        }

        self._doc_counter = 0
        self._project_counter = 0

    def _next_doc_id(self):
        self._doc_counter += 1
        return f"doc-{self._doc_counter:05d}"

    def _next_project_id(self):
        self._project_counter += 1
        return f"project-{self._project_counter:03d}"

    def _should_skip_dir(self, dirname):
        """目录名是否应跳过。"""
        low = dirname.lower()
        return (low in SKIP_DIR_NAMES
                or low.startswith('.')
                or low.endswith('.egg-info')
                or low.endswith('.dist-info'))

    def _should_skip_path(self, path):
        """路径是否包含应跳过的模式。"""
        normalized = path.replace('\\', '/').lower()
        return any(p in normalized for p in SKIP_PATH_PATTERNS)

    def scan(self):
        """执行扫描。"""
        print(f"扫描根目录: {self.scan_root}")
        print(f"扫描ID: {self.scan_id}")
        if self.extract_metadata:
            print(f"元数据提取: 已启用")
        print()

        source = {
            "type": "scan",
            "scan_id": self.scan_id,
            "target": "self",
            "path": self.scan_root.replace('\\', '/'),
        }

        for root, dirs, files in os.walk(self.scan_root):
            # 过滤目录
            original_count = len(dirs)
            dirs[:] = [d for d in dirs if not self._should_skip_dir(d)]
            self.stats['dirs_skipped'] += original_count - len(dirs)
            self.stats['dirs_scanned'] += 1

            if self._should_skip_path(root):
                dirs.clear()
                continue

            for filename in files:
                filepath = os.path.join(root, filename)
                category = classify_file(filepath)

                if category is None:
                    self.stats['files_skipped'] += 1
                    continue

                # 获取文件大小
                try:
                    size_bytes = os.path.getsize(filepath)
                except OSError:
                    self.stats['files_skipped'] += 1
                    continue

                # 重复检测
                if self.skip_duplicates and self.dup_detector.check(filepath, size_bytes):
                    self.stats['files_duplicate'] += 1
                    self.stats['files_skipped'] += 1
                    continue

                ns = self.ns_inferrer.infer(filepath)
                proj_name = self.ns_inferrer.infer_project_name(root)

                # 收集项目信息
                if ns not in self.projects:
                    self.projects[ns] = {
                        'name': proj_name,
                        'path': root.replace('\\', '/'),
                        'doc_count': 0,
                        'schema_count': 0,
                        'code_count': 0,
                    }

                if category == 'doc':
                    self.stats['files_doc'] += 1
                    self.projects[ns]['doc_count'] += 1
                    self._create_document(filepath, ns, source, size_bytes)
                elif category == 'schema':
                    self.stats['files_schema'] += 1
                    self.projects[ns]['schema_count'] += 1
                    self._create_document(filepath, ns, source, size_bytes)
                elif category == 'code':
                    self.stats['files_code'] += 1
                    self.projects[ns]['code_count'] += 1

        self._create_projects(source)
        print(f"\n扫描完成。")
        self._print_stats()

    def _create_document(self, filepath, namespace, source, size_bytes):
        """为一个文件创建 Document 实体。"""
        ext = os.path.splitext(filepath)[1].lower().lstrip('.')
        size_kb = round(size_bytes / 1024, 1)

        try:
            modified = datetime.fromtimestamp(os.path.getmtime(filepath)).strftime('%Y-%m-%d')
        except OSError:
            modified = "unknown"

        props = {
            "title": file_title(filepath),
            "path": filepath.replace('\\', '/'),
            "format": ext if ext else "unknown",
            "size_kb": size_kb,
            "modified": modified,
        }

        # 可选：提取 .docx/.pdf 元数据
        if self.extract_metadata:
            meta = {}
            if ext == 'docx':
                meta = extract_docx_metadata(filepath)
            elif ext == 'pdf':
                meta = extract_pdf_metadata(filepath)
            if meta:
                props['metadata'] = meta
                self.stats['metadata_extracted'] += 1

        entity = make_entity(
            entity_id=self._next_doc_id(),
            entity_type="Document",
            graph=namespace,
            source=source,
            properties=props,
        )
        self.entities.append(entity)

    def _create_projects(self, source):
        """从收集的目录信息创建 Project 实体。"""
        for ns, info in sorted(self.projects.items()):
            total = info['doc_count'] + info['schema_count'] + info['code_count']
            if total == 0:
                continue
            entity = make_entity(
                entity_id=self._next_project_id(),
                entity_type="Project",
                graph=ns,
                source=source,
                properties={
                    "name": info['name'],
                    "status": "active",
                    "path": info['path'],
                    "doc_count": info['doc_count'],
                    "schema_count": info['schema_count'],
                    "code_count": info['code_count'],
                },
            )
            self.entities.append(entity)

    def _print_stats(self):
        """打印扫描统计。"""
        s = self.stats
        total_indexed = s['files_doc'] + s['files_schema']
        total_all = total_indexed + s['files_code'] + s['files_skipped']

        print(f"\n{'='*60}")
        print(f"扫描统计")
        print(f"{'='*60}")
        print(f"目录: 扫描 {s['dirs_scanned']}, 跳过 {s['dirs_skipped']}")
        print(f"文件总计: {total_all}")
        print(f"  文档文件 (已索引):  {s['files_doc']}")
        print(f"  Schema文件 (已索引): {s['files_schema']}")
        print(f"  代码文件 (仅统计):  {s['files_code']}")
        print(f"  跳过:               {s['files_skipped']}")
        if s['files_duplicate'] > 0:
            print(f"  其中重复文件:       {s['files_duplicate']}")
        if s['metadata_extracted'] > 0:
            print(f"  元数据提取:         {s['metadata_extracted']}")
        print(f"\n实体生成:")
        print(f"  Document 实体: {self._doc_counter}")
        print(f"  Project 实体:  {self._project_counter}")
        print(f"  总计:          {len(self.entities)}")

        # 按命名空间统计
        ns_stats = defaultdict(lambda: {'docs': 0, 'schemas': 0, 'code': 0})
        for ns, info in self.projects.items():
            ns_stats[ns]['docs'] = info['doc_count']
            ns_stats[ns]['schemas'] = info['schema_count']
            ns_stats[ns]['code'] = info['code_count']

        print(f"\n{'命名空间':<45s} {'文档':>6s} {'Schema':>8s} {'代码':>6s}")
        print(f"{'-'*45} {'-'*6} {'-'*8} {'-'*6}")
        for ns in sorted(ns_stats.keys()):
            info = ns_stats[ns]
            total = info['docs'] + info['schemas'] + info['code']
            if total > 0:
                print(f"{ns:<45s} {info['docs']:>6d} {info['schemas']:>8d} {info['code']:>6d}")

        # 报告重复文件
        if self.dup_detector.duplicates:
            print(f"\n疑似重复文件 (前10条):")
            for dup, orig in self.dup_detector.duplicates[:10]:
                print(f"  {os.path.basename(dup)}")
                print(f"    ← {orig}")
                print(f"    → {dup}")

    def write_output(self):
        """写入 graph.jsonl。"""
        if self.dry_run:
            print(f"\n[DRY RUN] 跳过写入。共 {len(self.entities)} 条实体待写入 {GRAPH_FILE}")
            return

        write_errors = 0
        with open(GRAPH_FILE, 'a', encoding='utf-8') as f:
            for entity in self.entities:
                try:
                    line = json.dumps(entity, ensure_ascii=False)
                    # 清理 surrogate 字符（Windows 文件名偶尔包含）
                    line = line.encode('utf-8', errors='surrogateescape').decode('utf-8', errors='replace')
                    f.write(line + '\n')
                except (UnicodeEncodeError, UnicodeDecodeError):
                    write_errors += 1
        if write_errors:
            print(f"  编码错误跳过: {write_errors}")

        print(f"\n已写入 {len(self.entities)} 条实体到 {GRAPH_FILE}")
        size_kb = os.path.getsize(GRAPH_FILE) / 1024
        print(f"文件大小: {size_kb:.1f} KB")


# ============================================================
# 主入口
# ============================================================

def main():
    global GRAPH_FILE

    parser = argparse.ArgumentParser(
        description="Personal Knowledge Graph - Filesystem Scanner (Step 1)",
        epilog="""
示例:
  python scan_filesystem.py --root ~/Documents
  python scan_filesystem.py --root D:/ --config namespace_rules.yaml
  python scan_filesystem.py --root /home/user --extract-metadata --dry-run
        """,
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    parser.add_argument('--root', required=True, help='扫描根目录（必需）')
    parser.add_argument('--config', help='命名空间规则配置文件（YAML，可选）')
    parser.add_argument('--dry-run', action='store_true', help='只统计不写入')
    parser.add_argument('--extract-metadata', action='store_true',
                        help='从 .docx/.pdf 提取作者/标题等元数据（需要 python-docx 和 PyMuPDF）')
    parser.add_argument('--no-dedup', action='store_true', help='禁用重复文件检测')
    parser.add_argument('--output', help=f'输出文件路径（默认: {GRAPH_FILE}）')
    args = parser.parse_args()

    if args.output:
        GRAPH_FILE = args.output

    ns_map, expand = load_namespace_rules(args.config)

    scanner = FileScanner(
        scan_root=args.root,
        dry_run=args.dry_run,
        extract_metadata=args.extract_metadata,
        ns_map=ns_map,
        expand_children=expand,
        skip_duplicates=not args.no_dedup,
    )
    scanner.scan()
    scanner.write_output()


if __name__ == '__main__':
    main()


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)