Skill: 18874771327/knowledge-qa (version 1.0.0)

--- name: knowledge-qa description: 本地知识库问答技能。当用户需要基于个人知识库文档（PDF/Markdown/Word）进行问答、生成报告、制作思维导图、或上传文件到向量库时触发。触发词包括："基于知识库"、"基于mysql查询"、"基于某个分区"、"查一下知识库"、"帮我写报告"、"生成思维导图"、"根据文档"、"从我的资料"、"结合我的笔记"、"整理成报告"、"做个导图"、"上传知识库"、"有新文件"、"索引文档"、"建向量库"、"有哪些分区"、"分区列表"、"创建知识库"、"初始化知识库"、"知识库列表"。 --- # 知识库问答技能 (knowledge-qa) ## 概述 本技能支持**多知识库**架构，每个知识库对应一个独立的 DashVector Collection。用户可在同一工作目录下创建多个互不干扰的知识库（如"MySQLNotes"、"JavaNotes"等）。 **核心特性：** - 多知识库支持（一个知识库 = 一个 DashVector Collection） - 语义向量检索（基于阿里云 DashVector + 百炼 Embedding） - 自动分区（raw_docs 子目录自动映射为分区） - 多分区并行查询 + 结果合并 - 报告 + 思维导图双输出 - **需要 Python 环境**（用户需安装 Python 3.8+） --- ## 前置要求 ### 1. 安装 Python 环境 用户需在系统上安装 Python 3.8 或更高版本，以及以下依赖： ```bash pip install pdfplumber python-docx requests ``` ### 2. 安装 WorkBuddy 并打开工作目录 用户用 WorkBuddy 打开一个工作目录，在该目录下管理所有知识库。 --- ## 多知识库架构 ``` 工作空间 (WorkBuddy 打开的目录) │ ├── MySQLNotes/ ← 知识库 A → Collection: MySQLNotes │ ├── raw_docs/ │ │ ├── mysql/ ← 分区: mysql │ │ └── default/ │ ├── config.json ← 独立配置（不同的 DashVector Collection） │ ├── indexed_files.json │ └── README.md │ ├── JavaNotes/ ← 知识库 B → Collection: JavaNotes │ ├── raw_docs/ │ │ ├── java基础/ ← 分区: java_ │ │ └── default/ │ ├── config.json │ └── README.md │ └── scripts/ ← （由技能提供，用户不可见） ├── init_knowledge_base.py ├── upload_to_vector.py ├── query_knowledge_base.py └── partition_list.py ``` **映射关系：** - 1 个知识库文件夹 = 1 个 DashVector Collection（独立配置） - Collection 内按 raw_docs 子目录分区（mysql / oracle / java_ 等） - 切换知识库：由用户在提问时指定，如"基于 MySQLNotes 知识库回答..." --- ## 工作流程一：创建新知识库 **触发词：** "创建知识库"、"初始化一个知识库"、"新建知识库" ### 步骤 1：确认知识库名称和工作目录 用户告诉 AI 要创建的知识库名称，以及是否在当前工作目录下创建。 ### 步骤 2：运行初始化脚本 ```bash python <技能路径>/scripts/init_knowledge_base.py <知识库名称> --path <工作目录> ``` AI 自动执行脚本，在工作目录下创建以下结构： ``` 知识库名称/ ├── raw_docs/ │ └── default/ ├── config.json ← 模板文件，用户需填写阿里云凭证 ├── indexed_files.json └── README.md ``` ### 步骤 3：提示用户填写配置 告知用户打开 `config.json`，填入： - `dashvector.api_key` - `dashvector.endpoint` - `dashvector.collection_name`（建议与知识库名一致） - `bailian.api_key` 凭证获取地址： - DashVector: https://dashvector.console.aliyun.com/api-key - 百炼: https://bailian.console.aliyun.com/api-key --- ## 工作流程二：上传文件到向量库 **触发词：** "上传知识库"、"有新文件了"、"帮我索引文件" ### 步骤 1：确认目标知识库 用户必须指定知识库名称（如"MySQLNotes"）。AI 在工作目录下查找对应的知识库文件夹。 ### 步骤 2：运行上传脚本 ```bash python <技能路径>/scripts/upload_to_vector.py --kb-path <知识库路径> ``` 脚本自动： 1. 扫描 `raw_docs/` 下所有文件 2. 识别新增或变更的文件（对比 `indexed_files.json`） 3. 提取文本内容并分块 4. 调用百炼 API 生成向量 5. 调用 DashVector API 上传（自动创建分区） 6. 更新 `indexed_files.json` ### 分区识别规则 | 文件位置 | 分区名 | |---------|--------| | `raw_docs/文档.pdf` | `default` | | `raw_docs/mysql/文档.pdf` | `mysql` | | `raw_docs/MySQL实战/文档.pdf` | `mysql_`（小写、非字母数字转下划线）| --- ## 工作流程三：知识问答与报告生成 **触发词：** "基于知识库回答"、"查一下知识库"、"帮我写报告" ### 步骤 1：确认目标知识库和分区 用户必须指定知识库名称（如"基于 MySQLNotes 知识库"）。 可选指定分区（如"基于 MySQLNotes 的 mysql 和 oracle 分区"）。 ### 步骤 2：运行查询脚本 ```bash python <技能路径>/scripts/query_knowledge_base.py "<问题>" --kb-path "<知识库路径>" --partition <分区名> ``` 示例（单分区）： ```bash python scripts/query_knowledge_base.py "InnoDB 有哪些特性" --kb-path "MySQLNotes" --partition mysql ``` 示例（多分区）： ```bash python scripts/query_knowledge_base.py "MySQL 和 Oracle 有什么区别" --kb-path "MySQLNotes" --partition mysql,oracle ``` 示例（全分区）： ```bash python scripts/query_knowledge_base.py "介绍一下这个知识库" --kb-path "MySQLNotes" ``` **多分区查询说明：** DashVector 查询 API 的 `partition` 参数只接受字符串（不支持数组），脚本通过**串行多次查询**实现多分区，每个分区独立查询后合并结果，按相似度排序。 ### 步骤 3：读取原始文档补充上下文 根据检索结果中的 `source` 字段，用 `read_file` / `pdf` / `docx` 加载相关章节。 ### 步骤 4：网络搜索补充（如需要） 使用 `Web Search` 搜索实时信息或知识库未覆盖的背景知识。 ### 步骤 5：生成 Markdown 报告 报告格式规范： ```markdown # [报告标题] > 生成时间：[YYYY-MM-DD HH:mm] > 知识库：[知识库名称] > 来源：[x] 篇相关文档 + 网络补充 ## 摘要 [简要概述] ## 一、[章节标题] [内容...] ## 参考来源 - [文件名] - 分区: [分区名] ``` **报告保存路径：** `<工作目录>/<知识库名称>/reports/report_[时间戳]_[主题].md` ### 步骤 6：生成思维导图 使用 `visualizer:interactive` 工具生成交互式思维导图： - 提取标题层级（# / ## / ###） - 每个节点不超过 20 字 - 最多 3 级展示 **导图保存路径：** `<工作目录>/<知识库名称>/mindmaps/mindmap_[时间戳]_[主题].html` ### 步骤 7：输出结果 1. 对话中展示 Markdown 报告 2. `visualizer:interactive` 展示思维导图 3. 告知用户文件保存路径 --- ## 工作流程四：查看分区状态 **触发词：** "有哪些分区"、"分区列表"、"查看分区" ```bash python <技能路径>/scripts/partition_list.py --kb-path <知识库路径> ``` --- ## 文档格式支持 | 格式 | 读取方式 | |------|--------| | `.md` | 直接 `read_file` | | `.pdf` | `pdf` skill | | `.docx` / `.doc` | `docx` skill | --- ## 脚本汇总 | 脚本 | 用途 | 关键参数 | |------|------|--------| | `init_knowledge_base.py` | 创建新知识库 | `<名称> --path <工作目录>` | | `upload_to_vector.py` | 上传/索引文件 | `--kb-path <知识库路径>` | | `query_knowledge_base.py` | 多分区查询 | `<问题> --kb-path --partition` | | `partition_list.py` | 查看分区状态 | `--kb-path` | 所有脚本均支持 **自动发现知识库**：不传 `--kb-path` 时，自动在当前目录或父目录中查找第一个包含 `raw_docs/` + `config.json` 的文件夹。 --- ## 注意事项 1. **Python 环境**：用户需先安装 Python 3.8+ 和依赖包（pdfplumber, python-docx, requests） 2. **首次使用**：提醒用户先创建知识库并填写 `config.json` 3. **知识库为空**：告知用户先往 `raw_docs` 目录放入文档 4. **上传失败**：提示检查 API Key、Endpoint、Collection 名称是否正确 5. **分区路由**：上传时 `partition` 必须放在请求体顶层 6. **分区查询**：DashVector `partition` 参数只接受字符串，多分区通过串行查询实现 7. **多知识库切换**：由用户指定知识库名称，AI 负责在工作目录下查找

path: README.md
content: # 知识库问答技能 - 用户安装手册

> 适用对象：有 Word/PPT/PDF/Markdown 课件或笔记，希望基于自有文档进行 AI 问答的教师。

---

## 一、技能简介

本技能让你的 AI 助手能够：

1. **向量化存储**：将本地文档（PDF/MD/DOCX）上传到阿里云 DashVector 向量数据库
2. **语义问答**：基于文档内容回答问题，像聊天一样检索知识
3. **分区管理**：按主题分类管理（如 MySQL / Oracle / Java 分区隔离）
4. **多知识库**：在同一个工作目录下管理多个独立知识库
5. **报告生成**：将问答结果整理为 Markdown 报告 + 思维导图

---

## 二、系统要求

| 项目 | 要求 |
|------|------|
| Python | 3.8 或更高版本 |
| 依赖包 | `pdfplumber`, `python-docx`, `requests` |
| 阿里云账号 | DashVector + 百炼（Embedding）免费额度可用 |

### 安装 Python 依赖

```bash
pip install pdfplumber python-docx requests
```

---

## 三、阿里云凭证申请（免费）

### 3.1 DashVector（向量数据库）

1. 访问：https://dashvector.console.aliyun.com
2. 登录阿里云账号（可用支付宝账号）
3. 进入「API-KEY」页面，创建一个 API Key
4. 进入「集合」页面，创建一个 Collection：
   - 名称：任意（如 `my-knowledge-base`）
   - 向量维度：**1024**
   - 距离类型：Cosine（余弦相似度）
5. 复制 **Collection 访问地址**（格式如 `https://vrs-cn-xxxx.dashvector.cn-hangzhou.aliyuncs.com`）

### 3.2 百炼（向量生成）

1. 访问：https://bailian.console.aliyun.com
2. 登录后进入「API-KEY」页面
3. 创建一个 API Key

> **免费额度**：DashVector 新用户有免费额度，百炼 Embedding 也有免费调用次数，日常个人使用足够。

---

## 四、安装步骤

### 4.1 安装技能（复制文件夹）

将本技能包（`knowledge-qa-skill/`）完整复制到 WorkBuddy 的技能目录：

```
~/.workbuddy/skills/knowledge-qa/
```

> 如果你不清楚 WorkBuddy 的技能目录位置，请询问你的 AI 助手。

### 4.2 创建第一个知识库

假设你的工作目录下有课件文件，现在开始创建知识库：

**通过 AI 助手执行：**

```
请帮我创建一个名为 MyNotes 的知识库
```

AI 会运行初始化脚本，自动创建目录结构：

```
MyNotes/
├── raw_docs/
│   └── default/        ← 在这里放你的文档
├── config.json          ← 下一步要填写
├── indexed_files.json
└── README.md
```

### 4.3 填写配置

打开 `MyNotes/config.json`，将以下字段替换为你的阿里云凭证：

```json
{
  "knowledge_base": {
    "name": "MyNotes",
    "root_path": "MyNotes/raw_docs",
    "indexed_files_path": "MyNotes/indexed_files.json"
  },
  "dashvector": {
    "api_key": "在这里填入你的 DashVector API Key",
    "endpoint": "在这里填入 Collection 访问地址",
    "collection_name": "my-knowledge-base",
    "dimension": 1024,
    "model": "text-embedding-v3"
  },
  "bailian": {
    "api_key": "在这里填入你的百炼 API Key",
    "model": "text-embedding-v3"
  }
}
```

### 4.4 放入文档

将你的课件/笔记放到 `raw_docs/` 目录下：

```
MyNotes/raw_docs/
├── default/              ← 默认分区，直接放根目录
│   ├── 课程介绍.md
│   └── 第一章笔记.pdf
├── mysql/                ← MySQL 专题分区（创建子文件夹=创建分区）
│   └── InnoDB特性.pdf
└── oracle/
    └── 备份恢复.docx
```

---

## 五、使用流程

### 5.1 上传文档到向量库

告诉 AI：

```
帮我把 MyNotes 知识库的文件上传索引
```

AI 自动：
- 扫描 `raw_docs/` 下所有文件
- 提取文本内容
- 生成向量并上传到 DashVector
- 完成后告知结果

### 5.2 提问

告诉 AI：

```
基于 MyNotes 知识库回答：InnoDB 有哪些特性？
```

AI 自动：
- 在向量库中检索相关内容
- 结合原始文档给出回答
- 在对话中展示答案

### 5.3 生成报告

如果回答满意，告诉 AI：

```
把这个回答整理成报告并生成思维导图
```

AI 自动：
- 生成 Markdown 格式报告
- 生成交互式思维导图
- 保存到 `MyNotes/reports/` 和 `MyNotes/mindmaps/`

---

## 六、分区使用指南

### 什么是分区？

分区用于**按主题隔离检索范围**。例如：
- `raw_docs/mysql/` 下的文件 → 属于 `mysql` 分区
- `raw_docs/oracle/` 下的文件 → 属于 `oracle` 分区
- 直接放在 `raw_docs/` 根目录 → 属于 `default` 分区

### 查询指定分区

```
基于 MyNotes 知识库的 mysql 分区回答：主键索引和唯一索引有什么区别？
```

### 查询多个分区

```
基于 MyNotes 知识库的 mysql 和 oracle 分区回答：两者的索引机制有何不同？
```

### 查询全部分区

```
基于 MyNotes 知识库回答：这个知识库涵盖了哪些内容？
```

---

## 七、常见问题

### Q: 上传失败怎么办？
检查 `config.json` 中的 API Key、Endpoint、Collection 名称是否正确。

### Q: 文档格式支持哪些？
支持 `.md`、`.pdf`、`.docx`、`.doc`。

### Q: 可以管理多个知识库吗？
可以！在同一个工作目录下创建多个知识库文件夹（如 `MyNotes`、`JavaNotes`），每个有独立的配置和向量库。

### Q: 如何删除分区中的数据？
删除 `raw_docs/` 下对应子目录中的文件，然后重新运行上传脚本（会自动清理旧记录）。

---

## 八、文件结构一览

```
knowledge-qa-skill/              ← 技能包（复制到此目录）
├── SKILL.md                     ← 技能定义（AI 使用）
├── README.md                    ← 本手册（用户阅读）
├── config.template.json          ← 配置模板
└── scripts/
    ├── init_knowledge_base.py   ← 创建知识库
    ├── upload_to_vector.py      ← 上传/索引文件
    ├── query_knowledge_base.py  ← 语义问答
    └── partition_list.py        ← 查看分区状态

工作目录/
└── MyNotes/                    ← 用户创建的知识库
    ├── raw_docs/                ← 放文档
    ├── config.json              ← 填凭证
    ├── indexed_files.json       ← 自动维护
    └── README.md                ← 自动生成
```


path: config.template.json
content: {
  "_comment": "配置模板 - 请复制为 config.json 并填入你的凭证",
  "knowledge_base": {
    "name": "YOUR_KB_NAME",
    "root_path": "YOUR_KB_NAME/raw_docs",
    "indexed_files_path": "YOUR_KB_NAME/indexed_files.json"
  },
  "dashvector": {
    "_instructions": "请登录 https://dashvector.console.aliyun.com 获取以下信息",
    "api_key": "YOUR_DASHVECTOR_API_KEY",
    "endpoint": "https://vrs-cn-xxxx.dashvector.cn-hangzhou.aliyuncs.com",
    "collection_name": "YOUR_COLLECTION_NAME",
    "dimension": 1024,
    "model": "text-embedding-v3"
  },
  "bailian": {
    "_instructions": "请登录 https://bailian.console.aliyun.com 获取以下信息",
    "api_key": "YOUR_BAILIAN_API_KEY",
    "model": "text-embedding-v3"
  }
}


path: scripts/init_knowledge_base.py
content: # -*- coding: utf-8 -*-
"""
init_knowledge_base.py - 初始化一个新知识库
用法：
    python init_knowledge_base.py <知识库名称> [--path <工作目录>]
示例：
    python init_knowledge_base.py MySQLNotes
    python init_knowledge_base.py MySQLNotes --path "D:/workspace"
"""

import sys as _sys
_sys.stdout.reconfigure(encoding='utf-8')
_sys.stderr.reconfigure(encoding='utf-8')

import argparse
import json
import os
import sys


def create_config_template(collection_name, base_path):
    """
    创建 config.json 模板。
    base_path: 知识库的根目录
    """
    kb_root = base_path
    raw_docs_path = os.path.join(kb_root, "raw_docs")
    indexed_files_path = os.path.join(kb_root, "indexed_files.json")

    config = {
        "knowledge_base": {
            "name": os.path.basename(base_path),
            "root_path": raw_docs_path,
            "indexed_files_path": indexed_files_path
        },
        "dashvector": {
            "api_key": "YOUR_DASHVECTOR_API_KEY",
            "endpoint": "https://vrs-cn-xxxx.dashvector.cn-hangzhou.aliyuncs.com",
            "collection_name": collection_name,
            "dimension": 1024,
            "model": "text-embedding-v3"
        },
        "bailian": {
            "api_key": "YOUR_BAILIAN_API_KEY",
            "model": "text-embedding-v3"
        }
    }
    return config


def init_knowledge_base(kb_name, base_path=None):
    """
    在 base_path 下创建知识库 kb_name 的目录结构和配置模板。
    返回 (kb_path, config_path)
    """
    if base_path is None:
        base_path = os.getcwd()

    kb_path = os.path.join(base_path, kb_name)

    if os.path.exists(kb_path):
        raw_docs = os.path.join(kb_path, "raw_docs")
        config_json = os.path.join(kb_path, "config.json")
        if os.path.isdir(raw_docs) and os.path.isfile(config_json):
            print(f"[x] 知识库 '{kb_name}' 已存在（{kb_path}）")
            return None, None
        else:
            print(f"[!] 目录 '{kb_path}' 存在但不是有效知识库，将覆盖创建结构")
    else:
        os.makedirs(kb_path, exist_ok=True)

    dirs_to_create = [
        kb_path,
        os.path.join(kb_path, "raw_docs"),
        os.path.join(kb_path, "raw_docs", "default"),
    ]
    for d in dirs_to_create:
        os.makedirs(d, exist_ok=True)

    collection_name = kb_name.replace(" ", "_").replace("-", "_")
    config = create_config_template(collection_name, kb_path)
    config_path = os.path.join(kb_path, "config.json")
    with open(config_path, "w", encoding="utf-8") as f:
        json.dump(config, f, ensure_ascii=False, indent=2)

    indexed_path = os.path.join(kb_path, "indexed_files.json")
    with open(indexed_path, "w", encoding="utf-8") as f:
        json.dump({"last_updated": None, "files": []}, f, ensure_ascii=False, indent=2)

    readme_path = os.path.join(kb_path, "README.md")
    readme_content = f"""# {kb_name} 知识库

## 目录结构

```
{os.path.basename(kb_path)}/
├── raw_docs/           <- 放你的文档（PDF/MD/DOCX）
│   ├── default/        <- 默认分区（直接放根目录的文件）
│   ├── mysql/         <- 创建子文件夹 = 创建分区
│   ├── oracle/
│   └── (其他主题)/
├── config.json         <- 配置文件（请填写阿里云凭证）
└── indexed_files.json  <- 自动维护，无需手动编辑
```

## 分区使用方式

在 `raw_docs/` 下创建子文件夹，即可创建分区：
- `raw_docs/mysql/` -> 分区名 `mysql`
- `raw_docs/MySQL实战/` -> 分区名 `mysql_`（自动转小写）
- 直接放在 `raw_docs/` 根目录 -> 分区名 `default`

## 配置 config.json

请打开 `config.json`，填入你的阿里云凭证：
1. `dashvector.api_key` - DashVector API Key
2. `dashvector.endpoint` - Collection 访问地址
3. `dashvector.collection_name` - 集合名称（建议与知识库名一致）
4. `bailian.api_key` - 百炼 API Key（用于生成向量）

凭证获取地址：
- DashVector: https://dashvector.console.aliyun.com/api-key
- 百炼: https://bailian.console.aliyun.com/api-key
"""
    with open(readme_path, "w", encoding="utf-8") as f:
        f.write(readme_content)

    return kb_path, config_path


def main():
    parser = argparse.ArgumentParser(description="初始化一个新的知识库")
    parser.add_argument("kb_name", help="知识库名称（同时作为文件夹名和默认 Collection 名）")
    parser.add_argument("--path", dest="base_path", default=None,
                        help="工作目录路径（知识库将创建在此目录下），默认使用当前目录")
    args = parser.parse_args()

    kb_name = args.kb_name.strip()
    if not kb_name:
        print("[x] 知识库名称不能为空")
        sys.exit(1)

    base_path = args.base_path
    if base_path:
        if not os.path.isdir(base_path):
            print(f"[x] 目录不存在: {base_path}")
            sys.exit(1)
    else:
        base_path = os.getcwd()
        print(f"未指定 --path，使用当前目录: {base_path}")

    print(f"\n{'=' * 50}")
    print(f"初始化知识库：{kb_name}")
    print(f"工作目录：{base_path}")
    print(f"{'=' * 50}\n")

    kb_path, config_path = init_knowledge_base(kb_name, base_path)

    if kb_path is None:
        sys.exit(1)

    print(f"[+] 知识库初始化完成！\n")
    print(f"目录结构：")
    print(f"   {kb_path}/")
    print(f"   ├── raw_docs/")
    print(f"   |   └── default/")
    print(f"   ├── config.json       <- [!] 请填写阿里云凭证")
    print(f"   ├── indexed_files.json")
    print(f"   └── README.md")
    print(f"\n[!] 下一步：请编辑 config.json 填入你的阿里云凭证")
    print(f"   凭证获取方式：")
    print(f"   - DashVector: https://dashvector.console.aliyun.com/api-key")
    print(f"   - 百炼: https://bailian.console.aliyun.com/api-key")
    print(f"\n使用方式：")
    print(f"   1. 往 raw_docs/（及其子目录）放入文档")
    print(f"   2. 告诉 AI：'上传到知识库 {kb_name}'")
    print(f"   3. 问 AI：'基于 {kb_name} 知识库回答...'")


if __name__ == "__main__":
    main()


path: scripts/upload_to_vector.py
content: # -*- coding: utf-8 -*-
import sys as _sys
_sys.stdout.reconfigure(encoding='utf-8')
_sys.stderr.reconfigure(encoding='utf-8')

"""
知识库向量上传脚本
功能：
1. 扫描 raw_docs 目录下的所有文件（PDF/MD/DOCX）
2. 识别新文件（对比 indexed_files.json）
3. 提取文本内容并分块
4. 调用百炼 API 生成向量
5. 调用 DashVector API 上传
6. 更新 indexed_files.json
"""

import json
import os
import hashlib
import re
import time
from datetime import datetime

import pdfplumber
import requests
import argparse


def load_config(kb_path=None):
    """
    加载知识库配置。
    kb_path: 知识库根目录路径。
             不传 -> 自动发现（当前目录或父目录）
             传入 -> 直接在 kb_path 下找 config.json
    """
    if kb_path is None:
        kb_path = _find_knowledge_base()
        if kb_path is None:
            raise FileNotFoundError(
                "未找到知识库目录。请确保当前目录或其子目录包含 raw_docs/ 和 config.json，"
                "或使用 --kb-path 参数指定知识库路径。"
            )
    config_path = os.path.join(kb_path, "config.json")
    with open(config_path, "r", encoding="utf-8") as f:
        return json.load(f)


def _find_knowledge_base(start_path=None):
    """
    自动发现知识库目录。
    查找逻辑：当前目录 -> 当前目录子目录 -> 父目录子目录
    """
    if start_path is None:
        start_path = os.getcwd()

    if os.path.isdir(start_path):
        raw = os.path.join(start_path, "raw_docs")
        cfg = os.path.join(start_path, "config.json")
        if os.path.isdir(raw) and os.path.isfile(cfg):
            return start_path

    candidates = []
    if os.path.isdir(start_path):
        for item in os.listdir(start_path):
            sub = os.path.join(start_path, item)
            if os.path.isdir(sub):
                raw = os.path.join(sub, "raw_docs")
                cfg = os.path.join(sub, "config.json")
                if os.path.isdir(raw) and os.path.isfile(cfg):
                    candidates.append(sub)

    if candidates:
        return candidates[0]

    parent = os.path.dirname(start_path)
    if parent and parent != start_path and os.path.isdir(parent):
        for item in os.listdir(parent):
            sub = os.path.join(parent, item)
            if os.path.isdir(sub):
                raw = os.path.join(sub, "raw_docs")
                cfg = os.path.join(sub, "config.json")
                if os.path.isdir(raw) and os.path.isfile(cfg):
                    return sub

    return None


def load_indexed_files(config):
    path = config["knowledge_base"]["indexed_files_path"]
    if os.path.exists(path):
        with open(path, "r", encoding="utf-8") as f:
            return json.load(f)
    return {"last_updated": None, "files": []}


def save_indexed_files(config, data):
    path = config["knowledge_base"]["indexed_files_path"]
    data["last_updated"] = datetime.now().isoformat()
    os.makedirs(os.path.dirname(path), exist_ok=True)
    with open(path, "w", encoding="utf-8") as f:
        json.dump(data, f, ensure_ascii=False, indent=2)


def safe_doc_id(filename, chunk_index):
    """生成 DashVector 合法的文档 ID（只允许 [a-zA-Z0-9_-!@#$%+=.]）"""
    import urllib.parse
    name = os.path.splitext(filename)[0]
    safe_name = urllib.parse.quote(name, safe='')
    if len(safe_name) > 40:
        safe_name = safe_name[:40]
    return f"{safe_name}_{chunk_index}"


def scan_raw_docs(root_path):
    """
    递归扫描 raw_docs 目录下的所有文件。
    返回的每个文件记录包含 partition 字段：
    - 直接在 raw_docs 根目录 -> partition = "default"
    - 在 raw_docs/FOLDER/ 子目录 -> partition = folder_name（小写，字母数字下划线）
    """
    files = []
    for root, dirs, filenames in os.walk(root_path):
        rel_root = os.path.relpath(root, root_path)

        if rel_root == ".":
            partition = "default"
        else:
            parts = rel_root.split(os.sep)
            folder_name = parts[0]
            partition = re.sub(r"[^a-z0-9_]", "_", folder_name.lower())

        for filename in filenames:
            if filename.lower() == "readme.md":
                continue
            filepath = os.path.join(root, filename)
            rel_path = os.path.relpath(filepath, root_path)
            files.append({
                "path": filepath,
                "relative_path": rel_path,
                "name": filename,
                "size": os.path.getsize(filepath),
                "partition": partition
            })
    return files


def extract_text_pdf(filepath):
    """使用 pdfplumber 提取 PDF 文本"""
    texts = []
    try:
        with pdfplumber.open(filepath) as pdf:
            for i, page in enumerate(pdf.pages):
                text = page.extract_text()
                if text and text.strip():
                    texts.append(f"[Page {i+1}]\n{text}")
        return "\n\n".join(texts)
    except Exception as e:
        print(f"  PDF read failed: {e}")
        return ""


def extract_text_md(filepath):
    """读取 Markdown 文本"""
    with open(filepath, "r", encoding="utf-8", errors="ignore") as f:
        return f.read()


def extract_text_docx(filepath):
    """读取 Word 文档文本"""
    try:
        from docx import Document
        doc = Document(filepath)
        return "\n".join([p.text for p in doc.paragraphs if p.text.strip()])
    except ImportError:
        print("  Note: python-docx not installed, cannot read .docx files")
        return ""


def extract_content(filepath):
    """根据文件类型提取文本内容"""
    ext = os.path.splitext(filepath)[1].lower()
    if ext == ".pdf":
        return extract_text_pdf(filepath)
    elif ext == ".md":
        return extract_text_md(filepath)
    elif ext in [".docx", ".doc"]:
        return extract_text_docx(filepath)
    else:
        return ""


def chunk_text(text, chunk_size=600, overlap=50):
    """将长文本分块，便于向量检索"""
    paragraphs = re.split(r'\n\s*\n', text)
    chunks = []
    current_chunk = ""

    for para in paragraphs:
        if len(current_chunk) + len(para) < chunk_size:
            current_chunk += para + "\n\n"
        else:
            if current_chunk.strip():
                chunks.append(current_chunk.strip())
            overlap_text = current_chunk[-overlap:] if len(current_chunk) > overlap else current_chunk
            current_chunk = overlap_text + para + "\n\n"

    if current_chunk.strip():
        chunks.append(current_chunk.strip())

    return [c[:2000] for c in chunks]


def generate_embeddings_bailian(config, texts):
    """调用百炼 API 生成向量"""
    [REDACTED_SECRET]"bailian"]["api_key"]
    model = config["bailian"].get("model", "text-embedding-v3")
    dimension = config["dashvector"]["dimension"]

    url = "https://dashscope.aliyuncs.com/compatible-mode/v1/embeddings"
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }

    all_embeddings = []
    for i, text in enumerate(texts):
        payload = {
            "model": model,
            "input": text,
            "dimensions": dimension,
            "encoding_format": "float"
        }
        try:
            response = requests.post(url, headers=headers, json=payload, timeout=30)
            if response.status_code == 200:
                result = response.json()
                embedding = result["data"][0]["embedding"]
                all_embeddings.append(embedding)
            else:
                print(f"  Vector generation failed: {response.status_code} - {response.text[:200]}")
                return None
        except Exception as e:
            print(f"  Vector generation error: {e}")
            return None

    return all_embeddings


def create_partition_if_not_exists(config, partition_name):
    """
    如果 DashVector 分区不存在，则创建它。
    partition_name: 合法的分区名（小写字母、数字、下划线）
    """
    [REDACTED_SECRET]"dashvector"]["api_key"]
    endpoint = config["dashvector"]["endpoint"]
    collection = config["dashvector"]["collection_name"]
    url = f"{endpoint}/v1/collections/{collection}/partitions"
    headers = {
        "dashvector-auth-token": api_key,
        "Content-Type": "application/json"
    }

    payload = {"name": partition_name}
    try:
        response = requests.post(url, headers=headers, json=payload, timeout=30)
        resp_data = response.json()
        if response.status_code == 200 and resp_data.get("code") == 0:
            return True, f"Partition '{partition_name}' created"
        elif resp_data.get("code") == 60010:
            return True, f"Partition '{partition_name}' already exists"
        else:
            return False, f"Failed to create partition: code={resp_data.get('code')} msg={resp_data.get('message')}"
    except Exception as e:
        return False, f"Partition creation error: {e}"


def upload_with_retry(url, headers, payload, max_retries=4, base_delay=1.0):
    """
    带指数退避的 HTTP POST 请求。
    遇到 405 / 429 / 503 时自动重试。
    返回 (response, attempts, last_error_hint)，attempts 为尝试次数，
    last_error_hint 为连续 405 时的第一条错误信息（方便诊断）。
    """
    last_resp = None
    last_error_hint = None
    consecutive_405 = 0

    for attempt in range(max_retries):
        try:
            resp = requests.post(url, headers=headers, json=payload, timeout=60)
            last_resp = resp

            if resp.status_code == 200:
                if attempt > 0:
                    print(f"    [+] Attempt {attempt + 1}: HTTP 200 OK (recovered after {attempt} retries)")
                return resp, attempt + 1, None

            if resp.status_code in (405, 429, 503):
                consecutive_405 += 1
                # 记录第一条 405 的响应体用于诊断（通常包含了拒绝原因）
                if consecutive_405 == 1:
                    last_error_hint = f"HTTP {resp.status_code}: {resp.text[:300]}"
                # 连续 3 次 405 说明服务端明确拒绝，不必继续重试
                if consecutive_405 >= 3:
                    print(f"    [x] HTTP {resp.status_code} repeated {consecutive_405}x — request rejected by server, stopping retries")
                    break
                delay = base_delay * (2 ** attempt)
                print(f"    [!] HTTP {resp.status_code} (attempt {attempt + 1}/{max_retries}), "
                      f"waiting {delay:.1f}s... [reason: {resp.text[:100]}]")
                time.sleep(delay)
                continue

            # 其他 HTTP 错误（400/401/500 等）不重试
            print(f"    [x] HTTP {resp.status_code}: {resp.text[:200]}")
            return resp, attempt + 1, None

        except requests.exceptions.Timeout:
            print(f"    [!] Timeout (attempt {attempt + 1}/{max_retries}), retrying...")
            consecutive_405 = 0
            time.sleep(base_delay * (2 ** attempt))
        except Exception as e:
            print(f"    [!] Request error: {e} (attempt {attempt + 1}/{max_retries})")
            consecutive_405 = 0
            time.sleep(base_delay * (2 ** attempt))

    # 所有重试均失败
    return last_resp, max_retries, last_error_hint


def upload_to_dashvector(config, docs, partition, batch_size=1, max_retries=4):
    """
    调用 DashVector API 上传文档（含向量），小批量多次上传。
    partition: 物理分区名（放在请求体顶层，作用于整批文档）
    遇到 405/429/503 时自动指数退避重试（最多 4 次）。
    """
    [REDACTED_SECRET]"dashvector"]["api_key"]
    endpoint = config["dashvector"]["endpoint"]
    collection = config["dashvector"]["collection_name"]
    url = f"{endpoint}/v1/collections/{collection}/docs"
    headers = {
        "dashvector-auth-token": api_key,
        "Content-Type": "application/json"
    }

    all_success = True
    all_results = []
    total = len(docs)

    for i in range(0, total, batch_size):
        batch = docs[i:i+batch_size]
        batch_num = i // batch_size + 1
        batch_start = i + 1
        batch_end = min(i + batch_size, total)
        print(f"    Uploading {batch_start}-{batch_end}/{total}...")

        payload = {"partition": partition, "docs": batch}
        payload_size = len(json.dumps(payload, ensure_ascii=False).encode("utf-8"))
        print(f"    Payload size: {payload_size} bytes ({len(batch)} doc(s))")

        try:
            response, attempts, error_hint = upload_with_retry(url, headers, payload,
                                                               max_retries=max_retries, base_delay=1.0)

            if response is None or response.status_code != 200:
                # 所有重试全部失败
                all_success = False
                http_code = response.status_code if response is not None else "timeout"
                err_msg = f"HTTP {http_code} after {attempts} attempts"
                if error_hint:
                    err_msg += f" | first 405: {error_hint}"
                all_results.append({
                    "batch": batch_num,
                    "range": f"{batch_start}-{batch_end}",
                    "error": err_msg
                })
                print(f"    [x] Upload failed after {attempts} attempts")
                if error_hint:
                    print(f"    [i] First 405 hint: {error_hint}")
                continue

            # HTTP 200，检查业务层 code
            resp_data = response.json()

            if resp_data.get("code") != 0:
                all_success = False
                all_results.append({
                    "batch": batch_num,
                    "range": f"{batch_start}-{batch_end}",
                    "error": f"API code={resp_data.get('code')}: {resp_data.get('message')}"
                })
                print(f"    [x] API error: code={resp_data.get('code')}")
                continue

            outputs = resp_data.get("output", [])
            # -2027 = Duplicate Key，幂等跳过不算失败
            failed = [d for d in outputs if d.get("code") != 0 and d.get("code") != -2027]
            if failed:
                all_success = False
                all_results.append({"batch": batch_num, "range": f"{batch_start}-{batch_end}", "failed": failed})
                print(f"    [!] {len(failed)} chunk(s) failed")
            else:
                skipped = [d for d in outputs if d.get("code") == -2027]
                ok_count = len(outputs) - len(skipped)
                if skipped:
                    print(f"    [+] {ok_count} uploaded, {len(skipped)} duplicates skipped")
                else:
                    print(f"    [+] {ok_count} uploaded successfully")

        except Exception as e:
            all_success = False
            all_results.append({
                "batch": batch_num,
                "range": f"{batch_start}-{batch_end}",
                "exception": str(e)
            })
            print(f"    [x] Exception: {e}")

        # 批次间稍微间隔，降低触发限流概率
        if i + batch_size < total:
            time.sleep(0.3)

    return all_success, all_results


def main():
    parser = argparse.ArgumentParser(description="Knowledge base vector upload script")
    parser.add_argument("--kb-path", dest="kb_path", default=None,
                        help="Knowledge base path, auto-detect if not provided")
    args = parser.parse_args()

    print("=" * 50)
    print("Knowledge Base Vector Upload")
    print("=" * 50)

    kb_path = args.kb_path
    if kb_path:
        print(f"\nSpecified knowledge base: {kb_path}")
    else:
        print(f"\nAuto-detecting knowledge base...")

    config = load_config(kb_path)
    print(f"\nConfig loaded")
    print(f"  DashVector Collection: {config['dashvector']['collection_name']}")
    print(f"  Bailian Model: {config['bailian']['model']}")
    print(f"  Vector Dimension: {config['dashvector']['dimension']}")

    root_path = config["knowledge_base"]["root_path"]
    all_files = scan_raw_docs(root_path)
    print(f"\nScan complete, found {len(all_files)} files")

    from collections import Counter
    partition_counts = Counter(f["partition"] for f in all_files)
    print(f"  Partition distribution: {dict(partition_counts)}")

    indexed = load_indexed_files(config)
    indexed_names = {f["name"] for f in indexed["files"]}
    print(f"Already indexed: {len(indexed_names)} files")

    new_files = [f for f in all_files if f["name"] not in indexed_names]
    if not new_files:
        print("\nNo new files found, upload complete")
        return

    print(f"\nFound {len(new_files)} new files:")
    for f in new_files:
        print(f"  - [{f['partition']}] {f['relative_path']}")

    needed_partitions = set(f["partition"] for f in new_files)
    print(f"\nEnsuring partitions exist: {needed_partitions}")
    for pname in sorted(needed_partitions):
        if pname == "default":
            continue
        ok, msg = create_partition_if_not_exists(config, pname)
        print(f"  {msg}")

    total_uploaded = 0
    for file_info in new_files:
        partition = file_info["partition"]
        print(f"\nProcessing: [{partition}] {file_info['relative_path']}")

        content = extract_content(file_info["path"])
        if not content:
            print(f"  Skipped: unable to extract content")
            continue

        print(f"  Text length: {len(content)} chars")
        chunks = chunk_text(content)
        print(f"  Chunks: {len(chunks)}")

        print(f"  Generating vectors...")
        embeddings = generate_embeddings_bailian(config, chunks)
        if not embeddings:
            print(f"  Vector generation failed, skipping")
            continue

        print(f"  Vectors generated ({len(embeddings)} embeddings)")

        docs = []
        for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)):
            docs.append({
                "id": safe_doc_id(f"{partition}_{file_info['name']}", i),
                "vector": embedding,
                "fields": {
                    "content": chunk,
                    "source": file_info["name"],
                    "partition": partition,
                    "chunk_index": i
                }
            })

        print(f"  Uploading to DashVector ({len(docs)} chunks, partition={partition})...")
        success, result = upload_to_dashvector(config, docs, partition=partition, max_retries=4)
        if success:
            indexed["files"].append({
                "name": file_info["name"],
                "relative_path": file_info["relative_path"],
                "size": file_info["size"],
                "indexed_at": datetime.now().isoformat(),
                "chunks": len(chunks),
                "partition": partition
            })
            total_uploaded += 1
            print(f"  Upload successful! ({len(chunks)} chunks)")
        else:
            print(f"  Upload failed: {result}")

    save_indexed_files(config, indexed)
    print(f"\nIndex record updated: {len(indexed['files'])} files total")

    print(f"\n{'=' * 50}")
    print(f"Upload complete! {total_uploaded} files uploaded")
    print(f"{'=' * 50}")


if __name__ == "__main__":
    main()


path: scripts/query_knowledge_base.py
content: # -*- coding: utf-8 -*-
"""
query_knowledge_base.py - Multi-partition query script for knowledge base
Usage:
    # Single partition
    python query_knowledge_base.py "What are InnoDB features" --kb-path "MyNotes" --partition mysql

    # Multiple partitions (comma-separated)
    python query_knowledge_base.py "Difference between MySQL and Oracle" --kb-path "MyNotes" --partition mysql,oracle

    # All partitions
    python query_knowledge_base.py "What is in this knowledge base" --kb-path "MyNotes"

    # Auto-detect knowledge base
    python query_knowledge_base.py "What is xxx" --partition mysql

    # Custom topk
    python query_knowledge_base.py "xxx" --topk 10
"""

import sys as _sys
_sys.stdout.reconfigure(encoding='utf-8')
_sys.stderr.reconfigure(encoding='utf-8')

import argparse
import json
import os
import sys
import urllib.parse

import requests

# Import from upload_to_vector for shared config loading
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from upload_to_vector import load_config


def find_knowledge_base(start_path=None):
    """
    Auto-detect knowledge base.
    Logic:
    1. start_path has raw_docs/ + config.json -> return directly
    2. start_path subdir has raw_docs/ + config.json -> return first match
    3. Search parent directory (max 2 levels)
    """
    if start_path is None:
        start_path = os.getcwd()

    candidates = []

    if os.path.isdir(start_path):
        raw_docs = os.path.join(start_path, "raw_docs")
        config_json = os.path.join(start_path, "config.json")
        if os.path.isdir(raw_docs) and os.path.isfile(config_json):
            return start_path
        for item in os.listdir(start_path):
            sub_path = os.path.join(start_path, item)
            if os.path.isdir(sub_path):
                sub_raw = os.path.join(sub_path, "raw_docs")
                sub_cfg = os.path.join(sub_path, "config.json")
                if os.path.isdir(sub_raw) and os.path.isfile(sub_cfg):
                    candidates.append(sub_path)

    if candidates:
        return candidates[0]

    parent = os.path.dirname(start_path)
    if parent != start_path and os.path.isdir(parent):
        for item in os.listdir(parent):
            sub_path = os.path.join(parent, item)
            if os.path.isdir(sub_path):
                sub_raw = os.path.join(sub_path, "raw_docs")
                sub_cfg = os.path.join(sub_path, "config.json")
                if os.path.isdir(sub_raw) and os.path.isfile(sub_cfg):
                    return sub_path

    return None


def generate_embedding(config, text):
    """Call Bailian API to generate embedding for a single text"""
    [REDACTED_SECRET]"bailian"]["api_key"]
    model = config["bailian"].get("model", "text-embedding-v3")
    dimension = config["dashvector"]["dimension"]

    url = "https://dashscope.aliyuncs.com/compatible-mode/v1/embeddings"
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }
    payload = {
        "model": model,
        "input": text,
        "dimensions": dimension,
        "encoding_format": "float"
    }

    try:
        response = requests.post(url, headers=headers, json=payload, timeout=30)
        if response.status_code == 200:
            result = response.json()
            return result["data"][0]["embedding"]
        else:
            print(f"  [!] Vector generation failed: HTTP {response.status_code} - {response.text[:200]}")
            return None
    except Exception as e:
        print(f"  [!] Vector generation error: {e}")
        return None


def query_partition(config, vector, partition, topk=5, include_fields=None):
    """
    Query within a single partition.
    Returns (success, results_list)
    """
    dv = config["dashvector"]
    endpoint = dv["endpoint"]
    collection = dv["collection_name"]
    api_key = dv["api_key"]

    if include_fields is None:
        include_fields = ["content", "source", "partition", "chunk_index"]

    url = f"{endpoint}/v1/collections/{collection}/query"
    headers = {
        "dashvector-auth-token": api_key,
        "Content-Type": "application/json"
    }
    payload = {
        "vector": vector,
        "topk": topk,
        "partition": partition,
        "include_fields": True,
        "fields": include_fields
    }

    try:
        response = requests.post(url, headers=headers, json=payload, timeout=30)
        resp_data = response.json()

        if resp_data.get("code") == 0:
            outputs = resp_data.get("output", [])
            return True, outputs
        else:
            return False, {
                "code": resp_data.get("code"),
                "message": resp_data.get("message", "Unknown error")
            }
    except Exception as e:
        return False, {"exception": str(e)}


def list_partitions(config):
    """List all partition names in DashVector"""
    dv = config["dashvector"]
    endpoint = dv["endpoint"]
    collection = dv["collection_name"]
    api_key = dv["api_key"]

    url = f"{endpoint}/v1/collections/{collection}/partitions"
    headers = {"dashvector-auth-token": api_key}

    try:
        response = requests.get(url, headers=headers, timeout=30)
        resp_data = response.json()
        if response.status_code == 200 and resp_data.get("code") == 0:
            output = resp_data.get("output", [])
            if output and isinstance(output[0], str):
                return [p for p in output]
            return [p.get("name", "unknown") for p in output]
        return []
    except:
        return []


def query_multi_partition(config, question, partitions=None, topk=5):
    """
    Multi-partition query entry point.
    partitions: None (all partitions) or ["mysql", "oracle"] list
    Returns merged and sorted results
    """
    print(f"  Generating question embedding...")
    vector = generate_embedding(config, question)
    if vector is None:
        return None

    if partitions is None or partitions == ["all"] or partitions == []:
        print(f"  Getting all partitions...")
        all_parts = list_partitions(config)
        if not all_parts:
            print("  [!] No partitions found, trying 'default'...")
            all_parts = ["default"]
        partitions = all_parts
        print(f"  All partitions: {partitions}")
    else:
        if isinstance(partitions, str):
            partitions = [p.strip() for p in partitions.split(",")]
        print(f"  Querying partitions: {partitions}")

    all_results = []
    for pname in partitions:
        print(f"  Querying partition [{pname}]...")
        ok, result = query_partition(config, vector, pname, topk=topk)
        if ok:
            for item in result:
                item["_partition"] = pname
            all_results.extend(result)
            print(f"    -> [{pname}] returned {len(result)} results")
        else:
            print(f"    -> [{pname}] failed: {result}")

    all_results.sort(key=lambda x: x.get("score", 0), reverse=True)

    return all_results


def format_results(results, max_display=10):
    """Format query results as readable text"""
    if not results:
        return "No relevant results found"

    lines = []
    lines.append(f"\n{'=' * 60}")
    lines.append(f"Query Results ({len(results)} total)")
    lines.append(f"{'=' * 60}\n")

    for i, r in enumerate(results[:max_display], 1):
        score = r.get("score", 0)
        partition = r.get("_partition", "unknown")
        fields = r.get("fields", {})
        content = fields.get("content", "")[:300]
        source = fields.get("source", "Unknown source")

        lines.append(f"[{i}] Partition: {partition} | Similarity: {score:.4f}")
        lines.append(f"  Source: {source}")
        lines.append(f"  Content: {content}...")
        lines.append("")

    if len(results) > max_display:
        lines.append(f"({len(results) - max_display} more results not shown)")

    return "\n".join(lines)


def main():
    parser = argparse.ArgumentParser(description="Knowledge base multi-partition query")
    parser.add_argument("question", help="Query question (wrap in quotes)")
    parser.add_argument("--kb-path", dest="kb_path", default=None,
                        help="Knowledge base path, auto-detect if not provided")
    parser.add_argument("--partition", dest="partition", default=None,
                        help="Partition name(s), comma-separated. Omit for all partitions.")
    parser.add_argument("--topk", dest="topk", type=int, default=5,
                        help="Max results per partition, default 5")
    args = parser.parse_args()

    question = args.question.strip()
    if not question:
        print("[x] Question cannot be empty")
        sys.exit(1)

    kb_path = args.kb_path
    if kb_path is None:
        print("[*] Auto-detecting knowledge base...")
        kb_path = find_knowledge_base()
        if kb_path is None:
            print("[x] Knowledge base not found. Ensure current directory or subdirectory contains raw_docs/ and config.json")
            print("   or use --kb-path to specify the path")
            sys.exit(1)
        print(f"   Found: {kb_path}")

    config_path = os.path.join(kb_path, "config.json")
    if not os.path.isfile(config_path):
        print(f"[x] Config file not found: {config_path}")
        sys.exit(1)

    with open(config_path, "r", encoding="utf-8") as f:
        config = json.load(f)

    kb_name = config.get("knowledge_base", {}).get("name", os.path.basename(kb_path))
    collection = config["dashvector"]["collection_name"]

    print(f"\n{'=' * 60}")
    print(f"Knowledge Base Query")
    print(f"  KB Name: {kb_name}")
    print(f"  Collection: {collection}")
    print(f"  Question: {question}")
    print(f"{'=' * 60}\n")

    partitions = None
    if args.partition:
        partitions = [p.strip() for p in args.partition.split(",")]
        print(f"Specified partitions: {partitions}\n")

    results = query_multi_partition(config, question, partitions=partitions, topk=args.topk)

    if results is None:
        print("[x] Query failed (vector generation failed)")
        sys.exit(1)

    print(format_results(results))

    structured = {
        "question": question,
        "kb_name": kb_name,
        "collection": collection,
        "kb_path": kb_path,
        "total_results": len(results),
        "results": [
            {
                "score": r.get("score"),
                "partition": r.get("_partition"),
                "source": r.get("fields", {}).get("source"),
                "chunk_index": r.get("fields", {}).get("chunk_index"),
                "content": r.get("fields", {}).get("content", "")
            }
            for r in results
        ]
    }

    temp_json_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "query_result.json")
    with open(temp_json_path, "w", encoding="utf-8") as f:
        json.dump(structured, f, ensure_ascii=False, indent=2)
    print(f"\n[*] Structured results written to: {temp_json_path}")


if __name__ == "__main__":
    main()


path: scripts/partition_list.py
content: # -*- coding: utf-8 -*-
"""
partition_list.py - Check DashVector partition status
Usage:
    python partition_list.py
    python partition_list.py --kb-path "MyNotes"
"""

import sys
import os
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

import requests
import json
import argparse
from upload_to_vector import load_config


def list_partitions_with_counts(config):
    """
    Get partition list and document counts.
    DashVector has no direct count API, so we probe each partition
    and cross-reference with local indexed_files.json.
    """
    dv = config["dashvector"]
    endpoint = dv["endpoint"]
    collection = dv["collection_name"]
    api_key = dv["api_key"]
    dim = dv["dimension"]

    parts_url = f"{endpoint}/v1/collections/{collection}/partitions"
    r = requests.get(parts_url, headers={"dashvector-auth-token": api_key}, timeout=30)
    rd = r.json()

    if r.status_code != 200 or rd.get("code") != 0:
        print(f"Failed to get partition list: {rd}")
        return []

    partitions = rd.get("output", [])
    if partitions and isinstance(partitions[0], str):
        partitions = [{"name": p} for p in partitions]

    indexed_path = config["knowledge_base"]["indexed_files_path"]
    local_counts = {}
    if os.path.exists(indexed_path):
        with open(indexed_path, "r", encoding="utf-8") as f_read:
            idx = json.load(f_read)
        from collections import Counter
        local_counts = Counter(file_entry.get("partition", "default") for file_entry in idx.get("files", []))

    results = []
    for p in partitions:
        pname = p.get("name", "unknown")

        query_url = f"{endpoint}/v1/collections/{collection}/query"
        q = requests.post(query_url, headers={
            "dashvector-auth-token": api_key,
            "Content-Type": "application/json"
        }, json={
            "vector": [0.0] * dim,
            "topk": 100,
            "partition": pname,
            "include_fields": False
        }, timeout=30)
        qd = q.json()

        has_data = qd.get("code") == 0 and len(qd.get("output", [])) > 0
        local_count = local_counts.get(pname, 0)

        results.append({
            "partition": pname,
            "has_data": has_data,
            "local_indexed": local_count,
            "note": "Has data" if has_data else "Empty (no data in this partition)"
        })

    return results


def main():
    parser = argparse.ArgumentParser(description="Check DashVector partition status")
    parser.add_argument("--kb-path", dest="kb_path", default=None,
                        help="Knowledge base path, auto-detect if not provided")
    args = parser.parse_args()

    print("=" * 50)
    print("DashVector Partition Status")
    print("=" * 50)

    config = load_config(args.kb_path)
    print(f"\nCollection: {config['dashvector']['collection_name']}\n")

    results = list_partitions_with_counts(config)

    if not results:
        print("No partitions found")
        return

    print(f"{'Partition':<20} {'Local Index':<12} {'Cloud Status':<20} Note")
    print("-" * 70)
    for r in results:
        print(f"{r['partition']:<20} {r['local_indexed']:<12} {str(r['has_data']):<20} {r['note']}")

    print(f"\nTotal: {len(results)} partitions")
    print("\nNote: Local Index = count from indexed_files.json")
    print("      Cloud Status = probing via vector query")


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)