Skill: magicbook1108/test-ragflow (version 1.0.0)

--- name: ragflow-dataset-ingest description: "Use for RAGFlow dataset and retrieval tasks: create, list, inspect, update, or delete datasets; list, upload, update, or delete documents in a dataset; start or stop parsing uploaded documents; check parser status through `parse_status.py`; and retrieve relevant chunks from RAGFlow datasets with `search.py`." --- # RAGFlow Dataset And Retrieval Use only the bundled scripts in `scripts/`. Prefer `--json` for script execution so the returned fields can be relayed exactly. ## Trigger Phrases Use this skill when the user intent matches any of these actions, in either Chinese or English, even if the wording is informal. - List datasets Trigger phrases: "list datasets", "show datasets", "show all datasets", "what datasets do I have" "列出数据集", "查看数据集", "显示所有数据集", "我有哪些数据集" - Show dataset details Trigger phrases: "dataset details", "show dataset info", "inspect dataset", "describe this dataset" "数据集详情", "查看数据集信息", "检查数据集", "显示这个数据集的信息" - Create dataset Trigger phrases: "create dataset", "new dataset", "add a dataset" "创建数据集", "新建数据集", "添加数据集" - Update dataset Trigger phrases: "rename dataset", "update dataset", "change dataset description", "modify dataset" "重命名数据集", "更新数据集", "修改数据集描述", "编辑数据集" - Delete dataset Trigger phrases: "delete dataset", "remove dataset", "drop dataset" "删除数据集", "移除数据集", "清理数据集" - Upload documents Trigger phrases: "upload file", "upload document", "add file to dataset", "import files" "上传文件", "上传文档", "把文件加到数据集", "导入文件" - List documents Trigger phrases: "list documents", "show files", "show documents in dataset", "what files are in this dataset" "列出文档", "列出文件", "查看数据集里的文件", "这个数据集里有哪些文件" - Update document Trigger phrases: "rename document", "update document", "edit document metadata" "重命名文档", "更新文档", "修改文档元数据" - Delete document Trigger phrases: "delete document", "remove file", "delete file from dataset" "删除文档", "删除文件", "从数据集删除文件" - Start parsing Trigger phrases: "parse document", "start parsing", "run parsing", "re-parse document" "解析文档", "开始解析", "执行解析", "重新解析文档" - Stop parsing Trigger phrases: "stop parsing", "cancel parsing", "stop parse job" "停止解析", "取消解析", "停止解析任务" - Check parsing status or progress Trigger phrases: "check parsing status", "show progress", "what is still running", "parsing progress" "查看解析状态", "查看进度", "还有哪些在运行", "解析进度" - Search / retrieve Trigger phrases: "search knowledge base", "search dataset", "retrieve chunks", "find relevant content" "搜索知识库", "搜索数据集", "检索内容", "查找相关内容" - List models Trigger phrases: "list models", "show models", "available models", "what models are available", "list llms", "show llms", "model providers" "列出模型", "查看模型", "可用模型", "有哪些模型", "列出大模型", "查看大模型", "模型供应商" - Show model details or provider grouping Trigger phrases: "model details", "show model details", "group models by provider", "list all models including unavailable ones" "模型详情", "查看模型详情", "按供应商查看模型", "列出所有模型包括不可用模型" ## Workflow ```bash python scripts/datasets.py create "My Dataset" --description "Optional description" python scripts/datasets.py list python scripts/datasets.py info DATASET_ID python scripts/update_dataset.py DATASET_ID --name "Renamed Dataset" ``` 1. Create a dataset or confirm the target dataset. 2. Upload files. When asking the user to provide files, prefer explicit local file paths. If the user's client supports drag-and-drop, they may also drop files into the conversation, but local paths work best and large drag-and-drop uploads may fail. ```bash python scripts/upload.py list DATASET_ID --json python scripts/upload.py DATASET_ID /path/to/file1 [/path/to/file2 ...] python scripts/update_document.py DATASET_ID DOC_ID --name "Renamed Document" ``` Upload output returns `document_ids`. Pass those IDs into the next step. Use delete commands when the task is cleanup instead of ingest: ```bash python scripts/datasets.py delete --ids DATASET_ID1,DATASET_ID2 python scripts/upload.py delete DATASET_ID --ids DOC_ID1,DOC_ID2 ``` ⚠️ **DELETION REQUIRES CONFIRMATION**: Before executing any delete operation (datasets or documents): 1. List items to be deleted with details (names, IDs, counts) 2. Ask user for explicit confirmation (e.g., "yes", "confirm", "proceed") 3. Only proceed after user confirms For dataset deletion, execute only against explicit dataset IDs. For document deletion, execute only against explicit document IDs. If the user gives filenames or a fuzzy description, list documents first, resolve exact IDs, get confirmation, and only then run the delete command. Do not perform fuzzy batch delete operations. 3. Start parsing, or stop parsing when explicitly requested. ```bash python scripts/parse.py DATASET_ID DOC_ID1 [DOC_ID2 ...] python scripts/stop_parse_documents.py DATASET_ID DOC_ID1 [DOC_ID2 ...] ``` `parse.py` only sends the parse request and returns immediately. `stop_parse_documents.py` sends a stop request for explicit document IDs, then returns one current status snapshot for those documents. Use `parse_status.py` when the user asks to check progress or current parser status. If `parse_status.py` returns an error, return the error message directly and do not guess the cause. If a document status includes `progress_msg`, surface it automatically. For `FAIL` documents, treat `progress_msg` as the primary error detail. For later requests like "Check the progress" or "Which files are currently being parsed", resolve scope by specificity: - no dataset specified: inspect all datasets and all documents - dataset specified: inspect all documents in that dataset - document IDs specified: inspect only those documents 4. Retrieve chunks from one or more datasets when the user asks knowledge questions against RAGFlow. ```bash python scripts/search.py "What does the warranty policy say?" python scripts/search.py "What does the warranty policy say?" DATASET_ID python scripts/search.py --dataset-ids DATASET_ID1,DATASET_ID2 --doc-ids DOC_ID1,DOC_ID2 "What does the warranty policy say?" python scripts/search.py --threshold 0.7 --top-k 10 "query" python scripts/search.py --retrieval-test --kb-id DATASET_ID "query" ``` 5. Inspect configured LLM factories and models when the user asks what models are available. ```bash python scripts/list_models.py --json python scripts/list_models.py --include-details --json python scripts/list_models.py --group-by factory --json python scripts/list_models.py --all --group-by factory --include-details --json ``` ## Model Listing - default to listing only available models - default to grouping by model `type` - if multiple model groups or models are shown, prefer a table - if the user asks for details, provider grouping, or unavailable models, expand the output accordingly - prefer the grouped result in `groups` instead of reintroducing the raw server response shape ## Progress And Status Output - summarize `RUNNING` items first when reporting progress - status reporting should reflect the dataset document list API as-is; do not fabricate percentage progress ## Error Output - when returning raw script output, preserve error fields exactly as returned - if JSON output contains `api_error`, present that object directly rather than replacing it with a guessed explanation ## Scope Support only: - create datasets - list datasets - inspect datasets - update datasets - delete datasets - upload documents to a dataset - list documents in a dataset - update documents in a dataset - delete documents from a dataset - start parsing documents in a dataset - stop parsing documents in a dataset - list all currently parsing documents in a dataset for broad progress requests - aggregate parse progress across all datasets for broad progress requests - retrieve relevant chunks from one or more datasets - limit retrieval to specific dataset IDs or document IDs - use `retrieval_test` for single-dataset debugging when needed - list configured LLM factories and models through the web API Do not use this skill for chunk editing, memory APIs, or other RAGFlow capabilities outside dataset operations and retrieval. ## Environment Configure `.env` with: ```bash RAGFLOW_API_URL=http://127.0.0.1:9380 RAGFLOW_API_KEY=ragflow-your-api-key-here RAGFLOW_DATASET_IDS=["dataset-id-1", "dataset-id-2"] ``` `RAGFLOW_API_URL` is the primary base URL setting. `RAGFLOW_BASE_URL` is accepted as a legacy fallback for compatibility, but new setups should use `RAGFLOW_API_URL`. When loading the repository `.env`, the bundled scripts only import `RAGFLOW_` variables into the process environment. ## Endpoints - `GET /api/v1/datasets` - `POST /api/v1/datasets` - `PUT /api/v1/datasets/<dataset_id>` - `DELETE /api/v1/datasets` - `GET /api/v1/datasets/<dataset_id>/documents` - `POST /api/v1/datasets/<dataset_id>/documents` - `PUT /api/v1/datasets/<dataset_id>/documents/<document_id>` - `DELETE /api/v1/datasets/<dataset_id>/documents` - `POST /api/v1/datasets/<dataset_id>/chunks` - `DELETE /api/v1/datasets/<dataset_id>/chunks` - `POST /api/v1/retrieval` - `POST /api/v1/chunk/retrieval_test` - `GET /v1/llm/my_llms` ## Commands ```bash python scripts/datasets.py create "Example Dataset" --description "Quarterly reports" python scripts/datasets.py create "Example Dataset" --embedding-model bge-m3 --chunk-method naive --permission me python scripts/datasets.py list python scripts/datasets.py info DATASET_ID python scripts/update_dataset.py DATASET_ID --name "Updated Dataset" --description "Updated description" python scripts/datasets.py delete --ids DATASET_ID1,DATASET_ID2 --json python scripts/upload.py list DATASET_ID --json python scripts/upload.py DATASET_ID ./example.pdf --json python scripts/update_document.py DATASET_ID DOC_ID --name "Updated Document" --enabled 1 --json python scripts/upload.py delete DATASET_ID --ids DOC_ID1,DOC_ID2 --json python scripts/datasets.py list --json python scripts/parse.py DATASET_ID DOC_ID1 --json python scripts/stop_parse_documents.py DATASET_ID DOC_ID1 --json python scripts/parse_status.py DATASET_ID --json python scripts/search.py "query" python scripts/search.py "query" DATASET_ID --json python scripts/search.py --dataset-ids DATASET_ID1,DATASET_ID2 --doc-ids DOC_ID1,DOC_ID2 "query" --json python scripts/search.py --retrieval-test --kb-id DATASET_ID "query" --json python scripts/list_models.py --json python scripts/list_models.py --include-details --json ``` ## Notes - Dataset creation supports `--avatar`, `--description`, `--embedding-model`, `--permission`, `--chunk-method`, and `--language`. - Dataset update supports explicit flags or `--data` JSON payloads through `scripts/update_dataset.py`. - Upload does not start parsing by itself. - Prefer local file paths for uploads. Drag-and-drop is acceptable only when the client's UI supports it, and it may fail for large files. - Document update supports explicit flags or `--data` JSON payloads through `scripts/update_document.py`. - Parsing is asynchronous. - `parse.py` returns immediately after the start request. Do not wait for parse status in this command. - When a script returns an error, proactively include the error message in the same reply. Do not wait for the user to ask for the error details. - If JSON output contains `api_error`, return that API error object directly instead of replacing it with a guessed explanation. - If JSON output contains `error`, `api_error.message`, `status_error.message`, or `error_detail.message`, surface that message to the user immediately. - A stop request may not flip the document to `CANCEL` immediately. Use the returned snapshot or `scripts/parse_status.py` to confirm the terminal state. - For broad status/progress requests with no dataset specified, list datasets first and aggregate `scripts/parse_status.py DATASET_ID` across all datasets. - If a dataset is specified, prefer `scripts/parse_status.py DATASET_ID` without `--doc-ids`. - If document IDs are specified, pass `--doc-ids`. - Retrieval defaults to `POST /api/v1/retrieval`. - `scripts/search.py` accepts `RAGFLOW_DATASET_IDS` from `.env` as the default dataset scope when the user does not specify dataset IDs explicitly. - Use `--retrieval-test` only when the user wants single-dataset debugging or specifically asks for that endpoint. - `scripts/list_models.py` calls `GET /v1/llm/my_llms` and uses `RAGFLOW_API_KEY` Bearer auth only.

path: reference.md
content: # Output Format Reference

Style guide for consistent RAGFlow skill responses.
Apply this reference to all user-facing output for this skill.

## Format Decision Matrix

| Information Type | Format | Use Case |
|-----------------|--------|----------|
| Multiple items (3+) with attributes | **Table** | Datasets list, search results |
| Sequential steps | **Numbered List** | Upload workflow, procedures |
| Features/options | **Bullet List** | Capability overview |
| Structured data | **JSON Code Block** | API responses |
| Document content | **Quote Block** | Retrieved chunks |
| Single object properties | **Definition List** | Dataset details |
| Status | **Emoji + Text** | ✅ Done, 🟡 Running, ❌ Failed |

## Common Formats

### Tables (3+ items)
```markdown
| Dataset | Docs | Chunks | Status |
|---------|------|--------|--------|
| delete  | 4    | 53     | ✅     |
```
- Abbreviate long IDs: `abc123...`
- Use emojis for status: ✅ ❌ 🟡 ⚠️

### Bullet Lists
```markdown
- **Upload documents** to dataset
- **Start parsing** to generate chunks
```
- Start with verbs for actions
- Max 2 indent levels

### Numbered Lists
```markdown
1. Create dataset
2. Upload files
3. Start parsing
```
- Use for sequential procedures

### Status Icons
| Icon | Meaning |
|------|---------|
| ✅ | Success |
| ❌ | Failed |
| 🟡 | In Progress |
| ⚠️ | Warning |
| ⬜ | Empty |

## Response Templates

**List operations:**
```markdown
📋 **Datasets** (3 total)

| Name | ID | Status | Chunks |
|------|-----|--------|--------|
| test | abc... | ✅ | 152 |
```

**Search results:**
```markdown
🔍 **Results** (2 found)

| # | Source | Similarity | Content |
|---|--------|------------|---------|
| 1 | doc.pdf | 85% | excerpt... |
```

**Object details:**
```markdown
📊 **Dataset Details**

**ID:** `1ce917df20e411f191a984ba59bc54d9`
**Name:** delete
**Chunks:** 53
```



path: agents/openai.yaml
content: interface:
  display_name: "RAGFlow Dataset And Retrieval"
  short_description: "Manage datasets, docs, parsing, and retrieval"
  default_prompt: "Use $ragflow-dataset-ingest to manage RAGFlow datasets, documents, start or stop parsing, check parser status, and retrieve relevant chunks. For document delete or stop actions, require explicit document IDs."
environment:
  required:
    - name: "RAGFLOW_API_URL"
      description: "Trusted base URL for the target RAGFlow server."
    - name: "RAGFLOW_API_KEY"
      description: "Bearer API key for the target RAGFlow server."
      primary_credential: true


path: scripts/list_models.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-

import argparse
import os
import urllib.error
import urllib.parse
import urllib.request
from typing import Any

from common import (
    ApiError,
    ConfigError,
    DataError,
    ScriptError,
    configure_stdio_utf8,
    current_timestamp,
    decode_json_body,
    decode_json_response,
    decode_response_text,
    ensure_success,
    extract_error_message,
    format_json,
    load_repo_env,
    repo_root_from_path,
    require_api_key,
    resolve_base_url,
)

HTTP_TIMEOUT = 30
DEFAULT_API_PATH = "/v1/llm/my_llms"
DEFAULT_GROUP_BY = "type"
AVAILABLE_STATUSES = {"1", 1, True}


def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="List available LLMs from the RAGFlow web API.")
    parser.add_argument(
        "--include-details",
        action="store_true",
        help="Request the detailed response shape returned by include_details=true.",
    )
    parser.add_argument(
        "--group-by",
        choices=("type", "factory"),
        default=DEFAULT_GROUP_BY,
        help=f"Group models by type or factory (default: {DEFAULT_GROUP_BY})",
    )
    parser.add_argument(
        "--all",
        action="store_true",
        dest="include_unavailable",
        help="Include unavailable models. By default only available models are listed.",
    )
    parser.add_argument(
        "--api-path",
        default=DEFAULT_API_PATH,
        help=f"Endpoint path (default: {DEFAULT_API_PATH})",
    )
    parser.add_argument("--json", action="store_true", dest="json_output", help="Print JSON output")
    parser.add_argument(
        "--base-url",
        help="Base URL for the RAGFlow server (priority: --base-url > RAGFLOW_API_URL > RAGFLOW_BASE_URL > HOST_ADDRESS > default)",
    )
    return parser.parse_args(argv)


def _build_headers() -> dict[str, str]:
    return {
        "Accept": "application/json",
        "Authorization": f"Bearer {require_api_key()}",
    }


def _request_json(url: str, headers: dict[str, str]) -> dict[str, Any]:
    request_obj = urllib.request.Request(url, headers=headers, method="GET")

    try:
        with urllib.request.urlopen(request_obj, timeout=HTTP_TIMEOUT) as response:
            return decode_json_response(response.read())
    except urllib.error.HTTPError as exc:
        body = exc.read()
        response_payload = decode_json_body(body)
        response_text = decode_response_text(body)
        message = extract_error_message(body)
        if message:
            raise ApiError(
                message,
                http_status=exc.code,
                api_code=response_payload.get("code") if isinstance(response_payload, dict) else None,
                response_payload=response_payload,
                response_body=response_text,
            ) from None
        raise ApiError(
            f"HTTP request failed with status {exc.code}.",
            http_status=exc.code,
            api_code=response_payload.get("code") if isinstance(response_payload, dict) else None,
            response_payload=response_payload,
            response_body=response_text,
        ) from None
    except urllib.error.URLError as exc:
        reason = getattr(exc, "reason", exc)
        raise ApiError(f"HTTP request failed: {reason}") from None


def _normalize_llm(item: dict[str, Any]) -> dict[str, Any]:
    normalized = {
        "id": item.get("id"),
        "type": item.get("type"),
        "name": item.get("name"),
        "used_token": item.get("used_token"),
        "status": item.get("status"),
        "factory": item.get("factory"),
    }

    if "api_base" in item:
        normalized["api_base"] = item.get("api_base")
    if "max_tokens" in item:
        normalized["max_tokens"] = item.get("max_tokens")
    return normalized


def _normalize_data(data: Any) -> list[dict[str, Any]]:
    if not isinstance(data, dict):
        raise DataError("Response missing data object.")

    factories: list[dict[str, Any]] = []
    for factory_name, factory_payload in data.items():
        if not isinstance(factory_name, str) or not factory_name.strip():
            raise DataError("Response contains an invalid llm_factory key.")
        if not isinstance(factory_payload, dict):
            raise DataError(f"Factory {factory_name!r} payload is malformed.")

        llms = factory_payload.get("llm")
        if not isinstance(llms, list):
            raise DataError(f"Factory {factory_name!r} is missing llm list.")

        factories.append(
            {
                "name": factory_name,
                "tags": factory_payload.get("tags"),
                "llms": [
                    _normalize_llm({**item, "factory": factory_name})
                    for item in llms
                    if isinstance(item, dict)
                ],
            }
        )

    factories.sort(key=lambda item: item["name"])
    return factories


def _is_available(status: Any) -> bool:
    return status in AVAILABLE_STATUSES or str(status).strip() in {"1", "true", "True"}


def _group_models(
    factories: list[dict[str, Any]],
    *,
    group_by: str,
    include_details: bool,
    include_unavailable: bool,
) -> list[dict[str, Any]]:
    grouped: dict[str, dict[str, Any]] = {}

    for factory in factories:
        factory_name = factory["name"]
        factory_tags = factory.get("tags")
        for llm in factory["llms"]:
            if not include_unavailable and not _is_available(llm.get("status")):
                continue

            key = factory_name if group_by == "factory" else str(llm.get("type") or "unknown")
            if key not in grouped:
                grouped[key] = {
                    "name": key,
                    "factory": factory_name if group_by == "factory" else None,
                    "tags": factory_tags if group_by == "factory" else None,
                    "models": [],
                }

            model = {
                "id": llm.get("id"),
                "name": llm.get("name"),
                "type": llm.get("type"),
                "factory": llm.get("factory"),
                "used_token": llm.get("used_token"),
                "status": llm.get("status"),
            }
            if include_details:
                if "api_base" in llm:
                    model["api_base"] = llm.get("api_base")
                if "max_tokens" in llm:
                    model["max_tokens"] = llm.get("max_tokens")

            grouped[key]["models"].append(model)

    groups = list(grouped.values())
    groups.sort(key=lambda item: item["name"])
    for group in groups:
        group["models"].sort(key=lambda item: ((item.get("name") or ""), (item.get("id") or "")))
    return groups


def list_models(
    *,
    base_url: str,
    include_details: bool,
    api_path: str,
    group_by: str,
    include_unavailable: bool,
) -> dict[str, Any]:
    path = api_path.strip() or DEFAULT_API_PATH
    if not path.startswith("/"):
        raise ConfigError("--api-path must start with '/'.")

    query = urllib.parse.urlencode({"include_details": str(include_details).lower()})
    payload = ensure_success(
        _request_json(
            f"{base_url}{path}?{query}",
            _build_headers(),
        )
    )

    factories = _normalize_data(payload.get("data"))
    groups = _group_models(
        factories,
        group_by=group_by,
        include_details=include_details,
        include_unavailable=include_unavailable,
    )
    return {
        "checked_at": current_timestamp(),
        "include_details": include_details,
        "group_by": group_by,
        "available_only": not include_unavailable,
        "factory_count": len(factories),
        "llm_count": sum(len(group["models"]) for group in groups),
        "groups": groups,
    }


def _format_text(payload: dict[str, Any]) -> str:
    lines = [
        f"Checked at: {payload['checked_at']}",
        f"Factories: {payload['factory_count']}",
        f"Available models: {payload['llm_count']}",
        f"Group by: {payload['group_by']}",
        f"Available only: {str(payload['available_only']).lower()}",
        f"Include details: {str(payload['include_details']).lower()}",
    ]

    for group in payload["groups"]:
        lines.extend(
            [
                "",
                f"[{group['name']}]",
                f"models: {len(group['models'])}",
            ]
        )
        if payload["group_by"] == "factory":
            lines.append(f"tags: {group.get('tags') if group.get('tags') is not None else 'none'}")
        for llm in group["models"]:
            lines.append(
                f"- {llm.get('name') or 'unknown'} "
                f"(id: {llm.get('id') or 'unknown'}, "
                f"type: {llm.get('type') or 'unknown'}, "
                f"factory: {llm.get('factory') or 'unknown'})"
            )
            if payload["include_details"]:
                lines.append(
                    f"  used_token: {llm.get('used_token') if llm.get('used_token') is not None else 'unknown'}, "
                    f"status: {llm.get('status') or 'unknown'}, "
                    f"api_base: {llm.get('api_base') or ''}, "
                    f"max_tokens: {llm.get('max_tokens') if llm.get('max_tokens') is not None else 'unknown'}"
                )
    return "\n".join(lines)


def main(argv: list[str] | None = None) -> int:
    configure_stdio_utf8()
    load_repo_env(repo_root_from_path(__file__))
    args = _parse_args(argv)

    try:
        base_url = resolve_base_url(args.base_url)
        payload = list_models(
            base_url=base_url,
            include_details=args.include_details,
            api_path=args.api_path,
            group_by=args.group_by,
            include_unavailable=args.include_unavailable,
        )
        print(format_json(payload) if args.json_output else _format_text(payload))
        return 0
    except ScriptError as exc:
        if args.json_output:
            print(format_json({"checked_at": current_timestamp(), "error": str(exc)}))
        else:
            print(f"Error: {exc}")
        return 1


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/update_dataset.py
content: #!/usr/bin/env python3
#
#  Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
#
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.
#

import argparse
import json
import sys
import urllib.parse
from pathlib import Path
from typing import Any

from common import (
    ConfigError,
    DataError,
    ScriptError,
    configure_stdio_utf8,
    ensure_success,
    load_repo_env,
    repo_root_from_path,
    request_json,
    require_api_key,
    resolve_base_url,
)


def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Update a dataset through the RAGFlow HTTP API.")
    parser.add_argument("dataset_id", help="Dataset ID")
    parser.add_argument("--name", help="Updated dataset name")
    parser.add_argument("--avatar", help="Updated avatar as data URL/base64")
    parser.add_argument("--description", help="Updated dataset description")
    parser.add_argument("--embedding-model", help="Updated embedding model")
    parser.add_argument("--permission", choices=("me", "team"), help="Updated dataset permission")
    parser.add_argument("--chunk-method", help="Updated chunk method")
    parser.add_argument("--pagerank", type=int, help="Updated pagerank")
    parser.add_argument(
        "--parser-config",
        help="Parser config JSON object or @path/to/file.json",
    )
    parser.add_argument(
        "--data",
        help="Raw JSON object payload or @path/to/file.json. Explicit flags override the same keys.",
    )
    parser.add_argument("--json", action="store_true", dest="json_output", help="Print raw JSON response")
    parser.add_argument(
        "--base-url",
        help="Base URL for the RAGFlow server "
        "(priority: --base-url > RAGFLOW_API_URL > RAGFLOW_BASE_URL > HOST_ADDRESS > default)",
    )
    return parser.parse_args(argv)


def _load_json_object(raw_value: str, option_name: str) -> dict[str, Any]:
    value = raw_value
    if raw_value.startswith("@"):
        path = Path(raw_value[1:]).expanduser()
        try:
            value = path.read_text(encoding="utf-8")
        except OSError as exc:
            raise ConfigError(f"Failed to read {option_name} file {path}: {exc}") from exc

    try:
        payload = json.loads(value)
    except json.JSONDecodeError as exc:
        raise ConfigError(f"{option_name} must be valid JSON: {exc.msg}.") from exc

    if not isinstance(payload, dict):
        raise ConfigError(f"{option_name} must be a JSON object.")
    return payload


def _build_payload(args: argparse.Namespace) -> dict[str, Any]:
    payload: dict[str, Any] = {}
    if args.data:
        payload.update(_load_json_object(args.data, "--data"))

    field_values = {
        "name": args.name,
        "avatar": args.avatar,
        "description": args.description,
        "embedding_model": args.embedding_model,
        "permission": args.permission,
        "chunk_method": args.chunk_method,
        "pagerank": args.pagerank,
    }
    for key, value in field_values.items():
        if value is not None:
            payload[key] = value

    if args.parser_config is not None:
        payload["parser_config"] = _load_json_object(args.parser_config, "--parser-config")

    if not payload:
        raise ConfigError("No update fields provided. Use --data or at least one explicit update flag.")
    return payload


def _build_url(base_url: str, dataset_id: str) -> str:
    encoded_dataset_id = urllib.parse.quote(dataset_id, safe="")
    return f"{base_url}/api/v1/datasets/{encoded_dataset_id}"
def _normalize_payload(payload: dict[str, Any]) -> dict[str, Any]:
    payload = ensure_success(payload)

    data = payload.get("data")
    if not isinstance(data, dict):
        raise DataError("Response missing data object.")
    return payload


def _print_summary(payload: dict[str, Any]) -> None:
    data = payload["data"]
    dataset_id = str(data.get("id", "")).strip() or "<missing-id>"
    name = str(data.get("name", "")).strip() or "<missing-name>"
    print(f"Updated dataset: {name} ({dataset_id})")

    details = []
    if data.get("description"):
        details.append(f"description={data['description']}")
    if data.get("embd_id"):
        details.append(f"embedding_model={data['embd_id']}")
    if data.get("permission"):
        details.append(f"permission={data['permission']}")
    if data.get("parser_id"):
        details.append(f"chunk_method={data['parser_id']}")
    if data.get("pagerank") is not None:
        details.append(f"pagerank={data['pagerank']}")
    if details:
        print("  " + ", ".join(details))


def main(argv: list[str] | None = None) -> int:
    configure_stdio_utf8()
    load_repo_env(repo_root_from_path(__file__))

    try:
        args = _parse_args(argv)
        base_url = resolve_base_url(args.base_url)
        [REDACTED_SECRET])
        payload = _build_payload(args)
        response = request_json(
            _build_url(base_url, args.dataset_id),
            api_key,
            method="PUT",
            body=json.dumps(payload).encode("utf-8"),
            content_type="application/json",
        )
        normalized = _normalize_payload(response)
    except ScriptError as exc:
        print(f"Error: {exc}", file=sys.stderr)
        return 1

    if args.json_output:
        print(json.dumps(normalized, ensure_ascii=False, indent=2))
    else:
        _print_summary(normalized)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/parse_status.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-

import argparse
import json
import sys
import urllib.parse
from dataclasses import asdict, dataclass
from typing import Any

from common import (
    ApiError,
    ConfigError,
    DataError,
    ScriptError,
    configure_stdio_utf8,
    current_timestamp,
    ensure_success,
    format_json,
    load_repo_env,
    repo_root_from_path,
    request_json,
    require_api_key,
    resolve_base_url,
    serialize_script_error,
)

DEFAULT_PAGE_SIZE = 100
STATUS_ORDER = ("UNSTART", "RUNNING", "DONE", "FAIL", "CANCEL")
TERMINAL_STATES = {"DONE", "FAIL", "CANCEL"}
RUN_STATUS_MAP = {
    "0": "UNSTART",
    "1": "RUNNING",
    "2": "CANCEL",
    "3": "DONE",
    "4": "FAIL",
}


@dataclass
class DocumentStatus:
    id: str
    name: str
    run: str
    chunk_count: int
    token_count: int
    progress_msg: Any | None = None


def parse_doc_ids(raw_value: str | None) -> list[str] | None:
    if raw_value is None:
        return None

    doc_ids: list[str] = []
    seen: set[str] = set()
    for item in raw_value.split(","):
        doc_id = item.strip()
        if not doc_id or doc_id in seen:
            continue
        seen.add(doc_id)
        doc_ids.append(doc_id)

    if not doc_ids:
        raise ConfigError("--doc-ids must include at least one document ID.")
    return doc_ids


def _build_documents_url(base_url: str, dataset_id: str, page: int, page_size: int) -> str:
    encoded_dataset_id = urllib.parse.quote(dataset_id, safe="")
    query = urllib.parse.urlencode({"page": page, "page_size": page_size})
    return f"{base_url}/api/v1/datasets/{encoded_dataset_id}/documents?{query}"


def _fetch_documents_page(base_url: str, api_key: str, dataset_id: str, page: int, page_size: int) -> tuple[list[dict[str, Any]], int]:
    payload = ensure_success(request_json(_build_documents_url(base_url, dataset_id, page, page_size), api_key))
    data = payload.get("data")
    if not isinstance(data, dict):
        raise DataError("Response missing data object.")

    docs = data.get("docs")
    total = data.get("total")
    if not isinstance(docs, list):
        raise DataError("Response missing data.docs.")
    if not isinstance(total, int):
        raise DataError("Response missing data.total.")
    return docs, total


def _fetch_all_documents(base_url: str, api_key: str, dataset_id: str) -> list[dict[str, Any]]:
    all_docs: list[dict[str, Any]] = []
    page = 1
    total: int | None = None

    while True:
        docs, page_total = _fetch_documents_page(base_url, api_key, dataset_id, page, DEFAULT_PAGE_SIZE)
        if total is None:
            total = page_total
        all_docs.extend(docs)

        if len(all_docs) >= total or not docs:
            return all_docs[:total]
        page += 1


def _coerce_required_string(document_id: str, field_name: str, value: Any) -> str:
    if not isinstance(value, str) or not value.strip():
        raise DataError(f"Document {document_id} is missing a valid {field_name}.")
    return value.strip()


def _coerce_required_int(document_id: str, field_name: str, value: Any) -> int:
    if isinstance(value, bool):
        raise DataError(f"Document {document_id} is missing a valid {field_name}.")
    try:
        number = int(value)
    except (TypeError, ValueError):
        raise DataError(f"Document {document_id} is missing a valid {field_name}.") from None
    if number < 0:
        raise DataError(f"Document {document_id} is missing a valid {field_name}.")
    return number


def _normalize_run(value: Any, document_id: str) -> str:
    if isinstance(value, int):
        mapped = RUN_STATUS_MAP.get(str(value))
        if mapped:
            return mapped
    elif isinstance(value, str):
        raw_value = value.strip()
        mapped = RUN_STATUS_MAP.get(raw_value)
        if mapped:
            return mapped
        normalized = raw_value.upper()
        if normalized in STATUS_ORDER:
            return normalized

    raise DataError(f"Document {document_id} has an unsupported run status: {value!r}.")


def _normalize_progress_msg(value: Any) -> Any | None:
    if isinstance(value, str):
        normalized = value.strip()
        return normalized or None
    return value


def _normalize_document(raw_doc: dict[str, Any]) -> DocumentStatus:
    if not isinstance(raw_doc, dict):
        raise DataError("Response contains a malformed document entry.")

    raw_id = raw_doc.get("id")
    if not isinstance(raw_id, str) or not raw_id.strip():
        raise DataError("Response contains a document with a missing id.")
    document_id = raw_id.strip()

    return DocumentStatus(
        id=document_id,
        name=_coerce_required_string(document_id, "name", raw_doc.get("name")),
        run=_normalize_run(raw_doc.get("run"), document_id),
        chunk_count=_coerce_required_int(document_id, "chunk_count", raw_doc.get("chunk_count")),
        token_count=_coerce_required_int(document_id, "token_count", raw_doc.get("token_count")),
        progress_msg=_normalize_progress_msg(raw_doc.get("progress_msg")),
    )


def _select_documents(documents: list[DocumentStatus], target_ids: list[str] | None) -> list[DocumentStatus]:
    if not target_ids:
        return documents

    documents_by_id = {document.id: document for document in documents}
    missing_ids = [doc_id for doc_id in target_ids if doc_id not in documents_by_id]
    if missing_ids:
        raise DataError("--doc-ids contains document IDs that were not found in the dataset: " + ", ".join(missing_ids))
    return [documents_by_id[doc_id] for doc_id in target_ids]


def _build_payload(dataset_id: str, documents: list[DocumentStatus]) -> dict[str, Any]:
    summary = {"total": len(documents)}
    for status in STATUS_ORDER:
        summary[status] = 0

    for document in documents:
        summary[document.run] += 1

    return {
        "dataset_id": dataset_id,
        "checked_at": current_timestamp(),
        "summary": summary,
        "documents": [asdict(document) for document in documents],
        "all_terminal": all(document.run in TERMINAL_STATES for document in documents),
    }


def collect_status_payload(
    dataset_id: str,
    target_ids: list[str] | None = None,
    *,
    base_url: str | None = None,
    api_key: str | None = None,
) -> dict[str, Any]:
    resolved_base_url = base_url or resolve_base_url()
    resolved_api_key = api_key or require_api_key()
    raw_documents = _fetch_all_documents(resolved_base_url, resolved_api_key, dataset_id)
    documents = [_normalize_document(raw_doc) for raw_doc in raw_documents]
    return _build_payload(dataset_id, _select_documents(documents, target_ids))


def format_status_text(payload: dict[str, Any]) -> str:
    summary = payload["summary"]
    documents = payload["documents"]
    lines = [
        f"Dataset: {payload['dataset_id']}",
        f"Checked at: {payload['checked_at']}",
        f"Watching: {summary['total']} document(s)",
        "",
    ]

    for status in STATUS_ORDER:
        lines.append(f"{status}: {summary[status]}")

    for document in documents:
        lines.extend(
            [
                "",
                f"[{document['run']}] {document['name']}",
                f"id: {document['id']}",
                f"chunks: {document['chunk_count']}",
                f"tokens: {document['token_count']}",
            ]
        )
        progress_msg = document.get("progress_msg")
        if progress_msg is not None:
            rendered_progress = progress_msg if isinstance(progress_msg, str) else json.dumps(progress_msg, ensure_ascii=False)
            if rendered_progress:
                label = "error" if document["run"] == "FAIL" else "message"
                lines.append(f"{label}: {rendered_progress}")
    return "\n".join(lines)


def _write_error(
    exc: ScriptError,
    json_output: bool,
    dataset_id: str | None = None,
    payload: dict[str, Any] | None = None,
) -> None:
    if json_output:
        error_payload: dict[str, Any] = {}
        if payload:
            error_payload.update(payload)
        elif dataset_id:
            error_payload["dataset_id"] = dataset_id
        if "checked_at" not in error_payload:
            error_payload["checked_at"] = current_timestamp()
        error_payload["error"] = str(exc)
        error_detail = serialize_script_error(exc)
        if isinstance(exc, ApiError):
            error_payload["api_error"] = error_detail
        else:
            error_payload["error_detail"] = error_detail
        print(format_json(error_payload))
        return

    print(f"Error: {exc}", file=sys.stderr)


def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Show document parse status for a dataset.")
    parser.add_argument("dataset_id", help="Dataset ID")
    parser.add_argument("--doc-ids", help="Comma-separated document IDs to monitor")
    parser.add_argument("--json", action="store_true", dest="json_output", help="Print JSON output")
    parser.add_argument(
        "--base-url",
        help="Base URL for the RAGFlow server (priority: --base-url > RAGFLOW_API_URL > RAGFLOW_BASE_URL > HOST_ADDRESS > default)",
    )
    return parser.parse_args(argv)


def main(argv: list[str] | None = None) -> int:
    configure_stdio_utf8()
    load_repo_env(repo_root_from_path(__file__))
    args = _parse_args(argv)

    try:
        target_ids = parse_doc_ids(args.doc_ids)
        [REDACTED_SECRET])
        base_url = resolve_base_url(args.base_url)

        payload = collect_status_payload(
            args.dataset_id,
            target_ids,
            base_url=base_url,
            [REDACTED_SECRET],
        )
        print(format_json(payload) if args.json_output else format_status_text(payload))
        return 0
    except ScriptError as exc:
        _write_error(exc, args.json_output, dataset_id=args.dataset_id)
        return 1


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/parse.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-

import argparse
import json

from common import (
    ScriptError,
    configure_stdio_utf8,
    current_timestamp,
    ensure_success,
    format_json,
    load_repo_env,
    repo_root_from_path,
    request_json,
    require_api_key,
    resolve_base_url,
    serialize_script_error,
)


def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Start parsing uploaded RAGFlow documents and return immediately.")
    parser.add_argument("dataset_id", help="Dataset ID")
    parser.add_argument("document_ids", nargs="+", help="Document IDs to parse")
    parser.add_argument("--json", action="store_true", dest="json_output", help="Print JSON output")
    parser.add_argument(
        "--base-url",
        help="Base URL for the RAGFlow server (priority: --base-url > RAGFLOW_API_URL > RAGFLOW_BASE_URL > HOST_ADDRESS > default)",
    )
    return parser.parse_args(argv)


def start_parse(dataset_id: str, document_ids: list[str], *, base_url: str, api_key: str) -> dict[str, object]:
    url = f"{base_url}/api/v1/datasets/{dataset_id}/chunks"
    body = json.dumps({"document_ids": document_ids}).encode("utf-8")
    response = ensure_success(
        request_json(
            url,
            api_key,
            method="POST",
            body=body,
            content_type="application/json",
        )
    )
    return {
        "dataset_id": dataset_id,
        "document_ids": document_ids,
        "parse_requested_at": current_timestamp(),
        "api_response": response,
    }


def _format_payload(payload: dict[str, object]) -> str:
    lines = [
        f"Dataset: {payload['dataset_id']}",
        f"Parse requested at: {payload['parse_requested_at']}",
    ]
    api_response = payload.get("api_response")
    if isinstance(api_response, dict):
        message = api_response.get("message")
        if isinstance(message, str) and message.strip():
            lines.append(f"API message: {message.strip()}")
    return "\n".join(lines)


def _build_error_payload(
    args: argparse.Namespace,
    exc: ScriptError,
    *,
    parse_request: dict[str, object] | None,
) -> dict[str, object]:
    payload: dict[str, object] = (
        dict(parse_request)
        if parse_request is not None
        else {
            "dataset_id": args.dataset_id,
            "document_ids": args.document_ids,
            "parse_requested_at": current_timestamp(),
        }
    )
    payload["error"] = str(exc)
    error_detail = serialize_script_error(exc)
    if error_detail.get("type") == "ApiError":
        payload["api_error"] = error_detail
    else:
        payload["error_detail"] = error_detail
    return payload


def main(argv: list[str] | None = None) -> int:
    configure_stdio_utf8()
    load_repo_env(repo_root_from_path(__file__))
    args = _parse_args(argv)
    parse_request: dict[str, object] | None = None

    try:
        base_url = resolve_base_url(args.base_url)
        [REDACTED_SECRET])
        parse_request = start_parse(args.dataset_id, args.document_ids, base_url=base_url, [REDACTED_SECRET])
        print(format_json(parse_request) if args.json_output else _format_payload(parse_request))
        return 0
    except ScriptError as exc:
        if args.json_output:
            print(format_json(_build_error_payload(args, exc, parse_request=parse_request)))
        else:
            print(f"Error: {exc}")
        return 1


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/search.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-

import argparse
import json
import os
from typing import Any

from common import (
    ConfigError,
    DataError,
    ScriptError,
    configure_stdio_utf8,
    current_timestamp,
    ensure_success,
    format_json,
    load_repo_env,
    repo_root_from_path,
    request_json,
    require_api_key,
    resolve_base_url,
)

DEFAULT_TOP_K = 5
DEFAULT_THRESHOLD = 0.2
DEFAULT_PAGE = 1
DEFAULT_PAGE_SIZE = 30
PREVIEW_LIMIT = 240


def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Retrieve relevant chunks from RAGFlow datasets.")
    parser.add_argument("query", help="Search query text")
    parser.add_argument(
        "dataset_id",
        nargs="?",
        help="Optional dataset ID shortcut. Used for standard retrieval, or as the fallback kb_id for --retrieval-test.",
    )
    parser.add_argument("--dataset-ids", help="Comma-separated dataset IDs")
    parser.add_argument("--doc-ids", help="Comma-separated document IDs")
    parser.add_argument("--top-k", type=int, default=DEFAULT_TOP_K, help=f"Maximum results (default: {DEFAULT_TOP_K})")
    parser.add_argument(
        "--threshold",
        type=float,
        default=DEFAULT_THRESHOLD,
        help=f"Similarity threshold in the range [0, 1] (default: {DEFAULT_THRESHOLD})",
    )
    parser.add_argument("--vector-weight", type=float, help="Vector similarity weight in the range [0, 1]")
    parser.add_argument("--page", type=int, default=DEFAULT_PAGE, help=f"Page number (default: {DEFAULT_PAGE})")
    parser.add_argument(
        "--page-size",
        "--size",
        dest="page_size",
        type=int,
        default=DEFAULT_PAGE_SIZE,
        help=f"Page size (default: {DEFAULT_PAGE_SIZE})",
    )
    parser.add_argument("--keyword", action="store_true", help="Enable keyword extraction")
    parser.add_argument("--use-kg", action="store_true", help="Enable knowledge graph retrieval")
    parser.add_argument("--rerank-id", help="Optional rerank model ID")
    parser.add_argument("--search-id", help="Optional search session ID")
    parser.add_argument("--retrieval-test", action="store_true", help="Use /api/v1/chunk/retrieval_test")
    parser.add_argument("--kb-id", help="Dataset ID required by retrieval_test")
    parser.add_argument("--json", action="store_true", dest="json_output", help="Print JSON output")
    parser.add_argument(
        "--base-url",
        help="Base URL for the RAGFlow server (priority: --base-url > RAGFLOW_API_URL > RAGFLOW_BASE_URL > HOST_ADDRESS > default)",
    )
    return parser.parse_args(argv)


def _validate_range(name: str, value: float, *, min_value: float = 0.0, max_value: float = 1.0) -> None:
    if value < min_value or value > max_value:
        raise ConfigError(f"{name} must be between {min_value} and {max_value}.")


def _parse_ids(raw_value: str, *, label: str) -> list[str]:
    values: list[str] = []
    seen: set[str] = set()
    for item in raw_value.split(","):
        value = item.strip()
        if not value or value in seen:
            continue
        seen.add(value)
        values.append(value)

    if not values:
        raise ConfigError(f"{label} must include at least one ID.")
    return values


def _parse_dataset_ids_env() -> list[str]:
    raw_value = (os.getenv("RAGFLOW_DATASET_IDS") or "").strip()
    if not raw_value:
        return []

    try:
        parsed = json.loads(raw_value)
    except json.JSONDecodeError:
        return _parse_ids(raw_value, label="RAGFLOW_DATASET_IDS")

    if isinstance(parsed, list):
        values: list[str] = []
        seen: set[str] = set()
        for item in parsed:
            value = str(item).strip()
            if not value or value in seen:
                continue
            seen.add(value)
            values.append(value)
        if not values:
            raise ConfigError("RAGFLOW_DATASET_IDS must include at least one dataset ID when it is set.")
        return values

    if isinstance(parsed, str):
        return _parse_ids(parsed, label="RAGFLOW_DATASET_IDS")

    raise ConfigError("RAGFLOW_DATASET_IDS must be a JSON array or a comma-separated string.")


def _resolve_dataset_ids(args: argparse.Namespace) -> list[str]:
    if args.dataset_ids:
        return _parse_ids(args.dataset_ids, label="--dataset-ids")
    if args.dataset_id:
        dataset_id = args.dataset_id.strip()
        if not dataset_id:
            raise ConfigError("dataset_id must not be empty.")
        return [dataset_id]
    return _parse_dataset_ids_env()


def _resolve_kb_id(args: argparse.Namespace, dataset_ids: list[str]) -> str:
    if args.kb_id:
        kb_id = args.kb_id.strip()
        if not kb_id:
            raise ConfigError("--kb-id must not be empty.")
        return kb_id
    if dataset_ids:
        return dataset_ids[0]
    raise ConfigError("--kb-id is required for --retrieval-test when no dataset ID is provided.")


def _normalize_content(chunk: dict[str, Any]) -> str:
    for key in ("content_with_weight", "content", "answer", "chunk"):
        value = chunk.get(key)
        if isinstance(value, str):
            return value
        if isinstance(value, list):
            return " ".join(str(item) for item in value)
    return ""


def _normalize_chunk(chunk: dict[str, Any]) -> dict[str, Any]:
    return {
        "document_name": chunk.get("document_keyword") or chunk.get("docnm_kwd") or chunk.get("document_name"),
        "document_id": chunk.get("document_id") or chunk.get("doc_id"),
        "dataset_id": chunk.get("dataset_id") or chunk.get("kb_id"),
        "chunk_id": chunk.get("chunk_id") or chunk.get("id"),
        "similarity": chunk.get("similarity"),
        "vector_similarity": chunk.get("vector_similarity"),
        "term_similarity": chunk.get("term_similarity"),
        "positions": chunk.get("positions"),
        "content": _normalize_content(chunk),
    }


def _extract_chunks(payload: dict[str, Any]) -> list[dict[str, Any]]:
    data = payload.get("data")
    if data is None:
        return []
    if isinstance(data, dict):
        chunks = data.get("chunks")
        if chunks is None:
            return []
        if not isinstance(chunks, list):
            raise DataError("Retrieval response data.chunks must be a list.")
        return [_normalize_chunk(chunk) for chunk in chunks if isinstance(chunk, dict)]
    if isinstance(data, list):
        return [_normalize_chunk(chunk) for chunk in data if isinstance(chunk, dict)]
    raise DataError("Retrieval response data must be an object or array.")


def search(args: argparse.Namespace, *, base_url: str, api_key: str) -> dict[str, Any]:
    if args.top_k <= 0:
        raise ConfigError("--top-k must be greater than 0.")
    if args.page <= 0:
        raise ConfigError("--page must be greater than 0.")
    if args.page_size <= 0:
        raise ConfigError("--page-size must be greater than 0.")
    _validate_range("--threshold", args.threshold)
    if args.vector_weight is not None:
        _validate_range("--vector-weight", args.vector_weight)

    dataset_ids = _resolve_dataset_ids(args)
    doc_ids = _parse_ids(args.doc_ids, label="--doc-ids") if args.doc_ids else []

    body: dict[str, Any] = {
        "question": args.query,
        "top_k": args.top_k,
        "similarity_threshold": args.threshold,
        "page": args.page,
        "size": args.page_size,
    }
    if args.vector_weight is not None:
        body["vector_similarity_weight"] = args.vector_weight
    if doc_ids:
        body["document_ids"] = doc_ids
    if args.keyword:
        body["keyword"] = True
    if args.use_kg:
        body["use_kg"] = True
    if args.rerank_id:
        body["rerank_id"] = args.rerank_id
    if args.search_id:
        body["search_id"] = args.search_id

    api_name = "retrieval"
    kb_id: str | None = None
    if args.retrieval_test:
        api_name = "retrieval_test"
        kb_id = _resolve_kb_id(args, dataset_ids)
        body["kb_id"] = kb_id
        api_endpoint = f"{base_url}/api/v1/chunk/retrieval_test"
    else:
        if dataset_ids:
            body["dataset_ids"] = dataset_ids
        api_endpoint = f"{base_url}/api/v1/retrieval"

    payload = ensure_success(
        request_json(
            api_endpoint,
            api_key,
            method="POST",
            body=json.dumps(body).encode("utf-8"),
            content_type="application/json",
        )
    )
    chunks = _extract_chunks(payload)

    return {
        "checked_at": current_timestamp(),
        "query": args.query,
        "api": api_name,
        "dataset_ids": dataset_ids,
        "kb_id": kb_id,
        "doc_ids": doc_ids,
        "count": len(chunks),
        "chunks": chunks,
    }


def _format_similarity(value: Any) -> str:
    if isinstance(value, (int, float)):
        return f"{value:.2%}"
    return "unknown"


def _format_preview(content: str) -> str:
    compact = " ".join(content.split())
    if not compact:
        return "unknown"
    if len(compact) <= PREVIEW_LIMIT:
        return compact
    return f"{compact[: PREVIEW_LIMIT - 3]}..."


def _format_text(payload: dict[str, Any]) -> str:
    lines = [
        f"Checked at: {payload['checked_at']}",
        f"Query: {payload['query']}",
        f"API: {payload['api']}",
        f"Results: {payload['count']}",
    ]
    if payload["dataset_ids"]:
        lines.append(f"Dataset IDs: {', '.join(payload['dataset_ids'])}")
    if payload.get("kb_id"):
        lines.append(f"KB ID: {payload['kb_id']}")
    if payload["doc_ids"]:
        lines.append(f"Document IDs: {', '.join(payload['doc_ids'])}")

    if payload["count"] == 0:
        lines.append("No results found.")
        return "\n".join(lines)

    for index, chunk in enumerate(payload["chunks"], start=1):
        lines.extend(
            [
                "",
                f"[{index}] {chunk.get('document_name') or 'unknown'}",
                f"  similarity: {_format_similarity(chunk.get('similarity'))}",
                f"  document_id: {chunk.get('document_id') or 'unknown'}",
                f"  dataset_id: {chunk.get('dataset_id') or 'unknown'}",
                f"  chunk_id: {chunk.get('chunk_id') or 'unknown'}",
                f"  content: {_format_preview(chunk.get('content') or '')}",
            ]
        )
    return "\n".join(lines)


def main(argv: list[str] | None = None) -> int:
    configure_stdio_utf8()
    load_repo_env(repo_root_from_path(__file__))
    args = _parse_args(argv)

    try:
        base_url = resolve_base_url(args.base_url)
        [REDACTED_SECRET])
        payload = search(args, base_url=base_url, [REDACTED_SECRET])
        print(format_json(payload) if args.json_output else _format_text(payload))
        return 0
    except ScriptError as exc:
        if args.json_output:
            print(format_json({"checked_at": current_timestamp(), "error": str(exc)}))
        else:
            print(f"Error: {exc}")
        return 1


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/create_dataset.py
content: #!/usr/bin/env python3
#
#  Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
#
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.
#

import argparse
import json
import sys
from typing import Any

from common import (
    DataError,
    ScriptError,
    configure_stdio_utf8,
    ensure_success,
    load_repo_env,
    repo_root_from_path,
    request_json,
    require_api_key,
    resolve_base_url,
)


def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Create a dataset through the RAGFlow HTTP API.")
    parser.add_argument("name", help="Dataset name")
    parser.add_argument("--avatar", help="Dataset avatar")
    parser.add_argument("--description", help="Dataset description")
    parser.add_argument("--embedding-model", dest="embedding_model", help="Embedding model ID")
    parser.add_argument("--permission", help="Permission value, for example me or team")
    parser.add_argument("--chunk-method", dest="chunk_method", help="Chunking method / parser ID")
    parser.add_argument("--language", help="Dataset language")
    parser.add_argument("--json", action="store_true", dest="json_output", help="Print raw JSON response")
    parser.add_argument(
        "--base-url",
        help="Base URL for the RAGFlow server "
        "(priority: --base-url > RAGFLOW_API_URL > RAGFLOW_BASE_URL > HOST_ADDRESS > default)",
    )
    return parser.parse_args(argv)


def _build_payload(args: argparse.Namespace) -> dict[str, Any]:
    name = args.name.strip()
    if not name:
        raise ConfigError("Dataset name must not be empty.")

    payload: dict[str, Any] = {"name": name}
    if args.avatar:
        payload["avatar"] = args.avatar
    if args.description:
        payload["description"] = args.description
    if args.embedding_model:
        payload["embedding_model"] = args.embedding_model
    if args.permission:
        payload["permission"] = args.permission
    if args.chunk_method:
        payload["chunk_method"] = args.chunk_method
    if args.language:
        payload["language"] = args.language
    return payload
def _normalize_payload(payload: dict[str, Any]) -> dict[str, Any]:
    payload = ensure_success(payload)
    code = payload.get("code")

    data = payload.get("data")
    if not isinstance(data, dict):
        raise DataError("Response missing data object.")

    return {
        "code": code,
        "message": payload.get("message", ""),
        "data": data,
    }


def _format_text(payload: dict[str, Any]) -> str:
    data = payload["data"]
    lines = ["Dataset created"]

    dataset_id = data.get("id")
    name = data.get("name")
    if name:
        lines.append(f"name: {name}")
    if dataset_id:
        lines.append(f"id: {dataset_id}")

    for key in ("embedding_model", "permission", "chunk_method", "language", "avatar", "description"):
        value = data.get(key)
        if value not in (None, ""):
            lines.append(f"{key}: {value}")

    return "\n".join(lines)


def main(argv: list[str] | None = None) -> int:
    configure_stdio_utf8()
    load_repo_env(repo_root_from_path(__file__))

    args = _parse_args(argv)

    try:
        [REDACTED_SECRET])
        base_url = resolve_base_url(args.base_url)
        payload = request_json(
            f"{base_url}/api/v1/datasets",
            api_key,
            method="POST",
            body=json.dumps(_build_payload(args)).encode("utf-8"),
            content_type="application/json",
        )
        normalized = _normalize_payload(payload)

        if args.json_output:
            print(json.dumps(normalized, ensure_ascii=False, indent=2))
        else:
            print(_format_text(normalized))
        return 0
    except ScriptError as exc:
        print(f"Error: {exc}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    sys.exit(main())


path: scripts/update_document.py
content: #!/usr/bin/env python3
#
#  Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
#
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.
#

import argparse
import json
import sys
import urllib.parse
from pathlib import Path
from typing import Any

from common import (
    ConfigError,
    DataError,
    ScriptError,
    configure_stdio_utf8,
    ensure_success,
    load_repo_env,
    repo_root_from_path,
    request_json,
    require_api_key,
    resolve_base_url,
)


def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="Update a document through the RAGFlow HTTP API.")
    parser.add_argument("dataset_id", help="Dataset ID")
    parser.add_argument("document_id", help="Document ID")
    parser.add_argument("--name", help="Updated document name")
    parser.add_argument("--chunk-method", help="Updated chunk method")
    parser.add_argument(
        "--parser-config",
        help="Parser config JSON object or @path/to/file.json",
    )
    parser.add_argument(
        "--meta-fields",
        help="Document metadata JSON object or @path/to/file.json",
    )
    parser.add_argument(
        "--enabled",
        choices=("0", "1"),
        help="Set availability flag: 1 enabled, 0 disabled",
    )
    parser.add_argument(
        "--data",
        help="Raw JSON object payload or @path/to/file.json. Explicit flags override the same keys.",
    )
    parser.add_argument("--json", action="store_true", dest="json_output", help="Print raw JSON response")
    parser.add_argument(
        "--base-url",
        help="Base URL for the RAGFlow server "
        "(priority: --base-url > RAGFLOW_API_URL > RAGFLOW_BASE_URL > HOST_ADDRESS > default)",
    )
    return parser.parse_args(argv)


def _load_json_object(raw_value: str, option_name: str) -> dict[str, Any]:
    value = raw_value
    if raw_value.startswith("@"):
        path = Path(raw_value[1:]).expanduser()
        try:
            value = path.read_text(encoding="utf-8")
        except OSError as exc:
            raise ConfigError(f"Failed to read {option_name} file {path}: {exc}") from exc

    try:
        payload = json.loads(value)
    except json.JSONDecodeError as exc:
        raise ConfigError(f"{option_name} must be valid JSON: {exc.msg}.") from exc

    if not isinstance(payload, dict):
        raise ConfigError(f"{option_name} must be a JSON object.")
    return payload


def _build_payload(args: argparse.Namespace) -> dict[str, Any]:
    payload: dict[str, Any] = {}
    if args.data:
        payload.update(_load_json_object(args.data, "--data"))

    field_values = {
        "name": args.name,
        "chunk_method": args.chunk_method,
    }
    for key, value in field_values.items():
        if value is not None:
            payload[key] = value

    if args.parser_config is not None:
        payload["parser_config"] = _load_json_object(args.parser_config, "--parser-config")
    if args.meta_fields is not None:
        payload["meta_fields"] = _load_json_object(args.meta_fields, "--meta-fields")
    if args.enabled is not None:
        payload["enabled"] = int(args.enabled)

    if not payload:
        raise ConfigError("No update fields provided. Use --data or at least one explicit update flag.")
    return payload


def _build_url(base_url: str, dataset_id: str, document_id: str) -> str:
    encoded_dataset_id = urllib.parse.quote(dataset_id, safe="")
    encoded_document_id = urllib.parse.quote(document_id, safe="")
    return f"{base_url}/api/v1/datasets/{encoded_dataset_id}/documents/{encoded_document_id}"
def _normalize_payload(payload: dict[str, Any]) -> dict[str, Any]:
    payload = ensure_success(payload)

    data = payload.get("data")
    if not isinstance(data, dict):
        raise DataError("Response missing data object.")
    return payload


def _print_summary(payload: dict[str, Any]) -> None:
    data = payload["data"]
    document_id = str(data.get("id", "")).strip() or "<missing-id>"
    name = str(data.get("name", "")).strip() or "<missing-name>"
    dataset_id = str(data.get("dataset_id", "")).strip() or "<missing-dataset-id>"
    print(f"Updated document: {name} ({document_id})")

    details = [f"dataset_id={dataset_id}"]
    if data.get("chunk_method"):
        details.append(f"chunk_method={data['chunk_method']}")
    if data.get("run"):
        details.append(f"run={data['run']}")
    if data.get("chunk_count") is not None:
        details.append(f"chunks={data['chunk_count']}")
    if data.get("token_count") is not None:
        details.append(f"tokens={data['token_count']}")
    print("  " + ", ".join(details))


def main(argv: list[str] | None = None) -> int:
    configure_stdio_utf8()
    load_repo_env(repo_root_from_path(__file__))

    try:
        args = _parse_args(argv)
        base_url = resolve_base_url(args.base_url)
        [REDACTED_SECRET])
        payload = _build_payload(args)
        response = request_json(
            _build_url(base_url, args.dataset_id, args.document_id),
            api_key,
            method="PUT",
            body=json.dumps(payload).encode("utf-8"),
            content_type="application/json",
        )
        normalized = _normalize_payload(response)
    except ScriptError as exc:
        print(f"Error: {exc}", file=sys.stderr)
        return 1

    if args.json_output:
        print(json.dumps(normalized, ensure_ascii=False, indent=2))
    else:
        _print_summary(normalized)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: scripts/common.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-

import datetime
import io
import json
import os
import sys
import urllib.error
import urllib.parse
import urllib.request
from pathlib import Path
from typing import Any

DEFAULT_BASE_URL = "http://127.0.0.1"
HTTP_TIMEOUT = 30
ALLOWED_ENV_KEYS = {
    "RAGFLOW_API_URL",
    "RAGFLOW_BASE_URL",
    "RAGFLOW_API_KEY",
    "RAGFLOW_DATASET_IDS",
    "RAGFLOW_TOP_K",
    "RAGFLOW_SIMILARITY_THRESHOLD",
}


class ScriptError(Exception):
    pass


class ConfigError(ScriptError):
    pass


class ApiError(ScriptError):
    def __init__(
        self,
        message: str,
        *,
        http_status: int | None = None,
        api_code: Any | None = None,
        response_payload: Any | None = None,
        response_body: str | None = None,
    ):
        super().__init__(message)
        self.http_status = http_status
        self.api_code = api_code
        self.response_payload = response_payload
        self.response_body = response_body


class DataError(ScriptError):
    pass


def current_timestamp() -> str:
    return datetime.datetime.now().astimezone().isoformat(timespec="seconds")


def configure_stdio_utf8() -> None:
    if sys.platform == "win32":
        sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")
        sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding="utf-8", errors="replace")
        return

    for stream_name in ("stdout", "stderr"):
        stream = getattr(sys, stream_name, None)
        if stream is None or not hasattr(stream, "reconfigure"):
            continue
        try:
            stream.reconfigure(encoding="utf-8", errors="replace")
        except Exception:
            continue


def repo_root_from_path(file_path: str) -> Path:
    return Path(file_path).resolve().parents[1]


def load_repo_env(repo_root: Path) -> None:
    env_path = repo_root / ".env"
    if not env_path.is_file():
        return

    try:
        lines = env_path.read_text(encoding="utf-8").splitlines()
    except OSError as exc:
        raise ConfigError(f"Failed to read {env_path}: {exc}") from exc

    for raw_line in lines:
        line = raw_line.strip()
        if not line or line.startswith("#"):
            continue
        if line.startswith("export "):
            line = line[7:].strip()
        if "=" not in line:
            continue

        key, value = line.split("=", 1)
        key = key.strip()
        if not key or key in os.environ:
            continue
        if key not in ALLOWED_ENV_KEYS and not key.startswith("RAGFLOW_"):
            continue

        value = value.strip()
        if len(value) >= 2 and value[0] == value[-1] and value[0] in ("'", '"'):
            value = value[1:-1]
        os.environ[key] = value


def resolve_base_url(cli_base_url: str | None = None) -> str:
    base_url = (
        cli_base_url
        or os.getenv("RAGFLOW_API_URL")
        or os.getenv("RAGFLOW_BASE_URL")
        or os.getenv("HOST_ADDRESS")
        or DEFAULT_BASE_URL
    ).strip()

    parsed = urllib.parse.urlsplit(base_url)
    if not parsed.scheme or not parsed.netloc:
        raise ConfigError("Invalid base URL. Use an absolute URL such as http://127.0.0.1:9380.")
    return base_url.rstrip("/")


def require_api_key() -> str:
    [REDACTED_SECRET]"RAGFLOW_API_KEY") or "").strip()
    if not api_key:
        raise ConfigError("RAGFLOW_API_KEY is not configured. Set it in the environment or in the repository .env file.")
    return api_key


def decode_json_response(body: bytes) -> dict[str, Any]:
    try:
        payload = json.loads(body.decode("utf-8"))
    except Exception as exc:
        raise ApiError("Received a non-JSON response from the server.") from exc

    if not isinstance(payload, dict):
        raise DataError("Expected a JSON object from the server.")
    return payload


def decode_response_text(body: bytes) -> str | None:
    if not body:
        return None
    try:
        text = body.decode("utf-8", errors="replace").strip()
    except Exception:
        return None
    return text or None


def decode_json_body(body: bytes) -> Any | None:
    text = decode_response_text(body)
    if not text:
        return None
    try:
        return json.loads(text)
    except Exception:
        return None


def extract_error_message(body: bytes) -> str | None:
    payload = decode_json_body(body)
    if not isinstance(payload, dict):
        return None

    message = payload.get("message")
    if isinstance(message, str) and message.strip():
        return message.strip()
    return None


def serialize_script_error(exc: ScriptError) -> dict[str, Any]:
    payload: dict[str, Any] = {
        "type": exc.__class__.__name__,
        "message": str(exc),
    }
    if isinstance(exc, ApiError):
        if exc.http_status is not None:
            payload["http_status"] = exc.http_status
        if exc.api_code is not None:
            payload["api_code"] = exc.api_code
        if exc.response_payload is not None:
            payload["response"] = exc.response_payload
        elif exc.response_body:
            payload["response_body"] = exc.response_body
    return payload


def request_json(
    url: str,
    api_key: str,
    *,
    method: str = "GET",
    body: bytes | None = None,
    content_type: str | None = None,
    accept: str = "application/json",
) -> dict[str, Any]:
    headers = {"Authorization": f"Bearer {api_key}"}
    if accept:
        headers["Accept"] = accept
    if content_type:
        headers["Content-Type"] = content_type

    request_obj = urllib.request.Request(url, headers=headers, data=body, method=method)

    try:
        with urllib.request.urlopen(request_obj, timeout=HTTP_TIMEOUT) as response:
            return decode_json_response(response.read())
    except urllib.error.HTTPError as exc:
        body_bytes = exc.read()
        response_payload = decode_json_body(body_bytes)
        response_text = decode_response_text(body_bytes)
        message = extract_error_message(body_bytes)
        if message:
            raise ApiError(
                message,
                http_status=exc.code,
                api_code=response_payload.get("code") if isinstance(response_payload, dict) else None,
                response_payload=response_payload,
                response_body=response_text,
            ) from None
        raise ApiError(
            f"HTTP request failed with status {exc.code}.",
            http_status=exc.code,
            api_code=response_payload.get("code") if isinstance(response_payload, dict) else None,
            response_payload=response_payload,
            response_body=response_text,
        ) from None
    except urllib.error.URLError as exc:
        reason = getattr(exc, "reason", exc)
        raise ApiError(f"HTTP request failed: {reason}") from None


def ensure_success(payload: dict[str, Any]) -> dict[str, Any]:
    code = payload.get("code")
    if code != 0:
        message = payload.get("message") or f"API returned code {code}."
        raise ApiError(str(message), api_code=code, response_payload=payload)
    return payload


def format_json(payload: dict[str, Any]) -> str:
    return json.dumps(payload, ensure_ascii=False, indent=2)


path: scripts/list_documents.py
content: #!/usr/bin/env python3
#
#  Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
#
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.
#

import argparse
import json
import sys
import urllib.parse
from typing import Any

DEFAULT_PAGE = 1
DEFAULT_PAGE_SIZE = 10
DEFAULT_ORDERBY = "create_time"
from common import (
    DataError,
    ScriptError,
    configure_stdio_utf8,
    ensure_success,
    load_repo_env,
    repo_root_from_path,
    request_json,
    require_api_key,
    resolve_base_url,
)


def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    parser = argparse.ArgumentParser(description="List documents in a dataset through the RAGFlow HTTP API.")
    parser.add_argument("dataset_id", help="Dataset ID")
    parser.add_argument("--page", type=int, default=DEFAULT_PAGE, help=f"Page number (default: {DEFAULT_PAGE})")
    parser.add_argument(
        "--page-size",
        type=int,
        default=DEFAULT_PAGE_SIZE,
        help=f"Page size (default: {DEFAULT_PAGE_SIZE})",
    )
    parser.add_argument("--orderby", default=DEFAULT_ORDERBY, help=f"Sort field (default: {DEFAULT_ORDERBY})")
    parser.add_argument("--asc", action="store_true", help="Sort ascending. Descending is the default.")
    parser.add_argument("--keywords", help="Filter by keywords")
    parser.add_argument("--id", dest="document_id", help="Filter by document ID")
    parser.add_argument("--name", help="Filter by document name")
    parser.add_argument("--suffix", help="Filter by file suffix, for example pdf")
    parser.add_argument("--run", help="Filter by parse run status, for example DONE")
    parser.add_argument("--json", action="store_true", dest="json_output", help="Print raw JSON response")
    parser.add_argument(
        "--base-url",
        help="Base URL for the RAGFlow server "
        "(priority: --base-url > RAGFLOW_API_URL > RAGFLOW_BASE_URL > HOST_ADDRESS > default)",
    )
    return parser.parse_args(argv)


def _validate_positive(name: str, value: int) -> None:
    if value <= 0:
        raise DataError(f"{name} must be greater than 0.")


def _build_documents_url(base_url: str, args: argparse.Namespace) -> str:
    dataset_id = args.dataset_id.strip()
    if not dataset_id:
        raise ConfigError("dataset_id must not be empty.")

    query: dict[str, Any] = {
        "page": args.page,
        "page_size": args.page_size,
        "orderby": args.orderby,
        "desc": str(not args.asc).lower(),
    }
    if args.keywords:
        query["keywords"] = args.keywords
    if args.document_id:
        query["id"] = args.document_id
    if args.name:
        query["name"] = args.name
    if args.suffix:
        query["suffix"] = args.suffix
    if args.run:
        query["run"] = args.run

    encoded_dataset_id = urllib.parse.quote(dataset_id, safe="")
    encoded_query = urllib.parse.urlencode(query)
    return f"{base_url}/api/v1/datasets/{encoded_dataset_id}/documents?{encoded_query}"
def _normalize_payload(payload: dict[str, Any]) -> dict[str, Any]:
    payload = ensure_success(payload)
    code = payload.get("code")

    data = payload.get("data")
    if not isinstance(data, dict):
        raise DataError("Response missing data object.")

    docs = data.get("docs")
    total = data.get("total")
    if not isinstance(docs, list):
        raise DataError("Response missing data.docs.")
    if not isinstance(total, int):
        raise DataError("Response missing data.total.")

    return {
        "code": code,
        "message": payload.get("message", ""),
        "data": data,
    }


def _format_document_line(document: dict[str, Any]) -> str:
    document_id = str(document.get("id", "")).strip() or "<missing-id>"
    name = str(document.get("name", "")).strip() or "<missing-name>"
    lines = [f"{name} ({document_id})"]

    details = []
    for key, label in (
        ("run", "run"),
        ("type", "type"),
        ("chunk_count", "chunks"),
        ("token_count", "tokens"),
        ("size", "size"),
    ):
        value = document.get(key)
        if value not in (None, ""):
            details.append(f"{label}={value}")
    if details:
        lines.append("  " + ", ".join(details))

    return "\n".join(lines)


def _format_text(payload: dict[str, Any], dataset_id: str) -> str:
    data = payload["data"]
    docs = data["docs"]
    total = data["total"]

    lines = [f"Dataset: {dataset_id}", f"Documents: {len(docs)} / total={total}"]
    if not docs:
        return "\n".join(lines)

    for document in docs:
        lines.append("")
        lines.append(_format_document_line(document))
    return "\n".join(lines)


def main(argv: list[str] | None = None) -> int:
    configure_stdio_utf8()
    load_repo_env(repo_root_from_path(__file__))

    args = _parse_args(argv)

    try:
        _validate_positive("--page", args.page)
        _validate_positive("--page-size", args.page_size)
        [REDACTED_SECRET])
        base_url = resolve_base_url(args.base_url)
        payload = request_json(_build_documents_url(base_url, args), api_key)
        normalized = _normalize_payload(payload)

        if args.json_output:
            print(json.dumps(normalized, ensure_ascii=False, indent=2))
        else:
            print(_format_text(normalized, args.dataset_id.strip()))
        return 0
    except ScriptError as exc:
        print(f"Error: {exc}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    sys.exit(main())


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: malicious | SkillSpector: suspicious (CRITICAL)