Skill: samuelhsin/alicloud-dataworks (version 1.0.0)

--- name: dataworks-open-api description: Operate Alibaba Cloud DataWorks through dynamic API discovery and official SDKs (Node.js, Python, Java). Covers data development, workflow operations, data integration, data quality, metadata lineage, workspace management, and more. APIs are discovered at runtime from official docs and OpenAPI metadata — no hardcoded list. --- # DataWorks (dataworks-public) Use Alibaba Cloud OpenAPI (RPC) with official SDKs or OpenAPI Explorer to manage all DataWorks resources. MCP Server is available as an alternative integration. ## Workflow 1. Confirm region, project/resource identifiers, and desired action. 2. Discover available APIs and required parameters (see API discovery below). 3. Call API via SDK. 4. Verify results with describe/list/get APIs. ### Troubleshooting fallback chain (follow this order) If execution fails at any step, escalate to the next level: 1. **Official help docs** — read the API overview and per-API doc pages first. 2. **OpenAPI metadata** — fetch the raw JSON spec for exact parameter names, types, and required fields. 3. **SDK API docs** — check the typed SDK reference for correct method signatures and request classes. - Java: `https://aliyunsdk-pages.alicdn.com/apidocs/{PRODUCT_CODE}/{API_VERSION}/java-async-tea/8.0.3/index.html` - Node.js / Python: see `references/sources.md` 4. **GitHub source code** — read the Java SDK source for model/request class definitions: `https://github.com/aliyun/alibabacloud-java-sdk/tree/master/dataworks-public-20240518` (browse [REDACTED_SECRET] for request/response models). 5. **MCP Server** — use the MCP Server to call APIs directly (see `references/mcp_server.md`). 6. **Browser** — use browser-use to visit the DataWorks Homepage (`https://dataworks.data.aliyun.com/${ALICLOUD_REGION_ID}/#/index`) and inspect the UI behavior, network requests, or page content for clues. ## AccessKey priority (must follow) 1. Environment variables: `ALICLOUD_ACCESS_KEY_ID` / `ALICLOUD_ACCESS_KEY_SECRET` / `ALICLOUD_REGION_ID` Region policy: `ALICLOUD_REGION_ID` is an optional default. If unset, decide the most reasonable region for the task; if unclear, ask the user. 2. Shared config file: `~/.alibabacloud/credentials` ## API discovery Constants (use these values throughout): - **PRODUCT_CODE**: `dataworks-public` - **API_VERSION**: `2024-05-18` - Use OpenAPI metadata endpoints to list APIs and get schemas (see references). Do NOT rely on a hardcoded API list. Always discover APIs dynamically. Methods are listed in priority order — try them top-down: ### Method 1: Fetch from official help docs Run the discovery script to curl the official documentation page and extract the complete API overview: ```bash python scripts/fetch_api_overview.py ``` This fetches `https://help.aliyun.com/zh/dataworks/developer-reference/api-{PRODUCT_CODE}-{API_VERSION}-overview`, extracts the API list from `window.__ICE_PAGE_PROPS__.docDetailData.storeData.data.content`, and parses the HTML tables into a grouped API overview. ### Method 2: Fetch from OpenAPI metadata (full spec with parameters and responses) Fetch raw API schemas and parameter definitions: ```bash python scripts/list_openapi_meta_apis.py ``` This calls `https://next.api.aliyun.com/meta/v1/products/{PRODUCT_CODE}/versions/{API_VERSION}/api-docs.json` and outputs the full API docs JSON and a summary list. To get the schema of a single API (replace `{ApiName}` with a name from the list above, e.g. `ListProjects`): ``` https://next.api.aliyun.com/meta/v1/products/{PRODUCT_CODE}/versions/{API_VERSION}/apis/{ApiName}/api.json ``` ### Method 3: MCP Server tools (curated subset with category and parameter-position metadata) If the MCP Server is running, use the MCP `tools/list` capability, or fetch `https://dataworks.data.aliyun.com/pop-mcp-tools` directly for the tool catalog JSON. Includes some APIs not in Method 2 (e.g. data service, permission approval). ## Calling APIs via SDK Prefer the official Alibaba Cloud SDK. Two styles are supported: ### Style 1: Generalized call (recommended, covers all APIs) Use `@alicloud/openapi-client` to call any DataWorks API without importing product-specific SDK classes. This is the approach used by the MCP Server source code (`src/tools/callTool.ts`). ```bash npm install @alicloud/openapi-client @alicloud/openapi-util @alicloud/tea-util @alicloud/credentials ``` ```typescript import OpenApi from "@alicloud/openapi-client"; import Util from "@alicloud/tea-util"; // 1. Create client const config = new OpenApi.Config({ [REDACTED_SECRET], [REDACTED_SECRET], endpoint: `dataworks.${ process.env.ALICLOUD_REGION_ID || "cn-shanghai" }.aliyuncs.com`, }); const client = new OpenApi.default(config); // 2. Build request params (RPC style) const params = new OpenApi.Params({ style: "RPC", action: "ListProjects", // API name version: "{API_VERSION}", // see Constants above protocol: "HTTPS", method: "POST", // GET or POST per API spec authType: "AK", pathname: "/", bodyType: "json", }); // 3. Build query/body from input const query = { PageNumber: 1, PageSize: 10 }; const request = new OpenApi.OpenApiRequest({ query }); const runtime = new Util.RuntimeOptions({}); // 4. Call const res = await client.callApi(params, request, runtime); console.log(res.body); ``` Key rules from `src/tools/callTool.ts`: - `style`: `'RPC'` when API has no path; `'ROA'` when it has a path. - `method`: per API spec (`GET` / `POST` / `PUT` / `DELETE`). - Parameters with `in: 'query'` go to `query`; `in: 'body'` / `in: 'formData'` go to `body`. - For `GET` / `DELETE` requests, `body` must be `null` (otherwise signature fails). - Use `bodyType: 'string'` to avoid precision loss on large numbers. ### Style 2: Product-specific SDK (typed, per-API methods) ```bash npm install @alicloud/dataworks-public20240518 ``` ```typescript import DataWorks from "@alicloud/dataworks-public20240518"; import * as DataWorksClasses from "@alicloud/dataworks-public20240518"; import OpenApi from "@alicloud/openapi-client"; const config = new OpenApi.Config({ [REDACTED_SECRET], [REDACTED_SECRET], endpoint: `dataworks.cn-shanghai.aliyuncs.com`, }); const client = new DataWorks.default(config); // Each API has a typed Request class and a method const request = new DataWorksClasses.ListProjectsRequest({ pageNumber: 1, pageSize: 10, }); const runtime = new Util.RuntimeOptions({}); const resp = await client.listProjectsWithOptions(request, runtime); ``` API name → SDK method: `ListProjects` → `client.listProjectsWithOptions(request, runtime)`. ### Python SDK ```bash pip install alibabacloud_dataworks_public20240518 ``` ```python import os from alibabacloud_dataworks_public20240518.client import Client from alibabacloud_dataworks_public20240518 import models from alibabacloud_tea_openapi.models import Config config = Config( [REDACTED_SECRET]'ALICLOUD_ACCESS_KEY_ID'], [REDACTED_SECRET]'ALICLOUD_ACCESS_KEY_SECRET'], endpoint=f"dataworks.{os.environ.get('ALICLOUD_REGION_ID', 'cn-shanghai')}.aliyuncs.com", ) client = Client(config) request = models.ListProjectsRequest(page_number=1, page_size=10) resp = client.list_projects(request) print(resp.body) ``` ### Java SDK ```xml <dependency> <groupId>com.aliyun</groupId> <artifactId>alibabacloud-dataworks_public20240518</artifactId> <version>8.0.3</version> </dependency> ``` ```java import com.aliyun.dataworks_public20240518.Client; import com.aliyun.dataworks_public20240518.models.*; import com.aliyun.teaopenapi.models.Config; Config config = new Config() .setAccessKeyId(System.getenv("ALICLOUD_ACCESS_KEY_ID")) .setAccessKeySecret(System.getenv("ALICLOUD_ACCESS_KEY_SECRET")) .setEndpoint("dataworks." + System.getenv("ALICLOUD_REGION_ID") + ".aliyuncs.com"); Client client = new Client(config); ListProjectsRequest request = new ListProjectsRequest() .setPageNumber(1L) .setPageSize(10L); ListProjectsResponse resp = client.listProjects(request); System.out.println(resp.getBody()); ``` Java SDK API docs: `https://aliyunsdk-pages.alicdn.com/apidocs/{PRODUCT_CODE}/{API_VERSION}/java-async-tea/8.0.3/index.html` ## MCP Server integration (last resort) Only use MCP when SDK calls keep failing after consulting docs, metadata, and GitHub source: ```bash npm install -g alibabacloud-dataworks-mcp-server ``` See `references/mcp_server.md` for full configuration. ## API categories | Category | Description | Example APIs | | ---------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | 认证文件管理 | Certificate management | [REDACTED_SECRET] | | 空间管理 | Workspace, roles, members | `*Project`, `*ProjectMember`, `*ProjectRole` | | 数据源 | Data source CRUD and connectivity test | `*DataSource`, `*DataSourceSharedRule` | | 计算资源 | Compute resource management | `*ComputeResource` | | 资源组管理 | Resource groups, routes, networks | `*ResourceGroup`, `*Route`, `*Network` | | 数据开发（新版） | Nodes, workflows, resources, functions, pipelines | `*Node`, `*WorkflowDefinition`, `*Resource`, `*Function`, `*PipelineRun` | | 数据开发（旧版） | Files, folders, business flows, deployments | `*File`, `*Folder`, `*Business`, `*DeploymentPackage` | | 数据集成 | DI sync jobs and alarm rules | `*DIJob`, `*DIAlarmRule` | | 数据地图 | Catalogs, databases, tables, columns, lineage, datasets, meta collections | `*Catalog`, `*Table`, `*Column`, `*Lineage*`, `*Dataset*`, `*MetaCollection` | | 运维中心 | Alerts, tasks, task instances, workflows, workflow instances | `*AlertRule`, `*Task`, `*TaskInstance*`, `*Workflow`, `*WorkflowInstance*` | | 数据质量 | Quality templates, scans, alert rules, scan runs | `*DataQualityTemplate`, `*DataQualityScan`, `*DataQualityAlertRule` | | 安全中心 | Identity credentials | `CreateIdentifyCredential` | | 标签管理 | Data asset tags | `*DataAssetTag`, `TagDataAssets`, `UnTagDataAssets` | | 开放平台 | Async job status | `GetJobStatus` | Use dynamic discovery (above) to get the full list and parameter schemas. Common naming patterns: 1. Inventory/list: prefer `List*` / `Get*` / `Describe*` APIs. 2. Create/configure: prefer `Create*` / `Update*` / `Import*` APIs. 3. Remove: prefer `Delete*` / `Remove*` APIs. 4. Lifecycle: prefer `Start*` / `Stop*` / `Suspend*` / `Resume*` / `Rerun*` APIs. ## Destructive operation policy Before executing delete, stop, or suspend operations, always summarize the affected resources and confirm with the user. ## Timestamp handling DataWorks APIs use millisecond timestamps. Convert between human-readable dates and timestamps as needed. ## Selection questions (ask when unclear) 1. Which DataWorks project (project ID or name)? 2. Which region? (common: `cn-shanghai`, `cn-beijing`, `cn-hangzhou`, `cn-shenzhen`) 3. What type of operation? (data development / operations / data quality / metadata / workspace) 4. For task operations: target node ID, workflow, or instance ID? ## References - MCP Server integration: `references/mcp_server.md` - Sources: `references/sources.md`

path: README.md
content: # dataworks-open-api

Alibaba Cloud DataWorks skill for AI agents. Manage data development, workflow operations, data quality, metadata lineage, and workspace administration through a single skill.

## Skill structure

```
dataworks-open-api/
├── SKILL.md                          # Main skill instructions (read by AI agent)
├── agents/
│   └── openai.yaml                   # Agent interface definition
├── references/
│   ├── sources.md                    # API docs, SDK, and MCP Server links
│   └── mcp_server.md                 # MCP Server setup and configuration
└── scripts/
    ├── fetch_api_overview.py         # Fetch API list from official help docs
    └── list_openapi_meta_apis.py     # Fetch API specs from OpenAPI metadata
```

## Quick start

```bash
# Set credentials
export ALICLOUD_ACCESS_KEY_ID="your-ak"
export ALICLOUD_ACCESS_KEY_SECRET="your-sk"
export ALICLOUD_REGION_ID="cn-shanghai"

# Discover available APIs
python scripts/fetch_api_overview.py
python scripts/list_openapi_meta_apis.py
```

## SDK support

| Language   | Package                                        |
| ---------- | ---------------------------------------------- |
| Node.js    | `@alicloud/dataworks-public20240518`           |
| Python     | `alibabacloud_dataworks_public20240518`        |
| Java       | `com.aliyun:alibabacloud-dataworks_public20240518` |

## Troubleshooting fallback chain

When execution fails, escalate in this order:

1. Official help docs
2. OpenAPI metadata (raw JSON spec)
3. SDK API docs
4. GitHub SDK source code
5. MCP Server
6. Browser (DataWorks IDE)

See `SKILL.md` for full details on each level.

## Links

- [DataWorks documentation](https://www.alibabacloud.com/help/dataworks)
- [OpenAPI Explorer](https://api.aliyun.com/product/dataworks-public)
- [MCP Server (npm)](https://www.npmjs.com/package/alibabacloud-dataworks-mcp-server)
- [MCP Server (GitHub)](https://github.com/aliyun/alibabacloud-dataworks-mcp-server)


path: agents/openai.yaml
content: interface:
  display_name: "Alibaba Cloud DataWorks"
  short_description: "Dynamic API discovery + SDK-first approach for all DataWorks operations"
  default_prompt: "Use $dataworks-open-api to complete this data development/operations task on Alibaba Cloud DataWorks."


path: references/mcp_server.md
content: # MCP Server Integration

DataWorks MCP Server exports all DataWorks Open APIs as MCP tools, allowing AI agents to interact with DataWorks through the Model Context Protocol.

## Installation

```bash
npm install -g alibabacloud-dataworks-mcp-server
```

## Configuration

### For Cursor / Cline / MCP-compatible clients

```json
{
  "mcpServers": {
    "alibabacloud-dataworks-mcp-server": {
      "command": "npx",
      "args": ["alibabacloud-dataworks-mcp-server"],
      "env": {
        "REGION": "cn-shanghai",
        "ALIBABA_CLOUD_ACCESS_KEY_ID": "your_access_key_id",
        "ALIBABA_CLOUD_ACCESS_KEY_SECRET": "your_access_key_secret"
      }
    }
  }
}
```

### Environment Variables

| Variable | Required | Description |
|----------|----------|-------------|
| `REGION` | Yes | DataWorks API region (e.g. `cn-shanghai`) |
| `ALIBABA_CLOUD_ACCESS_KEY_ID` | Yes | Alibaba Cloud Access Key ID |
| `ALIBABA_CLOUD_ACCESS_KEY_SECRET` | Yes | Alibaba Cloud Access Key Secret |
| `TOOL_CATEGORIES` | No | Comma-separated category filter |
| `TOOL_NAMES` | No | Comma-separated API name filter |
| `TOOL_FILE_URI` | No | Local file URI for tools metadata (overrides remote) |

### Tool Categories

Filter tools by category using `TOOL_CATEGORIES`:

- `DATA_DEVELOP` — Data development (nodes, resources, workflows)
- `OPS_CENTER` — Operations center (task instances, alerts)
- `DATA_QUALITY` — Data quality rules and evaluation
- `DATA_MAP` — Metadata, lineage, catalogs
- `WORKSPACE` — Project and member management
- `DATA_INTEGRATION` — DI jobs
- `DATA_SOURCE` — Data source management
- `RESOURCE_GROUP` — Resource groups and networks
- `SERVER_IDE_DEFAULT` — Default IDE tool set

## Architecture

The MCP Server dynamically fetches tool definitions from:
- `https://dataworks.data.aliyun.com/pop-mcp-tools` (production)
- `https://pre-dataworks.data.aliyun.com/pop-mcp-tools` (pre-release, when `NODE_ENV=development`)

This means the tool list always reflects the latest available DataWorks APIs.

## API Invocation

The server uses Alibaba Cloud's generic OpenAPI client (RPC style) to invoke DataWorks APIs. Input parameters are validated against Zod schemas derived from the tool metadata.


path: references/sources.md
content: # Sources

> Constants: **PRODUCT_CODE** = `dataworks-public`, **API_VERSION** = `2024-05-18` (defined in SKILL.md)

## API & metadata
- OpenAPI product page: `https://api.aliyun.com/product/{PRODUCT_CODE}`
- API overview (help docs): `https://help.aliyun.com/zh/dataworks/developer-reference/api-{PRODUCT_CODE}-{API_VERSION}-overview`
- API list (metadata JSON): `https://next.api.aliyun.com/meta/v1/products/{PRODUCT_CODE}/versions/{API_VERSION}/api-docs.json`
- Single API definition: `https://next.api.aliyun.com/meta/v1/products/{PRODUCT_CODE}/versions/{API_VERSION}/apis/{ApiName}/api.json`

## SDK
- Node.js SDK (npm): `https://www.npmjs.com/package/@alicloud/dataworks-public20240518`
- Python SDK (PyPI): `https://pypi.org/project/alibabacloud-dataworks-public20240518/`
- Java SDK API docs: `https://aliyunsdk-pages.alicdn.com/apidocs/{PRODUCT_CODE}/{API_VERSION}/java-async-tea/8.0.3/index.html`
- Generalized call guide (Node.js): `https://help.aliyun.com/zh/sdk/developer-reference/generalized-call-node-js`

## MCP Server
- MCP Server tools discovery: `https://dataworks.data.aliyun.com/pop-mcp-tools`
- MCP Server metadata: `https://dataworks.data.aliyun.com/mcp`
- MCP Server tools overview: `https://dataworks.data.aliyun.com/dw-pop-mcptools`
- MCP Server GitHub: `https://github.com/aliyun/alibabacloud-dataworks-mcp-server`
- MCP Server npm: `https://www.npmjs.com/package/alibabacloud-dataworks-mcp-server`

## Documentation
- DataWorks documentation: `https://www.alibabacloud.com/help/dataworks`
- DataWorks MCP usage guide: `https://www.alibabacloud.com/help/dataworks/user-guide/dataworks-mcp-server-function-usage`


path: scripts/fetch_api_overview.py
content: #!/usr/bin/env python3
"""Fetch DataWorks API overview from the official Alibaba Cloud help docs.

Dynamically curls the help page, extracts the API list from
window.__ICE_PAGE_PROPS__.docDetailData.storeData.data.content,
parses the HTML tables, and outputs a structured markdown file.

Usage:
    python scripts/fetch_api_overview.py
    python scripts/fetch_api_overview.py --url <custom_help_url>
    python scripts/fetch_api_overview.py --output-dir output/alicloud-dataworks

Env:
- FETCH_TIMEOUT (seconds, default: 30)
"""

from __future__ import annotations

import argparse
import json
import os
import pathlib
import re
import urllib.request
from html.parser import HTMLParser

DEFAULT_PRODUCT_CODE = "dataworks-public"
DEFAULT_VERSION = "2024-05-18"
DEFAULT_URL = (
    f"https://help.aliyun.com/zh/dataworks/developer-reference"
    f"/api-{DEFAULT_PRODUCT_CODE}-{DEFAULT_VERSION}-overview"
)
OUTPUT_DIR = pathlib.Path("output/dataworks-open-api")


def fetch_page(url: str, timeout: int) -> str:
    """Fetch page HTML. Tries urllib first, falls back to curl on SSL errors."""
    import ssl
    import subprocess

    headers = {
        "User-Agent": (
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/131.0.0.0 Safari/537.36"
        ),
    }

    # Try with default SSL context first
    for ctx in [None, ssl.create_default_context(), ssl._create_unverified_context()]:
        try:
            req = urllib.request.Request(url, headers=headers)
            kwargs: dict = {"timeout": timeout}
            if ctx is not None:
                kwargs["context"] = ctx
            with urllib.request.urlopen(req, **kwargs) as resp:
                return resp.read().decode("utf-8")
        except (urllib.error.URLError, ssl.SSLError):
            continue

    # Fallback: use curl
    print("urllib SSL failed, falling back to curl...")
    result = subprocess.run(
        ["curl", "-fsSL", "-H", f"User-Agent: {headers['User-Agent']}", url],
        capture_output=True, text=True, timeout=timeout,
    )
    if result.returncode != 0:
        raise RuntimeError(f"curl failed ({result.returncode}): {result.stderr}")
    return result.stdout


def extract_doc_content(html: str) -> str:
    """Extract content from window.__ICE_PAGE_PROPS__."""
    match = re.search(
        r"window\.__ICE_PAGE_PROPS__\s*=\s*(\{.*?\});?\s*</script>",
        html,
        re.DOTALL,
    )
    if not match:
        raise ValueError("Could not find window.__ICE_PAGE_PROPS__ in page")

    data = json.loads(match.group(1))

    # Navigate: docDetailData.storeData.data.content (primary)
    #       or: docDetailData.data.content (fallback)
    detail = data.get("docDetailData", {})
    content = (
        detail.get("storeData", {}).get("data", {}).get("content", "")
        or detail.get("data", {}).get("content", "")
    )
    if not content:
        raise ValueError(
            "content field is empty. "
            f"Available top-level keys: {list(detail.keys())}"
        )
    return content


class _APITableParser(HTMLParser):
    """Parse API tables from the help doc HTML content."""

    def __init__(self) -> None:
        super().__init__()
        self._in_h2 = False
        self._in_h3 = False
        self._in_td = False
        self._text = ""
        self._row: list[str] = []
        self._in_tr = False
        self._current_section: str | None = None
        self._rows: list[tuple[list[str], str | None]] = []

    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
        if tag == "h2":
            self._in_h2 = True
            self._text = ""
        elif tag == "h3":
            self._in_h3 = True
            self._text = ""
        elif tag == "tr":
            self._in_tr = True
            self._row = []
        elif tag == "td":
            self._in_td = True
            self._text = ""

    def handle_endtag(self, tag: str) -> None:
        if tag == "h2" and self._in_h2:
            self._in_h2 = False
            t = self._text.strip()
            if t and "SDK" not in t:
                self._current_section = t
        elif tag == "h3" and self._in_h3:
            self._in_h3 = False
            t = self._text.strip()
            if t:
                self._current_section = t
        elif tag == "td":
            self._in_td = False
            self._row.append(self._text.strip())
        elif tag == "tr" and self._in_tr:
            self._in_tr = False
            if self._row and len(self._row) >= 2:
                self._rows.append((list(self._row), self._current_section))

    def handle_data(self, data: str) -> None:
        if self._in_h2 or self._in_h3 or self._in_td:
            self._text += data

    def get_grouped(self) -> dict[str, list[list[str]]]:
        from collections import OrderedDict
        grouped: dict[str, list[list[str]]] = OrderedDict()
        for row, section in self._rows:
            if not section:
                continue
            if section not in grouped:
                grouped[section] = []
            grouped[section].append(row)
        return grouped


def parse_api_tables(html_content: str) -> dict[str, list[list[str]]]:
    parser = _APITableParser()
    parser.feed(html_content)
    return parser.get_grouped()


def build_markdown(grouped: dict[str, list[list[str]]], version: str = DEFAULT_VERSION) -> str:
    lines = [f"# DataWorks API Overview ({version})", ""]

    total_apis = 0
    for section, rows in grouped.items():
        api_rows = [r for r in rows if len(r) >= 2 and r[0] != "API"]
        if not api_rows:
            continue
        total_apis += len(api_rows)
        lines.append(f"## {section} ({len(api_rows)} APIs)")
        lines.append("")
        lines.append("| API | 描述 |")
        lines.append("|-----|------|")
        for row in api_rows:
            lines.append(f"| {row[0]} | {row[1]} |")
        lines.append("")

    lines.insert(2, f"Total: {total_apis} APIs across {len(grouped)} categories")
    lines.insert(3, "")
    return "\n".join(lines) + "\n"


def build_json(grouped: dict[str, list[list[str]]]) -> list[dict]:
    result = []
    for section, rows in grouped.items():
        apis = []
        for row in rows:
            if len(row) >= 2 and row[0] != "API":
                apis.append({"name": row[0], "description": row[1]})
        if apis:
            result.append({"category": section, "count": len(apis), "apis": apis})
    return result


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Fetch DataWorks API overview from official help docs"
    )
    parser.add_argument("--url", default=DEFAULT_URL, help="Help doc URL")
    parser.add_argument("--output-dir", default=str(OUTPUT_DIR))
    args = parser.parse_args()

    timeout = int(os.getenv("FETCH_TIMEOUT", "30"))
    output_dir = pathlib.Path(args.output_dir)
    output_dir.mkdir(parents=True, exist_ok=True)

    print(f"Fetching: {args.url}")
    page_html = fetch_page(args.url, timeout)
    print(f"Page size: {len(page_html):,} bytes")

    doc_content = extract_doc_content(page_html)
    print(f"Doc content: {len(doc_content):,} chars")

    grouped = parse_api_tables(doc_content)

    total = sum(
        len([r for r in rows if len(r) >= 2 and r[0] != "API"])
        for rows in grouped.values()
    )
    print(f"Found: {total} APIs across {len(grouped)} categories")
    for section, rows in grouped.items():
        count = len([r for r in rows if len(r) >= 2 and r[0] != "API"])
        print(f"  {section}: {count} APIs")

    md_file = output_dir / "api_overview.md"
    md_file.write_text(build_markdown(grouped), encoding="utf-8")
    print(f"\nSaved: {md_file}")

    json_file = output_dir / "api_overview.json"
    json_data = build_json(grouped)
    json_file.write_text(
        json.dumps(json_data, ensure_ascii=False, indent=2), encoding="utf-8"
    )
    print(f"Saved: {json_file}")


if __name__ == "__main__":
    main()


path: scripts/list_openapi_meta_apis.py
content: #!/usr/bin/env python3
"""Fetch OpenAPI metadata API list for DataWorks and save to output/.

Env:
- OPENAPI_META_TIMEOUT (seconds, default: 20)
"""

from __future__ import annotations

import argparse
import json
import os
import pathlib
import urllib.request

DEFAULT_PRODUCT_CODE = "dataworks-public"
DEFAULT_VERSION = "2024-05-18"
OUTPUT_DIR = pathlib.Path("output/dataworks-open-api")


def fetch_json(url: str, timeout: int) -> dict:
    """Fetch JSON from URL. Tries urllib first, falls back to curl on SSL errors."""
    import ssl
    import subprocess
    
    headers = {"User-Agent": "codex-skill"}
    
    # Try with default SSL context first
    for ctx in [None, ssl.create_default_context(), ssl._create_unverified_context()]:
        try:
            req = urllib.request.Request(url, headers=headers)
            kwargs: dict = {"timeout": timeout}
            if ctx is not None:
                kwargs["context"] = ctx
            with urllib.request.urlopen(req, **kwargs) as resp:
                return json.loads(resp.read().decode("utf-8"))
        except (urllib.error.URLError, ssl.SSLError):
            continue
    
    # Fallback: use curl
    print("urllib SSL failed, falling back to curl...")
    result = subprocess.run(
        ["curl", "-fsSL", "-H", f"User-Agent: {headers['User-Agent']}", url],
        capture_output=True, text=True, timeout=timeout,
    )
    if result.returncode != 0:
        raise RuntimeError(f"curl failed ({result.returncode}): {result.stderr}")
    return json.loads(result.stdout)


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--product-code", default=DEFAULT_PRODUCT_CODE)
    parser.add_argument("--version", default=DEFAULT_VERSION)
    parser.add_argument("--output-dir", default=str(OUTPUT_DIR))
    args = parser.parse_args()

    timeout = int(os.getenv("OPENAPI_META_TIMEOUT", "20"))
    output_dir = pathlib.Path(args.output_dir)
    output_dir.mkdir(parents=True, exist_ok=True)

    url = (
        f"https://next.api.aliyun.com/meta/v1/products/{args.product_code}"
        f"/versions/{args.version}/api-docs.json"
    )
    payload = fetch_json(url, timeout)

    raw_apis = payload.get("apis", {})
    if isinstance(raw_apis, dict):
        api_names = sorted(raw_apis.keys())
    elif isinstance(raw_apis, list):
        names = []
        for item in raw_apis:
            if isinstance(item, dict):
                name = item.get("name") or item.get("apiName")
                if name:
                    names.append(name)
            elif isinstance(item, str):
                names.append(item)
        api_names = sorted(set(names))
    else:
        api_names = []

    json_file = output_dir / f"{args.product_code}_{args.version}_api_docs.json"
    md_file = output_dir / f"{args.product_code}_{args.version}_api_list.md"

    json_file.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
    md_lines = [
        f"# {args.product_code} {args.version} API List",
        "",
        f"- Source: {url}",
        f"- API count: {len(api_names)}",
        "",
    ]
    md_lines.extend([f"- `{name}`" for name in api_names])
    md_file.write_text("\n".join(md_lines) + "\n", encoding="utf-8")

    print(f"Saved: {json_file}")
    print(f"Saved: {md_file}")


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)