Skill: dataify-server/dataify-github-repository-by-repo-url (version 1.0.0)

--- name: [REDACTED_SECRET] description: "Prepare Dataify builder requests for the github.com scraper family rooted at github_repository_by-repo-url. Use when needs to work with the successful Dataify scraper detail entry for github_repository_by-repo-url, let the user choose one of its available tools, read saved getToolParams options, and generate a scraperapi.dataify.com/builder curl request with DATAIFY_API_TOKEN." --- # Dataify Builder Skill Use this skill to prepare Dataify builder requests for the scraper family rooted at `github_repository_by-repo-url` on `github.com`. ## Workflow 1. Check whether `DATAIFY_API_TOKEN` exists in the environment. 2. If the token is missing, stop and tell the user to sign in at Dataify Dashboard](https://dataify.com/dashboard) to obtain it. 3. Ask the user to choose exactly one tool from the following Chinese list: - 通过仓库URL采集 (github_repository_by-repo-url) - 通过搜索URL采集 (github_repository_by-search-url) - 通过URL采集 (github_repository_by-url) 4. Read `references/tool-params.json` and find the chosen tool by `tool_sign` or Chinese tool name. 5. For each parameter in the chosen tool: - If `input_mode` is `user_input`, ask the user for the value. - If `input_mode` is `select`, present the saved options to the user. 6. Use `scripts/build-dataify-request.py` as the default cross-platform helper. 7. Use `scripts/build-dataify-request.ps1` as the Windows PowerShell helper when needed. 8. When a selectable parameter has a human-readable Chinese label, keep that label in `spider_parameters`. Do not replace it with a code such as `HK` unless the user explicitly asks for the coded value. 9. Build `spider_parameters` as a JSON array. 10. If every parameter has only one final value, build one object such as `[{"searchurl":"...","country":"Hong Kong"}]`. 11. If one or more parameters have multiple aligned values, zip them by index and build one object per row. Example: `[{"search_url":"url1","page_turning":"1","max_num":"15"},{"search_url":"url2","page_turning":"1","max_num":"15"}]`. 12. If a parameter has one value while another parameter has multiple values, reuse the single value across every generated row. 13. Set `spider_name` to `github.com`. 14. Set `spider_id` to the selected tool's `tool_sign`. 15. Always include `spider_errors=true` and `file_name={{TasksID}}`. 16. Return a curl command for `https://scraperapi.dataify.com/builder`. ## Set DATAIFY_API_TOKEN Prefer a permanent environment-variable setup instead of setting the token only for the current terminal session. Windows PowerShell, permanent for the current user: ```powershell [Environment]::SetEnvironmentVariable("DATAIFY_API_TOKEN", "your_token_here", "User") ``` Then reopen PowerShell. If the current session also needs the token immediately, run: ```powershell $env:DATAIFY_API_TOKEN = "your_token_here" ``` macOS or Linux, permanent for bash: ```bash echo 'export DATAIFY_API_TOKEN="your_token_here"' >> ~/.bashrc source ~/.bashrc ``` macOS or Linux, permanent for zsh: ```bash echo 'export DATAIFY_API_TOKEN="your_token_here"' >> ~/.zshrc source ~/.zshrc ``` ## Script usage Python: ```bash python scripts/build-dataify-request.py --tool-sign <selected_tool_sign> --values-file values.json ``` PowerShell: ```powershell & ".\scripts\build-dataify-request.ps1" -ToolSign "<selected_tool_sign>" -ValuesFile ".\values.json" ``` The `values.json` file should contain either one object or an array of objects. Example: ```json [{"searchurl":"https://www.airbnb.com/s/Greece/homes?...","country":"Hong Kong"}] ``` ## Required output shape Generate a curl command in this form: ```bash curl -X POST 'https://scraperapi.dataify.com/builder' \ -H "Authorization: Bearer $DATAIFY_API_TOKEN" \ -H 'Content-Type: application/x-www-form-urlencoded' \ -d 'spider_name=github.com' \ -d 'spider_id=<selected_tool_sign>' \ -d 'spider_parameters=[{"param":"value"}]' \ -d 'spider_errors=true' \ -d 'file_name={{TasksID}}' ``` ## Reference usage - `references/tool-params.json` stores the full saved parameter catalog for every available tool in this scraper family. - `scripts/build-dataify-request.py` is the portable implementation and should be preferred. - `scripts/build-dataify-request.ps1` mirrors the same behavior for Windows users. - If a parameter has no options, the user must provide the value. - If a parameter has options, present those options back to the user before building the final request. - Do not assume `spider_parameters` always contains exactly one object. Multi-value tools may require multiple objects zipped by index. - Use the saved `url_example` only as a reference example. Do not assume the user wants the example values unless they explicitly confirm them.

path: SKILL.zh-CN.md
content: # Dataify Builder Skill 中文版

这个 skill 用于为 `github.com` 下、以 `github_repository_by-repo-url` 为入口的 Dataify scraper 工具族生成 builder 请求。

## 工作流程

1. 先检查环境变量中是否存在 `DATAIFY_API_TOKEN`。
2. 如果 token 缺失，提示用户前往 <a href="https://www.dataify.com/">dataify&#23448;&#32593;</a> 获取。
3. 先让用户从下面的中文工具列表中明确选择一个工具：
- 通过仓库URL采集 (github_repository_by-repo-url)
- 通过搜索URL采集 (github_repository_by-search-url)
- 通过URL采集 (github_repository_by-url)
4. 再读取 `references/tool-params.json`，根据 `tool_sign` 或中文工具名找到对应工具。
5. 对所选工具的每个参数分别处理：
   - 如果 `input_mode` 是 `user_input`，让用户提供值。
   - 如果 `input_mode` 是 `select`，把已保存的可选项展示给用户，让用户选择。
6. 默认优先使用 `scripts/build-dataify-request.py`，因为它是跨平台版本。
7. Windows 下也可以使用 `scripts/build-dataify-request.ps1`。
8. 对于可选型参数，如果存在人类可读标签，优先把该标签写入 `spider_parameters`。
9. `spider_parameters` 必须是一个 JSON 数组。
10. 像搜索 URL 这类多值工具，需要按索引生成多个对象。
11. `spider_name` 固定取 `github.com`。
12. `spider_id` 固定取用户所选工具的 `tool_sign`。
13. 始终包含 `spider_errors=true` 和 `file_name={{TasksID}}`。

## 设置 DATAIFY_API_TOKEN

推荐使用永久环境变量，而不是只在当前终端临时设置。

Windows PowerShell，当前用户永久设置：

```powershell
[Environment]::SetEnvironmentVariable("DATAIFY_API_TOKEN", "your_token_here", "User")
```

然后重新打开 PowerShell。如果当前会话也要立即生效，再执行：

```powershell
$env:DATAIFY_API_TOKEN = "your_token_here"
```

macOS 或 Linux，bash 永久设置：

```bash
echo 'export DATAIFY_API_TOKEN="your_token_here"' >> ~/.bashrc
source ~/.bashrc
```

macOS 或 Linux，zsh 永久设置：

```bash
echo 'export DATAIFY_API_TOKEN="your_token_here"' >> ~/.zshrc
source ~/.zshrc
```

## 脚本用法

Python：

```bash
python scripts/build-dataify-request.py --tool-sign <selected_tool_sign> --values-file values.json
```

PowerShell：

```powershell
& ".\scripts\build-dataify-request.ps1" -ToolSign "<selected_tool_sign>" -ValuesFile ".\values.json"
```

`values.json` 可以是单个对象，也可以是对象数组。

## 输出格式

最终 `curl` 命令应为：

```bash
curl -X POST 'https://scraperapi.dataify.com/builder' \
  -H "Authorization: Bearer $DATAIFY_API_TOKEN" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'spider_name=github.com' \
  -d 'spider_id=<selected_tool_sign>' \
  -d 'spider_parameters=[{"param":"value"}]' \
  -d 'spider_errors=true' \
  -d 'file_name={{TasksID}}'
```

## 参考文件

- `references/tool-params.json` 保存了这个 skill 下所有工具及参数选项。
- `scripts/build-dataify-request.py` 是首选的跨平台实现。
- `scripts/build-dataify-request.ps1` 是 Windows PowerShell 版本。
- 如果参数没有预设选项，必须向用户要值。
- 如果参数有预设选项，先把选项展示给用户，再生成最终请求。
- 不要假设 `spider_parameters` 永远只有一个对象；多值工具可能需要按索引生成多个对象。
- `url_example` 仅作为参考，不要默认用户就要用示例值，除非用户明确确认。


path: scripts/build-dataify-request.py
content: #!/usr/bin/env python3

import argparse
import json
import os
import sys
from pathlib import Path
from typing import Any, Dict, List


def load_catalog(script_dir: Path) -> List[Dict[str, Any]]:
    catalog_path = script_dir.parent / "references" / "tool-params.json"
    return json.loads(catalog_path.read_text(encoding="utf-8"))


def find_tool(catalog: List[Dict[str, Any]], tool_sign: str) -> Dict[str, Any]:
    for tool in catalog:
        if tool.get("tool_sign") == tool_sign:
            return tool
    raise SystemExit(f"Unknown tool_sign: {tool_sign}")


def load_values(values_file: Path) -> List[Dict[str, Any]]:
    payload = json.loads(values_file.read_text(encoding="utf-8"))
    if isinstance(payload, dict):
        return [payload]
    if isinstance(payload, list):
        return payload
    raise SystemExit("values file must be a JSON object or JSON array")


def map_select_labels(tool: Dict[str, Any], rows: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
    param_defs = {item["param"]: item for item in tool.get("params", [])}
    mapped_rows: List[Dict[str, Any]] = []
    for row in rows:
        mapped: Dict[str, Any] = {}
        for key, value in row.items():
            param_def = param_defs.get(key)
            final_value = value
            if param_def and param_def.get("input_mode") == "select":
                for option in param_def.get("options", []):
                    if value in {
                        option.get("label"),
                        option.get("submitted_value"),
                        option.get("raw_value"),
                        option.get("raw_type_value"),
                    }:
                        final_value = option.get("submitted_value")
                        break
            mapped[key] = final_value
        mapped_rows.append(mapped)
    return mapped_rows


def build_curl(tool: Dict[str, Any], spider_parameters_json: str) -> str:
    [REDACTED_SECRET]"DATAIFY_API_TOKEN", "").strip()
    if not token:
        raise SystemExit(
            "DATAIFY_API_TOKEN is not set. Sign in at https://www.dataify.com/ to obtain it, then export it as an environment variable."
        )

    parts = [
        "curl -X POST 'https://scraperapi.dataify.com/builder'",
        f"  -H 'Authorization: Bearer {token}'",
        "  -H 'Content-Type: application/x-www-form-urlencoded'",
        f"  -d 'spider_name={tool['spider_name']}'",
        f"  -d 'spider_id={tool['tool_sign']}'",
        f"  -d 'spider_parameters={spider_parameters_json}'",
        "  -d 'spider_errors=true'",
        "  -d 'file_name={{TasksID}}'",
    ]
    return " \\\n".join(parts)


def main() -> int:
    parser = argparse.ArgumentParser(description="Build a Dataify builder curl request.")
    parser.add_argument("--tool-sign", required=True, help="Selected tool sign.")
    parser.add_argument("--values-file", required=True, help="Path to a JSON object or array with spider parameter values.")
    parser.add_argument("--print-spider-parameters-only", action="store_true", help="Print only the normalized spider_parameters JSON.")
    args = parser.parse_args()

    script_dir = Path(__file__).resolve().parent
    catalog = load_catalog(script_dir)
    tool = find_tool(catalog, args.tool_sign)
    rows = load_values(Path(args.values_file).resolve())
    normalized = map_select_labels(tool, rows)
    spider_parameters_json = json.dumps(normalized, ensure_ascii=False, separators=(",", ":"))

    if args.print_spider_parameters_only:
        print(spider_parameters_json)
        return 0

    print(build_curl(tool, spider_parameters_json))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())


path: agents/openai.yaml
content: interface:
  display_name: "Dataify Github Builder"
  short_description: "Prepare Dataify builder calls for github.com"
  default_prompt: "Use $dataify-github-repository-by-repo-url to prepare a Dataify builder curl request."


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)