Skill: wangzhiming1999/felo-web-fetch (version 1.0.1)

--- name: felo-web-fetch description: "Fetch web page content from a URL using Felo Web Extract API. Use when users ask to scrape/capture/fetch webpage content, get article text from URL, convert page to markdown/text, or when explicit commands like /felo-web-fetch are used. Supports html, text, markdown output and readability mode." --- # Felo Web Fetch Skill ## When to Use Trigger this skill when the user wants to: - Fetch or scrape content from a webpage URL - Get article/main text from a link - Convert a webpage to Markdown or plain text - Capture readable content from a URL for summarization or processing Trigger keywords (examples): - fetch webpage, scrape URL, fetch page content, web fetch, url to markdown - Explicit: `/felo-web-fetch`, "use felo web fetch" - Same intent in other languages (e.g. 网页抓取, 提取网页内容) also triggers this skill Do NOT use for: - Real-time search or Q&A (use `felo-search`) - Generating slides (use `felo-slides`) - Local file content (read files directly) ## Setup ### 1. Get API key 1. Visit [felo.ai](https://felo.ai) 2. Open Settings -> API Keys 3. Create and copy your API key ### 2. Configure environment variable Linux/macOS: ```bash export FELO_API_KEY="your-api-key-here" ``` Windows PowerShell: ```powershell $env:FELO_API_KEY="your-api-key-here" ``` ## How to Execute ### Option A: Use the bundled script or packaged CLI **Script** (from repo): ```bash node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com/article" [options] ``` **Packaged CLI** (after `npm install -g felo-ai`): same options, with short forms allowed: ```bash felo web-fetch -u "https://example.com" [options] # Short forms: -u (url), -f (format), -t (timeout, seconds), -j (json) ``` Options: | Option | Default | Description | |--------|---------|-------------| | `--url` | (required) | Webpage URL to fetch | | `--format` | markdown | Output format: `html`, `text`, `markdown` | | `--target-selector` | - | CSS selector: fetch only this element (e.g. `article.main`, `#content`) | | `--wait-for-selector` | - | Wait for this selector before fetching (e.g. dynamic content) | | `--readability` | false | Enable readability processing (main content only) | | `--crawl-mode` | fast | `fast` or `fine` | | `--timeout` | 60000 (script) / 60 (CLI) | Request timeout: script uses **milliseconds**, CLI uses **seconds** (e.g. `-t 90`) | | `--json` / `-j` | false | Print full API response as JSON | ### How to write instructions (target_selector + output_format) When the user wants a **specific part** of the page or a **specific output format**, phrase the command like this: - **Output format**: "Fetch as **text**" / "Get **markdown**" / "Return **html**" → use `--format text`, `--format markdown`, or `--format html`. - **Target one element**: "Only the **main article**" / "Just the **content inside** `#main`" / "Fetch only **article.main-content**" → use `--target-selector "article.main"` or the selector they give (e.g. `#main`, `.main-content`, `article .post`). Examples of user intents and equivalent commands: | User intent | Command | |-------------|---------| | "Fetch this page as plain text" | `--url "..." --format text` | | "Get only the main content area" | `--url "..." --target-selector "main"` or `article` | | "Fetch the div with id=content as markdown" | `--url "..." --target-selector "#content" --format markdown` | | "Just the article body, as HTML" | `--url "..." --target-selector "article .body" --format html` | Examples: ```bash # Basic: fetch as Markdown node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" # Article-style with readability node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com/article" --readability --format markdown # Raw HTML node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --format html --json # Only the element matching a CSS selector (e.g. main article) node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --target-selector "article.main" --format markdown # Specific output format + target selector node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --target-selector "#content" --format text ``` ### Option B: Call API with curl ```bash curl -X POST "https://openapi.felo.ai/v2/web/extract" \ -H "Authorization: Bearer $FELO_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "output_format": "markdown", "with_readability": true}' ``` ## API Reference (summary) - **Endpoint**: `POST /v2/web/extract` - **Base URL**: `https://openapi.felo.ai`. Override with `FELO_API_BASE` env if needed. - **Auth**: `[REDACTED_SECRET]` ### Request body (JSON) | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | url | string | Yes | - | Webpage URL to fetch | | crawl_mode | string | No | fast | `fast` or `fine` | | output_format | string | No | html | `html`, `text`, `markdown` | | with_readability | boolean | No | - | Use readability (main content) | | with_links_summary | boolean | No | - | Include links summary | | with_images_summary | boolean | No | - | Include images summary | | target_selector | string | No | - | CSS selector for target element | | wait_for_selector | string | No | - | Wait for selector before fetch | | timeout | integer | No | - | Timeout in milliseconds | | with_cache | boolean | No | true | Use cache | ### Response Success (200): ```json { "code": 0, "message": "success", "data": { "content": { ... } } } ``` Fetched content is in `data.content`; structure depends on `output_format`. ### Error codes | HTTP | Code | Description | |------|------|-------------| | 400 | - | Parameter validation failed | | 401 | INVALID_API_KEY | API key invalid or revoked | | 500/502 | WEB_EXTRACT_FAILED | Fetch failed (server or page error) | ## Output Format On success (script without `--json`): - Print the fetched content only (for direct use or piping). With `--json`: - Print full API response including `code`, `message`, `data`. Error response to user: ```markdown ## Web Fetch Failed - Error: <code or message> - URL: <requested url> - Suggestion: <e.g. check URL, retry, or use --timeout> ``` ## Important Notes - Always check `FELO_API_KEY` before calling; if missing, return setup instructions. - For long articles or slow sites, consider `--timeout` or `timeout` in request body. - Use `output_format: "markdown"` and `with_readability: true` for clean article text. - API may cache results; use `with_cache: false` in body only when fresh content is required (script does not expose this by default). ## References - [Felo Web Extract API](https://openapi.felo.ai/docs/api-reference/v2/web-extract.html) - [Felo Open Platform](https://openapi.felo.ai/docs/)

path: README.md
content: # Felo Web Fetch Skill

Fetch webpage content from a URL using the [Felo Web Extract API](https://openapi.felo.ai/docs/api-reference/v2/web-extract.html).

## Features

- Fetch content from any URL as **html**, **text**, or **markdown** (`--format`)
- **Target one element** with a CSS selector (`--target-selector`, e.g. `article.main`, `#content`)
- Optional **readability** mode for main article content only
- Crawl modes: `fast` (default) or `fine`
- Same `FELO_API_KEY` as other Felo skills

## Quick Start

### 1) Configure API key

At [felo.ai](https://felo.ai) -> Settings -> API Keys, create a key, then:

```bash
# Linux/macOS
export FELO_API_KEY="your-api-key-here"
```

```powershell
# Windows PowerShell
$env:FELO_API_KEY="your-api-key-here"
```

### 2) Run fetch

```bash
# Fetch as Markdown (default)
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com/article"

# With readability for clean article text
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --readability --format markdown

# Full JSON response
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --json

# Only a specific element (CSS selector) and output format
node felo-web-fetch/scripts/run_web_fetch.mjs --url "https://example.com" --target-selector "article.main" --format markdown
```

## Using the packaged CLI (`felo web-fetch`)

After `npm install -g felo-ai`, you can run:

```bash
felo web-fetch --url "https://example.com"
```

**All parameters (how to pass)**

| Parameter | Option | Example |
|-----------|--------|---------|
| URL (required) | `-u`, `--url` | `--url "https://example.com"` |
| Output format | `-f`, `--format` | `--format text`, `-f markdown`, `-f html` |
| Target element (CSS selector) | `--target-selector` | `--target-selector "article.main"` |
| Wait for selector | `--wait-for-selector` | `--wait-for-selector ".content"` |
| Readability (main content only) | `--readability` | `--readability` |
| Crawl mode | `--crawl-mode` | `--crawl-mode fine` (default: `fast`) |
| Timeout (seconds) | `-t`, `--timeout` | `--timeout 120`, `-t 90` |
| Full JSON response | `-j`, `--json` | `-j` or `--json` |

**Examples with multiple options**

```bash
felo web-fetch -u "https://example.com" -f text --readability
felo web-fetch --url "https://example.com" --target-selector "#content" --format markdown --timeout 90
felo web-fetch --url "https://example.com" --wait-for-selector "main" --readability -j
```

## When to use (Agent)

Trigger keywords: fetch webpage, scrape URL, fetch page content, url to markdown, `/felo-web-fetch`.

See [SKILL.md](SKILL.md) for full agent instructions and API parameters.


path: scripts/run_web_fetch.mjs
content: #!/usr/bin/env node

const DEFAULT_API_BASE = 'https://openapi.felo.ai';
const DEFAULT_FORMAT = 'markdown';
const DEFAULT_TIMEOUT_MS = 60_000;
const SPINNER_FRAMES = ['⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'];
const SPINNER_INTERVAL_MS = 80;
const STATUS_PAD = 56;

function startSpinner(message) {
  const start = Date.now();
  let i = 0;
  const id = setInterval(() => {
    const elapsed = Math.floor((Date.now() - start) / 1000);
    const line = `${message} ${SPINNER_FRAMES[i % SPINNER_FRAMES.length]} ${elapsed}s`;
    process.stderr.write(`\r${line.padEnd(STATUS_PAD, ' ')}`);
    i += 1;
  }, SPINNER_INTERVAL_MS);
  return id;
}

function stopSpinner(id) {
  if (id != null) clearInterval(id);
  process.stderr.write(`\r${' '.repeat(STATUS_PAD)}\r`);
}

function usage() {
  console.error(
    [
      'Usage:',
      '  node felo-web-fetch/scripts/run_web_fetch.mjs --url <url> [options]',
      '',
      'Options:',
      '  --url <url>           Page URL to fetch (required)',
      '  --format <format>     Output format: html, text, markdown (default: markdown)',
      '  --target-selector 
 CSS selector for target element only',
      '  --wait-for-selector 
 Wait for selector before fetch',
      '  --readability         Enable readability (main content only)',
      '  --crawl-mode <mode>   fast or fine (default: fast)',
      '  --timeout <ms>        Request timeout in ms (default: 60000)',
      '  --json                Print full API response as JSON',
      '  --help                Show this help',
    ].join('\n')
  );
}

function parseArgs(argv) {
  const out = {
    url: '',
    format: DEFAULT_FORMAT,
    targetSelector: '',
    waitForSelector: '',
    readability: false,
    crawlMode: 'fast',
    timeoutMs: DEFAULT_TIMEOUT_MS,
    json: false,
  };

  for (let i = 0; i < argv.length; i += 1) {
    const a = argv[i];
    if (a === '--help' || a === '-h') {
      out.help = true;
    } else if (a === '--json') {
      out.json = true;
    } else if (a === '--readability') {
      out.readability = true;
    } else if (a === '--url') {
      const next = argv[i + 1];
      if (next === undefined || next === null || String(next).trim() === '' || String(next).startsWith('-')) {
        out.url = '';
      } else {
        out.url = String(next).trim();
      }
      i += 1;
    } else if (a === '--format') {
      const f = (argv[i + 1] ?? '').toLowerCase();
      out.format = ['html', 'text', 'markdown'].includes(f) ? f : DEFAULT_FORMAT;
      i += 1;
    } else if (a === '--target-selector') {
      out.targetSelector = (argv[i + 1] ?? '').trim();
      i += 1;
    } else if (a === '--wait-for-selector') {
      out.waitForSelector = (argv[i + 1] ?? '').trim();
      i += 1;
    } else if (a === '--crawl-mode') {
      const m = (argv[i + 1] ?? '').toLowerCase();
      out.crawlMode = ['fast', 'fine'].includes(m) ? m : 'fast';
      i += 1;
    } else if (a === '--timeout') {
      const n = Number.parseInt(argv[i + 1] ?? '', 10);
      if (Number.isFinite(n) && n > 0) out.timeoutMs = n;
      i += 1;
    }
  }

  return out;
}

function getMessage(payload) {
  return (
    payload?.message ||
    payload?.error ||
    payload?.msg ||
    payload?.code ||
    'Unknown error'
  );
}

async function fetchJson(url, init, timeoutMs) {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), timeoutMs);
  try {
    const res = await fetch(url, { ...init, signal: controller.signal });
    let body = {};
    try {
      body = await res.json();
    } catch {
      body = {};
    }

    if (!res.ok) {
      throw new Error(`HTTP ${res.status}: ${getMessage(body)}`);
    }
    const code = body.code;
    const hasData = body?.data != null;
    const successCodes = [0, 200];
    const hasSuccessCode =
      successCodes.includes(Number(code)) ||
      code === undefined ||
      code === null ||
      (hasData && res.ok);
    if (!hasSuccessCode) {
      throw new Error(getMessage(body));
    }
    return body;
  } finally {
    clearTimeout(timer);
  }
}

function stringifyContent(content) {
  if (content == null) return '';
  if (typeof content === 'string') return content;
  if (typeof content === 'object') {
    if (content.markdown) return content.markdown;
    if (content.text) return content.text;
    if (content.html) return content.html;
    return JSON.stringify(content, null, 2);
  }
  return String(content);
}

async function main() {
  const args = parseArgs(process.argv.slice(2));
  if (args.help) {
    usage();
    process.exit(0);
  }
  if (!args.url) {
    usage();
    process.exit(1);
  }

  const [REDACTED_SECRET]);
  if (!apiKey) {
    console.error('ERROR: FELO_API_KEY not set');
    process.exit(1);
  }

  const apiBase = (process.env.FELO_API_BASE?.trim() || DEFAULT_API_BASE).replace(/\/$/, '');

  const shortUrl = args.url.length > 45 ? args.url.slice(0, 42) + '...' : args.url;
  const spinnerId = startSpinner(`Fetching ${shortUrl}`);

  try {
    const body = {
      url: args.url,
      output_format: args.format,
      crawl_mode: args.crawlMode,
      with_readability: args.readability,
      timeout: args.timeoutMs,
    };
    if (args.targetSelector) body.target_selector = args.targetSelector;
    if (args.waitForSelector) body.wait_for_selector = args.waitForSelector;

    const payload = await fetchJson(
      `${apiBase}/v2/web/extract`,
      {
        method: 'POST',
        headers: {
          Accept: 'application/json',
          Authorization: `Bearer ${apiKey}`,
          'Content-Type': 'application/json',
        },
        body: JSON.stringify(body),
      },
      args.timeoutMs
    );

    const data = payload?.data ?? {};
    const content = data?.content;

    if (args.json) {
      console.log(JSON.stringify(payload, null, 2));
      return;
    }

    const out = stringifyContent(content);
    const isEmpty = out == null || String(out).trim() === '';
    if (isEmpty) {
      stopSpinner(spinnerId);
      process.stderr.write(
        `No content fetched from ${args.url}. The page may be empty, blocked, or the selector did not match.\n`
      );
      process.exit(1);
    }
    console.log(out);
  } finally {
    stopSpinner(spinnerId);
  }
}

main().catch((err) => {
  let url = '';
  const argv = process.argv.slice(2);
  const i = argv.findIndex((a) => a === '--url' || a === '-u');
  if (i >= 0 && argv[i + 1]) url = argv[i + 1];
  process.stderr.write(
    `Web fetch failed${url ? ` for ${url}` : ''}: ${err?.message || err}\n`
  );
  process.exit(1);
});


Security scan verdicts — ClawScan: clean | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)