Skill: justserpapi/justserpapi-web (version 1.0.1)

--- name: Web Crawling API description: Fetch raw HTML, rendered HTML, or clean Markdown from public webpages through Just Serp API. author: Just Serp API homepage: https://justserpapi.com/?utm_source=clawhub.ai&utm_medium=referral&utm_campaign=justserpapi_web&utm_content=project_link metadata: {"openclaw":{"homepage":"https://justserpapi.com/?utm_source=clawhub.ai&utm_medium=referral&utm_campaign=justserpapi_web&utm_content=project_link","primaryEnv":"JUST_SERP_API_KEY","requires":{"bins":["node"],"env":["JUST_SERP_API_KEY"]},"skillKey":"justserpapi_web"}} --- # Web Crawling Use this skill when the user needs webpage retrieval rather than a search engine result. It fits crawling, scraping preparation, readable content extraction, and page structure inspection for a known URL. ## When To Use It - The user already has a target webpage URL and wants its raw HTML, rendered HTML, or cleaned Markdown. - The task is about content extraction, page inspection, scrape preparation, or converting a page into LLM-friendly text. - The user can provide a direct `url` to crawl. - The user needs page content from the source URL itself, not Google search results about that URL. ## Representative Operations - `html`: Crawl Webpage (HTML) — Retrieve the raw HTML response for a page. - `renderedHtml`: Crawl Webpage (Rendered HTML) — Retrieve DOM output after rendering for JavaScript-heavy pages. - `markdown`: Crawl Webpage (Markdown) — Extract the main readable content as clean Markdown for summarization or downstream processing. ## Request Pattern - 3 read-only `GET` operations are available in this skill. - All operations require a direct `url` query parameter. - No operation in this skill requires a request body. - Choose `renderedHtml` for dynamic pages, `html` for raw source, and `markdown` for readable content extraction. ## How To Work 1. Read `generated/operations.md` before choosing an endpoint. 2. Start with one of these operations when it matches the user's request: `html`, `renderedHtml`, `markdown`. 3. Pick the smallest matching operation instead of guessing. 4. Ask the user for any missing required parameter. Do not invent values. 5. Call the helper with: ```bash node {baseDir}/bin/run.mjs --operation "<operation-id>" --api-key "$JUST_SERP_API_KEY" --params-json '{"key":"value"}' ``` ## Environment - Required: `JUST_SERP_API_KEY` - This skill uses `JUST_SERP_API_KEY` only for authenticated Just Serp API requests. - Keep `JUST_SERP_API_KEY` private. Do not paste it into chat messages, screenshots, or logs. - Project site: [Just Serp API](https://justserpapi.com/?utm_source=clawhub.ai&utm_medium=referral&utm_campaign=justserpapi_web&utm_content=project_link). - Authentication details: [Just Serp API Docs](https://docs.justserpapi.com/?utm_source=clawhub.ai&utm_medium=referral&utm_campaign=justserpapi_web&utm_content=project_link). ## Output Rules - Start with what was fetched: raw HTML, rendered HTML, or cleaned Markdown. - Echo the target URL so the crawl scope is explicit. - For `markdown`, surface the extracted readable content or key sections before raw JSON. - For HTML-oriented requests, mention whether the user asked for source HTML or rendered output. - If the backend errors, include the backend payload and the exact operation ID.

path: bin/run.mjs
content: #!/usr/bin/env node
const manifest = {
  "baseUrl": "https://api.justserpapi.com",
  "description": "Fetch raw HTML, rendered HTML, or clean Markdown from public webpages through Just Serp API.",
  "displayName": "Web Crawling",
  "openapi": "3.1.0",
  "platformKey": "web",
  "primaryTag": "Web Crawling",
  "skillName": "justserpapi_web",
  "slug": "justserpapi-web",
  "sourceTitle": "API Shop Documentation",
  "operations": [
    {
      "description": "Get webpage crawl data, including returns full raw HTML content, fast and cost-efficient, and optimized for static page crawling, for scraping, metadata extraction, and page structure analysis.",
      "method": "GET",
      "operationId": "html",
      "parameters": [
        {
          "defaultValue": null,
          "description": "The full URL of the webpage to crawl (e.g., 'https://www.example.com').",
          "enumValues": [],
          "location": "query",
          "name": "url",
          "required": true,
          "schemaType": "string"
        }
      ],
      "path": "/api/v1/web/html",
      "requestBody": null,
      "responses": [
        {
          "description": "Authentication failed: API Key is invalid or missing",
          "statusCode": "401"
        },
        {
          "description": "Access denied: Insufficient credits or quota exceeded",
          "statusCode": "403"
        },
        {
          "description": "Internal server error or upstream service exception",
          "statusCode": "500"
        },
        {
          "description": "default response",
          "statusCode": "default"
        }
      ],
      "summary": "Crawl Webpage (HTML)",
      "tags": [
        "Web Crawling"
      ]
    },
    {
      "description": "Get webpage crawl data, including removing boilerplate, for readable extraction, documentation workflows, and LLM input.",
      "method": "GET",
      "operationId": "markdown",
      "parameters": [
        {
          "defaultValue": null,
          "description": "The full URL of the webpage to crawl and convert to Markdown (e.g., 'https://www.example.com').",
          "enumValues": [],
          "location": "query",
          "name": "url",
          "required": true,
          "schemaType": "string"
        }
      ],
      "path": "/api/v1/web/markdown",
      "requestBody": null,
      "responses": [
        {
          "description": "Authentication failed: API Key is invalid or missing",
          "statusCode": "401"
        },
        {
          "description": "Access denied: Insufficient credits or quota exceeded",
          "statusCode": "403"
        },
        {
          "description": "Internal server error or upstream service exception",
          "statusCode": "500"
        },
        {
          "description": "default response",
          "statusCode": "default"
        }
      ],
      "summary": "Crawl Webpage (Markdown)",
      "tags": [
        "Web Crawling"
      ]
    },
    {
      "description": "Get webpage crawl data, including returns full raw Rendered HTML content, fast and cost-efficient, and optimized for static page crawling, for scraping, metadata extraction, and page structure analysis.",
      "method": "GET",
      "operationId": "renderedHtml",
      "parameters": [
        {
          "defaultValue": null,
          "description": "The full URL of the webpage to crawl (e.g., 'https://www.example.com').",
          "enumValues": [],
          "location": "query",
          "name": "url",
          "required": true,
          "schemaType": "string"
        }
      ],
      "path": "/api/v1/web/rendered-html",
      "requestBody": null,
      "responses": [
        {
          "description": "Authentication failed: API Key is invalid or missing",
          "statusCode": "401"
        },
        {
          "description": "Access denied: Insufficient credits or quota exceeded",
          "statusCode": "403"
        },
        {
          "description": "Internal server error or upstream service exception",
          "statusCode": "500"
        },
        {
          "description": "default response",
          "statusCode": "default"
        }
      ],
      "summary": "Crawl Webpage (Rendered HTML)",
      "tags": [
        "Web Crawling"
      ]
    }
  ]
};
const args = parseArgs(process.argv.slice(2));

if (!args.operation) {
  fail("Missing required --operation argument.");
}

const operation = manifest.operations.find((item) => item.operationId === args.operation);
if (!operation) {
  fail(`Unknown operation "${args.operation}".`, { availableOperations: manifest.operations.map((item) => item.operationId) });
}

if (!args.apiKey) {
  fail("Missing required --api-key argument.");
}

const params = parseParams(args.paramsJson);
applyDefaults(operation, params);
validateRequired(operation, params);

const baseUrl = manifest.baseUrl;
const url = new URL(operation.path, ensureBaseUrl(baseUrl));
applyPathParams(operation, params, url);
applyQueryParams(operation, params, url);

const requestInit = {
  headers: {
    "accept": "application/json",
    "X-API-Key": args.apiKey,
  },
  method: operation.method,
};

if (operation.requestBody && params.body !== undefined) {
  requestInit.body = JSON.stringify(params.body);
  requestInit.headers["content-type"] = operation.requestBody.contentType || "application/json";
}

let response;
try {
  response = await fetch(url, requestInit);
} catch (error) {
  fail("Network request failed.", {
    cause: error instanceof Error ? error.message : String(error),
    operationId: operation.operationId,
  });
}

const rawBody = await response.text();
let parsedBody;
try {
  parsedBody = rawBody ? JSON.parse(rawBody) : null;
} catch (error) {
  if (!response.ok) {
    fail("Backend returned a non-JSON error response.", {
      body: rawBody,
      operationId: operation.operationId,
      status: response.status,
      statusText: response.statusText,
    });
  }
  fail("Backend returned invalid JSON.", {
    body: rawBody,
    operationId: operation.operationId,
    status: response.status,
    statusText: response.statusText,
  });
}

if (!response.ok) {
  fail("Backend request failed.", {
    body: parsedBody,
    operationId: operation.operationId,
    status: response.status,
    statusText: response.statusText,
  });
}

process.stdout.write(`${JSON.stringify(parsedBody, null, 2)}\n`);

function parseArgs(argv) {
  const parsed = { apiKey: null, operation: null, paramsJson: "{}" };
  for (let index = 0; index < argv.length; index += 1) {
    const flag = argv[index];
    const value = argv[index + 1];
    if (flag === "--operation") {
      parsed.operation = value;
      index += 1;
      continue;
    }
    if (flag === "--params-json") {
      parsed.paramsJson = value;
      index += 1;
      continue;
    }
    if (flag === "--api-key") {
      parsed.apiKey = value;
      index += 1;
      continue;
    }
    fail(`Unknown argument "${flag}".`);
  }
  return parsed;
}

function parseParams(input) {
  try {
    const parsed = JSON.parse(input || "{}");
    if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
      fail("--params-json must decode to a JSON object.");
    }
    return parsed;
  } catch (error) {
    fail("Failed to parse --params-json.", {
      cause: error instanceof Error ? error.message : String(error),
    });
  }
}

function applyDefaults(operation, params) {
  for (const parameter of operation.parameters) {
    if (params[parameter.name] === undefined && parameter.defaultValue !== null) {
      params[parameter.name] = parameter.defaultValue;
    }
  }
}

function validateRequired(operation, params) {
  const missing = [];
  for (const parameter of operation.parameters) {
    if (parameter.required && params[parameter.name] === undefined) {
      missing.push(parameter.name);
    }
  }
  if (operation.requestBody?.required && params.body === undefined) {
    missing.push("body");
  }
  if (missing.length) {
    fail("Missing required parameters.", {
      missing,
      operationId: operation.operationId,
    });
  }
}

function applyPathParams(operation, params, url) {
  let pathname = url.pathname;
  for (const parameter of operation.parameters.filter((item) => item.location === "path")) {
    const value = params[parameter.name];
    if (value === undefined) {
      continue;
    }
    pathname = pathname.replace(`{${parameter.name}}`, encodeURIComponent(String(value)));
  }
  url.pathname = pathname;
}

function applyQueryParams(operation, params, url) {
  for (const parameter of operation.parameters.filter((item) => item.location === "query")) {
    const value = params[parameter.name];
    if (value === undefined) {
      continue;
    }
    appendValue(url.searchParams, parameter.name, value);
  }
}

function appendValue(searchParams, name, value) {
  if (Array.isArray(value)) {
    for (const item of value) {
      appendValue(searchParams, name, item);
    }
    return;
  }
  if (value && typeof value === "object") {
    searchParams.append(name, JSON.stringify(value));
    return;
  }
  searchParams.append(name, String(value));
}

function ensureBaseUrl(value) {
  return value.endsWith("/") ? value : `${value}/`;
}

function fail(message, details = null) {
  const payload = { message };
  if (details) {
    payload.details = details;
  }
  process.stderr.write(`${JSON.stringify(payload, null, 2)}\n`);
  process.exit(1);
}


path: generated/operations.json
content: {
  "baseUrl": "https://api.justserpapi.com",
  "description": "Fetch raw HTML, rendered HTML, or clean Markdown from public webpages through Just Serp API.",
  "displayName": "Web Crawling",
  "openapi": "3.1.0",
  "operations": [
    {
      "description": "Get webpage crawl data, including returns full raw HTML content, fast and cost-efficient, and optimized for static page crawling, for scraping, metadata extraction, and page structure analysis.",
      "method": "GET",
      "operationId": "html",
      "parameters": [
        {
          "defaultValue": null,
          "description": "The full URL of the webpage to crawl (e.g., 'https://www.example.com').",
          "enumValues": [],
          "location": "query",
          "name": "url",
          "required": true,
          "schemaType": "string"
        }
      ],
      "path": "/api/v1/web/html",
      "requestBody": null,
      "responses": [
        {
          "description": "Authentication failed: API Key is invalid or missing",
          "statusCode": "401"
        },
        {
          "description": "Access denied: Insufficient credits or quota exceeded",
          "statusCode": "403"
        },
        {
          "description": "Internal server error or upstream service exception",
          "statusCode": "500"
        },
        {
          "description": "default response",
          "statusCode": "default"
        }
      ],
      "summary": "Crawl Webpage (HTML)",
      "tags": [
        "Web Crawling"
      ]
    },
    {
      "description": "Get webpage crawl data, including removing boilerplate, for readable extraction, documentation workflows, and LLM input.",
      "method": "GET",
      "operationId": "markdown",
      "parameters": [
        {
          "defaultValue": null,
          "description": "The full URL of the webpage to crawl and convert to Markdown (e.g., 'https://www.example.com').",
          "enumValues": [],
          "location": "query",
          "name": "url",
          "required": true,
          "schemaType": "string"
        }
      ],
      "path": "/api/v1/web/markdown",
      "requestBody": null,
      "responses": [
        {
          "description": "Authentication failed: API Key is invalid or missing",
          "statusCode": "401"
        },
        {
          "description": "Access denied: Insufficient credits or quota exceeded",
          "statusCode": "403"
        },
        {
          "description": "Internal server error or upstream service exception",
          "statusCode": "500"
        },
        {
          "description": "default response",
          "statusCode": "default"
        }
      ],
      "summary": "Crawl Webpage (Markdown)",
      "tags": [
        "Web Crawling"
      ]
    },
    {
      "description": "Get webpage crawl data, including returns full raw Rendered HTML content, fast and cost-efficient, and optimized for static page crawling, for scraping, metadata extraction, and page structure analysis.",
      "method": "GET",
      "operationId": "renderedHtml",
      "parameters": [
        {
          "defaultValue": null,
          "description": "The full URL of the webpage to crawl (e.g., 'https://www.example.com').",
          "enumValues": [],
          "location": "query",
          "name": "url",
          "required": true,
          "schemaType": "string"
        }
      ],
      "path": "/api/v1/web/rendered-html",
      "requestBody": null,
      "responses": [
        {
          "description": "Authentication failed: API Key is invalid or missing",
          "statusCode": "401"
        },
        {
          "description": "Access denied: Insufficient credits or quota exceeded",
          "statusCode": "403"
        },
        {
          "description": "Internal server error or upstream service exception",
          "statusCode": "500"
        },
        {
          "description": "default response",
          "statusCode": "default"
        }
      ],
      "summary": "Crawl Webpage (Rendered HTML)",
      "tags": [
        "Web Crawling"
      ]
    }
  ],
  "platformKey": "web",
  "primaryTag": "Web Crawling",
  "skillName": "justserpapi_web",
  "slug": "justserpapi-web",
  "sourceTitle": "API Shop Documentation"
}


path: generated/operations.md
content: # Web Crawling operations

Generated from Just Serp API OpenAPI for group key `web`.

## `html`

- Method: `GET`
- Path: `/api/v1/web/html`
- Summary: Crawl Webpage (HTML)
- Description: Get webpage crawl data, including returns full raw HTML content, fast and cost-efficient, and optimized for static page crawling, for scraping, metadata extraction, and page structure analysis.
- Tags: `Web Crawling`

### Parameters

| Name | In | Required | Type | Default | Description |
| --- | --- | --- | --- | --- | --- |
| `url` | `query` | yes | `string` | n/a | The full URL of the webpage to crawl (e.g., 'https://www.example.com'). |

### Request body

No request body.

### Responses

- `401`: Authentication failed: API Key is invalid or missing
- `403`: Access denied: Insufficient credits or quota exceeded
- `500`: Internal server error or upstream service exception
- `default`: default response

## `markdown`

- Method: `GET`
- Path: `/api/v1/web/markdown`
- Summary: Crawl Webpage (Markdown)
- Description: Get webpage crawl data, including removing boilerplate, for readable extraction, documentation workflows, and LLM input.
- Tags: `Web Crawling`

### Parameters

| Name | In | Required | Type | Default | Description |
| --- | --- | --- | --- | --- | --- |
| `url` | `query` | yes | `string` | n/a | The full URL of the webpage to crawl and convert to Markdown (e.g., 'https://www.example.com'). |

### Request body

No request body.

### Responses

- `401`: Authentication failed: API Key is invalid or missing
- `403`: Access denied: Insufficient credits or quota exceeded
- `500`: Internal server error or upstream service exception
- `default`: default response

## `renderedHtml`

- Method: `GET`
- Path: `/api/v1/web/rendered-html`
- Summary: Crawl Webpage (Rendered HTML)
- Description: Get webpage crawl data, including returns full raw Rendered HTML content, fast and cost-efficient, and optimized for static page crawling, for scraping, metadata extraction, and page structure analysis.
- Tags: `Web Crawling`

### Parameters

| Name | In | Required | Type | Default | Description |
| --- | --- | --- | --- | --- | --- |
| `url` | `query` | yes | `string` | n/a | The full URL of the webpage to crawl (e.g., 'https://www.example.com'). |

### Request body

No request body.

### Responses

- `401`: Authentication failed: API Key is invalid or missing
- `403`: Access denied: Insufficient credits or quota exceeded
- `500`: Internal server error or upstream service exception
- `default`: default response


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)