Skill: chdlc/web-to-md (version 1.0.0)

--- name: web-to-md description: "Extracts readable markdown from user-provided URLs via a deterministic fallback chain (markdown.new → r.jina.ai). Use when the user supplies specific URLs and wants reliable extraction, summarization, or analysis." version: 1.0.0 author: chdlc license: MIT-0 metadata: openclaw: requires: bins: ["curl"] hermes: tags: [web, extraction, markdown, url, content] related_skills: [use-tinyfish, browser-automation] category: utility --- # Web to Markdown Deterministic, console-first extraction workflow for user-provided URLs. Enforces a fixed fallback chain to maximize content quality without open-ended browsing. ## When to Use - The user provides one or more **specific URLs**. - The task requires reading, extracting, summarizing, or analyzing those URLs. - A deterministic fallback order is preferred over open-ended browsing. **Do not use** for open-ended web discovery unless the user explicitly asks for discovery first. ## Fallback Chain For each URL, attempt in order. Stop at the first sufficient result. ### 1. markdown.new (AI mode) ```bash curl -s "https://markdown.new/{URL}?method=ai" ``` ### 2. markdown.new (Auto mode) Only if step 1 is insufficient or timed out: ```bash curl -s "https://markdown.new/{URL}?method=auto" ``` ### 3. r.jina.ai (Browser engine) Only if steps 1–2 are insufficient or timed out: ```bash curl -s "https://r.jina.ai/{URL}" -H "X-Engine: browser" ``` ### 4. Agent tools (last resort) If all three prefixes fail, report the failure and fall back to the agent's own extraction tools. This is outside the skill's chain — acknowledge it as a fallback. ## Quality Gate After each step, content is **insufficient** when any condition is true: - Main article or body text is missing - Content is clearly truncated - Output is mostly navigation, boilerplate, placeholders, or login walls - Useful text is too short for the task - Important sections requested by the user are absent **Rule of thumb:** Under ~1,200 useful characters for an article page is almost certainly truncated. Naturally short pages (announcements, status updates) may be legitimately brief — use judgment. ## URL Handling - Preserve the protocol when present. - Ensure the URL is shell-safe and **quoted** in all curl commands. - Process each URL independently when multiple are provided. ## Provenance Reporting Report exactly one final source label per extracted URL in your response: | Label | When | |---|---| | `markdown.new:ai` | method=ai was sufficient | | `markdown.new:auto` | method=auto was sufficient (ai failed) | | `r.jina.ai` | r.jina.ai was sufficient (both markdown.new failed) | | `agent-tools` | All three prefixes failed; agent used own tools | ## Workflow 1. **Scope gate** — Only process URLs explicitly provided by the user. If discovery is needed, use web search first and confirm candidate URLs before extraction. 2. **Normalize** — Quote URLs, preserve protocol. 3. **Extract** — Run the fallback chain per URL. 4. **Quality gate** — Check each result against the insufficiency conditions. 5. **Continue** — Use the richest sufficient source for the task. 6. **Report** — Include provenance labels in the final response. ## Best Practices - Keep extraction deterministic — explicit fallback transitions, state why each happened. - Prefer reproducible commands with quoted URLs. - Conservative timeout handling: continue immediately to the next fallback when blocked. - Preserve source traceability via provenance labels. - Avoid tool-specific assumptions beyond curl and standard HTTP endpoints. ## Edge Cases - **Page blocks automated access:** Skip to next fallback immediately. - **Multiple URLs:** Apply the same sequence to each independently. - **Naturally short pages:** Accept shorter content when it satisfies the request. - **All prefixes fail:** Report failure clearly, then use agent tools as last resort. ## Common Pitfalls 1. **Output format must be markdown.** If any level returns raw HTML or another format, it breaks the contract. Test each level independently. 2. **Don't skip testing lower fallback levels** just because the top level works. A chain is only as reliable as its weakest link. 3. **Quality is subjective** — the 1,200-char heuristic is a guideline, not a hard rule. Apply judgment for short-form content. ## Verification Checklist - [ ] curl is installed (`which curl`) - [ ] Extraction starts with `markdown.new?method=ai` - [ ] `method=auto` is tried only after ai fails - [ ] `r.jina.ai` is tried only after both markdown.new attempts fail - [ ] All three prefixes failing → report + fall back to agent tools - [ ] Quality checks include: missing body, truncation, boilerplate, too-short content - [ ] Final response includes provenance label per URL


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)