Skill: multicul-silver-wolf/sawana-multicul-official-docs-to-mdx (version 1.0.0)

--- name: official-docs-to-mdx description: Download and normalize official documentation pages into local .mdx files at user-specified paths using markdown.new. Use when user asks to fetch docs from a URL, archive official docs locally, sync docs snapshots, or convert docs pages into clean MDX with frontmatter (title, description, sourceUrl, retrievedAt). --- # Official Docs to MDX Convert a docs URL into a cleaned local `.mdx` file at a path the user specifies. Use this skill when the user wants reproducible, file-based docs snapshots for local knowledge bases, RAG prep, or project docs folders. ## Quick Start Run: ```bash scripts/mdnew_to_mdx.sh <source_url> <output_mdx_path> ``` Example: ```bash scripts/mdnew_to_mdx.sh \ https://docs.convex.dev/agents/agent-usage \ docs/convex/agent-usage.mdx ``` ## Workflow 1. Validate inputs. - Require exactly 2 args: source URL and output path. - Keep URL as full `https://...` form. 2. Convert with script. - Execute `scripts/mdnew_to_mdx.sh`. 3. Confirm output. - Verify file exists. - Print or inspect frontmatter for `title`, `description`, `sourceUrl`, `retrievedAt`. 4. Add indexes for folder-based docs (required). - Any docs folder must include `index.mdx`. - If `docs/` exists, `docs/index.mdx` must exist. - Index files should link child docs/subfolders. - Keep frontmatter: `title`, `description`, `sourceUrl` (`local://...` allowed), `retrievedAt`. 5. Report result. - Return output path(s), overwrite status, and any created/updated index files. ## Output Contract The generated MDX must include frontmatter: - `title` - `description` - `sourceUrl` - `retrievedAt` (UTC `YYYY-MM-DD`) Body is cleaned to remove common docs-site artifacts and excess blank lines. ## Script Details For behavior details and cleanup rules, read: - `references/mdnew_to_mdx.md` Use the bundled script directly: - `scripts/mdnew_to_mdx.sh` ## Notes - Dependencies: `bash`, `curl`, `awk`, `mktemp`. - Output file is overwritten if it already exists. - Re-run anytime to refresh docs snapshot from upstream.

path: references/mdnew_to_mdx.md
content: ---
title: mdnew_to_mdx.sh
description: Convert docs pages into cleaned MDX via markdown.new.
updatedAt: 2026-02-25
---

# `mdnew_to_mdx.sh`

Converts a webpage into local `.mdx` using Cloudflare `markdown.new`, then cleans common docs-site UI artifacts.

## Location

- Script path: `scripts/mdnew_to_mdx.sh`

## Usage

```bash
scripts/mdnew_to_mdx.sh <source_url> <output_mdx_path>
```

Example:

```bash
scripts/mdnew_to_mdx.sh \
  https://docs.convex.dev/agents/agent-usage \
  docs/convex/agent-usage.mdx
```

## What it does

1. Fetches `https://markdown.new/<source_url>`.
2. Extracts only the `Markdown Content` section.
3. Reads source `title` and `description` from upstream frontmatter (if present).
4. Writes normalized frontmatter:
   - `title`
   - `description`
   - `sourceUrl`
   - `retrievedAt` (UTC date, `YYYY-MM-DD`)
5. Removes common artifacts:
   - "Skip to main content"
   - docs nav/header copy text
   - auto-generated on-page anchor list items
   - Docusaurus direct-link anchor fragments inside headings
6. Collapses excessive blank lines.

## Notes

- Requires: `bash`, `curl`, `awk`, `mktemp`.
- Overwrites output file if it already exists.
- Rerun the script to refresh local MDX when source docs change.


path: scripts/mdnew_to_mdx.sh
content: #!/usr/bin/env bash
set -euo pipefail

if [ "$#" -ne 2 ]; then
  echo "Usage: $0 <source_url> <output_mdx_path>" >&2
  exit 1
fi

source_url="$1"
out_file="$2"
retrieved_at="$(date -u +%F)"

raw_file="$(mktemp)"
body_file="$(mktemp)"
content_file="$(mktemp)"
clean_file="$(mktemp)"
trap 'rm -f "$raw_file" "$body_file" "$content_file" "$clean_file"' EXIT

curl -sS -D - "https://markdown.new/${source_url}" > "$raw_file"

awk 'BEGIN{p=0} /^Markdown Content:/{p=1; next} p{print}' "$raw_file" > "$body_file"

src_title=""
src_description=""
if awk 'NR==1{exit ($0=="---" ? 0 : 1)}' "$body_file"; then
  src_title="$(awk 'BEGIN{inside=0} /^---$/{if(inside==0){inside=1; next}else{exit}} inside && /^title:[[:space:]]/{sub(/^title:[[:space:]]*/, ""); print; exit}' "$body_file")"
  src_description="$(awk 'BEGIN{inside=0} /^---$/{if(inside==0){inside=1; next}else{exit}} inside && /^description:[[:space:]]/{sub(/^description:[[:space:]]*/, ""); print; exit}' "$body_file")"
fi

if [ -z "$src_title" ]; then
  src_title="Official Doc"
fi
if [ -z "$src_description" ]; then
  src_description="Converted from docs via markdown.new"
fi

awk '
BEGIN { in_fm=0 }
NR==1 && $0=="---" { in_fm=1; next }
in_fm && $0=="---" { in_fm=0; next }
in_fm { next }
{
  if ($0 ~ /^\[Skip to main content\]/) next
  if ($0 ~ /^\[\]\(https:\/\/[^)]*\)\[Docs\]\(\/home\)$/) next
  if ($0 ~ /^Copy as Markdown$/) next
  if ($0 ~ /^Copied!$/) next
  if ($0 ~ /^On this page$/) next
  if ($0 ~ /^[[:space:]]*\* \[[^]]+\]\(#[^)]+\)[[:space:]]*$/) next
  line=$0
  gsub(/\[​\]\(#[^)]+ "Direct link to [^"]+"\)/, "", line)
  print line
}
' "$body_file" > "$content_file"

awk '
{ if ($0 ~ /^[[:space:]]*$/) { blank++; if (blank <= 1) print ""; next } blank=0; print }
' "$content_file" > "$clean_file"

mkdir -p "$(dirname "$out_file")"
{
  echo "---"
  echo "title: ${src_title}"
  echo "description: ${src_description}"
  echo "sourceUrl: ${source_url}"
  echo "retrievedAt: ${retrieved_at}"
  echo "---"
  echo
  cat "$clean_file"
} > "$out_file"

echo "Wrote ${out_file}"


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: suspicious | SkillSpector: suspicious (MEDIUM)