npm package: @se-studio/site-check
Description: Validate SE marketing sites (sitemap, llms.txt) and download markdown files preserving structure
Latest version: 1.0.32

# @se-studio/site-check

Validate SE marketing sites (sitemap.xml, sitemap-unindexed.xml, llms.txt) and download all markdown files into a local directory, preserving URL path structure.

## Usage

```bash
npx @se-studio/site-check <baseUrl> [-o dir] [-H "Name: value"] [--vercel-bypass [secret]] [--check-against <devUrl>] [--check-rewrites] [--ignore-missing-child-sitemaps]
```

### Examples

```bash
# Local site
npx @se-studio/site-check http://localhost:3015

# Custom output directory
npx @se-studio/site-check http://localhost:3015 -o ./out

# Compare production vs development: use production sitemap, check pages exist on dev
npx @se-studio/site-check https://example.com --check-against http://localhost:3010

# Enable rewrite check (optional; off by default)
npx @se-studio/site-check https://example.com --check-rewrites

# Vercel Deployment Protection: use env secret
VERCEL_AUTOMATION_BYPASS_SECRET=your-secret npx @se-studio/site-check https://preview.vercel.app --vercel-bypass

# Vercel bypass with explicit secret
npx @se-studio/site-check https://preview.vercel.app --vercel-bypass your-secret

# Custom headers
npx @se-studio/site-check https://example.com -H "Authorization: Bearer token" -H "X-Custom: value"

# Sitemap index lists a child that does not exist locally (404) — skip it and use remaining children
npx @se-studio/site-check http://localhost:3016 --ignore-missing-child-sitemaps -o ./out
```

## What it does

1. **Validates** the site by fetching:
   - `sitemap.xml` (required) — must return 200 and contain `<loc>` entries; if it is a sitemap index, child sitemaps are fetched and must return valid urlset content (or use `--ignore-missing-child-sitemaps` to skip child URLs that return **404**)
   - `sitemap-unindexed.xml` (optional) — if 404, a warning is printed and the run continues. When it exists (returns 200), its page URLs are included in all checks (existence on dev, optional rewrite check, markdown collection).
   - `llms.txt` (optional) — if 404, a warning is printed and the run continues; if it returns 200 it is validated

2. **Collects markdown URLs** from **sitemap.xml** and **sitemap-unindexed.xml** (Option B): when either URL returns a **sitemap index** (e.g. `<sitemapindex>` with child `<sitemap><loc>...</loc></sitemap>`), the tool follows those links and collects page URLs from each child sitemap’s urlset (one level only). When the response is a normal urlset, every `<loc>` is treated as a page URL. Each page URL is then converted to a `.md` URL (path + `.md`). This ensures all indexed and unindexed pages are checked.

3. **Checks for unexpected rewrites** (only when `--check-rewrites` is set): For each page URL from the sitemaps, the tool sends a HEAD request and reads the `x-nextjs-rewritten-path` response header. If the header is present and the rewritten path is different from the requested path (after normalisation), the run **fails** (exit 1). This catches cases where the sitemap lists one URL (e.g. `/learning-hub/blog/`) but the app rewrites to another (e.g. `/articles/blog`). Rewrites that only add or remove a locale prefix are allowed when `SITE_CHECK_LOCALES` is set (see [Options](#options)). By default the rewrite check is **disabled**.

4. **Checks and downloads**: For each markdown URL, the tool fetches it. If any return non-2xx, the run **fails** (exit 1) and reports which URLs are missing or errored. Otherwise it saves each response to the output directory, preserving path structure (e.g. `blog/foo.md` → `./markdown-export/blog/foo.md`, `es-US/about.md` → `./markdown-export/es-US/about.md`).

### Compare mode (`--check-against`)

When `--check-against <devUrl>` is set, the tool runs in **compare mode**:

- **Production** (the `<baseUrl>`) is the source of the URL list: sitemap.xml and sitemap-unindexed.xml are fetched from production. Validation (and llms.txt) applies to production.
- **Development** (`<devUrl>`) is the site checked: each production sitemap page URL (including from sitemap-unindexed.xml when present) is mapped to the same path on the development origin. The tool then checks that each of those development URLs returns 2xx (HEAD). If `--check-rewrites` is set, it also runs the rewrite check against the development site. Markdown collection and download are **not** performed in compare mode.

Use this to ensure a development or staging build has the same pages as production (e.g. before release). Page title comparison is not yet implemented; a future option may add it.

## Options

| Option | Description |
|--------|-------------|
| `baseUrl` | Base URL of the site (required). In compare mode this is the **production** URL (sitemap source). Trailing slash is stripped. |
| `-o`, `--out <dir>` | Output directory for markdown files (default: `./markdown-export`). Ignored in compare mode. |
| `-H`, `--header "Name: value"` | Add a request header (repeatable). Applied to both production and development requests in compare mode. |
| `--check-against <devUrl>` | Compare mode: use production sitemap(s) but check that each page exists on development (same path on `<devUrl>`). No markdown download. |
| `--check-rewrites` | Enable checking for unexpected rewrites (x-nextjs-rewritten-path). Default: off. When set, the run fails if any sitemap page URL rewrites to a different path (locale-only rewrites allowed with `SITE_CHECK_LOCALES`). |
| `--ignore-missing-child-sitemaps` | When the root sitemap (or `sitemap-unindexed.xml`) is a **sitemap index**, any child `<loc>` that returns **404** is skipped instead of failing validation. Other non-2xx responses still fail. After skipping, at least one page URL must still be collected from the remaining children. |
| `--vercel-bypass [secret]` | Set `x-vercel-protection-bypass` for [Vercel Deployment Protection](https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/protection-bypass-automation). If `secret` is omitted, uses `VERCEL_AUTOMATION_BYPASS_SECRET` from the environment. |

Headers from `-H` override the Vercel bypass header if the same name is used.

### Environment

| Variable | Description |
|----------|-------------|
| `SITE_CHECK_LOCALES` | Comma-separated list of locale path segments (e.g. `en,en-gb,de`). When set, a rewrite is considered acceptable if the only difference between the requested path and the rewritten path is a leading locale segment. If unset, any rewrite to a different path fails. |

## Exit codes

- `0` — Validation passed (sitemap.xml required; llms.txt and sitemap-unindexed.xml optional). If `--check-rewrites` was set, no unexpected rewrites. Every sitemap page returned 200 for its `.md` URL and files were saved (or in compare mode: all production sitemap pages exist on development and, if `--check-rewrites`, no unexpected rewrites).
- `1` — Usage error (missing baseUrl), validation failed (sitemap.xml), one or more sitemap pages rewrite to a different path when `--check-rewrites` was set, one or more markdown URLs returned non-2xx (pages missing or broken), or in compare mode one or more development URLs did not return 2xx. Validation does not fail when llms.txt returns 404.