npm package: @pointsharp/antora-llm-generator
Description: An Antora extension to generate llms.txt files for LLM consumption following llmstxt.org specification.
Author: Pointsharp AB
Latest version: 1.2.0

# Antora LLM Generator Extension

An [Antora](https://antora.org) extension that creates an auxiliary text file after each site build following the [llmstxt.org specification](https://llmstxt.org/):

- **`llms.txt`** - A structured index with links to all documentation pages, organized into sections

The extension also generates compatibility and discovery files for crawlers and tools:

- **`llm.txt`** - Same content as `llms.txt`
- **`sitemap-llms.xml`** - Sitemap containing the `llms.txt` URL
- Updates the main `sitemap.xml` sitemap index to include `sitemap-llms.xml`

These files help large-language models ingest your documentation with proper structure, URLs, and context.

---

## Installation

```bash
npm install @pointsharp/antora-llm-generator
```

---

## Playbook configuration

Add the extension to your `antora-playbook.yml`:

```yaml
antora:
  extensions:
    - require: "@pointsharp/antora-llm-generator"
      summary: "Brief summary about your documentation site"
      details: |
        Optional longer description or important notes.
        Can be multi-line markdown text.
      skippaths:
        - "someGlob/**/path"
```

### Configuration options

- **`summary`** - Optional. Appears as a blockquote at the top of `llms.txt` (following llmstxt.org spec).
- **`details`** - Optional. Appears as regular text after the summary. Can be multi-line markdown.
- **`skippaths`** - Optional. Array of glob patterns. Files matching these patterns are omitted from `llms.txt`.
- **`debug`** - Optional. Set to `true` to enable verbose logging during build. Default: `false`.

### Navigation-based organization (default)

By default, the extension uses your Antora navigation structure from `nav.adoc` files to organize pages in `llms.txt`. This works automatically with:

- **Native Antora navigation** - Always works
- **Navigator extension** - If you use it, the same navigation structure is used
- **Any custom navigation** - As long as it's in your `nav.adoc` files

Pages are organized by component titles as H2 sections, and the navigation hierarchy is preserved.

**Example output:**

```markdown
# Your Site Title

> Optional summary

## Product A

- [What is Product A?](https://example.com/product-a/latest/index.html)
- **Installation**
  - [Install on Windows](https://example.com/product-a/latest/installation-windows.html)
  - [Install on Linux](https://example.com/product-a/latest/installation-linux.html)

## Product B

- [Introduction to Product B](https://example.com/product-b/index.html)
```

---

## Page-level attributes

You can control how individual pages are included using AsciiDoc page attributes:

```adoc
:page-llms-ignore: true
:description: Brief description of this page
```

- **`:page-llms-ignore:`** - Omit this page from `llms.txt` entirely
- **`:description:`** - Standard AsciiDoc attribute used for both HTML metadata and page descriptions in `llms.txt`

### Page descriptions

The extension uses the standard `:description:` attribute to add optional descriptions to page links in `llms.txt`, following the [llmstxt.org specification](https://llmstxt.org/):

**Without description:**
```markdown
- [Getting Started](https://example.com/getting-started)
```

**With description:**
```markdown  
- [Getting Started](https://example.com/getting-started): Quick start guide for new users
```

**Example:**

```adoc
= System Requirements
:description: This section describes the system requirements for Product A.
```

The `:description:` attribute is a standard AsciiDoc attribute that serves dual purposes:
- **HTML metadata** - Used in `<meta name="description">` tags for SEO
- **LLM files** - Provides context for each page link in `llms.txt`

The extension automatically resolves any attribute references within the description using the standard `{attribute}` syntax, so you can compose descriptions from other attributes if needed.

---

## Output structure

### llms.txt structure

Following the [llmstxt.org specification](https://llmstxt.org/), the file contains:

```markdown
# Your Site Title

> Optional summary from playbook config

Optional details from playbook config

## Section Name

- [Page Title](https://url): Optional description
- [Another Page](https://url)

## Another Section

- [Page Title](https://url): Description
```

### Sitemap integration

The extension also adds `llms.txt` to sitemap discovery without pretending it is an Antora page:

- Generates `sitemap-llms.xml` with the `llms.txt` URL
- Updates the main `sitemap.xml` sitemap index to reference `sitemap-llms.xml`

This matches Antora best practice by keeping `llms.txt` in the site catalog instead of inserting it into the content catalog as a fake page.

---

## Building the site

Run your Antora build as usual:

```bash
antora antora-playbook.yml
```

After completion, these files appear in the build output directory:

- `llms.txt` and `llm.txt` (both contain the structured index with links)
- `sitemap-llms.xml` (contains the `llms.txt` URL)
- `sitemap.xml` is updated to include `sitemap-llms.xml`

The duplicate `llms.txt` and `llm.txt` filenames ensure compatibility with different naming conventions.

---

## Logging

The extension runs **silently** by default, showing only a success message when complete:

```
Generated llms.txt
```

### Enabling verbose output

To see detailed processing information (pages collected, components processed, etc.), enable the `debug` flag in your playbook:

```yaml
antora:
  extensions:
    - require: "@pointsharp/antora-llm-generator"
      debug: true
      summary: "Your summary"
```

With `debug: true`, you'll see:
- Navigation data source detection
- Number of pages in the page map
- Components and sections being processed
- Item counts for each section

### Error handling

The extension always reports errors and critical warnings, regardless of the debug setting:
- **Errors** - Processing failures or file write errors
- **Warnings** - Navigator data parsing issues

---

## Example configuration

Here's a complete example following the [llmstxt.org](https://llmstxt.org/) best practices:

```yaml
antora:
  extensions:
    - require: "@pointsharp/antora-llm-generator"
      summary: "Comprehensive documentation for the Acme API, including authentication, endpoints, and best practices."
      details: |
        Important notes:
        
        - All API endpoints require authentication via API key
        - Rate limits apply to all endpoints (1000 requests/hour)
        - WebSocket connections are available for real-time updates
      skippaths:
        - "admin/**"
        - "**/internal/**"
```

**This produces:** The summary appears as a blockquote and details as regular text at the top of `llms.txt`, providing important context for LLMs before they see the documentation links.

---

## Tips for effective llms.txt files

Following [llmstxt.org guidelines](https://llmstxt.org/):

- **Use concise, clear language** in summaries and descriptions
- **Include brief, informative descriptions** for important pages
- **Avoid ambiguous terms** or unexplained jargon
- **Use the "Optional" section** for secondary information that can be skipped for shorter context
- **Organize related pages** into logical sections
- **Test with actual LLMs** to ensure they can answer questions about your content

---

## Compatibility

This extension follows the official [llmstxt.org specification](https://llmstxt.org/) for maximum compatibility with LLM tools and agents.

For more details about the specification, visit [https://llmstxt.org/](https://llmstxt.org/).

---

## Copyright and License

Copyright © 2006-present Pointsharp AB.

Use of this software is granted under the terms of the Apache License Version 2.0 (Apache-2.0).  
See the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) for the full license text.