npm package: @snaf/butt-rag
Description: fetch, filter, and compact documentation into your local commode
Author: @snaf
Latest version: 0.1.2

# 🧻 butt-rag

**Bundler** for **Unified** **Text** **Transform**, **Retrieval**-**Augmented** **Generation**.

A CLI that fetches documentation from git repos and URLs, filters it, optionally compacts it, and deposits it in a local `.commode/` directory for your AI tools to consume. Think of it as plumbing for your project's docs.

Also works as a declarative docs manifest. Instead of scattering links across READMEs or bookmarking docs you'll lose in a week, declare all your dependency docs in one `.turd.yaml` and pull them down with a single command. Sometimes you just want local docs you can `cmd+f` through without fighting a search bar that thinks it knows better. The RAG stuff is a bonus, not a prerequisite.

## 📦 Install

```bash
pnpm add -g @snaf/butt-rag
```

Or, for the commitment-averse:

```bash
pnpx @snaf/butt-rag
```

## 🚽 Usage

Drop a `docs.turd.yaml` (**Tree** for **Unified** **RAG** **Dump**) in your project root:

```yaml
compact: false # set true to also produce a single load.md with everything

defaults:
  compact: false # per-source default: squash files into one .md
  wipe: false # per-source default: delete raw files after compaction
  bidet: false # per-source default: delete compact file after top-level compaction

sources:
  tailwind:
    type: git
    url: https://github.com/tailwindlabs/tailwindcss.com.git
    subDirectories: ['src/docs']
    extensions: ['.mdx']
    compact: true
    wipe: true # only keep the compacted file

  svelte:
    type: txt
    url: https://svelte.dev/llms-medium.txt
```

Then let 'er rip:

```bash
butt-rag
```

Your docs land in `.commode/`. You're welcome.

## 🪠 The Pipeline

Every good system needs a flow.

1. **Fetch** each source (sparse git clone or HTTP fetch)
2. **Filter** git sources by subdirectory and file extension
3. **Compact** per-source if configured (concatenate with file markers)
4. **Wipe** raw intermediates if configured
5. **Compact all** into `load.md` if top-level compact is on
6. **Bidet** per-source compact files if configured

## 🗺️ What Goes Where

| scenario                     | output                                                         |
| ---------------------------- | -------------------------------------------------------------- |
| git source, `compact: false` | `.commode/<name>/` (preserves dir structure)                   |
| git source, `compact: true`  | `.commode/<name>.md` (one file, file markers between sections) |
| txt source                   | `.commode/<name>.txt`                                          |
| top-level `compact: true`    | `.commode/load.md` (the whole load in one file)                |

## 🔧 CLI Options

```
butt-rag                             # run with ./docs.turd.yaml
butt-rag --config-file my.turd.yaml  # use a different config
butt-rag --dry-run                   # print the plan without touching disk (sounds painful, isn't)
butt-rag --flush                     # nuke the commode
```

## 📋 Config Reference

### Top-level

- `compact` (boolean, default `false`): produce a combined `load.md` from all sources
- `defaults` (object): default values for per-source flags

### Defaults

- `compact` (boolean, default `false`): squash source files into a single `.md`
- `wipe` (boolean, default `false`): delete raw files after per-source compaction
- `bidet` (boolean, default `false`): remove per-source compactions after top-level compaction

### Source (git)

- `type`: `git`
- `url`: git clone URL
- `subDirectories` (string[]): only checkout these paths (uses sparse checkout)
- `extensions` (string[]): only keep files with these extensions
- `skip` (boolean, default `false`): skip this source without commenting it out
- `compact`, `wipe`, `bidet`: override defaults

### Source (txt)

- `type`: `txt`
- `url`: URL to fetch (typically an `llms.txt` endpoint)
- `skip` (boolean, default `false`): skip this source without commenting it out
- `compact`, `wipe`, `bidet`: no-ops for txt sources (already single files, nothing to squeeze)

## 💩 Glossary

Because you can't spell "documentation" without... actually, you can. Anyway.

- **TURD** 📜 Tree for Unified RAG Dump. Your config file. Treat it with respect.
- **commode** 🚽 The `.commode/` output directory. Where processed docs end up. Classy name for a classy destination.
- **compact** 🧱 Squash multiple files into one. Fiber for your docs.
- **flush** 🌊 Nuclear option. Delete everything in the commode. Start fresh. No debris left behind.
- **wipe** 🧻 Clean up raw files after compaction. Basic hygiene.
- **bidet** 💦 Clean up per-source compact files after top-level compaction. _Fancy_ hygiene.
- **load** 📦 The final combined output file. The whole payload.
- **skip** ⏭️ Leave a source in your config but don't process it. For when you're not ready to commit.