npm package: @markwharton/flatten
Description: Simple codebase flattening for LLMs
Author: Mark Wharton
Latest version: 1.5.0

# @markwharton/flatten

[![npm version](https://img.shields.io/npm/v/@markwharton/flatten.svg)](https://www.npmjs.com/package/@markwharton/flatten)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Node.js Version](https://img.shields.io/node/v/@markwharton/flatten.svg)](https://nodejs.org)

Simple, intelligent codebase flattening for LLMs. Creates a structured YAML representation of your repository and a single text file containing all your code - perfect for providing context to AI tools.

## Installation

```bash
# Install globally
npm install -g @markwharton/flatten

# Or use directly with npx
npx @markwharton/flatten
```

## Usage

### Basic Flattening

```bash
# Basic usage - creates timestamped output files
flatten

# Process specific directories
flatten src/ lib/

# Include specific files (even if normally ignored)
flatten src/ README.md package.json

# Exclude patterns
flatten --exclude "*.png,*.jpg,infra/sql"

# Structure only (no file contents)
flatten --no-contents

# Allow non-git directories
flatten --no-git

# Override file limit
flatten --max-files 5000

# Custom output file (single file, format from extension)
flatten --output context.txt
flatten -o structure.yaml

# Show version
flatten --version

# Show help
flatten --help
```

### Cleanup Old Files

```bash
# Show what would be removed (dry run)
flatten cleanup --dry-run

# Remove all flatten output files
flatten cleanup

# Remove files older than 7 days
flatten cleanup --days 7

# Get help on cleanup options
flatten cleanup --help
```

## Output Files

By default, flatten creates two files with smart, timestamped naming:

- `flatten-YYYYMMDD-HHMMSS-<git-hash>-<status>.yaml` - YAML structure of your repository
- `flatten-YYYYMMDD-HHMMSS-<git-hash>-<status>.txt` - Flattened content of all files

Where:
- `YYYYMMDD-HHMMSS` is the timestamp when flatten was run
- `<git-hash>` is the short git commit hash (7 characters)
- `<status>` is either 'clean' or 'dirty' based on git status

For non-git directories (with `--no-git`):
- `flatten-YYYYMMDD-HHMMSS.yaml` - YAML structure
- `flatten-YYYYMMDD-HHMMSS.txt` - Flattened content

### Custom Output

Use `--output` (or `-o`) to specify a custom filename and create a single file:

```bash
# Create a single text file with flattened content
flatten --output context.txt

# Create a single YAML file with structure only
flatten -o structure.yaml
```

The format is inferred from the extension:
- `.yaml` or `.yml` → YAML structure
- Any other extension → Flattened text content

Parent directories are created automatically if needed:
```bash
flatten --output output/dir/context.txt
```

### Example Output

In a git repository:
```
flatten-20250610-121900-abc1234-clean.yaml    # Repository structure
flatten-20250610-121900-abc1234-clean.txt     # Flattened content
```

After making changes:
```
flatten-20250610-143200-abc1234-dirty.yaml    # Shows uncommitted changes exist
flatten-20250610-143200-abc1234-dirty.txt     # Includes uncommitted changes
```

With exclusions:
```bash
$ flatten --exclude "*.png,infra/sql"

✓ Created flatten-20250610-151230-abc1234-clean.yaml
✓ Created flatten-20250610-151230-abc1234-clean.txt (487.2 KB)
✓ Excluded 23 files matching patterns
✓ Omitted 3 binary files from content

Largest files:
  src/runtime/services/database/mssql-database.ts (31.4 KB)
  tools/generators/schema-generator.ts (28.7 KB)
  docs/architecture/component-model.md (24.3 KB)
  src/runtime/components/extractors/html-extractor.ts (19.8 KB)
  tests/integration/full-pipeline.test.ts (17.2 KB)
```

### Ignoring Flatten Output Files

To prevent flatten output files from being committed to your repository, add these patterns to your `.gitignore`:

```gitignore
# Flatten output files
flatten-*.txt
flatten-*.yaml
```

This will ignore all flatten-generated files while keeping your repository clean.

## Managing Output Files

The new timestamped naming makes it easy to manage multiple flatten outputs:

```bash
# List all flatten outputs (sorted by timestamp)
$ ls -l flatten-*.{yaml,txt}
-rw-r--r--  1 user  staff    869 Jun  8 12:29 flatten-20250608-122900-9e228ad-clean.yaml
-rw-r--r--  1 user  staff  52459 Jun  8 12:29 flatten-20250608-122900-9e228ad-clean.txt
-rw-r--r--  1 user  staff    318 Jun  8 14:54 flatten-20250608-145400-74706ff-dirty.yaml
-rw-r--r--  1 user  staff  15592 Jun  8 14:54 flatten-20250608-145400-74706ff-dirty.txt
-rw-r--r--  1 user  staff    869 Jun 10 12:19 flatten-20250610-121900-74706ff-clean.yaml
-rw-r--r--  1 user  staff  53297 Jun 10 12:19 flatten-20250610-121900-74706ff-clean.txt

# Clean up old files
$ flatten cleanup --dry-run
Would remove 4 files:
  - flatten-20250608-122900-9e228ad-clean.yaml (2 days old)
  - flatten-20250608-122900-9e228ad-clean.txt (2 days old)
  - flatten-20250608-145400-74706ff-dirty.yaml (2 days old)
  - flatten-20250608-145400-74706ff-dirty.txt (2 days old)

$ flatten cleanup --days 1
✓ Removed 4 files older than 1 days
```

## What Gets Included?

Flatten includes all files except:

1. **Git ignored files** (respects `.gitignore`)
2. **Lock files** that don't add value for LLMs:
   - `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`
   - `composer.lock`, `Gemfile.lock`, `poetry.lock`, `Cargo.lock`
3. **Minified files**: `*.min.js`, `*.min.css`
4. **Flatten output files**: `flatten-*.txt`, `flatten-*.yaml`
5. **Files matching your exclude patterns** (when using `--exclude`)

### Binary File Handling

Binary files (images, PDFs, executables, etc.) are intelligently detected using a two-stage approach:
- **Stage 1**: Fast classification by file extension (99% of files)
- **Stage 2**: Content analysis for unknown file types (null byte detection, printability ratio)
- **Included** in the YAML structure (so you can see what files exist)
- **Omitted** from the text content (replaced with `[Binary file omitted]`)
- **Performance optimized**: Large files (>1MB) are classified as binary without content reading

This keeps your output focused on readable code while maintaining awareness of all project files.

## Performance Features

- **Fast binary detection**: 99% of files classified instantly by extension
- **Smart content analysis**: Only unknown file types get content inspection
- **Large file optimization**: Files >1MB assumed binary without reading
- **Efficient directory scanning**: Respects ignore patterns early in the process

## Exclude Patterns

The `--exclude` option accepts comma-separated patterns following gitignore-style rules:

```bash
# Exclude by extension
flatten --exclude "*.png,*.jpg"

# Exclude specific directories
flatten --exclude "dist,build"

# Exclude directory at any level
flatten --exclude "node_modules"

# Exclude from root only
flatten --exclude "/temp"

# Complex patterns
flatten --exclude "*.log,temp/,/cache,docs/*.pdf"
```

## Smart Git Integration

When run inside a git repository, flatten:
- Automatically switches to the repository root
- Preserves your intended paths (relative to where you run it)
- Includes the commit hash in output filenames
- Detects uncommitted changes (dirty state)

```bash
# Example: Running from a subdirectory
cd ~/projects/MyApp/frontend
flatten src/

# Output:
# Switches to git root: ~/projects/MyApp
# Processes: frontend/src/
# Creates: flatten-20250610-152000-abc1234-clean.yaml and .txt
```

## Safety Features

- **File count limit**: Default 1000 files (override with `--max-files`)
- **Git repository boundary**: Won't accidentally flatten your entire home directory
- **Clear error messages**: Tells you exactly what to do when limits are hit
- **File size insights**: Shows largest files when output exceeds 250KB

## Use Cases

- **LLM Context**: Provide complete codebase context to ChatGPT, Claude, or other AI assistants
- **Code Reviews**: Share entire project structure and content easily
- **Documentation**: Generate file inventories and snapshots
- **Archiving**: Create readable snapshots tied to specific commits
- **Version Tracking**: Timestamped outputs make it easy to track changes over time

## Example Session

```bash
$ cd my-awesome-project
$ flatten

✓ Created flatten-20250610-152000-abc1234-clean.yaml
✓ Created flatten-20250610-152000-abc1234-clean.txt (2.3 MB)

$ # Make some changes...
$ echo "TODO: fix this" >> src/index.js
$ flatten

✓ Created flatten-20250610-153045-abc1234-dirty.yaml
✓ Created flatten-20250610-153045-abc1234-dirty.txt (2.3 MB)

$ # Exclude large SQL files
$ flatten --exclude "*.sql"

✓ Created flatten-20250610-154500-abc1234-dirty.yaml
✓ Created flatten-20250610-154500-abc1234-dirty.txt (1.8 MB)
✓ Excluded 15 files matching patterns
✓ Omitted 2 binary files from content

Largest files:
  src/services/data-processor.ts (89.2 KB)
  docs/images/architecture.png (45.3 KB) [binary]
  src/utils/validators.ts (38.1 KB)
  tests/integration/full-suite.test.ts (35.2 KB)
  src/models/complex-model.ts (31.7 KB)

$ # Check what we've created over time
$ ls -lt flatten-*.txt | head -5
-rw-r--r--  1 user  staff  1887232 Jun 10 15:45 flatten-20250610-154500-abc1234-dirty.txt
-rw-r--r--  1 user  staff  2412032 Jun 10 15:30 flatten-20250610-153045-abc1234-dirty.txt
-rw-r--r--  1 user  staff  2412001 Jun 10 15:20 flatten-20250610-152000-abc1234-clean.txt

$ # Clean up files older than 1 day
$ flatten cleanup --days 1
✓ Removed 6 files older than 1 days

$ # Too many files?
$ cd ~/large-monorepo
$ flatten

Error: Found 1,234 files (max: 1000)
Try: flatten --max-files 2000
```

## Output Format Examples

### YAML Structure (flatten-20250610-152000-abc1234-clean.yaml)
```yaml
codebase:
   - path: .
     type: directory
     contents:
        - path: src
          type: directory
          contents:
             - path: src/index.js
               type: file
             - path: src/utils.js
               type: file
        - path: README.md
          type: file
        - path: package.json
          type: file
```

### Flattened Content (flatten-20250610-152000-abc1234-clean.txt)
```
<src/index.js>
import { helper } from './utils';

console.log('Hello world');
console.log(helper());

</src/index.js>

<src/utils.js>
export function helper() {
  return 'helping!';
}

</src/utils.js>

<README.md>
# My Project

This is my awesome project.

</README.md>

<package.json>
{
  "name": "my-project",
  "version": "1.0.0"
}

</package.json>
```

## Philosophy

This tool follows a minimalist philosophy:
- **Simple** - No configuration files or complex options
- **Smart** - Intelligent defaults that just work
- **Safe** - Built-in protections against common mistakes
- **Useful** - Solves real problems for developers working with LLMs

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

MIT - see [LICENSE](LICENSE) for details.

## Acknowledgments

This project was inspired by the need to easily provide codebase context to LLMs. The original version was a simple bash script that evolved into this more robust npm package.