npm package: @sharc-code/splitter
Description: SHARC Splitter - AST and recursive text chunking for semantic search
Author: SHARC
Latest version: 0.2.7

# @sharc-code/splitter

Code chunking library for semantic search, featuring AST-based splitting with context injection and recursive character fallback.

[![npm version](https://img.shields.io/npm/v/@sharc-code/splitter.svg)](https://www.npmjs.com/package/@sharc-code/splitter)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)

## Overview

`@sharc-code/splitter` provides intelligent code chunking for RAG (Retrieval-Augmented Generation) and semantic code search applications. It extracts meaningful code units while preserving semantic context.

### Key Features

- **AST-Based Splitting**: Tree-sitter powered parsing for 9 languages
- **Context Injection**: Automatically adds class/module context to extracted methods
- **Decorator/Annotation Support**: Preserves decorators (`@Get()`, `#[derive]`, etc.) in context
- **Recursive Fallback**: Character-based splitting for unsupported languages
- **Syntax Error Detection**: Validate code before indexing
- **Memory Safe**: Proper cleanup of native tree-sitter resources

## Installation

```bash
# npm
npm install @sharc-code/splitter

# bun
bun add @sharc-code/splitter --trust

# pnpm
pnpm add @sharc-code/splitter
```

**Note:** This package includes native tree-sitter bindings. Ensure you have a C++ compiler available:
- **Windows**: Visual Studio Build Tools
- **macOS**: Xcode Command Line Tools (`xcode-select --install`)
- **Linux**: `build-essential` package

If you install with Bun, verify the native bindings were trusted:

```bash
bun pm untrusted
```

If `tree-sitter` or `tree-sitter-*` packages still appear there, AST chunking may degrade to the recursive fallback splitter until those dependencies are trusted.

## Quick Start

```typescript
import { AstCodeSplitter, LangChainCodeSplitter } from '@sharc-code/splitter';

// AST-based splitting (recommended for code)
const astSplitter = new AstCodeSplitter(3500, 0);

const chunks = await astSplitter.split(
  `class UserService {
    async authenticate(user: string): Promise<boolean> {
      return this.validateCredentials(user);
    }
  }`,
  'typescript',
  'src/services/user.ts'
);

console.log(chunks[0].content);
// Output:
// // Context: class UserService (services/user.ts)
// async authenticate(user: string): Promise<boolean> {
//   return this.validateCredentials(user);
// }

// Recursive splitting (for docs/config)
const langchainSplitter = new LangChainCodeSplitter(1500, 150);
const docChunks = await langchainSplitter.split(markdownContent, 'markdown', 'README.md');
```

## Splitters

### AstCodeSplitter

Tree-sitter based splitter that extracts complete semantic units (functions, classes, methods) with automatic context injection.

**Supported Languages:**
- TypeScript / JavaScript
- Python
- Java
- C++ / C
- Go
- Rust
- C#
- Scala

**Features:**
- Extracts functions, classes, methods, interfaces as complete units
- Injects context comments (e.g., `// Context: class UserService > module auth`)
- Automatic fallback to recursive splitting for unsupported languages
- Syntax error detection via `checkSyntaxErrors()`
- Memory-safe with `dispose()` for cleanup

```typescript
import { AstCodeSplitter } from '@sharc-code/splitter';

const splitter = new AstCodeSplitter(
  3500,  // chunkSize (max characters per chunk)
  0      // chunkOverlap (0 for AST - semantic units don't need overlap)
);

// Split code
const chunks = await splitter.split(code, 'typescript', 'src/auth.ts');

// Check for syntax errors before indexing
const { hasErrors, errorCount } = splitter.checkSyntaxErrors(code, 'typescript');

// Check if language is supported
if (AstCodeSplitter.isLanguageSupported('rust')) {
  // Use AST splitter
}

// Clean up when done (important for long-running processes)
splitter.dispose();
```

### LangChainCodeSplitter

Character-based splitter using SHARC's vendored recursive character splitter. Best for documentation, configuration files, and languages without AST support.

This class name is kept for backward compatibility even though SHARC no longer depends on LangChain internally.

**Supported Languages:**
- JavaScript/TypeScript (as `js`)
- Python, Java, C++, Go, Rust, PHP, Ruby, Swift, Scala
- Markdown, HTML, LaTeX
- Solidity

```typescript
import { LangChainCodeSplitter } from '@sharc-code/splitter';

const splitter = new LangChainCodeSplitter(
  1500,  // chunkSize
  150    // chunkOverlap (overlap preserves context across chunks)
);

const chunks = await splitter.split(content, 'markdown', 'docs/README.md');
```

## API Reference

### CodeChunk

```typescript
interface CodeChunk {
  content: string;
  metadata: {
    startLine: number;
    endLine: number;
    language?: string;
    filePath?: string;
    chunkKind?: 'wrapper' | 'member' | 'container' | 'data_container' | 'standalone';
    documentClass?: 'implementation' | 'test' | 'docs_examples';
  };
}
```

### Splitter Interface

```typescript
interface Splitter {
  split(code: string, language: string, filePath?: string): Promise<CodeChunk[]>;
  setChunkSize(chunkSize: number): void;
  setChunkOverlap(chunkOverlap: number): void;
}
```

### AstCodeSplitter

```typescript
class AstCodeSplitter implements Splitter {
  constructor(chunkSize?: number, chunkOverlap?: number);

  // Split code into chunks
  split(code: string, language: string, filePath?: string): Promise<CodeChunk[]>;

  // Check for syntax errors
  checkSyntaxErrors(code: string, language: string): { hasErrors: boolean; errorCount: number };

  // Configuration
  setChunkSize(chunkSize: number): void;
  setChunkOverlap(chunkOverlap: number): void;

  // Cleanup native resources
  dispose(): void;

  // Static utilities
  static getSupportedLanguages(): string[];
  static isLanguageSupported(language: string): boolean;
}
```

### LangChainCodeSplitter

```typescript
class LangChainCodeSplitter implements Splitter {
  constructor(chunkSize?: number, chunkOverlap?: number);

  split(code: string, language: string, filePath?: string): Promise<CodeChunk[]>;
  setChunkSize(chunkSize: number): void;
  setChunkOverlap(chunkOverlap: number): void;
}
```

## Context Injection

The AST splitter automatically adds context comments to extracted code chunks, improving search relevance:

### TypeScript/JavaScript/Java/C++/Go/Rust/C#/Scala

```typescript
// Context: class UserService > module auth (services/user.ts)
async authenticate(user: string): Promise<boolean> {
  return this.validateCredentials(user);
}
```

### Python

```python
# Context: class UserService (services/user.py)
def authenticate(self, user: str) -> bool:
    return self.validate_credentials(user)
```

### Context Hierarchy

The splitter tracks nested containers and builds a context path:

```typescript
// Input: deeply nested method
namespace App {
  module Auth {
    class UserService {
      authenticate() { ... }
    }
  }
}

// Output chunk:
// Context: namespace App > module Auth > class UserService (auth/user.ts)
authenticate() { ... }
```

## Recommended Chunk Sizes

| Content Type | Chunk Size | Overlap | Splitter | Rationale |
|--------------|------------|---------|----------|-----------|
| **Code (AST)** | 3500 | 0 | AST | Complete semantic units, no overlap needed |
| **Documentation** | 1500 | 150 | Recursive | Prose flows between sections |
| **Config/Data** | 1500 | 100 | Recursive | Related keys grouped together |
| **Fallback Code** | 1500 | 100 | Recursive | Conservative chunking |

## Memory Management

Tree-sitter uses native C++ bindings that allocate memory outside Node.js's garbage collector. For long-running processes:

```typescript
const splitter = new AstCodeSplitter();

try {
  // Process many files...
  for (const file of files) {
    const chunks = await splitter.split(file.content, file.language, file.path);
    // ... use chunks
  }
} finally {
  // Free native memory when done
  splitter.dispose();
}
```

The splitter also automatically cleans up parse trees after each `split()` call.

## Error Handling

### Syntax Error Detection

```typescript
const splitter = new AstCodeSplitter();

// Check for errors before indexing
const { hasErrors, errorCount } = splitter.checkSyntaxErrors(code, 'typescript');

if (hasErrors) {
  console.warn(`File has ${errorCount} syntax errors, skipping...`);
} else {
  const chunks = await splitter.split(code, 'typescript');
}
```

### Automatic Fallback

If AST parsing fails or the language isn't supported, the AST splitter automatically falls back to recursive splitting:

```typescript
const splitter = new AstCodeSplitter();

// Vue files aren't AST-supported, will use the recursive fallback
const chunks = await splitter.split(vueCode, 'vue', 'App.vue');
// Console: "Language vue not supported by AST, using recursive fallback splitter for: App.vue"
```

## Development

```bash
# Install dependencies
bun install

# Build
bun run build

# Type check
bun run typecheck

# Watch mode
bun run dev

# Clean build artifacts
bun run clean
```

## Dependencies

| Package | Purpose |
|---------|---------|
| `tree-sitter` | AST parsing engine |
| `tree-sitter-*` | Language grammars (9 languages) |
| vendored recursive splitter | Text splitting utilities |

## Use in SHARC

This package is used by:

- **[@sharc-code/mcp](https://www.npmjs.com/package/@sharc-code/mcp)** - MCP server for AI assistants
- **@sharc/core** - Core indexing engine (internal)

For end-to-end semantic code search, see the main [SHARC documentation](https://docs.sharc.sh).

## License

MIT - See [LICENSE](https://github.com/yxanul/SHARC/blob/master/LICENSE) for details.

## Contributing

All code modifications must be done via Pull Request. See [CLAUDE.md](https://github.com/yxanul/SHARC/blob/master/CLAUDE.md) for guidelines.