p-doom/crowd-code-dataset-0.2 huggingface dataset:
[tab] Cargo.toml (plaintext)
[workspace]
resolver = "2"
members = [
    "crates/core",
    "crates/napi",
    "crates/cli",
]

[workspace.package]
version = "0.1.0"
edition = "2021"
license = "MIT"
repository = "https://github.com/p-doom/crowd-pilot-serializer"

[workspace.dependencies]
# Core dependencies
regex = "1.10"
thiserror = "1.0"

# Serialization
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"

# Python bindings
pyo3 = { version = "0.20", features = ["extension-module"] }

# The core crate
crowd-pilot-serializer-core = { path = "crates/core" }
[tab] README.md (markdown)
# crowd-pilot-serializer

Core serialization library for crowd-pilot IDE interaction data. Converts IDE events (tab switches, edits, terminal commands, etc.) into conversation format for training language models.

## Architecture

This is a Rust library with bindings for:
- **Node.js/TypeScript** (via napi-rs) - for the VS Code extension
- **Python** (via PyO3) - for batch preprocessing

The serialization logic is the single source of truth, ensuring consistency between runtime inference and training data preprocessing.

## Crates

- `crates/core` - Core serialization logic
- `crates/napi` - Node.js bindings (`@crowd-pilot/serializer` npm package)
- `crates/python` - Python bindings (`crowd-pilot-serializer` PyPI package)

## Building

### Prerequisites

- Rust 1.70+
- Node.js 18+ (for napi bindings)
- Python 3.9+ (for Python bindings)
- maturin (`pip install maturin`)

### Build all

```bash
cargo build --release
```

### Build Node.js bindings

```bash
cd crates/napi
npm install
npm run build
```

### Build Python bindings

```bash
cd crates/python
maturin develop
```

## Usage

### Node.js/TypeScript

```typescript
import { ConversationStateManager } from '@crowd-pilot/serializer';

const manager = new ConversationStateManager({
  countTokens: (text) => Math.ceil(text.length / 4),
  maxTokensPerConversation: 8192,
  maxTokensPerMessage: 2048,
});

manager.handleTabEvent('/path/to/file.ts', 'file contents...');
manager.handleContentEvent('/path/to/file.ts', 10, 0, 'inserted text');

const messages = manager.finalizeForModel();
```

### Python

```python
from crowd_pilot_serializer import ConversationStateManager
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-7B")

manager = ConversationStateManager(
    count_tokens=lambda text: len(tokenizer.encode(text)),
    max_tokens_per_conversation=8192,
    max_tokens_per_message=2048,
)

manager.handle_tab_event("/path/to/file.ts", "file contents...")
manager.handle_content_event("/path/to/file.ts", 10, 0, "inserted text")

messages = manager.finalize_for_model()
```

## License

MIT
[content] LICENSE (plaintext)
Copyright 2025 Franz Srambical/p(doom)

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
[tab] .gitignore (ignore)
# Rust
/target/
Cargo.lock

# Node.js
node_modules/
*.node
*.js
*.d.ts
!crates/napi/index.js
!crates/napi/index.d.ts

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
*.egg-info/
dist/
build/
.eggs/

# IDE
.idea/
.vscode/
*.swp
*.swo

# OS
.DS_Store
Thumbs.db
[tab] crates/napi/package-lock.json (json)
{
  "name": "@crowd-pilot/serializer",
  "version": "0.1.0",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "@crowd-pilot/serializer",
      "version": "0.1.0",
      "license": "MIT",
      "devDependencies": {
        "@napi-rs/cli": "^2.18.0"
      }
    },
    "node_modules/@napi-rs/cli": {
      "version": "2.18.4",
      "resolved": "https://registry.npmjs.org/@napi-rs/cli/-/cli-2.18.4.tgz",
      "integrity": "sha512-SgJeA4df9DE2iAEpr3M2H0OKl/yjtg1BnRI5/JyowS71tUWhrfSu2LT0V3vlHET+g1hBVlrO60PmEXwUEKp8Mg==",
      "dev": true,
      "bin": {
        "napi": "scripts/index.js"
      },
      "engines": {
        "node": ">= 10"
      },
      "funding": {
        "type": "github",
        "url": "https://github.com/sponsors/Brooooooklyn"
      }
    }
  }
}
[content] README.md (markdown)
This is a Rust library with:
- **Node.js/TypeScript bindings** (via napi-rs) - for the VS Code extension (uses character approximation for token counting)
- **CLI binary** - for batch preprocessing (uses HuggingFace tokenizer via embedded Python)

The serialization logic is the single source of truth, ensuring consistency between runtime inference and training data preprocessing.

## Crates

- `crates/core` - Core serialization logic
- `crates/napi` - Node.js bindings (`@crowd-pilot/serializer` npm package)
- `crates/cli` - CLI binary for preprocessing (`crowd-pilot-serialize`)

## Building

### Prerequisites

- Rust 1.70+
- Node.js 18+ (for napi bindings)
- Python 3.9+ with `transformers` installed (for CLI tokenizer)

### Build all

```bash
cargo build --release
```

### Build Node.js bindings

```bash
cd crates/napi
npm install
npm run build
```

### Build CLI

```bash
cargo build --release -p crowd-pilot-serialize
```

## Usage

### Node.js/TypeScript (VS Code Extension)

```typescript
import { ConversationStateManager } from '@crowd-pilot/serializer';

const manager = new ConversationStateManager({
  viewportRadius: 10,
  coalesceRadius: 5,
  maxTokensPerMessage: 2048,
  maxTokensPerTerminalOutput: 256,
});

manager.handleTabEvent('/path/to/file.ts', 'file contents...');
manager.handleContentEvent('/path/to/file.ts', 10, 0, 'inserted text');

const messages = manager.finalizeForModel();
```

### CLI (Preprocessing)

```bash
crowd-pilot-serialize \
    --csv-root ./data/sessions \
    --output-dir ./output \
    --tokenizer "Qwen/Qwen2-7B" \
    --max-tokens-per-conversation 8192 \
    --max-tokens-per-message 2048 \
    --val-ratio 0.1
```

This reads CSV session files, processes them through the Rust serializer, and outputs `training.jsonl` and `validation.jsonl` in NeMo conversation format.

#### CLI Options

| Option | Default | Description |
|--------|---------|-------------|
| `--csv-root` | required | Root directory containing per-session CSV files |
| `--output-dir` | required | Output directory for JSONL files |
| `--tokenizer` | required | HuggingFace tokenizer name or path |
| `--max-tokens-per-conversation` | 8192 | Maximum tokens per conversation chunk |
| `--max-tokens-per-message` | 2048 | Maximum tokens per message |
| `--max-tokens-per-terminal-output` | 256 | Maximum tokens for terminal output |
| `--min-conversation-messages` | 5 | Minimum messages to keep a conversation |
| `--viewport-radius` | 10 | Lines above/below cursor to show |
| `--coalesce-radius` | 5 | Radius for grouping nearby edits |
| `--val-ratio` | 0.10 | Fraction of sessions for validation |

## License

MIT
[tab] crates/core/Cargo.toml (plaintext)
[package]
name = "crowd-pilot-serializer-core"
version.workspace = true
edition.workspace = true
license.workspace = true
description = "Core serialization logic for crowd-pilot's IDE interaction data"

[dependencies]
regex = { workspace = true }
thiserror = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
csv = "1.3"
walkdir = "2.5"

[dev-dependencies]
tempfile = "3.10"
[tab] crates/cli/Cargo.toml (plaintext)
[package]
name = "crowd-pilot-serialize"
version.workspace = true
edition.workspace = true
license.workspace = true
description = "CLI tool for serializing crowd-pilot IDE interaction data"

[[bin]]
name = "crowd-pilot-serialize"
path = "src/main.rs"

[dependencies]
crowd-pilot-serializer-core = { path = "../core" }
clap = { version = "4.5", features = ["derive"] }
pyo3 = { version = "0.20", features = ["auto-initialize"] }  # Different features from workspace
serde_json = { workspace = true }
[tab] crates/core/src/lib.rs (rust)
//! Core serialization logic for crowd-pilot IDE interaction data.
//!
//! This crate provides the `ConversationStateManager` which converts IDE events
//! (tab switches, edits, terminal commands, etc.) into conversation format
//! suitable for training language models.

mod conversation;
mod diff;
mod helpers;
pub mod pipeline;

pub use conversation::{ConversationMessage, ConversationStateManager, ConversationStateManagerConfig, FinalizedConversation};
pub use pipeline::{
    discover_csv_files, process_all_sessions, process_session, write_jsonl_output,
    NemoMessage, NemoRecord, PipelineConfig, PipelineResult, SessionResult,
};
pub use diff::{compute_changed_block_lines, ChangedBlock};
pub use helpers::{
    apply_backspaces, apply_change, clean_text, escape_single_quotes_for_sed, fenced_block,
    line_numbered_output, normalize_terminal_output, serialize_compute_viewport, Viewport,
};

/// Default viewport radius (lines above/below cursor to show)
pub const VIEWPORT_RADIUS: usize = 10;

/// Default coalesce radius for grouping nearby edits
pub const COALESCE_RADIUS: usize = 5;

/// Default maximum tokens per message (approximate)
pub const MAX_TOKENS_PER_MESSAGE: usize = 2048;

/// Default maximum tokens per terminal output
pub const MAX_TOKENS_PER_TERMINAL_OUTPUT: usize = 256;
[content] crates/core/src/lib.rs (rust)
/// Trait for tokenization operations.
/// 
/// Implementors provide token counting and truncation capabilities.
/// For exact tokenization (preprocessing), use a real tokenizer.
/// For approximate tokenization (runtime), use character-based estimation.
pub trait Tokenizer {
    /// Count the number of tokens in the given text.
    fn count_tokens(&self, text: &str) -> usize;
    
    /// Truncate text to at most `max_tokens` tokens.
    /// Returns the truncated text.
    fn truncate_to_max_tokens(&self, text: &str, max_tokens: usize) -> String;
}
[content] crates/core/src/conversation.rs (rust)
            message.value = self.tokenizer.truncate_to_max_tokens(
                &message.value,
                self.config.max_tokens_per_message,
            );
            tokens = self.config.max_tokens_per_message;
[content] crates/core/src/conversation.rs (rust)
            let max_chars = self.config.max_tokens_per_message * 4;
            if message.value.len() > max_chars {
                message.value = message.value.chars().take(max_chars).collect();
                tokens = self.config.max_tokens_per_message;
            }
[content] crates/core/src/conversation.rs (rust)
            let truncated = self.tokenizer.truncate_to_max_tokens(
                &cleaned,
                self.config.max_tokens_per_terminal_output,
            );
            cleaned = format!("{}
... [truncated]", truncated);
[content] crates/core/src/conversation.rs (rust)
            let max_chars = self.config.max_tokens_per_terminal_output * 4;
            if cleaned.len() > max_chars {
                cleaned = format!(
                    "{}
... [truncated]",
                    cleaned.chars().take(max_chars).collect::<String>()
                );
            }
[content] crates/core/src/conversation.rs (rust)
    /// Character-based approximate tokenizer for tests.
    struct CharApproxTokenizer;

    impl Tokenizer for CharApproxTokenizer {
        fn count_tokens(&self, text: &str) -> usize {
            text.len() / 4
        }

        fn truncate_to_max_tokens(&self, text: &str, max_tokens: usize) -> String {
            text.chars().take(max_tokens * 4).collect()
        }
[content] crates/core/src/conversation.rs (rust)
            message.value = self.tokenizer.truncate_to_max_tokens(
                &message.value,
                self.config.max_tokens_per_message,
            );
            tokens = self.config.max_tokens_per_message;
        }
[content] crates/cli/src/main.rs (rust)
                    text.len() / 4
                }
            }
        })
    }

    fn truncate_to_max_tokens(&self, text: &str, max_tokens: usize) -> String {
        Python::with_gil(|py| {
            let tokenizer = self.tokenizer.as_ref(py);
            
            // Encode text to token IDs
            let tokens = match tokenizer.call_method1("encode", (text,)) {
                Ok(t) => t,
                Err(_) => {
                    // Fallback to character approximation on error
                    return text.chars().take(max_tokens * 4).collect();
                }
            };
            
            // Get the length
            let token_count = tokens.len().unwrap_or(0);
            if token_count <= max_tokens {
                return text.to_string();
            }
            
            // Slice tokens to max_tokens using Python slicing: tokens[:max_tokens]
            let truncated_tokens = match tokens.get_item(pyo3::types::PySlice::new(py, 0, max_tokens as isize, 1)) {
                Ok(t) => t,
                Err(_) => {
                    // Fallback to character approximation on error
                    return text.chars().take(max_tokens * 4).collect();
                }
            };
            
            // Decode back to text
            match tokenizer.call_method1("decode", (truncated_tokens,)) {
                Ok(decoded) => decoded.extract().unwrap_or_else(|_| {
                    text.chars().take(max_tokens * 4).collect()
                }),
                Err(_) => {
                    // Fallback to character approximation on error
                    text.chars().take(max_tokens * 4).collect()
[content] crates/cli/src/main.rs (rust)
            
            let truncated_tokens = tokens
                .get_item(pyo3::types::PySlice::new(py, 0, max_tokens as isize, 1))
                .expect("Failed to slice tokens");
            
            tokenizer
                .call_method1("decode", (truncated_tokens,))
                .expect("Failed to decode truncated tokens")
                .extract()
                .expect("Failed to extract decoded string")
[content] crates/cli/src/main.rs (rust)
            let tokens = tokenizer
                .call_method1("encode", (text,))
                .expect("Failed to encode text with tokenizer");
            tokens.len().expect("Failed to get token count")
        })
    }

    fn truncate_to_max_tokens(&self, text: &str, max_tokens: usize) -> String {
        Python::with_gil(|py| {
            let tokenizer = self.tokenizer.as_ref(py);
            
            let tokens = tokenizer
                .call_method1("encode", (text,))
                .expect("Failed to encode text with tokenizer");
            
            let token_count = tokens.len().expect("Failed to get token count");
            if token_count <= max_tokens {
                return text.to_string();
[content] crates/cli/src/main.rs (rust)
            match tokenizer.call_method1("encode", (text,)) {
                Ok(tokens) => tokens.len().unwrap_or(0),
                Err(_) => {
                    // Fallback to character approximation on error
                    (text.len() + 3) / 4
                }
[content] crates/cli/src/main.rs (rust)
            
            let truncated_tokens = tokens
                .get_item(pyo3::types::PySlice::new(py, 0, max_tokens as isize, 1))
                .unwrap();
            
            tokenizer
                .call_method1("decode", (truncated_tokens,))
                .expect("Failed to decode truncated tokens")
                .extract()
                .unwrap()
[content] crates/cli/src/main.rs (rust)
            let tokens = tokenizer
                .call_method1("encode", (text,))
                .expect("Failed to encode text with tokenizer");
            tokens.len().unwrap()
        })
    }

    fn truncate_to_max_tokens(&self, text: &str, max_tokens: usize) -> String {
        Python::with_gil(|py| {
            let tokenizer = self.tokenizer.as_ref(py);
            
            let tokens = tokenizer
                .call_method1("encode", (text,))
                .expect("Failed to encode text with tokenizer");
            
            let token_count = tokens.len().unwrap();
            if token_count <= max_tokens {
                return text.to_string();
[content] crates/napi/src/lib.rs (rust)
        // Use core defaults as the single source of truth
        let defaults = ConversationStateManagerConfig::default();
        
        let config = match options {
            Some(opts) => ConversationStateManagerConfig {
                viewport_radius: opts.viewport_radius.map(|v| v as usize).unwrap_or(defaults.viewport_radius),
                coalesce_radius: opts.coalesce_radius.map(|v| v as usize).unwrap_or(defaults.coalesce_radius),
                max_tokens_per_message: opts.max_tokens_per_message.map(|v| v as usize).unwrap_or(defaults.max_tokens_per_message),
                max_tokens_per_terminal_output: opts.max_tokens_per_terminal_output.map(|v| v as usize).unwrap_or(defaults.max_tokens_per_terminal_output),
                // Extension-specific: no chunking (single ongoing conversation)
                max_tokens_per_conversation: None,
                min_conversation_messages: defaults.min_conversation_messages,
            },
            None => ConversationStateManagerConfig {
                // Extension-specific: no chunking
                max_tokens_per_conversation: None,
                ..defaults
            },
[content] crates/napi/src/lib.rs (rust)
        let opts = options.unwrap_or(ConversationStateManagerOptions {
            viewport_radius: None,
            coalesce_radius: None,
            max_tokens_per_message: None,
            max_tokens_per_terminal_output: None,
        });

        let config = ConversationStateManagerConfig {
            viewport_radius: opts.viewport_radius.map(|v| v as usize).unwrap_or(10),
            coalesce_radius: opts.coalesce_radius.map(|v| v as usize).unwrap_or(5),
            max_tokens_per_message: opts.max_tokens_per_message.map(|v| v as usize).unwrap_or(2048),
            max_tokens_per_terminal_output: opts
                .max_tokens_per_terminal_output
                .map(|v| v as usize)
                .unwrap_or(256),
            // No chunking for the extension - it's a single ongoing conversation
            max_tokens_per_conversation: None,
            min_conversation_messages: 5,
[content] crates/napi/src/lib.rs (rust)
        let defaults = ConversationStateManagerConfig::default();
        
        let config = match options {
            Some(opts) => ConversationStateManagerConfig {
                viewport_radius: opts.viewport_radius.map(|v| v as usize).unwrap_or(defaults.viewport_radius),
                coalesce_radius: opts.coalesce_radius.map(|v| v as usize).unwrap_or(defaults.coalesce_radius),
                max_tokens_per_message: opts.max_tokens_per_message.map(|v| v as usize).unwrap_or(defaults.max_tokens_per_message),
                max_tokens_per_terminal_output: opts.max_tokens_per_terminal_output.map(|v| v as usize).unwrap_or(defaults.max_tokens_per_terminal_output),
                // Extension-specific: no chunking (single ongoing conversation)
                max_tokens_per_conversation: None,
                min_conversation_messages: defaults.min_conversation_messages,
            },
            None => ConversationStateManagerConfig {
                // Extension-specific: no chunking
                max_tokens_per_conversation: None,
                ..defaults
            },
[content] crates/napi/src/lib.rs (rust)
    /// Maximum tokens per conversation (for context length limits). Default: None (no limit)
    pub max_tokens_per_conversation: Option<u32>,
    /// Minimum messages required to keep a conversation. Default: 5
    pub min_conversation_messages: Option<u32>,
[content] crates/napi/src/lib.rs (rust)
        let defaults = ConversationStateManagerConfig::default();
        
        let config = match options {
            Some(opts) => ConversationStateManagerConfig {
                viewport_radius: opts.viewport_radius.map(|v| v as usize).unwrap_or(defaults.viewport_radius),
                coalesce_radius: opts.coalesce_radius.map(|v| v as usize).unwrap_or(defaults.coalesce_radius),
                max_tokens_per_message: opts.max_tokens_per_message.map(|v| v as usize).unwrap_or(defaults.max_tokens_per_message),
                max_tokens_per_terminal_output: opts.max_tokens_per_terminal_output.map(|v| v as usize).unwrap_or(defaults.max_tokens_per_terminal_output),
                max_tokens_per_conversation: opts.max_tokens_per_conversation.map(|v| v as usize),
                min_conversation_messages: opts.min_conversation_messages.map(|v| v as usize).unwrap_or(defaults.min_conversation_messages),
            },
            None => defaults,
[tab] crates/napi/Cargo.toml (plaintext)
[package]
name = "crowd-pilot-serializer-napi"
version.workspace = true
edition.workspace = true
license.workspace = true
description = "Node.js bindings for crowd-pilot serializer"

[lib]
crate-type = ["cdylib"]

[dependencies]
crowd-pilot-serializer-core = { workspace = true }
napi = { version = "2", features = ["napi4"] }
napi-derive = "2"
serde = { workspace = true }
serde_json = { workspace = true }

[build-dependencies]
napi-build = "2"
[content] crates/core/src/pipeline.rs (rust)
                let offset = row.range_offset.expect("content event missing RangeOffset") as usize;
                let length = row.range_length.expect("content event missing RangeLength") as usize;
                let text = row.text.as_deref().expect("content event missing Text");
[tab] crates/napi/package.json (json)
{
  "name": "@crowd-pilot/serializer",
  "version": "0.1.0",
  "description": "Node.js bindings for crowd-pilot serializer",
  "main": "index.js",
  "types": "index.d.ts",
  "napi": {
    "name": "serializer",
    "triples": {
      "defaults": true,
      "additional": [
        "aarch64-apple-darwin",
        "aarch64-unknown-linux-gnu",
        "aarch64-unknown-linux-musl",
        "x86_64-unknown-linux-musl"
      ]
    }
  },
  "files": [
    "index.js",
    "index.d.ts"
  ],
  "devDependencies": {
    "@napi-rs/cli": "^2.18.0"
  },
  "scripts": {
    "artifacts": "napi artifacts",
    "build": "napi build --platform --release",
    "build:debug": "napi build --platform",
    "prepublishOnly": "napi prepublish -t npm",
    "universal": "napi universal",
    "version": "napi version"
  },
  "license": "MIT",
  "repository": {
    "type": "git",
    "url": "https://github.com/your-org/crowd-pilot-serializer"
  }
}
[tab] .gitignore (ignore)
# Rust
/target/
Cargo.lock

# Node.js
node_modules/
*.node
*.js
*.d.ts
package-lock.json

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
*.egg-info/
dist/
build/
.eggs/

# IDE
.idea/
.vscode/
*.swp
*.swo

# OS
.DS_Store
Thumbs.db
[content] crates/core/src/conversation.rs (rust)
            // Only emit messages if file hasn't been opened in this conversation yet
            if !self.files_opened_in_conversation.contains(file_path) {
                let cmd = format!("cat -n {}", file_path);
                self.append_message(ConversationMessage::assistant(fenced_block(
                    Some("bash"),
                    &clean_text(&cmd),
                )));
                let output = line_numbered_output(&content, None, None);
                self.append_message(ConversationMessage::user(format!(
                    "<stdout>
{}
</stdout>",
                    output
                )));
                self.files_opened_in_conversation.insert(file_path.to_string());
            }
[content] crates/core/src/conversation.rs (rust)
            let cmd = format!("cat -n {}", file_path);
            self.append_message(ConversationMessage::assistant(fenced_block(
                Some("bash"),
                &clean_text(&cmd),
            )));
            let output = line_numbered_output(&content, None, None);
            self.append_message(ConversationMessage::user(format!(
                "<stdout>
{}
</stdout>",
                output
            )));
            self.files_opened_in_conversation.insert(file_path.to_string());