npm package: @neutrondev/kalla-oi-wasm-compress
Description: Lossless compression library with adaptive algorithm selection using Rust + WASM. Supports Brotli, Deflate, and smart detection for already-compressed files.
Author: Neutrondev
Latest version: 0.1.2

# @neutrondev/kalla-oi-wasm-compress

A high-performance lossless compression library built with Rust and WebAssembly. Features adaptive algorithm selection that automatically chooses the optimal compression strategy based on file type and content entropy analysis.

## Table of Contents

- [Features](#features)
- [Installation](#installation)
- [Bundler Configuration](#bundler-configuration)
  - [Next.js (Turbopack)](#nextjs-app-router-with-turbopack)
  - [Next.js (Webpack)](#nextjs-webpack---pages-router-or-app-router-without-turbopack)
  - [Vite](#vite)
  - [Webpack 5](#webpack-5)
  - [Troubleshooting](#troubleshooting)
- [Quick Start](#quick-start)
- [API Reference](#api-reference)
  - [Initialization](#initialization)
  - [Compression](#compression)
  - [Decompression](#decompression)
  - [File Info](#file-info)
  - [Metrics](#metrics)
  - [Synchronous API](#synchronous-api)
- [Compression Algorithms](#compression-algorithms)
- [Adaptive Strategy](#adaptive-strategy)
- [Binary Format Specification](#binary-format-specification)
- [Web Worker Support](#web-worker-support)
- [Error Handling](#error-handling)
- [Performance](#performance)
- [Browser & Node.js Compatibility](#browser--nodejs-compatibility)
- [License](#license)

## Features

- **Adaptive Algorithm Selection** - Automatically selects the best compression algorithm based on file type detection and entropy analysis
- **Multiple Algorithms** - Supports Brotli (best ratio), Deflate (balanced), and Store (passthrough for pre-compressed files)
- **Smart File Detection** - Detects file types via magic bytes and extension hints to skip compression on already-compressed formats
- **Shannon Entropy Analysis** - Calculates data entropy to determine compressibility for unknown file types
- **Chunked Processing** - Memory-efficient handling of large files up to 100MB with configurable chunk sizes
- **Web Worker Support** - Non-blocking compression for large files in browser environments
- **Cross-Platform** - Works seamlessly in both browser and Node.js environments
- **Type-Safe** - Full TypeScript support with comprehensive type definitions
- **Zero Dependencies** - Pure Rust/WASM implementation with no runtime dependencies

## Installation

```bash
npm install @neutrondev/kalla-oi-wasm-compress
```

```bash
yarn add @neutrondev/kalla-oi-wasm-compress
```

```bash
pnpm add @neutrondev/kalla-oi-wasm-compress
```

```bash
bun add @neutrondev/kalla-oi-wasm-compress
```

## Bundler Configuration

WASM files require special configuration in modern bundlers. Follow the guide for your setup:

### Next.js (App Router with Turbopack)

Next.js with Turbopack requires copying WASM files to the public folder and using dynamic imports:

**1. Copy WASM files to public folder:**

```bash
# Add to your package.json scripts
"postinstall": "cp -r node_modules/@neutrondev/kalla-oi-wasm-compress/src/wasm public/wasm"
```

Or manually copy `node_modules/@neutrondev/kalla-oi-wasm-compress/src/wasm/` to `public/wasm/`.

**2. Create a client-side hook:**

```typescript
// hooks/use-compression.ts
'use client';

import { useCallback, useEffect, useRef, useState } from 'react';

export function useCompression() {
  const [isReady, setIsReady] = useState(false);
  const [error, setError] = useState<Error | null>(null);
  const moduleRef = useRef<typeof import('@neutrondev/kalla-oi-wasm-compress') | null>(null);

  useEffect(() => {
    let mounted = true;

    async function init() {
      try {
        // Dynamic import to avoid SSR issues
        const mod = await import('@neutrondev/kalla-oi-wasm-compress');

        // Initialize with explicit WASM path from public folder
        await mod.initWasm('/wasm/compression_wasm_bg.wasm');

        if (mounted) {
          moduleRef.current = mod;
          setIsReady(true);
        }
      } catch (err) {
        if (mounted) {
          setError(err instanceof Error ? err : new Error(String(err)));
        }
      }
    }

    init();
    return () => { mounted = false; };
  }, []);

  const compress = useCallback(async (data: Uint8Array, options?: Parameters<typeof import('@neutrondev/kalla-oi-wasm-compress').compress>[1]) => {
    if (!moduleRef.current) throw new Error('Compression not initialized');
    return moduleRef.current.compress(data, options);
  }, []);

  const decompress = useCallback(async (data: Uint8Array) => {
    if (!moduleRef.current) throw new Error('Compression not initialized');
    return moduleRef.current.decompress(data);
  }, []);

  return { isReady, error, compress, decompress };
}
```

**3. Use in components:**

```typescript
'use client';

import { useCompression } from '@/hooks/use-compression';

export function MyComponent() {
  const { isReady, error, compress, decompress } = useCompression();

  if (error) return <div>Error: {error.message}</div>;
  if (!isReady) return <div>Loading compression...</div>;

  const handleCompress = async (file: File) => {
    const buffer = await file.arrayBuffer();
    const data = new Uint8Array(buffer);
    const result = await compress(data, { filename: file.name });
    console.log(`Compressed: ${result.savings * 100}% savings`);
  };

  return <button onClick={() => handleCompress(file)}>Compress</button>;
}
```

### Next.js (Webpack - Pages Router or App Router without Turbopack)

**1. Configure next.config.js:**

```javascript
// next.config.js
/** @type {import('next').NextConfig} */
const nextConfig = {
  webpack: (config, { isServer }) => {
    // Handle WASM files
    config.experiments = {
      ...config.experiments,
      asyncWebAssembly: true,
    };

    // Fix for WASM imports
    config.module.rules.push({
      test: /\.wasm$/,
      type: 'asset/resource',
    });

    return config;
  },
};

module.exports = nextConfig;
```

**2. Use dynamic import with ssr disabled:**

```typescript
// components/compression-provider.tsx
'use client';

import dynamic from 'next/dynamic';

const CompressionComponent = dynamic(
  () => import('./compression-component'),
  { ssr: false }
);

export default CompressionComponent;
```

### Vite

Vite has built-in WASM support but may need configuration for optimal loading:

**1. Install vite-plugin-wasm (optional, for better compatibility):**

```bash
npm install vite-plugin-wasm vite-plugin-top-level-await
```

**2. Configure vite.config.ts:**

```typescript
// vite.config.ts
import { defineConfig } from 'vite';
import wasm from 'vite-plugin-wasm';
import topLevelAwait from 'vite-plugin-top-level-await';

export default defineConfig({
  plugins: [
    wasm(),
    topLevelAwait(),
  ],
  optimizeDeps: {
    exclude: ['@neutrondev/kalla-oi-wasm-compress'],
  },
});
```

**3. Use in your app:**

```typescript
import { initWasm, compress, decompress } from '@neutrondev/kalla-oi-wasm-compress';

// Initialize once
await initWasm();

// Use compression
const result = await compress(data);
```

### Webpack 5

**1. Configure webpack.config.js:**

```javascript
// webpack.config.js
module.exports = {
  experiments: {
    asyncWebAssembly: true,
  },
  module: {
    rules: [
      {
        test: /\.wasm$/,
        type: 'asset/resource',
      },
    ],
  },
};
```

### Troubleshooting

#### Error: "Module not found: Can't resolve './wasm/compression_wasm_bg.wasm'"

This occurs when the bundler can't resolve the WASM file path. Solutions:

1. **Copy WASM to public folder** and use explicit URL:
   ```typescript
   await initWasm('/wasm/compression_wasm_bg.wasm');
   ```

2. **Use dynamic imports** to load the module only on the client side.

3. **Configure your bundler** to handle `.wasm` files (see configurations above).

#### Error: "WebAssembly is not defined" (SSR)

WASM only works in browser environments. Solutions:

1. Use dynamic imports with `ssr: false` (Next.js)
2. Check for browser environment before initializing:
   ```typescript
   if (typeof window !== 'undefined') {
     await initWasm();
   }
   ```

#### Error: "CompileError: WebAssembly.instantiate()"

The WASM file may be corrupted or incorrectly served. Ensure:

1. WASM file is served with `Content-Type: application/wasm`
2. File is not being processed/transformed by the bundler incorrectly

## Quick Start

```typescript
import { initWasm, compress, decompress } from '@neutrondev/kalla-oi-wasm-compress';

// Initialize WASM module (required once before any operations)
await initWasm();

// Compress text data
const text = 'Hello, World! '.repeat(1000);
const data = new TextEncoder().encode(text);

const compressed = await compress(data, { filename: 'greeting.txt' });

console.log(`Original size: ${compressed.originalSize} bytes`);
console.log(`Compressed size: ${compressed.compressedSize} bytes`);
console.log(`Algorithm used: ${compressed.algo}`);
console.log(`Space savings: ${(compressed.savings * 100).toFixed(1)}%`);

// Decompress back to original
const decompressed = await decompress(compressed.data);
const originalText = new TextDecoder().decode(decompressed.data);

console.log(originalText); // "Hello, World! Hello, World! ..."
```

## API Reference

### Initialization

#### `initWasm(wasmUrl?: string | URL): Promise<void>`

Initializes the WebAssembly module. This function must be called once before using any compression functions. It is idempotent - subsequent calls return immediately.

```typescript
import { initWasm } from '@neutrondev/kalla-oi-wasm-compress';

// Default initialization (auto-detects WASM location)
await initWasm();

// Custom WASM URL (useful for CDN deployment)
await initWasm('https://cdn.example.com/compression_wasm_bg.wasm');
```

#### `isWasmInitialized(): boolean`

Checks if the WASM module has been initialized.

```typescript
import { isWasmInitialized } from '@neutrondev/kalla-oi-wasm-compress';

if (!isWasmInitialized()) {
  await initWasm();
}
```

### Compression

#### `compress(data: Uint8Array, options?: CompressOptions): Promise<CompressResult>`

Compresses data using adaptive algorithm selection.

**Parameters:**

| Parameter | Type | Description |
|-----------|------|-------------|
| `data` | `Uint8Array` | Raw bytes to compress (max 100MB) |
| `options` | `CompressOptions` | Optional compression settings |

**CompressOptions:**

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `filename` | `string` | `undefined` | Filename hint for file type detection (e.g., `"data.csv"`) |
| `level` | `CompressionLevel` | `Default` | Compression level: `Fast`, `Default`, or `Best` |
| `chunkSize` | `number` | `4194304` (4MB) | Chunk size for large files (64KB - 16MB) |
| `forceAlgo` | `Algorithm` | `undefined` | Force a specific algorithm instead of auto-detection |

**CompressResult:**

| Field | Type | Description |
|-------|------|-------------|
| `data` | `Uint8Array` | Compressed data with header |
| `algo` | `Algorithm` | Algorithm that was used (`Brotli`, `Deflate`, or `Store`) |
| `originalSize` | `number` | Original data size in bytes |
| `compressedSize` | `number` | Compressed data size in bytes |
| `ratio` | `number` | Compression ratio (compressed / original) |
| `savings` | `number` | Space savings as decimal (0.0 - 1.0) |

**Examples:**

```typescript
import { compress, CompressionLevel, Algorithm } from '@neutrondev/kalla-oi-wasm-compress';

// Basic compression with filename hint
const csvData = new TextEncoder().encode('name,age,city\nAlice,30,NYC\n'.repeat(1000));
const result = await compress(csvData, { filename: 'users.csv' });

// Maximum compression for archival
const archiveResult = await compress(data, {
  filename: 'important.json',
  level: CompressionLevel.Best
});

// Fast compression for real-time use
const fastResult = await compress(data, {
  level: CompressionLevel.Fast
});

// Force specific algorithm
const brotliResult = await compress(data, {
  forceAlgo: Algorithm.Brotli
});

// Custom chunk size for memory-constrained environments
const chunkedResult = await compress(largeData, {
  chunkSize: 1024 * 1024 // 1MB chunks
});
```

### Decompression

#### `decompress(data: Uint8Array): Promise<DecompressResult>`

Decompresses data that was compressed with this library.

**DecompressResult:**

| Field | Type | Description |
|-------|------|-------------|
| `data` | `Uint8Array` | Decompressed original data |
| `algo` | `Algorithm` | Algorithm that was used |
| `originalSize` | `number` | Size after decompression |
| `compressedSize` | `number` | Size before decompression |

```typescript
import { decompress } from '@neutrondev/kalla-oi-wasm-compress';

const decompressed = await decompress(compressedData);
const text = new TextDecoder().decode(decompressed.data);
```

### File Info

#### `getFileInfo(data: Uint8Array): Promise<FileInfo>`

Extracts metadata from compressed data without performing full decompression. Useful for displaying file information before decompression.

**FileInfo:**

| Field | Type | Description |
|-------|------|-------------|
| `version` | `number` | Format version (currently 1) |
| `algo` | `Algorithm` | Compression algorithm used |
| `originalSize` | `number` | Original uncompressed size |
| `isChunked` | `boolean` | Whether data was chunked |
| `chunkSize` | `number` | Chunk size if chunked, 0 otherwise |

```typescript
import { getFileInfo } from '@neutrondev/kalla-oi-wasm-compress';

const info = await getFileInfo(compressedData);

console.log(`Format version: ${info.version}`);
console.log(`Algorithm: ${info.algo}`);
console.log(`Original size: ${info.originalSize} bytes`);
console.log(`Chunked: ${info.isChunked ? `Yes (${info.chunkSize} bytes)` : 'No'}`);
```

### Metrics

#### `getMetrics(): CompressionMetrics | null`

Returns performance metrics from the last compression/decompression operation.

**CompressionMetrics:**

| Field | Type | Description |
|-------|------|-------------|
| `inputBytes` | `number` | Input data size |
| `outputBytes` | `number` | Output data size |
| `timeMs` | `number` | Processing time in milliseconds |
| `throughputMbps` | `number` | Throughput in MB/s |

```typescript
import { compress, getMetrics } from '@neutrondev/kalla-oi-wasm-compress';

await compress(largeData);
const metrics = getMetrics();

if (metrics) {
  console.log(`Processed ${metrics.inputBytes} bytes in ${metrics.timeMs.toFixed(2)}ms`);
  console.log(`Throughput: ${metrics.throughputMbps.toFixed(2)} MB/s`);
}
```

### Synchronous API

After WASM initialization, synchronous versions are available for scenarios where async/await is not suitable.

#### `compressSync(data: Uint8Array, options?: CompressOptions): CompressResult`

```typescript
import { initWasm, compressSync } from '@neutrondev/kalla-oi-wasm-compress';

await initWasm();

// Now sync API is available
const result = compressSync(data, { filename: 'data.txt' });
```

#### `decompressSync(data: Uint8Array): DecompressResult`

```typescript
import { decompressSync } from '@neutrondev/kalla-oi-wasm-compress';

const result = decompressSync(compressedData);
```

## Compression Algorithms

### Brotli

Brotli is the default algorithm for text and structured data. It provides excellent compression ratios, especially for text content.

| Aspect | Details |
|--------|---------|
| **Best for** | Text, JSON, CSV, HTML, CSS, JavaScript, logs |
| **Compression ratio** | Excellent (typically 70-90% reduction for text) |
| **Speed** | Moderate (optimized for ratio over speed) |
| **Use case** | Archival, network transfer, storage optimization |

### Deflate

Deflate provides a balance between compression ratio and speed. It's the algorithm behind ZIP and gzip.

| Aspect | Details |
|--------|---------|
| **Best for** | General binary data, real-time compression |
| **Compression ratio** | Good (typically 50-70% reduction) |
| **Speed** | Fast |
| **Use case** | Real-time compression, streaming |

### Store (Passthrough)

Store mode passes data through without compression. Used automatically for already-compressed formats.

| Aspect | Details |
|--------|---------|
| **Best for** | JPEG, PNG, MP4, ZIP, PDF, and other compressed formats |
| **Compression ratio** | 1:1 (no compression) |
| **Speed** | Instant |
| **Use case** | Avoiding double-compression overhead |

## Adaptive Strategy

The library automatically selects the optimal compression strategy based on multiple factors:

### File Type Detection

Files are detected using magic bytes (file signatures) and extension hints:

| File Type | Magic Bytes | Extensions | Algorithm |
|-----------|-------------|------------|-----------|
| **Text/Structured** | | `.txt`, `.csv`, `.json`, `.xml`, `.html`, `.css`, `.js`, `.ts`, `.log` | Brotli |
| **JPEG** | `FF D8 FF` | `.jpg`, `.jpeg` | Store |
| **PNG** | `89 50 4E 47` | `.png` | Store |
| **GIF** | `47 49 46 38` | `.gif` | Store |
| **WebP** | `52 49 46 46 ... 57 45 42 50` | `.webp` | Store |
| **MP4** | `00 00 00 .. 66 74 79 70` | `.mp4`, `.m4v`, `.m4a` | Store |
| **WebM** | `1A 45 DF A3` | `.webm` | Store |
| **MP3** | `FF FB`, `FF FA`, `49 44 33` | `.mp3` | Store |
| **ZIP** | `50 4B 03 04` | `.zip` | Store |
| **GZIP** | `1F 8B` | `.gz`, `.gzip` | Store |
| **PDF** | `25 50 44 46` | `.pdf` | Store |
| **XLSX** | `50 4B 03 04` + `xl/` | `.xlsx` | Store |
| **DOCX** | `50 4B 03 04` + `word/` | `.docx` | Store |

### Entropy Analysis

For unknown file types, Shannon entropy is calculated to determine compressibility:

```
H = -Σ p(x) × log₂(p(x))
```

Where `p(x)` is the probability of each byte value.

| Entropy (bits/byte) | Interpretation | Algorithm |
|---------------------|----------------|-----------|
| 0.0 - 4.0 | Highly compressible (repetitive data) | Brotli |
| 4.0 - 7.0 | Moderately compressible | Brotli/Deflate |
| 7.0 - 8.0 | Low compressibility (random/encrypted) | Store |

### Minimum Savings Threshold

Compression is only applied if it achieves meaningful space savings:

| Content Type | Minimum Savings |
|--------------|-----------------|
| Text/Structured data | ≥ 5% |
| Unknown data | ≥ 2% |
| Already compressed | Skipped (Store) |

## Binary Format Specification

Compressed data uses a 27-byte header followed by the payload:

```
┌─────────────────────────────────────────────────────────────────┐
│                        HEADER (27 bytes)                        │
├────────┬──────┬───────────────────────────────────────────────┤
│ Offset │ Size │ Field                                          │
├────────┼──────┼───────────────────────────────────────────────┤
│ 0      │ 4    │ Magic bytes: "CWZ1" (0x43, 0x57, 0x5A, 0x31)  │
│ 4      │ 1    │ Version (currently 1)                         │
│ 5      │ 1    │ Flags (bit0=stored, bit1=chunked, bit2=dict)  │
│ 6      │ 1    │ Algorithm (0=Store, 1=Brotli, 2=Deflate)      │
│ 7      │ 4    │ Chunk size (u32 LE, 0 if not chunked)         │
│ 11     │ 8    │ Original size (u64 LE)                        │
│ 19     │ 4    │ Dictionary ID (u32 LE, reserved)              │
│ 23     │ 4    │ CRC32 checksum of original data               │
├────────┴──────┴───────────────────────────────────────────────┤
│                       PAYLOAD (N bytes)                        │
├───────────────────────────────────────────────────────────────┤
│ Compressed data (single block or multiple chunks)             │
└───────────────────────────────────────────────────────────────┘
```

### Chunked Format

For large files, the payload contains multiple chunks:

```
┌─────────────────────────────────────────────────────────────────┐
│                     CHUNKED PAYLOAD                             │
├────────┬──────┬───────────────────────────────────────────────┤
│ For each chunk:                                                │
│ - 4 bytes: Compressed chunk size (u32 LE)                     │
│ - N bytes: Compressed chunk data                               │
└───────────────────────────────────────────────────────────────┘
```

## Web Worker Support

For large files, use Web Workers to avoid blocking the main thread:

```typescript
import { createCompressionWorker } from '@neutrondev/kalla-oi-wasm-compress/worker';

// Create and initialize worker
const worker = await createCompressionWorker();

// Compress in background (non-blocking)
const compressed = await worker.compress(largeData, {
  filename: 'large-file.csv',
  level: CompressionLevel.Best
});

// Decompress in background
const decompressed = await worker.decompress(compressed.data);

// Get file info
const info = await worker.getFileInfo(compressed.data);

// Clean up when done
worker.terminate();
```

### Worker with Custom WASM URL

```typescript
import { createCompressionWorker } from '@neutrondev/kalla-oi-wasm-compress/worker';

const worker = await createCompressionWorker('https://cdn.example.com/wasm/compression.wasm');
```

## Error Handling

The library throws `CompressionError` with specific error codes:

```typescript
import { compress, decompress, CompressionError, ErrorCode } from '@neutrondev/kalla-oi-wasm-compress';

try {
  const result = await decompress(invalidData);
} catch (error) {
  if (error instanceof CompressionError) {
    switch (error.code) {
      case ErrorCode.InvalidHeader:
        console.error('Invalid or corrupted compressed data');
        break;
      case ErrorCode.ChecksumMismatch:
        console.error('Data integrity check failed');
        break;
      case ErrorCode.SizeLimit:
        console.error('Input exceeds 100MB limit');
        break;
      case ErrorCode.RatioExceeded:
        console.error('Decompression bomb detected');
        break;
      default:
        console.error(`Compression error: ${error.message}`);
    }
  }
}
```

### Error Codes

| Code | Name | Description |
|------|------|-------------|
| `1` | `InvalidHeader` | Missing or malformed header |
| `2` | `UnsupportedAlgo` | Unknown compression algorithm |
| `3` | `TruncatedInput` | Incomplete compressed data |
| `4` | `SizeLimit` | Input exceeds 100MB maximum |
| `5` | `ChecksumMismatch` | CRC32 verification failed |
| `6` | `OutputMismatch` | Decompressed size doesn't match header |
| `7` | `CompressionFailed` | Algorithm-specific compression error |
| `8` | `DecompressionFailed` | Algorithm-specific decompression error |
| `9` | `RatioExceeded` | Decompression ratio exceeds 10x (bomb protection) |

## Performance

### Benchmarks

Typical performance on modern hardware (M1/Intel i7):

| Operation | Data Type | Size | Time | Throughput |
|-----------|-----------|------|------|------------|
| Compress (Brotli) | Text/JSON | 1 MB | ~15ms | ~67 MB/s |
| Compress (Deflate) | Binary | 1 MB | ~8ms | ~125 MB/s |
| Compress (Store) | JPEG | 1 MB | ~1ms | ~1000 MB/s |
| Decompress | Any | 1 MB | ~5ms | ~200 MB/s |

### Optimization Tips

1. **Use filename hints** - Providing `filename` enables instant file type detection without entropy analysis
2. **Match compression level to use case** - Use `Fast` for real-time, `Best` for archival
3. **Use Web Workers for large files** - Prevents UI blocking in browsers
4. **Adjust chunk size** - Smaller chunks use less memory but have more overhead

## Browser & Node.js Compatibility

### Browser Support

| Browser | Minimum Version |
|---------|-----------------|
| Chrome | 57+ |
| Firefox | 52+ |
| Safari | 11+ |
| Edge | 79+ |

### Node.js Support

| Node.js | Support |
|---------|---------|
| 18.x | Full support |
| 20.x | Full support |
| 22.x | Full support |

### ESM Only

This package is ESM-only and requires:

```json
{
  "type": "module"
}
```

Or use `.mjs` extension for your files.

## License

MIT License - see [LICENSE](./LICENSE) for details.

---

Built with Rust and WebAssembly for maximum performance and reliability.