npm package: @openai-hce/encode
Description: HCE (Hierarchical Columnar Encoding) encoder for efficient data compression
Author: OpenAI HCE Team
Latest version: 1.0.3

# @openai-hce/encode

Hierarchical Columnar Encoding (HCE) encoder for compressing structured JSON payloads. Converts JSON arrays into a column-oriented string representation that typically delivers **40–60 % smaller payloads** compared to raw JSON, while keeping the layout deterministic for fast decoding.

## Installation

```bash
npm install @openai-hce/encode
```

```bash
pnpm add @openai-hce/encode
yarn add @openai-hce/encode
```

## Quick start

```ts
import { HCEEncoder } from '@openai-hce/encode';

const data = [
	{ type: 'user', id: 1, name: 'Alice', role: 'admin' },
	{ type: 'user', id: 2, name: 'Bob', role: 'user' }
];

const encoder = new HCEEncoder();
const hce = encoder.encode(data, 'users');

console.log(hce);
// users(user)[2]:
//   user(id,name,role)[2]:
//     1,Alice,admin|2,Bob,user
```

## Why HCE

- Columnar compression for repeated schemas.
- Automatic grouping of similar records.
- Fully typed TypeScript API.
- Deterministic output suitable for caching.
- Zero runtime dependencies.

## API overview

### `new HCEEncoder(options?: HCEOptions)`

```ts
interface HCEOptions {
	fieldDelimiter?: string;      // default: ','
	recordDelimiter?: string;     // default: '|'
	nestedDelimiter?: string;     // default: ';'
	missingValue?: string;        // default: ' '
	flattenNested?: boolean;      // default: true
	typeField?: string;           // default: 'type'
	autoDetectGrouping?: boolean; // default: true
	preferTypeGrouping?: boolean; // default: true
	minGroupSizeForSecondaryGrouping?: number; // default: 5
	schemaUniformityThreshold?: number;        // default: 0.9
}
```

### `encoder.encode(data, rootKey?)`

- `data`: array of objects, or `{ key: [...] }`.
- `rootKey`: optional explicit group name.
- Returns an HCE string.

## Examples

### Type-only grouping is implicit

The encoder never prints `by type`; type grouping is the default.

```ts
const products = {
	products: [
		{ type: 'product', name: 'Laptop', price: 999 },
		{ type: 'product', name: 'Phone', price: 599 }
	]
};

console.log(new HCEEncoder().encode(products));
```

Output:
```
products(product)[2]:
	product(name,price)[2]:
		Laptop,999|Phone,599
```

Notice the absence of `by type` in the header—`type` is implicit.

### Secondary grouping adds `by {field}`

When a good secondary field (category, role, status, …) exists, the encoder emits the `by` suffix and suppresses that field inside each group.

```ts
const groupedProducts = {
	products: [
		{ type: 'product', category: 'Electronics', name: 'Laptop', price: 999 },
		{ type: 'product', category: 'Electronics', name: 'Phone', price: 599 },
		{ type: 'product', category: 'Books', name: 'JS Guide', price: 39 },
		{ type: 'product', category: 'Books', name: 'TS Handbook', price: 45 }
	]
};

console.log(new HCEEncoder().encode(groupedProducts));
```

Output:
```
products(product by category)[4]:
	Electronics(name,price)[2]:
		Laptop,999|Phone,599
	Books(name,price)[2]:
		JS Guide,39|TS Handbook,45
```

### Uniform schemas stay type-only

If every record shares the same shape, the encoder honours `preferTypeGrouping` and keeps a single `type` group—even if a secondary field exists.

```ts
const uniformUsers = {
	users: [
		{ type: 'user', role: 'admin', name: 'Alice', age: 30 },
		{ type: 'user', role: 'admin', name: 'Bob', age: 25 },
		{ type: 'user', role: 'user', name: 'Charlie', age: 35 }
	]
};

console.log(new HCEEncoder().encode(uniformUsers));
```

Output:
```
users(user)[3]:
	user(age,name,role)[3]:
		30,Alice,admin|25,Bob,admin|35,Charlie,user
```

### Multi-type collections list every type

```ts
const items = {
	items: [
		{ type: 'book', title: 'HCE Guide', pages: 200 },
		{ type: 'product', name: 'Laptop', price: 999 },
		{ type: 'service', name: 'Consulting', rate: 150 }
	]
};

console.log(new HCEEncoder().encode(items));
```

Output:
```
items(book,product,service)[3]:
	book(pages,title)[1]:
		200,HCE Guide
	product(name,price)[1]:
		Laptop,999
	service(name,rate)[1]:
		Consulting,150
```

### Edge case: single-valued secondary field

```ts
const adminsOnly = {
	users: [
		{ type: 'user', role: 'admin', name: 'Alice' },
		{ type: 'user', role: 'admin', name: 'Bob' }
	]
};

console.log(new HCEEncoder().encode(adminsOnly));
```

Output:
```
users(user)[2]:
	user(name,role)[2]:
		Alice,admin|Bob,admin
```

The encoder keeps type-only grouping because `role` has only one value.

### Edge case: missing type field

Objects without a `type` field still encode safely—the encoder collapses the output to a single header and preserves your original schema.

```ts
const unnamed = {
	products: [
		{ name: 'Laptop', price: 999 },
		{ name: 'Phone', price: 599 }
	]
};

console.log(new HCEEncoder().encode(unnamed));
```

Output:
```
products(name,price)[2]:
    Laptop,999|Phone,599
```

### Nested objects and arrays

```ts
const posts = [
	{
		type: 'post',
		id: 1,
		title: 'Hello World',
		author: { name: 'Alice', team: 'Platform' },
		tags: ['intro', 'hce']
	},
	{
		type: 'post',
		id: 2,
		title: 'Encoder Tips',
		author: { name: 'Bob', team: 'SDK' },
		tags: ['guide']
	}
];

console.log(new HCEEncoder().encode(posts, 'posts'));
```

Output:
```
posts(post)[2]:
	post(id,title,.author,.tags)[2]:
		1,'Hello World'|2,'Encoder Tips'
		.author(name,team)[2]:
			Alice,Platform|Bob,SDK
		.tags: intro;hce|guide
```

### Custom delimiters

```ts
const encoder = new HCEEncoder({
	fieldDelimiter: '\t',
	recordDelimiter: '\n',
	nestedDelimiter: ',',
});

console.log(encoder.encode([{ type: 'row', id: 1, value: 'A' }], 'rows'));
```

Output:
```
rows(row)[1]:
	row(id\tvalue)[1]:
		1\tA
```

## Tips

- Provide `{ users: [...] }` if you want the root key to match an existing JSON property automatically.
- Disable `autoDetectGrouping` when deterministic single-group output is required.
- Decode using [`@openai-hce/decode`](../decode/README.md) for round-trip conversions.

### Grouping configuration cheat sheet

| Option | Default | Effect |
|--------|---------|--------|
| `autoDetectGrouping` | `true` | Finds the best grouping field automatically. Set `false` to force type-only grouping. |
| `preferTypeGrouping` | `true` | Keeps uniform data in a single type group. Set `false` to allow aggressive secondary grouping. |
| `schemaUniformityThreshold` | `0.9` | Minimum proportion of records sharing the same schema before the encoder prefers type-only grouping. |
| `minGroupSizeForSecondaryGrouping` | `5` | Average group size needed to justify a secondary field split. Lower it to accept smaller groups. |
| `typeField` | `'type'` | Name of the discriminator field. Change when the source data uses `kind`, `category`, etc. |

## License

MIT © OpenAI HCE Team