npm package: @qezor/chunkysys
Description: Bounded-budget chunked file processing for object storage and constrained runtimes.
Author: Abdul Ahad
Latest version: 1.0.2

# @qezor/chunkysys

`@qezor/chunkysys` is a low-memory chunk processing system for files sitting on top of fs-like object storage adapters such as `@qezor/objsys`.

It is meant to be the foundation for:
- bounded file transforms
- line-based rewrites
- text streaming
- delimiter and paired-frame iteration
- guard-controlled buffer lifetimes
- live rewriters later

It is not a scoped parser like `jsonsys`. It stays generic and works on bytes, text, and line streams.

## What It Prioritizes

- bounded buffer budgets
- chunk leases that release memory as soon as they are no longer needed
- async iterator APIs instead of callback pyramids
- no recursion-heavy internals
- direct streaming into object storage writes
- precise byte windows via `startByte` and `endByte`
- range-aware source reads when the underlying fs adapter exposes `supportsRanges`

## Install

```bash
npm install @qezor/chunkysys
```

## Main Helpers

- `createChunkySystem(options)`
- `createChunkySystemFromEnv(options)`
- `createChunkGuard(options)`
- `chunks(path, options)`
- `text(path, options)`
- `lines(path, options)`
- `delimit(path, options)`
- `frames(path, options)`
- `copy(sourcePath, destinationPath, options)`
- `rewriteBytes(sourcePath, destinationPath, transformer, options)`
- `rewriteText(sourcePath, destinationPath, transformer, options)`
- `rewriteLines(sourcePath, destinationPath, transformer, options)`

## Core Idea

`chunkysys` does not try to store whole files in memory.

Instead it:
- reads from an fs-like stream
- exposes a bounded chunk lease
- releases that chunk automatically on the next step unless you explicitly retain it
- streams transformed output straight back into the destination writer

That makes it usable in tight environments where holding the whole file is not acceptable.

## Usage

```js
const { createChunkySystem } = require("@qezor/chunkysys")

const chunky = createChunkySystem({
  maxBufferBytes: 64 * 1024,
  chunkBytes: 16 * 1024,
})

for await (const chunk of chunky.chunks("catalogs/v1.txt")) {
  console.log(chunk.index, chunk.byteStart, chunk.byteEnd, chunk.text())
}
```

## Text And Line Rewrites

`rewriteText()` and `rewriteLines()` accept either:
- a function
- a small text plan object or array of plans

The plan path uses `@qezor/structkit` string helpers under the hood.

Example:

```js
await chunky.rewriteLines(
  "drafts/input.txt",
  "drafts/output.txt",
  [
    { op: "insert", after: "bulk", value: "-" },
  ]
)
```

## Precise Byte Windows

You can limit reading to a byte range without loading the whole object:

```js
for await (const chunk of chunky.chunks("big.bin", {
  startByte: 1024,
  endByte: 4096,
  chunkBytes: 512,
})) {
  // process only the requested window
}
```

## Delimiters And Frames

`delimit()` is for precise streaming splits without loading the full object:

```js
for await (const item of chunky.delimit("queues.ndjson", {
  delimiter: "\n",
  maxSegmentBytes: 128 * 1024,
})) {
  console.log(item.text)
}
```

`frames()` is for paired delimiters such as `()`, `<>`, `{{ }}`, or block markers:

```js
for await (const item of chunky.frames("template.txt", {
  open: "{{",
  close: "}}",
  emitOutside: true,
  allowNested: true,
})) {
  console.log(item.kind, item.text)
}
```

## Guard System

`createChunkGuard()` gives you a resource governor above plain byte budgeting:

- byte limits
- step limits
- rules
- lifetimes
- scheduled purging

Example:

```js
const { createChunkGuard, createChunkySystem } = require("@qezor/chunkysys")

const guard = createChunkGuard({
  maxBytes: 64 * 1024,
  maxSteps: 10_000,
  autoPurgeEverySteps: 100,
  defaultLifetimeMs: 5_000,
  purgeIntervalMs: 1_000,
})

const chunky = createChunkySystem({
  guard,
})
```

## Notes

- `retain()` is for rare cases. If you retain too much and your budget is exhausted, the system will fail fast by default.
- retained chunks can now be given explicit lifetimes, and the guard can purge expired resources on schedule or step intervals
- when the source adapter supports ranged reads, byte-window iteration avoids pulling unrelated bytes through the stream path
- `rewriteBytes()` is the right base for binary or parser-specific transforms later.
- `rewriteLines()` is a good fit for markdown, config, logs, NDJSON, and annotation passes.
- this package is designed to be the low-memory base for later DOM parsers and LiveRewriter systems

## Versioning

`@qezor/chunkysys` follows practical semver:

- patch: safe fixes, doc cleanup, and low-risk performance or correctness improvements
- minor: additive stream, guard, or rewrite capabilities that do not break existing callers
- major: contract changes to chunk leases, iterator shapes, guard semantics, or rewrite behavior

Release notes live in [CHANGELOG.md](./CHANGELOG.md).