open-index/open-npm huggingface dataset:
npm package: @santerijps/mion
Description: Minimal Object Notation
Author: Santeri Sydänmetsä
Latest version: 1.0.2

# mion

`mion` (Minimal Object Notation) is a clean and lean alternative to JSON, which is easily converted to JSON.
It looks familiar to JSON with a minimalist syntax to ensure easy readability and a small footprint.
As an additional feature, `mion` supports list schemas, which makes it delightful to declare lists without repetition.

```text
# This is a comment!

name      "Alice"   # string
age       50        # number
is_adult  true      # boolean
naughty   null      # null

address { # White space is irrelevant
  country   "UK"
  city      "London"
  street    "123 Baker Street"
}

fruits [
  "apple"
  "banana"
  "orange"
]

people (id name age) [
  1 "Alice" 50
  2 "Bob" 60
  3 "Charlie" 45
]
```
npm package: @santerijps/tokenizer
Description: A simple generic string tokenizer
Author: Santeri Sydänmetsä
Latest version: 1.0.3

# tokenizer

A simple generic string tokenizer written in `TypeScript`, this module exposes the following functions and classes:

```typescript
function* token_iterator<T extends string>(text: string, token_spec: TokenSpecification<T>);
function tokenize<T extends string>(text: string, token_spec: TokenSpecification<T>);
class TokenStream<T extends string>;
```

## Usage

The API functions always take in a string as a parameter that should be tokenized, as well as a token specification that specifies the different tokens that will be matched. If the token specification is empty, then nothing will be returned, as only matched tokens are returned. This should be remembered as there might be some tokens that are "forgotten" and not checked, causing there to be characters in the string that are never handled.

In the example below, `Elizabeth` and the whitespace between `"King"` and `true` is ignored completely:

```typescript
import { Matchers, TokenStream } from "@santerijps/tokenizer";

const text = `{Charles}{Elizabeth}"King" true`;

const spec = {
  lbrace: "{",
  rbrace: "}",
  boolean:  /(?:true|false)/,
  string: Matchers.DoubleQuotedMultilineString,
  royalty: (text) => {
    if (text.startsWith("Charles")) {
      return "Charles";
    }
    return null;
  },
};

const stream = new TokenStream(text, spec);

for (const token of stream) {
  console.log(token);
}
```

To catch all characters, it's a good idea to include a catch-all token specification:

```typescript
const spec = {
  undefined: /./s // the s flag makes the dot match newlines as well
};
```