open-index/open-npm huggingface dataset:
npm package: @lenml/tokenizer-aya_expanse
Description: aya_expanse tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-aya_expanse

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-aya_expanse";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-baichuan2
Description: baichuan2 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-baichuan2

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-baichuan2";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-chatglm3
Description: chatglm3 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-chatglm3

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-chatglm3";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-claude
Description: claude tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-claude

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-claude";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-claude1
Description: claude1 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-claude1

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-claude1";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-command_r_plus
Description: command_r_plus tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-command_r_plus

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-command_r_plus";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-deepseek_v3
Description: deepseek_v3 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-deepseek_v3

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-deepseek_v3";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-gemini
Description: gemini tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-gemini

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-gemini";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-gemma
Description: gemma tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-gemma

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-gemma";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-gemma2
Description: gemma2 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-gemma2

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-gemma2";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-gemma3
Description: gemma3 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-gemma3

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-gemma3";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-gpt2
Description: gpt2 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-gpt2

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-gpt2";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-gpt3
Description: gpt3 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-gpt3

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-gpt3";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-gpt35turbo
Description: gpt35turbo tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-gpt35turbo

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-gpt35turbo";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-gpt35turbo16k
Description: gpt35turbo16k tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-gpt35turbo16k

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-gpt35turbo16k";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-gpt4
Description: gpt4 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-gpt4

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-gpt4";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-gpt4o
Description: gpt4o tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-gpt4o

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-gpt4o";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-gptoss
Description: gptoss tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-gptoss

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-gptoss";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-internlm2
Description: internlm2 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-internlm2

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-internlm2";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-llama2
Description: llama2 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-llama2

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-llama2";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-llama3
Description: llama3 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-llama3

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-llama3";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-llama3_1
Description: llama3_1 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-llama3_1

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-llama3_1";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-llama3_2
Description: llama3_2 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-llama3_2

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-llama3_2";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-minicpm_v4_5
Description: minicpm_v4_5 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-minicpm_v4_5

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-minicpm_v4_5";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-mistral_nemo
Description: mistral_nemo tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-mistral_nemo

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-mistral_nemo";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-qwen1_5
Description: qwen1_5 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-qwen1_5

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-qwen1_5";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-qwen2_5
Description: qwen2_5 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-qwen2_5

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-qwen2_5";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-qwen3
Description: qwen3 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-qwen3

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-qwen3";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-text_davinci002
Description: text_davinci002 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-text_davinci002

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-text_davinci002";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-text_davinci003
Description: text_davinci003 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-text_davinci003

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-text_davinci003";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-text_embedding_ada002
Description: text_embedding_ada002 tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-text_embedding_ada002

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-text_embedding_ada002";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0
npm package: @lenml/tokenizer-yi
Description: yi tokenizer for NodeJS/Browser
Author: zhzluke96
Latest version: 3.7.2

# @lenml/tokenizer-yi

a tokenizer.

> based on `@lenml/tokenizers`

# Usage

```ts
import { fromPreTrained } from "@lenml/tokenizer-yi";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));
```

# Full Tokenizer API

Complete api parameters and usage can be found in [transformer.js tokenizers document](https://huggingface.co/docs/transformers.js/v3.0.0/api/tokenizers)

# License

Apache-2.0