Skill: zhaobudaoyuema/prompt-token-counter (version 1.0.11)

--- name: prompt-token-counter version: 1.0.10 description: "Count tokens and estimate costs for 300+ LLM models. Primary use: audit OpenClaw workspace token consumption (memory, persona, skills)." trigger: "token count, cost estimate, prompt length, API cost, OpenClaw audit, workspace token usage, memory/persona/skills tokens, context window limit" --- # Prompt Token Counter (toksum) > **First load reminder:** This skill provides the `scripts` CLI (toksum). Use it when the user asks to count tokens, estimate API costs, or **audit OpenClaw component token consumption** (memory, persona, skills). ## Before Installing — Security & Privacy - **What will be read:** The audit workflow reads files under `~/.openclaw/workspace` and `~/.openclaw/skills` (AGENTS.md, SOUL.md, MEMORY.md, SKILL.md, etc.). Those files may contain personal data or secrets. Only install if you accept that access. - **URL fetching:** The CLI can fetch HTTP(S) URLs via `-u`. SKILL.md requires the agent to confirm each URL with the user before fetching. Insist the agent follow that rule; never allow automatic fetching of unknown URLs. - **Source verification:** Source: [https://github.com/Zhaobudaoyuema/prompt-token-counter](https://github.com/Zhaobudaoyuema/prompt-token-counter). Review `scripts/core.py` and `scripts/cli.py` before use. The code performs local file reads and optional HTTP GETs only; no other network calls or data exfiltration. - **Run locally first:** If unsure, run the CLI manually in an isolated environment against safe test files to verify behavior. ## Primary Use: OpenClaw Token Consumption Audit **Goal:** Help users identify which OpenClaw components consume tokens and how much. ### 1. Memory & Persona Files These files are injected into sessions and consume tokens. Search and count them: | File | Purpose | Typical Location | |------|---------|------------------| | `AGENTS.md` | Operating instructions, workflow, priorities | `~/.openclaw/workspace/` | | `SOUL.md` | Persona, tone, values, behavioral guidelines | `~/.openclaw/workspace/` | | `IDENTITY.md` | Name, role, goals, visual description | `~/.openclaw/workspace/` | | `USER.md` | User preferences, communication style | `~/.openclaw/workspace/` | | `MEMORY.md` | Long-term memory, persistent facts | `~/.openclaw/workspace/` | | `TOOLS.md` | Tool quirks, path conventions | `~/.openclaw/workspace/` | | `HEARTBEAT.md` | Periodic maintenance checklist | `~/.openclaw/workspace/` | | `BOOT.md` | Startup ritual (when hooks enabled) | `~/.openclaw/workspace/` | | `memory/YYYY-MM-DD.md` | Daily memory logs | `~/.openclaw/workspace/memory/` | **Workspace path:** Default `~/.openclaw/workspace`; may be overridden in `~/.openclaw/openclaw.json` via `agent.workspace`. ### 2. Skill Files (SKILL.md) Skills are loaded per session. Count each `SKILL.md`: | Location | Scope | |----------|-------| | `~/.openclaw/skills/*/SKILL.md` | OpenClaw managed skills | | `~/.openclaw/workspace/skills/*/SKILL.md` | Workspace-specific skills (override) | ### 3. Audit Workflow 1. **Locate workspace:** Resolve `~/.openclaw/workspace` (or config override). 2. **Collect files:** List all memory/persona files and `SKILL.md` paths above. 3. **Count tokens:** Run `python -m scripts.cli <path1> <path2> ... -m <model> -c` (batch mode). 4. **Summarize:** Group by category (memory, persona, skills), report total and per-file. **Example audit command (PowerShell):** ```powershell $ws = "$env:USERPROFILE\.openclaw\workspace" python -m scripts.cli -m gpt-4o -c "$ws\AGENTS.md" "$ws\SOUL.md" "$ws\USER.md" "$ws\IDENTITY.md" "$ws\MEMORY.md" "$ws\TOOLS.md" ``` **Example audit (Bash):** ```bash WS=~/.openclaw/workspace python -m scripts.cli -m gpt-4o -c "$WS/AGENTS.md" "$WS/SOUL.md" "$WS/USER.md" "$WS/IDENTITY.md" "$WS/MEMORY.md" "$WS/TOOLS.md" ``` --- ## Project Layout ``` prompt_token_counter/ ├── SKILL.md ├── package.json # npm package (OpenClaw skill) ├── publish_npm.py # Publish to npm; syncs version └── scripts/ # Python package, CLI + examples ├── cli.py # Entry point ├── core.py # TokenCounter, estimate_cost ├── registry/ │ ├── models.py # 300+ models │ └── pricing.py # Pricing data └── examples/ # Script examples ├── count_prompt.py ├── estimate_cost.py ├── batch_compare.py └── benchmark_token_ratio.py ``` Invoke: `python -m scripts.cli` from project root. ### Version Sync (publish_npm.py) When publishing to npm, `publish_npm.py` bumps the patch version and syncs it to: - `package.json` — `version` - `SKILL.md` — frontmatter `version` - `scripts/__init__.py` — `__version__` Run: `python publish_npm.py` (after `npm login`). --- ## Runtime Dependencies - **Python 3** — required - **tiktoken** (optional) — `pip install tiktoken` for exact OpenAI counts --- ## Language Rule **Respond in the user's language.** Match the user's language (e.g. Chinese if they write in Chinese, English if they write in English). --- ## URL Usage — Mandatory Agent Rule **Before using `-u` / `--url` to fetch content from any URL, you MUST:** 1. **Explicitly warn the user** that the CLI will make an outbound HTTP/HTTPS request to the given URL. 2. **Confirm the URL is trusted** — tell the user: "Only use URLs you fully trust. Untrusted URLs may expose your IP, leak data, or be used for SSRF. Do you confirm this URL is safe?" 3. **Prefer alternatives** — if the user can provide the content via `-f` (local file) or inline text, suggest that instead of URL fetch. 4. **Never auto-fetch** — do not invoke `-u` without the user having explicitly provided the URL and acknowledged the risk. **If the user insists on using a URL:** Proceed only after they confirm. State clearly: "I will fetch from [URL] to count tokens. Proceed?" --- ## Model Name — Mandatory Agent Rule **Before invoking the CLI, you MUST have a concrete model name from the user.** 1. **Require explicit model** — `-m` / `--model` is required. Do not guess or assume; the user must provide the exact name (e.g. gpt-4o, claude-3-5-sonnet-20241022). 2. **If unclear, ask** — if the user says "GPT" or "Claude" or "the latest model" without a specific name, ask: "Please specify the exact model name (e.g. gpt-4o, claude-3-5-sonnet-20241022). Run `python -m scripts.cli -l` to list supported models." 3. **Do not auto-pick** — never substitute a model on behalf of the user without their confirmation. 4. **Validate when possible** — if the model name seems ambiguous, offer `-l` output or confirm: "I'll use [model]. Is that correct?" --- ## CLI Usage **Default:** Read from local file(s). No segmentation. Supports multiple file paths for batch execution. ```bash python -m scripts.cli [OPTIONS] [FILE ...] ``` | Option | Short | Description | |--------|-------|-------------| | `--model` | `-m` | Model name (required unless `--list-models`) — **Agent must obtain exact name from user; ask if unclear** | | `--file` | `-f` | Read from file (repeatable) | | `--url` | `-u` | Read from URL (repeatable) — **Agent must warn user before use; only trusted URLs** | | `--list-models` | `-l` | List supported models | | `--cost` | `-c` | Show cost estimate | | `--output-tokens` | | Use output token pricing | | `--currency` | | USD or INR | | `--verbose` | `-v` | Detailed output | ### Examples ```bash # Multiple local files (default batch mode) python -m scripts.cli file1.txt file2.txt -m gpt-4 python -m scripts.cli AGENTS.md SOUL.md MEMORY.md -m gpt-4o -c # Single file with -f python -m scripts.cli -f input.txt -m claude-3-opus -c # Inline text (when arg is not an existing file path) python -m scripts.cli -m gpt-4 "Hello, world!" # List models python -m scripts.cli -l # Run bundled example scripts python scripts/examples/count_prompt.py file1.txt file2.txt -m gpt-4 python scripts/examples/estimate_cost.py "Your text" gpt-4 python scripts/examples/batch_compare.py file1.txt -m gpt-4 claude-3-opus ``` --- ## Python API ```python from scripts import TokenCounter, count_tokens, estimate_cost, get_supported_models tokens = count_tokens("Hello!", "gpt-4") counter = TokenCounter("claude-3-opus") tokens = counter.count_messages([ {"role": "system", "content": "..."}, {"role": "user", "content": "..."} ]) cost = estimate_cost(tokens, "gpt-4", input_tokens=True) ``` --- ## Supported Models 300+ models across 34+ providers: OpenAI, Anthropic, Google, Meta, Mistral, Cohere, xAI, DeepSeek, etc. Use `python -m scripts.cli -l` for full list. - **OpenAI:** exact via tiktoken - **Others:** ~85–95% approximation --- ## Response Output — Agent Guideline **After returning token count or cost estimate results, the agent MUST:** 1. **Include the project link** — e.g. > Source: [prompt-token-counter](https://github.com/Zhaobudaoyuema/prompt-token-counter) 2. **Briefly explain how tokens are calculated** — e.g. > **How tokens are counted:** OpenAI models use tiktoken (exact). Other models use provider-specific formulas calibrated from benchmark data. For CJK-heavy text, the ratio is blended by CJK character ratio so that Chinese gets fewer chars per token. --- ## Common Issues | Issue | Action | |-------|--------| | "tiktoken is required" | `pip install tiktoken` | | UnsupportedModelError | Use `-l` for valid names | | Cost "NA" | Model has no pricing; count still valid | | User provides URL | **Agent must warn:** outbound request, SSRF risk, only trusted URLs; confirm before `-u` | | Model unclear / vague | **Agent must ask:** user to specify exact model name; offer `-l` to list; do not guess | --- ## When to Trigger This Skill Activate this skill when the user: | Trigger | Example phrases | |---------|-----------------| | **Token count** | "How many tokens?", "Count tokens in this prompt", "Token length of X" | | **Cost estimate** | "Estimate API cost", "How much for this text?", "Cost for GPT-4" | | **Prompt size** | "Check prompt length", "Is this too long?", "Context window limit" | | **OpenClaw audit** | "How many tokens does my workspace use?", "Audit OpenClaw memory/persona/skills", "Which components consume tokens?", "Token usage of AGENTS.md / SOUL.md / skills" | | **Model comparison** | "Compare token cost across models", "Which model is cheaper?" | Also trigger when the agent needs to count tokens or estimate cost before/after generating content. --- ## Quick Reference | Item | Command | |------|---------| | Invoke | `python -m scripts.cli` | | List models | `python -m scripts.cli -l` | | Cost | `-c` (input) / `--output-tokens` (output) | | Currency | `--currency USD` or `INR` |

path: package.json
content: {
  "name": "prompt-token-counter",
  "version": "1.0.10",
  "description": "OpenClaw skill: Count tokens and estimate costs for 300+ LLM models. Audit workspace token consumption (memory, persona, skills).",
  "keywords": [
    "openclaw",
    "skill",
    "token",
    "toksum",
    "llm"
  ],
  "license": "MIT",
  "repository": {
    "type": "git",
    "url": "https://github.com/Zhaobudaoyuema/prompt-token-counter"
  },
  "homepage": "https://github.com/Zhaobudaoyuema/prompt-token-counter",
  "bugs": {
    "url": "https://github.com/Zhaobudaoyuema/prompt-token-counter/issues"
  },
  "files": [
    "SKILL.md",
    "publish_npm.py",
    "scripts"
  ]
}

path: README.md
content: # prompt-token-counter

[中文](README.zh.md)

An **OpenClaw skill** for counting tokens and estimating API costs.

## How It Works

This project provides **local token counting** without API calls:

| Approach | OpenAI | Other models |
|----------|--------|--------------|
| **Method** | tiktoken (exact) | Formula-based approximation |
| **Accuracy** | Exact | ~85–95% typical |

For non-OpenAI models, we use **provider-specific formulas** calibrated from benchmark data. When text contains CJK (Chinese/Japanese/Korean), we **blend the ratio** by CJK character ratio so that Chinese-heavy text gets fewer chars-per-token (more tokens per character).

### Benchmark: Main Models (8927 chars, English/mixed)

| Model | Chars | Tokens | 1 token ≈ chars | Status |
|-------|-------|--------|-----------------|--------|
| anthropic/claude-sonnet-4-6 | 8927 | 2763 | 3.23 | ✓ |
| anthropic/claude-sonnet-4-5 | 8927 | 2763 | 3.23 | ✓ |
| anthropic/claude-opus-4.6 | 8927 | 2763 | 3.23 | ✓ |
| openai/gpt-5.2-codex | 8927 | 2459 | 3.63 | ✓ |
| google/gemini-3.1-pro-preview | 8927 | 2627 | 3.40 | ✓ |
| z-ai/glm-5 | 8927 | 2457 | 3.63 | ✓ |
| volcengine/doubao-seed-2-0-pro | 8927 | 2702 | 3.30 | ✓ |
| moonshot/kimi-k2.5 | 8927 | 2402 | 3.72 | ✓ |
| minimax/MiniMax-M2.5 | 8927 | 2428 | 3.68 | ✓ |
| deepseek-v3.2 | 8927 | 2578 | 3.46 | ✓ |

### Benchmark: Main Models (3050 chars, Chinese-mixed 混杂中文)

| Model | Chars | Tokens | 1 token ≈ chars | Status |
|-------|-------|--------|-----------------|--------|
| anthropic/claude-sonnet-4-6 | 3050 | 1913 | 1.59 | ✓ |
| anthropic/claude-sonnet-4-5 | 3050 | 1913 | 1.59 | ✓ |
| anthropic/claude-opus-4.6 | 3050 | 1913 | 1.59 | ✓ |
| openai/gpt-5.2-codex | 3050 | 1564 | 1.95 | ✓ |
| google/gemini-3.1-pro-preview | 3050 | 1473 | 2.07 | ✓ |
| z-ai/glm-5 | 3050 | 1318 | 2.31 | ✓ |
| volcengine/doubao-seed-2-0-pro | 3050 | 1494 | 2.04 | ✓ |
| moonshot/kimi-k2.5 | 3050 | 1257 | 2.43 | ✓ |
| minimax/MiniMax-M2.5 | 3050 | 1289 | 2.37 | ✓ |
| deepseek-v3.2 | 3050 | 1361 | 2.24 | ✓ |

*Benchmark data from `scripts/examples/benchmark_token_ratio.py` (API mode).*

## What This Skill Does

When loaded, the agent can:

| Capability | Use case |
|------------|----------|
| **Count tokens** | "How many tokens in this prompt?", "Token length of X" |
| **Estimate cost** | "How much for this text on GPT-4?", "API cost for Claude" |
| **Audit OpenClaw workspace** | "How many tokens does my workspace use?", "Which memory/persona/skills consume tokens?" |
| **Compare models** | "Compare token cost across models", "Which model is cheaper?" |

### OpenClaw Token Audit

The skill helps identify token consumption of workspace components:

- **Memory & persona**: AGENTS.md, SOUL.md, IDENTITY.md, USER.md, MEMORY.md, TOOLS.md, etc.
- **Skills**: Each SKILL.md under `~/.openclaw/skills/` or `workspace/skills/`

Example audit (batch mode, multiple files):
```bash
python -m scripts.cli -m gpt-4o -c AGENTS.md SOUL.md MEMORY.md
```

## When to Trigger

- User asks about token count, prompt length, API cost
- User mentions OpenClaw context size or workspace token usage
- Agent needs to audit token consumption before/after changes

## Copy and send to OpenClaw

**ClawHub (recommended, international)**
```text
Please run clawhub install prompt-token-counter to install this skill, and help me count tokens and estimate API costs.
```

**npm**
```text
Please run npm i prompt-token-counter to install this skill, and help me count tokens and estimate API costs.
```

## Quick Reference

```bash
python -m scripts.cli -m gpt-4 "Hello, world!"
python -m scripts.cli -f input.txt -m claude-3-opus -c
python -m scripts.cli -l   # list 300+ models
```

## Benchmark Script

Run `scripts/examples/benchmark_token_ratio.py` to test token ratios across models:

- **API mode** (default): Uses model API to get exact `prompt_tokens`. Set `API_KEY` and `BASE_URL` in the script.
- **Local mode** (`--local`): Uses this project's TokenCounter (no API). Good for quick comparison.

```bash
python scripts/examples/benchmark_token_ratio.py           # API mode
python scripts/examples/benchmark_token_ratio.py --local   # local approximation
```

MIT


path: README.zh.md
content: # prompt-token-counter

[English](README.md)

一个 **OpenClaw skill**，用于统计 token 数量并估算 API 成本。

## 实现原理

本项目提供**本地 token 计数**，无需调用 API：

| 方式 | OpenAI | 其他模型 |
|------|--------|----------|
| **方法** | tiktoken（精确） | 公式近似 |
| **精度** | 精确 | 约 85–95% |

对非 OpenAI 模型，使用**按 provider 校准的公式**。当文本包含 CJK（中/日/韩）时，会根据 CJK 字符占比**混合比例**，使中文为主的文本获得更低的 chars/token（每字符更多 token）。

### 基准测试：主力模型（8927 字符，英文/混合）

| 模型 | 字符数 | Token 数 | 1 token ≈ 多少字符 | 状态 |
|------|--------|----------|-------------------|------|
| anthropic/claude-sonnet-4-6 | 8927 | 2763 | 3.23 | ✓ |
| anthropic/claude-sonnet-4-5 | 8927 | 2763 | 3.23 | ✓ |
| anthropic/claude-opus-4.6 | 8927 | 2763 | 3.23 | ✓ |
| openai/gpt-5.2-codex | 8927 | 2459 | 3.63 | ✓ |
| google/gemini-3.1-pro-preview | 8927 | 2627 | 3.40 | ✓ |
| z-ai/glm-5 | 8927 | 2457 | 3.63 | ✓ |
| volcengine/doubao-seed-2-0-pro | 8927 | 2702 | 3.30 | ✓ |
| moonshot/kimi-k2.5 | 8927 | 2402 | 3.72 | ✓ |
| minimax/MiniMax-M2.5 | 8927 | 2428 | 3.68 | ✓ |
| deepseek-v3.2 | 8927 | 2578 | 3.46 | ✓ |

### 基准测试：主力模型（3050 字符，混杂中文）

| 模型 | 字符数 | Token 数 | 1 token ≈ 多少字符 | 状态 |
|------|--------|----------|-------------------|------|
| anthropic/claude-sonnet-4-6 | 3050 | 1913 | 1.59 | ✓ |
| anthropic/claude-sonnet-4-5 | 3050 | 1913 | 1.59 | ✓ |
| anthropic/claude-opus-4.6 | 3050 | 1913 | 1.59 | ✓ |
| openai/gpt-5.2-codex | 3050 | 1564 | 1.95 | ✓ |
| google/gemini-3.1-pro-preview | 3050 | 1473 | 2.07 | ✓ |
| z-ai/glm-5 | 3050 | 1318 | 2.31 | ✓ |
| volcengine/doubao-seed-2-0-pro | 3050 | 1494 | 2.04 | ✓ |
| moonshot/kimi-k2.5 | 3050 | 1257 | 2.43 | ✓ |
| minimax/MiniMax-M2.5 | 3050 | 1289 | 2.37 | ✓ |
| deepseek-v3.2 | 3050 | 1361 | 2.24 | ✓ |

*基准数据来自 `scripts/examples/benchmark_token_ratio.py`（API 模式）。*

## 这个 Skill 能做什么

加载后，Agent 可以：

| 能力 | 使用场景 |
|------|----------|
| **统计 token** | 「这段 prompt 有多少 token？」、「X 的 token 长度」 |
| **估算成本** | 「这段文字用 GPT-4 要多少钱？」、「Claude 的 API 成本」 |
| **审计 OpenClaw 工作区** | 「我的工作区用了多少 token？」、「哪些 memory/persona/skills 消耗 token？」 |
| **对比模型** | 「对比不同模型的 token 成本」、「哪个模型更便宜？」 |

### OpenClaw Token 审计

该 skill 帮助识别工作区各组件的 token 消耗：

- **Memory 与 persona**：AGENTS.md、SOUL.md、IDENTITY.md、USER.md、MEMORY.md、TOOLS.md 等
- **Skills**：`~/.openclaw/skills/` 或 `workspace/skills/` 下的每个 SKILL.md

审计示例（批量多文件）：
```bash
python -m scripts.cli -m gpt-4o -c AGENTS.md SOUL.md MEMORY.md
```

## 何时触发

- 用户询问 token 数量、prompt 长度、API 成本
- 用户提到 OpenClaw 上下文大小或工作区 token 使用
- Agent 需要在变更前后审计 token 消耗

## 复制发送给 OpenClaw

**ClawHub（推荐，国际）**
```text
Please run clawhub install prompt-token-counter to install this skill, and help me count tokens and estimate API costs.
```

**npm**
```text
Please run npm i prompt-token-counter to install this skill, and help me count tokens and estimate API costs.
```

## 快速参考

```bash
python -m scripts.cli -m gpt-4 "Hello, world!"
python -m scripts.cli -f input.txt -m claude-3-opus -c
python -m scripts.cli -l   # 列出 300+ 模型
```

## 基准测试脚本

运行 `scripts/examples/benchmark_token_ratio.py` 可批量测试各模型的 token 比例：

- **API 模式**（默认）：调用模型 API 获取精确 `prompt_tokens`。需在脚本中设置 `API_KEY` 和 `BASE_URL`。
- **本地模式**（`--local`）：使用本项目的 TokenCounter 近似计算，无需 API。

```bash
python scripts/examples/benchmark_token_ratio.py           # API 模式
python scripts/examples/benchmark_token_ratio.py --local   # 本地近似
```

MIT


path: scripts/cli.py
content: """
Command-line interface for scripts.

This module provides a comprehensive command-line interface for the scripts library,
allowing users to count tokens and estimate costs for various LLM models directly
from the terminal.

The CLI supports:
    - Token counting for text input or files
    - Cost estimation with detailed breakdowns
    - Listing all supported models by provider
    - Verbose output with detailed information
    - Support for both input and output token pricing

Examples:
    Basic token counting:
    
    .. code-block:: bash
    
        scripts "Hello, world!" gpt-4
        scripts --file input.txt claude-3-opus-20240229
    
    Cost estimation:
    
    .. code-block:: bash
    
        scripts --cost "Your text here" gpt-4
        scripts --cost --output-tokens "Response text" gpt-4
    
    List supported models:
    
    .. code-block:: bash
    
        scripts --list-models
    
    Verbose output:
    
    .. code-block:: bash
    
        scripts --verbose --cost --file large_document.txt gpt-4

Functions:
    main: Main CLI entry point that handles argument parsing and execution
    list_models: Display all supported models organized by provider

The CLI provides comprehensive error handling and user-friendly output formatting
for both simple token counting and detailed cost analysis workflows.
"""

import argparse
import sys
import urllib.parse
import urllib.request
from pathlib import Path
from typing import List, Optional, Tuple

from .core import TokenCounter, get_supported_models, estimate_cost
from .exceptions import UnsupportedModelError, TokenizationError


def _read_file(path: str) -> str:
    with open(path, "r", encoding="utf-8") as file_handle:
        return file_handle.read()


def _read_url(url: str) -> str:
    parsed = urllib.parse.urlparse(url)
    if parsed.scheme not in {"http", "https"}:
        raise ValueError("URL must start with http:// or https://")
    with urllib.request.urlopen(url, timeout=10) as response:
        return response.read().decode("utf-8", errors="replace")


def _collect_inputs(argv: List[str]) -> List[Tuple[str, str, Optional[str]]]:
    """Collect inputs. Returns list of (source, value, label) where label is path for display."""
    items: List[Tuple[str, str, Optional[str]]] = []
    i = 0
    while i < len(argv):
        arg = argv[i]
        if arg in {"--file", "-f"}:
            if i + 1 >= len(argv):
                raise ValueError("--file requires a path")
            path = argv[i + 1]
            items.append(("file", path, path))
            i += 2
            continue
        if arg in {"--url", "-u"}:
            if i + 1 >= len(argv):
                raise ValueError("--url requires a URL")
            items.append(("url", argv[i + 1], argv[i + 1]))
            i += 2
            continue
        if arg in {"--model", "-m", "--currency"}:
            i += 2
            continue
        if arg in {"--cost", "-c", "--output-tokens", "--verbose", "-v", "--list-models", "-l"}:
            i += 1
            continue
        if arg.startswith("-"):
            i += 1
            continue
        # Positional: treat as file path if it exists, else inline text
        p = Path(arg)
        if p.is_file():
            items.append(("file", arg, arg))
        else:
            items.append(("text", arg, None))
        i += 1
    return items


def _format_cost(cost: float, currency: str) -> str:
    if cost <= 0:
        return "NA"
    symbol = "₹" if currency.upper() == "INR" else "$"
    precision = 2 if currency.upper() == "INR" else 6
    return f"{symbol}{cost:.{precision}f}"


def main() -> None:
    """
    Main CLI entry point.
    
    Parses command-line arguments and executes the appropriate scripts functionality.
    Supports token counting, cost estimation, model listing, and file input processing.
    
    The function handles:
        - Argument parsing and validation
        - Text input from command line or file
        - Token counting for specified models
        - Cost estimation with input/output token differentiation
        - Model listing with provider organization
        - Comprehensive error handling and user feedback
        - Verbose output formatting
    
    Command-line Arguments:
        text (str, optional): Text to count tokens for
        model (str, optional): Model name (required unless using --list-models)
        --file, -f (str): Read text from file instead of command line
        --list-models, -l: List all supported models by provider
        --cost, -c: Show cost estimation along with token count
        --output-tokens: Calculate cost for output tokens instead of input
        --verbose, -v: Show detailed output with additional information
    
    Exit Codes:
        0: Success
        1: Error (unsupported model, file not found, tokenization failure, etc.)
    
    Raises:
        SystemExit: On error conditions or user interruption
    
    Examples:
        Basic usage:
        
        .. code-block:: bash
        
            scripts "Hello, world!" gpt-4
            scripts --file document.txt claude-3-opus-20240229
        
        With cost estimation:
        
        .. code-block:: bash
        
            scripts --cost --verbose "Long text content" gpt-4
            scripts --cost --output-tokens "Response text" gpt-4
        
        List models:
        
        .. code-block:: bash
        
            scripts --list-models
    """
    parser = argparse.ArgumentParser(
        description="Count tokens for various LLM models",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  scripts file1.txt file2.txt --model gpt-4
  scripts --file input.txt --model claude-3-opus-20240229
  scripts --cost --model gpt-4 AGENTS.md SOUL.md MEMORY.md
  scripts --list-models
        """
    )

    parser.add_argument(
        "paths",
        nargs="*",
        metavar="FILE",
        help="Local file path(s) to count tokens (default input mode). Multiple paths supported for batch."
    )

    parser.add_argument(
        "--model", "-m",
        help="Model name (e.g., gpt-4, claude-3-opus-20240229)"
    )

    parser.add_argument(
        "--file", "-f",
        action="append",
        help="Read text from file (can be used multiple times)"
    )

    parser.add_argument(
        "--url", "-u",
        action="append",
        help="Read text from URL (http/https, can be used multiple times)"
    )

    parser.add_argument(
        "--list-models", "-l",
        action="store_true",
        help="List all supported models"
    )

    parser.add_argument(
        "--cost", "-c",
        action="store_true",
        help="Show cost estimation along with token count"
    )

    parser.add_argument(
        "--currency",
        default="USD",
        help="Currency for cost output (USD or INR)"
    )

    parser.add_argument(
        "--output-tokens",
        action="store_true",
        help="Calculate cost for output tokens instead of input tokens"
    )

    parser.add_argument(
        "--verbose", "-v",
        action="store_true",
        help="Show verbose output"
    )

    args = None

    try:
        args = parser.parse_args()
        if args.list_models:
            list_models()
            return
        
        if not args.model:
            parser.error("Model name is required unless using --list-models")

        inputs = _collect_inputs(sys.argv[1:])
        # Also add positional paths from argparse (in case -m was before paths)
        for p in (args.paths or []):
            if Path(p).is_file():
                if not any(src == "file" and val == p for src, val, _ in inputs):
                    inputs.append(("file", p, p))
            else:
                if not any(src == "text" and val == p for src, val, _ in inputs):
                    inputs.append(("text", p, None))

        if not inputs:
            parser.error("Provide local file path(s), --file, or --url inputs")

        items: List[Tuple[str, str, Optional[str]]] = []
        for source, value, label in inputs:
            if source == "text":
                items.append((source, value, label))
                continue
            try:
                if source == "file":
                    content = _read_file(value)
                    items.append((source, content, value))
                elif source == "url":
                    content = _read_url(value)
                    items.append((source, content, value))
                else:
                    continue
            except FileNotFoundError:
                print(f"Error: File '{value}' not found", file=sys.stderr)
                sys.exit(1)
            except Exception as e:
                print(f"Error reading {source} '{value}': {e}", file=sys.stderr)
                sys.exit(1)

        try:
            counter = TokenCounter(args.model)
            for index, (source, content, label) in enumerate(items, start=1):
                token_count = counter.count(content)
                prefix = f"{label}: " if label else f"{index}) "
                if args.cost:
                    cost = estimate_cost(
                        token_count,
                        args.model,
                        input_tokens=not args.output_tokens,
                        currency=args.currency,
                    )
                    cost_display = _format_cost(cost, args.currency)
                    print(f"{prefix}tokens={token_count} cost={cost_display}")
                else:
                    print(f"{prefix}tokens={token_count}")

        except UnsupportedModelError as e:
            print(f"Error: {e}", file=sys.stderr)
            if args.verbose:
                print("\nUse --list-models to see supported models", file=sys.stderr)
            sys.exit(1)

        except TokenizationError as e:
            print(f"Error: {e}", file=sys.stderr)
            sys.exit(1)
    
    except KeyboardInterrupt:
        print("\nInterrupted by user", file=sys.stderr)
        sys.exit(1)
    
    except Exception as e:
        print(f"Unexpected error: {e}", file=sys.stderr)
        if args is not None and args.verbose:
            import traceback
            traceback.print_exc()
        sys.exit(1)


def list_models() -> None:
    """
    List all supported models organized by provider.
    
    Displays a comprehensive list of all supported models grouped by their
    respective providers (OpenAI, Anthropic, Google, Meta, etc.). The output
    includes model counts per provider and a total count across all providers.
    
    The function:
        - Retrieves all supported models using get_supported_models()
        - Groups models by provider with clear section headers
        - Sorts models alphabetically within each provider
        - Shows model counts for each provider and overall total
        - Formats output for easy readability
    
    Output Format:
        .. code-block:: text
        
            Supported models:
            ==================================================
            
            OPENAI (25 models):
            ------------------------------
              gpt-3.5-turbo
              gpt-4
              gpt-4o
              ...
            
            ANTHROPIC (12 models):
            ------------------------------
              claude-3-haiku-20240307
              claude-3-opus-20240229
              ...
            
            Total: 200+ models
    
    Note:
        This function is typically called when the --list-models CLI flag is used.
        It provides users with a complete overview of available models for token
        counting and cost estimation.
    """
    models = get_supported_models()
    
    print("Supported models:")
    print("=" * 50)
    
    for provider, model_list in models.items():
        print(f"\n{provider.upper()} ({len(model_list)} models):")
        print("-" * 30)
        
        for model in sorted(model_list):
            print(f"  {model}")
    
    print(f"\nTotal: {sum(len(models) for models in models.values())} models")


if __name__ == "__main__":
    main()


path: scripts/core.py
content: """
Core functionality for token counting across different LLM providers.

This module contains the main TokenCounter class and supporting functions that provide
token counting capabilities for 200+ Large Language Models from 25+ providers.

The module implements:
    - **Precise tokenization** for OpenAI models using the tiktoken library
    - **Intelligent approximation algorithms** for all other providers
    - **Provider detection** with case-insensitive model name matching
    - **Message format support** for chat-based interactions
    - **Comprehensive error handling** with detailed error messages
    - **Cost estimation** for supported models with USD/INR currency support

Key Components:
    - :class:`TokenCounter`: Main class for token counting operations
    - :func:`count_tokens`: Convenience function for quick token counting
    - :func:`get_supported_models`: Returns all supported models by provider
    - :func:`estimate_cost`: Calculates estimated costs for token usage

Provider Support:
    The module supports models from major providers including:
    
    - **OpenAI**: GPT-4, GPT-3.5, GPT-4o, O1 models, embeddings
    - **Anthropic**: Claude 3/3.5 (Opus, Sonnet, Haiku), Claude 2, Instant
    - **Google**: Gemini Pro/Flash, Gemini 1.5/2.0, PaLM models
    - **Meta**: LLaMA 2/3/3.1/3.2/3.3 in various sizes
    - **Mistral**: Mistral 7B, Mixtral, Mistral Large variants
    - **Cohere**: Command, Command-R, Command-R+ models
    - **xAI**: Grok 1/1.5/2 and beta models
    - **Chinese providers**: Alibaba Qwen, Baidu ERNIE, Huawei PanGu, Tsinghua ChatGLM
    - **Code-specialized**: DeepSeek Coder, Replit Code, BigCode StarCoder
    - **Open source**: EleutherAI, Stability AI, TII Falcon, RWKV
    - **Enterprise**: Databricks DBRX, Microsoft Phi, Amazon Titan, IBM Granite

Tokenization Approach:
    - **OpenAI models**: Uses official tiktoken encodings (cl100k_base, p50k_base, r50k_base)
    - **Other providers**: Intelligent approximation based on:
        - Character count analysis
        - Whitespace and punctuation detection
        - Provider-specific adjustment factors
        - Language-optimized calculations (Chinese, Russian, etc.)

The approximation algorithms are calibrated to provide reasonable accuracy for:
    - Cost estimation and budgeting
    - Rate limit planning
    - Content length assessment
    - Comparative analysis across providers

Note:
    For production applications requiring exact token counts, use OpenAI models
    with tiktoken. For other providers, the approximations are suitable for
    planning and estimation purposes.
"""

import re
from typing import Dict, List, Optional, Union, TYPE_CHECKING, Any

if TYPE_CHECKING:
    import tiktoken
    from anthropic import Anthropic
else:
    try:
        import tiktoken
    except ImportError:
        tiktoken = None

    try:
        from anthropic import Anthropic
    except ImportError:
        Anthropic = None

from .exceptions import UnsupportedModelError, TokenizationError
from .registry.models import (
    ModelInfo,
    get_all_supported_models,
    get_model_info,
    get_supported_models_by_provider,
    normalize_model_name,
)
from .registry.pricing import USD_TO_INR, get_pricing_rate


DEFAULT_FORMULA = (4.0, 0.3)
FORMULAS = {
    "anthropic": (4.0, 0.3),
    "google": (3.8, 0.25),
    "meta": (3.5, 0.2),
    "mistral": (3.7, 0.25),
    "cohere": (4.2, 0.3),
    "perplexity": (3.6, 0.2),
    "huggingface": (4.0, 0.25),
    "ai21": (3.8, 0.25),
    "together": (3.9, 0.25),
    "xai": (3.8, 0.25),
    "alibaba": (3.2, 0.2),
    "baidu": (3.3, 0.2),
    "huawei": (3.4, 0.2),
    "yandex": (3.6, 0.2),
    "stability": (3.8, 0.25),
    "tii": (3.7, 0.25),
    "eleutherai": (3.6, 0.2),
    "mosaicml": (3.7, 0.25),
    "replit": (3.5, 0.2),
    "minimax": (3.3, 0.2),
    "aleph_alpha": (3.9, 0.25),
    "deepseek": (3.6, 0.2),
    "tsinghua": (3.2, 0.2),
    "rwkv": (3.8, 0.25),
    "community": (3.6, 0.2),
    "microsoft": (3.7, 0.25),
    "amazon": (3.9, 0.25),
    "nvidia": (3.6, 0.2),
    "ibm": (3.8, 0.25),
    "salesforce": (3.5, 0.2),
    "bigcode": (3.4, 0.2),
    "databricks": (4.0, 0.25),
    "voyage": (3.8, 0.25),
    "volcengine": (3.3, 0.2),
    "moonshot": (3.7, 0.2),
    "glm5": (3.63, 0.15),
    "minimax_m25": (3.68, 0.15),
}

# Chars per token for CJK-heavy text (3050 chars benchmark). Used when cjk_ratio > 0.2.
CJK_RATIO = {
    "anthropic": 1.59,
    "google": 2.07,
    "volcengine": 2.04,
    "moonshot": 2.43,
    "glm5": 2.31,
    "minimax_m25": 2.37,
    "minimax": 2.37,
    "deepseek": 2.24,
}
DEFAULT_CJK_RATIO = 2.0


def _cjk_ratio(text: str) -> float:
    """Return fraction of CJK characters (0..1). CJK chars tokenize ~1 char/token vs ~4 for English."""
    if not text:
        return 0.0
    cjk = sum(1 for c in text if "\u4e00" <= c <= "\u9fff" or "\u3400" <= c <= "\u4dbf" or "\u3000" <= c <= "\u303f")
    return cjk / len(text)


class TokenCounter:
    """
    A comprehensive token counter for various Large Language Model (LLM) providers.

    This class provides functionality to count tokens for 200+ different LLMs from 25+ providers,
    including OpenAI, Anthropic, Google, Meta, Mistral, and many others. It supports both 
    individual text strings and lists of messages (for chat-like interactions).

    The token counting is precise for OpenAI models using the official tiktoken library, 
    and provides reasonable approximations for other providers using intelligent algorithms
    calibrated for each provider's tokenization characteristics.

    Attributes:
        model (str): The model name (converted to lowercase)
        provider (str): The detected provider name
        tokenizer (Optional[Any]): The tokenizer instance (tiktoken for OpenAI, None for others)

    Supported Providers:
        - **OpenAI**: GPT-4, GPT-3.5, GPT-4o, O1 models, embeddings (25+ models)
        - **Anthropic**: Claude 3/3.5 (Opus, Sonnet, Haiku), Claude 2, Instant (12+ models)
        - **Google**: Gemini Pro/Flash, Gemini 1.5/2.0, PaLM (10+ models)
        - **Meta**: LLaMA 2/3/3.1/3.2/3.3 in various sizes (15+ models)
        - **Mistral**: Mistral 7B, Mixtral, Mistral Large variants (10+ models)
        - **Cohere**: Command, Command-R, Command-R+ (8+ models)
        - **xAI**: Grok 1/1.5/2 and beta models (4+ models)
        - **Alibaba**: Qwen 1.5/2.0/2.5 and vision models (20+ models)
        - **Baidu**: ERNIE 3.0/3.5/4.0 and variants (8+ models)
        - **Huawei**: PanGu Alpha and Coder models (5+ models)
        - **Yandex**: YaLM and YaGPT models (4+ models)
        - **DeepSeek**: Coder, VL, and LLM models (8+ models)
        - **Tsinghua**: ChatGLM and GLM models (5+ models)
        - **And 15+ more providers with specialized models**

    Examples:
        Basic usage:

        .. code-block:: python

            # Count tokens for a single text string
            counter = TokenCounter("gpt-4")
            token_count = counter.count("This is a test string.")
            print(f"Token count: {token_count}")

        Chat message format:

        .. code-block:: python

            # Count tokens for a list of messages (chat format)
            messages = [
                {"role": "user", "content": "Hello!"},
                {"role": "assistant", "content": "How can I help you?"},
            ]
            token_count = counter.count_messages(messages)
            print(f"Token count (messages): {token_count}")

        Different providers:

        .. code-block:: python

            # Compare token counts across providers
            models = ["gpt-4", "claude-3-opus", "gemini-pro", "llama-3-70b"]
            text = "Compare tokenization across different models."
            
            for model in models:
                counter = TokenCounter(model)
                tokens = counter.count(text)
                print(f"{model}: {tokens} tokens")

        Cost estimation:

        .. code-block:: python

            from scripts.core import estimate_cost
            
            counter = TokenCounter("gpt-4")
            tokens = counter.count("Your text here")
            cost = estimate_cost(tokens, "gpt-4", input_tokens=True)
            print(f"Estimated cost: ${cost:.4f}")

    Tokenization Accuracy:
        - **OpenAI models**: Exact token counts using official tiktoken encodings
        - **Other providers**: Approximations with typical accuracy of ±10-20%
        - **Approximation factors**: Calibrated per provider based on tokenization patterns
        - **Language optimization**: Adjusted for Chinese, Russian, and other languages

    Note:
        For production applications requiring exact token counts, use OpenAI models.
        For other providers, approximations are suitable for cost estimation,
        rate limit planning, and comparative analysis.

    Raises:
        UnsupportedModelError: If the specified model is not supported
        TokenizationError: If tokenization fails or required dependencies are missing
    """
    def __init__(self, model: str):
        """
        Initialize the TokenCounter with a specific model.
        
        Sets up the appropriate tokenizer based on the model's provider. For OpenAI models,
        initializes the tiktoken tokenizer with the correct encoding. For other providers,
        sets up approximation-based token counting.

        Args:
            model (str): The model name (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro').
                        Model names are case-insensitive and will be converted to lowercase.

        Raises:
            UnsupportedModelError: If the model is not supported. The exception includes
                                 a list of all supported models for reference.
            TokenizationError: If required dependencies are missing (e.g., tiktoken for OpenAI models)
                             or if tokenizer initialization fails.

        Examples:
            .. code-block:: python

                # OpenAI model (requires tiktoken)
                counter = TokenCounter("gpt-4")
                
                # Anthropic model (uses approximation)
                counter = TokenCounter("claude-3-opus-20240229")
                
                # Case-insensitive model names
                counter = TokenCounter("GPT-4")  # Same as "gpt-4"
                
                # Google model
                counter = TokenCounter("gemini-pro")
                
                # Meta model
                counter = TokenCounter("llama-3-70b")

        Note:
            The constructor automatically detects the provider based on the model name
            and sets up the appropriate tokenization method. OpenAI models use precise
            tiktoken-based counting, while other providers use calibrated approximations.
        """
        self.tokenizer: Optional[Any] = None
        self.model = normalize_model_name(model)
        self.model_info = self._get_model_info()
        self.provider = self.model_info.provider
        self._setup_tokenizer()

    def _get_model_info(self) -> ModelInfo:
        model_info = get_model_info(self.model)
        if not model_info:
            raise UnsupportedModelError(self.model, get_all_supported_models())
        return model_info

    def _detect_provider(self) -> str:
        return self.model_info.provider

    def _setup_tokenizer(self) -> None:
        """
        Setup the appropriate tokenizer for the model based on its provider.

        For OpenAI models, initializes the tiktoken tokenizer with the correct encoding
        (cl100k_base, p50k_base, or r50k_base). For all other providers, sets the
        tokenizer to None to indicate approximation-based counting will be used.

        OpenAI Encodings:
            - **cl100k_base**: GPT-4, GPT-3.5-turbo, GPT-4o, embeddings (most models)
            - **p50k_base**: text-davinci-003, text-davinci-002 (legacy completion models)
            - **r50k_base**: GPT-3, davinci, curie, babbage, ada (oldest models)

        Raises:
            TokenizationError: If tiktoken is not installed for OpenAI models, or if
                             the tokenizer fails to initialize for any reason.

        Examples:
            The tokenizer setup is automatic and transparent:

            .. code-block:: python

                # OpenAI model - sets up tiktoken with cl100k_base encoding
                counter = TokenCounter("gpt-4")

                # Anthropic model - sets tokenizer to None for approximation
                counter = TokenCounter("claude-3-opus")

        Note:
            This method is called automatically during initialization. Users should not
            call this method directly. The tokenizer instance is stored in self.tokenizer
            and used by the count() method.

        Dependencies:
            - **tiktoken**: Required for OpenAI models. Install with: ``pip install tiktoken``
            - **No dependencies**: Required for other providers (uses built-in approximation)
        """
        if self.provider == "openai":
            if tiktoken is None:
                raise TokenizationError(
                    "tiktoken is required for OpenAI models. Install with: pip install tiktoken",
                    model=self.model
                )

            encoding_name = self.model_info.encoding or "cl100k_base"
            try:
                self.tokenizer = tiktoken.get_encoding(encoding_name)
            except Exception as e:
                raise TokenizationError(f"Failed to load tokenizer: {str(e)}", model=self.model)
        else:
            # For all other providers, we'll use approximation since they don't provide public tokenizers
            self.tokenizer = None
    
    def count(self, text: str) -> int:
        """
        Count tokens in the given text.
        
        Performs token counting using the appropriate method for the model's provider.
        For OpenAI models, uses precise tiktoken-based counting. For other providers,
        uses intelligent approximation algorithms calibrated for each provider.

        Args:
            text (str): The text to count tokens for. Must be a string.

        Returns:
            int: The number of tokens in the text. Returns 0 for empty strings.

        Raises:
            TokenizationError: If tokenization fails, input is invalid, or required
                             dependencies are missing. Includes detailed error context
                             with model name and text preview.

        Input Validation:
            The method performs comprehensive input validation:
            
            - **None check**: Rejects None input with clear error message
            - **Type check**: Ensures input is a string, not int/float/list/dict/etc.
            - **Empty string**: Returns 0 for empty strings (valid case)

        Tokenization Methods:
            - **OpenAI models**: Uses tiktoken.encode() for exact token counts
            - **Other providers**: Uses _approximate_tokens() with provider-specific calibration

        Provider-Specific Accuracy:
            - **OpenAI**: 100% accurate (official tokenizer)
            - **Anthropic**: ~90-95% accurate (well-calibrated approximation)
            - **Google**: ~85-90% accurate (Gemini-optimized approximation)
            - **Meta**: ~85-90% accurate (LLaMA-optimized approximation)
            - **Chinese models**: ~80-90% accurate (character-optimized for Chinese)
            - **Code models**: ~85-95% accurate (code-pattern optimized)
            - **Other providers**: ~80-90% accurate (general approximation)

        Examples:
            Basic usage:

            .. code-block:: python

                counter = TokenCounter("gpt-4")
                
                # Simple text
                tokens = counter.count("Hello, world!")
                print(f"Tokens: {tokens}")  # Exact count for OpenAI
                
                # Empty string
                tokens = counter.count("")
                print(f"Tokens: {tokens}")  # Always returns 0
                
                # Longer text
                text = "This is a longer text that will be tokenized."
                tokens = counter.count(text)
                print(f"Tokens: {tokens}")

            Comparing providers:

            .. code-block:: python

                text = "Compare tokenization across different models."
                models = ["gpt-4", "claude-3-opus", "gemini-pro"]
                
                for model in models:
                    counter = TokenCounter(model)
                    tokens = counter.count(text)
                    print(f"{model}: {tokens} tokens")

            Error handling:

            .. code-block:: python

                try:
                    counter = TokenCounter("gpt-4")
                    tokens = counter.count("Valid text")
                except TokenizationError as e:
                    print(f"Tokenization failed: {e}")

        Performance:
            - **OpenAI models**: Fast (native tiktoken performance)
            - **Other providers**: Very fast (lightweight approximation algorithms)
            - **Typical speed**: 10,000+ texts per second for approximation methods

        Note:
            For production applications requiring exact token counts, use OpenAI models.
            For cost estimation, rate limiting, and comparative analysis, approximations
            provide sufficient accuracy with much better performance.
        """
        # Comprehensive input validation
        if text is None:
            raise TokenizationError("Input cannot be None", model=self.model)
        
        if not isinstance(text, str):
            # Handle common invalid types explicitly
            if isinstance(text, (int, float, bool)):
                raise TokenizationError(f"Input must be a string, got {type(text).__name__}", model=self.model)
            elif isinstance(text, (list, tuple, dict, set)):
                raise TokenizationError(f"Input must be a string, got {type(text).__name__}", model=self.model)
            else:
                raise TokenizationError("Input must be a string", model=self.model)
        
        # Handle empty string case
        if not text:
            return 0
        
        try:
            if self.provider == "openai":
                if self.tokenizer is None:
                    raise TokenizationError("Tokenizer not initialized", model=self.model)
                if not text:
                    return 0
                return len(self.tokenizer.encode(text))
            else:
                # Use approximation for all other providers
                return self._approximate_tokens(text)
        except TokenizationError:
            # Re-raise TokenizationError as-is
            raise
        except Exception as e:
            raise TokenizationError(str(e), model=self.model, text_preview=text)
    
    def _approximate_tokens(self, text: str) -> int:
        """
        Approximate token count for non-OpenAI models.
        
        Uses intelligent approximation algorithms calibrated for each provider's
        tokenization characteristics. The approximation considers character count,
        whitespace patterns, punctuation density, and provider-specific factors.

        Args:
            text (str): The text to approximate tokens for. Must be a string.

        Returns:
            int: Approximated number of tokens. Minimum return value is 1 for non-empty text.

        Raises:
            TokenizationError: If text processing fails or input is invalid.

        Algorithm Components:
            The approximation algorithm analyzes several text characteristics:
            
            1. **Character count**: Base measurement for token estimation
            2. **Whitespace analysis**: Spaces and newlines often become separate tokens
            3. **Punctuation analysis**: Special characters frequently tokenize separately
            4. **Provider calibration**: Adjustment factors based on tokenizer characteristics

        Provider-Specific Calibrations:
            Each provider has calibrated ratios based on empirical analysis:

            **Major Providers:**
            
            - **Anthropic**: ~4 chars/token (Claude guidance), +30% punctuation adjustment
            - **Google**: ~3.8 chars/token (Gemini-optimized), +25% adjustment
            - **Meta**: ~3.5 chars/token (LLaMA-optimized), +20% adjustment
            - **Mistral**: ~3.7 chars/token (GPT-similar), +25% adjustment
            - **Cohere**: ~4.2 chars/token (conservative), +30% adjustment

            **Regional/Language-Optimized:**
            
            - **Alibaba/Qwen**: ~3.2 chars/token (Chinese-optimized)
            - **Baidu/ERNIE**: ~3.3 chars/token (Chinese-optimized)
            - **Huawei/PanGu**: ~3.4 chars/token (Chinese-optimized)
            - **Yandex/YaLM**: ~3.6 chars/token (Russian-optimized)
            - **Tsinghua/ChatGLM**: ~3.2 chars/token (Chinese-optimized)

            **Code-Specialized:**
            
            - **DeepSeek Coder**: ~3.6 chars/token (code-optimized)
            - **Replit Code**: ~3.5 chars/token (code-optimized)
            - **BigCode StarCoder**: ~3.4 chars/token (code-optimized)
            - **Salesforce CodeGen**: ~3.5 chars/token (code-optimized)

        Accuracy Expectations:
            - **Well-calibrated providers**: ±10-15% accuracy
            - **Language-optimized**: ±15-20% for target languages
            - **General approximation**: ±20-25% accuracy
            - **Code models**: ±10-20% for code content

        Examples:
            This method is called automatically by count() for non-OpenAI models:

            .. code-block:: python

                # Automatic approximation for Anthropic
                counter = TokenCounter("claude-3-opus")
                tokens = counter.count("Hello, world!")  # Uses _approximate_tokens()
                
                # Different providers give different approximations
                text = "This is a test sentence with punctuation!"
                
                anthropic_counter = TokenCounter("claude-3-opus")
                google_counter = TokenCounter("gemini-pro")
                meta_counter = TokenCounter("llama-3-70b")
                
                print(f"Anthropic: {anthropic_counter.count(text)} tokens")
                print(f"Google: {google_counter.count(text)} tokens")
                print(f"Meta: {meta_counter.count(text)} tokens")

        Performance:
            - **Speed**: Very fast, 10,000+ approximations per second
            - **Memory**: Minimal memory usage, no model loading required
            - **Dependencies**: No external dependencies required

        Note:
            This method should not be called directly. Use the count() method instead,
            which automatically selects the appropriate tokenization method based on
            the model's provider.
        """
        if not isinstance(text, str):
            raise TokenizationError(f"Input must be a string, got {type(text).__name__}", model=self.model)

        if not text:
            return 0
        
        try:
            # Basic character-based approximation
            char_count = len(text)
            
            # Adjust for whitespace (spaces and newlines are often separate tokens)
            whitespace_count = len(re.findall(r'\s+', text))
            
            # Adjust for punctuation (often separate tokens)
            punctuation_count = len(re.findall(r'[^\w\s]', text))
        except Exception as e:
            raise TokenizationError(f"Failed to process text: {str(e)}", model=self.model, text_preview=text)
        
        ratio, adjustment_factor = FORMULAS.get(self.model_info.formula or self.provider, DEFAULT_FORMULA)
        formula_key = self.model_info.formula or self.provider
        cjk = _cjk_ratio(text)
        if cjk > 0.2:
            cjk_chars_per_token = CJK_RATIO.get(formula_key) or CJK_RATIO.get(self.provider) or DEFAULT_CJK_RATIO
            ratio = ratio * (1 - cjk) + cjk_chars_per_token * cjk
        base_tokens = char_count / ratio
        adjustment = (whitespace_count + punctuation_count) * adjustment_factor

        return max(1, int(base_tokens + adjustment))
    
    def count_messages(self, messages: List[Dict[str, str]]) -> int:
        """
        Count tokens for a list of messages in chat format.
        
        Processes a list of message dictionaries (typical chat/conversation format)
        and returns the total token count including any formatting overhead. This
        method is essential for chat-based applications and conversation analysis.

        Args:
            messages (List[Dict[str, str]]): List of message dictionaries. Each message
                                           must contain 'role' and 'content' keys.
                                           
                                           Expected format:
                                           
                                           .. code-block:: python
                                           
                                               [
                                                   {"role": "system", "content": "You are a helpful assistant."},
                                                   {"role": "user", "content": "Hello!"},
                                                   {"role": "assistant", "content": "Hi there!"}
                                               ]

        Returns:
            int: Total token count for all messages including formatting overhead.

        Raises:
            TokenizationError: If messages format is invalid, contains non-string content,
                             or if tokenization of individual messages fails. Includes
                             detailed error context with message index and content preview.

        Message Format Validation:
            The method performs comprehensive validation:
            
            - **Input type**: Must be a list, not string/dict/int/etc.
            - **Message structure**: Each message must be a dictionary
            - **Required keys**: Each message must have 'role' and 'content' keys
            - **Content type**: Message content must be a string, not None/int/list/etc.
            - **Role type**: Message role must be a string if present

        Formatting Overhead:
            Different providers handle message formatting differently:
            
            - **OpenAI**: Minimal overhead (~1 token per role)
            - **Anthropic**: No additional formatting overhead
            - **Other providers**: No additional overhead assumed

        Common Message Roles:
            - **system**: System instructions or context
            - **user**: User input or questions
            - **assistant**: AI assistant responses
            - **function**: Function call results (some providers)

        Examples:
            Basic chat conversation:

            .. code-block:: python

                counter = TokenCounter("gpt-4")
                
                messages = [
                    {"role": "system", "content": "You are a helpful assistant."},
                    {"role": "user", "content": "What is the capital of France?"},
                    {"role": "assistant", "content": "The capital of France is Paris."}
                ]
                
                total_tokens = counter.count_messages(messages)
                print(f"Total conversation tokens: {total_tokens}")

            Comparing individual vs. message counting:

            .. code-block:: python

                counter = TokenCounter("gpt-4")
                
                # Count individual messages
                individual_total = 0
                for msg in messages:
                    tokens = counter.count(msg["content"])
                    individual_total += tokens
                    print(f"{msg['role']}: {tokens} tokens")
                
                # Count as message format (includes formatting overhead)
                message_total = counter.count_messages(messages)
                
                print(f"Individual sum: {individual_total}")
                print(f"Message format: {message_total}")
                print(f"Formatting overhead: {message_total - individual_total}")

            Error handling:

            .. code-block:: python

                try:
                    counter = TokenCounter("gpt-4")
                    
                    # Invalid format - missing content
                    invalid_messages = [{"role": "user"}]
                    tokens = counter.count_messages(invalid_messages)
                    
                except TokenizationError as e:
                    print(f"Message format error: {e}")

            Multi-provider comparison:

            .. code-block:: python

                messages = [
                    {"role": "user", "content": "Hello!"},
                    {"role": "assistant", "content": "Hi there! How can I help?"}
                ]
                
                models = ["gpt-4", "claude-3-opus", "gemini-pro"]
                for model in models:
                    counter = TokenCounter(model)
                    tokens = counter.count_messages(messages)
                    print(f"{model}: {tokens} tokens")

        Performance:
            - **Speed**: Processes thousands of message lists per second
            - **Memory**: Minimal additional memory overhead
            - **Scalability**: Handles conversations with hundreds of messages

        Use Cases:
            - **Chat applications**: Calculate conversation costs
            - **API rate limiting**: Plan request sizes for chat endpoints
            - **Conversation analysis**: Analyze dialogue token patterns
            - **Cost estimation**: Budget for chat-based AI applications
            - **Content moderation**: Assess conversation length and complexity

        Note:
            This method is specifically designed for chat/conversation formats.
            For simple text token counting, use the count() method instead.
        """
        # Comprehensive input validation
        if messages is None:
            raise TokenizationError("Messages cannot be None", model=self.model)
        
        if not isinstance(messages, list):
            # Handle common invalid types explicitly
            if isinstance(messages, str):
                raise TokenizationError("Messages must be a list, got string", model=self.model)
            elif isinstance(messages, (int, float, bool)):
                raise TokenizationError(f"Messages must be a list, got {type(messages).__name__}", model=self.model)
            elif isinstance(messages, (dict, tuple, set)):
                raise TokenizationError(f"Messages must be a list, got {type(messages).__name__}", model=self.model)
            else:
                raise TokenizationError("Messages must be a list", model=self.model)
        
        total_tokens = 0
        
        for i, message in enumerate(messages):
            if not isinstance(message, dict):
                raise TokenizationError(f"Message at index {i} must be a dict, got {type(message).__name__}", model=self.model)

            if 'role' not in message:
                raise TokenizationError(f"Message at index {i} must have 'role' key", model=self.model)
            
            if 'content' not in message:
                raise TokenizationError(f"Message at index {i} must have 'content' key", model=self.model)
            
            # Validate content is a string
            if not isinstance(message['content'], str):
                if message['content'] is None:
                    raise TokenizationError(f"Message content at index {i} cannot be None", model=self.model)
                else:
                    raise TokenizationError(f"Message content at index {i} must be a string, got {type(message['content']).__name__}", model=self.model)
            
            # Validate role if present
            if 'role' in message and not isinstance(message['role'], str):
                if message['role'] is None:
                    raise TokenizationError(f"Message role at index {i} cannot be None", model=self.model)
                else:
                    raise TokenizationError(f"Message role at index {i} must be a string, got {type(message['role']).__name__}", model=self.model)
            
            # Count content tokens
            try:
                content_tokens = self.count(message['content'])
                total_tokens += content_tokens
            except TokenizationError:
                # Re-raise TokenizationError as-is
                raise
            except Exception as e:
                raise TokenizationError(f"Failed to count tokens for message at index {i}: {str(e)}", model=self.model)
            
            # Add overhead for message formatting (extremely minimal overhead)
            if self.provider == "openai":
                # OpenAI adds minimal formatting overhead
                if 'role' in message:
                    total_tokens += 1  # Role is typically 1 token
            elif self.provider == "anthropic":
                # Claude has no additional formatting overhead beyond content
                pass
            else:
                # Other providers have no additional overhead
                pass
        
        # No additional final message overhead
        
        return total_tokens


def count_tokens(text: str, model: str) -> int:
    """
    Convenience function to count tokens for a given text and model.
    
    This is a simplified interface that creates a TokenCounter instance and
    performs token counting in a single function call. Ideal for one-off
    token counting operations without needing to manage TokenCounter instances.

    Args:
        text (str): The text to count tokens for. Must be a string.
        model (str): The model name (e.g., 'gpt-4', 'claude-3-opus-20240229').
                    Model names are case-insensitive.

    Returns:
        int: The number of tokens in the text.

    Raises:
        UnsupportedModelError: If the specified model is not supported.
        TokenizationError: If tokenization fails or input is invalid.

    Examples:
        Basic usage:

        .. code-block:: python

            from scripts import count_tokens
            
            # OpenAI model
            tokens = count_tokens("Hello, world!", "gpt-4")
            print(f"GPT-4 tokens: {tokens}")
            
            # Anthropic model
            tokens = count_tokens("Hello, world!", "claude-3-opus")
            print(f"Claude tokens: {tokens}")
            
            # Case-insensitive model names
            tokens = count_tokens("Hello, world!", "GPT-4")  # Same as "gpt-4"

        Comparing models:

        .. code-block:: python

            text = "This is a sample text for comparison."
            models = ["gpt-4", "gpt-3.5-turbo", "claude-3-opus", "gemini-pro"]
            
            for model in models:
                tokens = count_tokens(text, model)
                print(f"{model}: {tokens} tokens")

        Error handling:

        .. code-block:: python

            try:
                tokens = count_tokens("Hello!", "unsupported-model")
            except UnsupportedModelError as e:
                print(f"Model not supported: {e}")
            except TokenizationError as e:
                print(f"Tokenization failed: {e}")

    Performance:
        This function creates a new TokenCounter instance for each call.
        For multiple operations with the same model, consider using
        TokenCounter directly for better performance:

        .. code-block:: python

            # Less efficient for multiple calls
            for text in texts:
                tokens = count_tokens(text, "gpt-4")
            
            # More efficient for multiple calls
            counter = TokenCounter("gpt-4")
            for text in texts:
                tokens = counter.count(text)

    Note:
        This function is equivalent to:
        
        .. code-block:: python
        
            counter = TokenCounter(model)
            return counter.count(text)
    """
    counter = TokenCounter(model)
    return counter.count(text)


def get_supported_models() -> Dict[str, List[str]]:
    """
    Get a comprehensive dictionary of supported models organized by provider.
    
    Returns all 200+ supported models grouped by their respective providers,
    making it easy to discover available models and understand the scope
    of scripts's capabilities.

    Returns:
        Dict[str, List[str]]: Dictionary with provider names as keys and lists
                             of model names as values. Providers include:
                             
                             - **openai**: GPT-4, GPT-3.5, GPT-4o, O1, embeddings (25+ models)
                             - **anthropic**: Claude 3/3.5, Claude 2, Instant (12+ models)
                             - **google**: Gemini Pro/Flash, Gemini 1.5/2.0, PaLM (10+ models)
                             - **meta**: LLaMA 2/3/3.1/3.2/3.3 variants (15+ models)
                             - **mistral**: Mistral 7B, Mixtral, Large variants (10+ models)
                             - **cohere**: Command, Command-R, Command-R+ (8+ models)
                             - **xai**: Grok 1/1.5/2 and beta models (4+ models)
                             - **alibaba**: Qwen 1.5/2.0/2.5 and vision models (20+ models)
                             - **baidu**: ERNIE 3.0/3.5/4.0 variants (8+ models)
                             - **huawei**: PanGu Alpha and Coder models (5+ models)
                             - **yandex**: YaLM and YaGPT models (4+ models)
                             - **deepseek**: Coder, VL, and LLM models (8+ models)
                             - **tsinghua**: ChatGLM and GLM models (5+ models)
                             - **databricks**: DBRX and Dolly models (6+ models)
                             - **voyage**: Voyage embedding models (6+ models)
                             - **And 10+ more providers**

    Examples:
        Basic usage:

        .. code-block:: python

            from scripts import get_supported_models
            
            models = get_supported_models()
            
            # List all providers
            print("Supported providers:")
            for provider in models.keys():
                print(f"  {provider}")

        Explore specific providers:

        .. code-block:: python

            models = get_supported_models()
            
            # OpenAI models
            print("OpenAI models:")
            for model in models["openai"]:
                print(f"  {model}")
            
            # Anthropic models
            print("\\nAnthropic models:")
            for model in models["anthropic"]:
                print(f"  {model}")

        Count models by provider:

        .. code-block:: python

            models = get_supported_models()
            
            print("Model counts by provider:")
            total_models = 0
            for provider, model_list in models.items():
                count = len(model_list)
                total_models += count
                print(f"  {provider}: {count} models")
            
            print(f"\\nTotal: {total_models} models")

        Find models by pattern:

        .. code-block:: python

            models = get_supported_models()
            
            # Find all GPT-4 variants
            gpt4_models = []
            for model in models["openai"]:
                if "gpt-4" in model:
                    gpt4_models.append(model)
            
            print("GPT-4 variants:")
            for model in gpt4_models:
                print(f"  {model}")

        Validate model support:

        .. code-block:: python

            models = get_supported_models()
            
            def is_model_supported(model_name):
                model_lower = model_name.lower()
                for provider_models in models.values():
                    if model_lower in [m.lower() for m in provider_models]:
                        return True
                return False
            
            # Check if models are supported
            test_models = ["gpt-4", "claude-3-opus", "unknown-model"]
            for model in test_models:
                supported = is_model_supported(model)
                print(f"{model}: {'✓' if supported else '✗'}")

        Integration with TokenCounter:

        .. code-block:: python

            from scripts import TokenCounter, get_supported_models
            
            models = get_supported_models()
            text = "Test tokenization across providers."
            
            # Test a few models from each major provider
            test_models = {
                "openai": models["openai"][0],      # First OpenAI model
                "anthropic": models["anthropic"][0], # First Anthropic model
                "google": models["google"][0],       # First Google model
                "meta": models["meta"][0]            # First Meta model
            }
            
            for provider, model in test_models.items():
                counter = TokenCounter(model)
                tokens = counter.count(text)
                print(f"{provider} ({model}): {tokens} tokens")

    Provider Categories:
        The returned dictionary includes models from these categories:
        
        **Major Cloud Providers:**
        - OpenAI, Anthropic, Google, Microsoft, Amazon
        
        **AI-First Companies:**
        - Mistral, Cohere, xAI, Perplexity, AI21
        
        **Regional/Language-Specific:**
        - Alibaba (Chinese), Baidu (Chinese), Huawei (Chinese)
        - Yandex (Russian), Tsinghua (Chinese)
        
        **Open Source/Research:**
        - EleutherAI, Stability AI, TII, RWKV, Community models
        
        **Enterprise/Specialized:**
        - Databricks, Voyage, DeepSeek, BigCode, Replit
        - Nvidia, IBM, Salesforce

    Note:
        The model lists are comprehensive but may not include every variant
        or the very latest models. The library is regularly updated to
        include new models as they become available.

    See Also:
        - :class:`TokenCounter`: For creating token counters with specific models
        - :func:`count_tokens`: For quick token counting with model validation
        - :exc:`UnsupportedModelError`: Exception raised for unsupported models
    """
    return get_supported_models_by_provider()


def estimate_cost(
    token_count: int,
    model: str,
    input_tokens: bool = True,
    currency: str = "USD"
) -> float:
    """
    Estimate the cost for a given number of tokens and model.

    Calculates estimated costs based on current pricing for supported models.
    Supports both input and output token pricing, as many models have different
    rates for input vs. output tokens. Provides costs in USD or INR currency.

    Args:
        token_count (int): Number of tokens to estimate cost for. Must be non-negative.
        model (str): Model name (e.g., "gpt-4", "gpt-4o", "claude-3-opus-20240229").
                    Model names are case-insensitive.
        input_tokens (bool, optional): True for input token pricing, False for output
                                     token pricing. Defaults to True. Many models charge
                                     more for output tokens than input tokens.
        currency (str, optional): Currency code ("USD" or "INR"). Defaults to "USD".
                                 Uses current conversion rate for INR.

    Returns:
        float: Estimated cost in the specified currency. Returns 0.0 if the model
               is not in the pricing database or if pricing is not available.

    Pricing Coverage:
        The function includes pricing for major models:
        
        **OpenAI Models:**
        - GPT-4: $0.03/$0.06 per 1K tokens (input/output)
        - GPT-4 Turbo: $0.01/$0.03 per 1K tokens
        - GPT-4o: $0.005/$0.015 per 1K tokens
        - GPT-4o Mini: $0.00015/$0.0006 per 1K tokens
        - GPT-3.5 Turbo: $0.001/$0.002 per 1K tokens
        
        **Anthropic Models:**
        - Claude-3 Opus: $0.015/$0.075 per 1K tokens
        - Claude-3 Sonnet: $0.003/$0.015 per 1K tokens
        - Claude-3 Haiku: $0.00025/$0.00125 per 1K tokens
        - Claude-3.5 Sonnet: $0.003/$0.015 per 1K tokens
        - Claude-3.5 Haiku: $0.001/$0.005 per 1K tokens
        
        **Databricks Models:**
        - DBRX Instruct: $0.001/$0.002 per 1K tokens
        - Dolly models: $0.001/$0.002 per 1K tokens
        
        **Voyage AI Models:**
        - All Voyage models: $0.0001/$0.0001 per 1K tokens

    Examples:
        Basic cost estimation:

        .. code-block:: python

            from scripts import count_tokens, estimate_cost
            
            text = "This is a sample text for cost estimation."
            model = "gpt-4"
            
            # Count tokens and estimate cost
            tokens = count_tokens(text, model)
            input_cost = estimate_cost(tokens, model, input_tokens=True)
            output_cost = estimate_cost(tokens, model, input_tokens=False)
            
            print(f"Text: '{text}'")
            print(f"Tokens: {tokens}")
            print(f"Input cost: ${input_cost:.4f}")
            print(f"Output cost: ${output_cost:.4f}")

        Compare costs across models:

        .. code-block:: python

            text = "Compare costs across different models." * 100  # Longer text
            models = ["gpt-4", "gpt-4o", "gpt-3.5-turbo", "claude-3-opus", "claude-3-haiku"]
            
            print(f"Text length: {len(text)} characters")
            print("\\nCost comparison:")
            
            for model in models:
                try:
                    tokens = count_tokens(text, model)
                    input_cost = estimate_cost(tokens, model, input_tokens=True)
                    output_cost = estimate_cost(tokens, model, input_tokens=False)
                    
                    print(f"{model}:")
                    print(f"  Tokens: {tokens}")
                    print(f"  Input: ${input_cost:.4f}")
                    print(f"  Output: ${output_cost:.4f}")
                except Exception as e:
                    print(f"{model}: Error - {e}")

        Currency conversion:

        .. code-block:: python

            tokens = 1000
            model = "gpt-4"
            
            # USD pricing
            cost_usd = estimate_cost(tokens, model, currency="USD")
            print(f"Cost in USD: ${cost_usd:.4f}")
            
            # INR pricing
            cost_inr = estimate_cost(tokens, model, currency="INR")
            print(f"Cost in INR: ₹{cost_inr:.2f}")

        Batch cost estimation:

        .. code-block:: python

            texts = [
                "Short text",
                "Medium length text with more content",
                "Much longer text that will cost more to process" * 10
            ]
            
            model = "gpt-4o"
            total_cost = 0
            
            print("Individual text costs:")
            for i, text in enumerate(texts, 1):
                tokens = count_tokens(text, model)
                cost = estimate_cost(tokens, model)
                total_cost += cost
                print(f"Text {i}: {tokens} tokens, ${cost:.4f}")
            
            print(f"\\nTotal estimated cost: ${total_cost:.4f}")

        Chat conversation costing:

        .. code-block:: python

            from scripts import TokenCounter
            
            messages = [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": "Explain quantum computing."},
                {"role": "assistant", "content": "Quantum computing is a revolutionary..."}
            ]
            
            counter = TokenCounter("gpt-4")
            total_tokens = counter.count_messages(messages)
            
            # Estimate costs for the conversation
            input_cost = estimate_cost(total_tokens, "gpt-4", input_tokens=True)
            output_cost = estimate_cost(total_tokens, "gpt-4", input_tokens=False)
            
            print(f"Conversation tokens: {total_tokens}")
            print(f"If all input: ${input_cost:.4f}")
            print(f"If all output: ${output_cost:.4f}")

    Currency Conversion:
        - **USD to INR rate**: 83.0 (as of July 2025)
        - **Rate updates**: The conversion rate is periodically updated
        - **Precision**: INR costs are calculated from USD base prices

    Limitations:
        - **Pricing accuracy**: Based on publicly available pricing, may not reflect
          current rates or enterprise discounts
        - **Model coverage**: Only includes models with known pricing
        - **Rate changes**: Pricing may change without notice
        - **Approximation**: For non-OpenAI models, token counts are approximated

    Note:
        This function provides cost estimates for planning and budgeting purposes.
        Actual costs may vary based on current pricing, volume discounts, and
        exact tokenization. Always verify current pricing with the model provider
        for production applications.

    See Also:
        - :func:`count_tokens`: For getting token counts to use with this function
        - :class:`TokenCounter`: For more complex token counting scenarios
        - :func:`get_supported_models`: For checking which models are available
    """
    model = normalize_model_name(model)
    model_info = get_model_info(model)
    pricing_key = model_info.pricing_id if model_info else model
    rate = get_pricing_rate(pricing_key, input_tokens)
    if rate is None:
        return 0.0

    cost_usd: float = (token_count / 1000) * rate

    return cost_usd * USD_TO_INR if currency.upper() == "INR" else cost_usd


path: scripts/examples/batch_compare.py
content: #!/usr/bin/env python3
"""
Compare token counts across multiple models for local file(s).
Run from project root:
  python scripts/examples/batch_compare.py file1.txt file2.txt
  python scripts/examples/batch_compare.py -f input.txt -m gpt-4 claude-3-opus
"""
import argparse
import sys
from pathlib import Path

# Ensure project root is on path
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))

from scripts.core import TokenCounter
from scripts.exceptions import UnsupportedModelError, TokenizationError


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Compare token counts across models for local file(s)"
    )
    parser.add_argument(
        "paths",
        nargs="*",
        metavar="FILE",
        help="Local file path(s) to compare",
    )
    parser.add_argument(
        "-f", "--file",
        action="append",
        dest="files",
        help="Read from file (repeatable)",
    )
    parser.add_argument(
        "-m", "--models",
        nargs="*",
        default=["gpt-4", "claude-3-opus", "gemini-pro"],
        help="Model names (default: gpt-4 claude-3-opus gemini-pro)",
    )
    args = parser.parse_args()

    paths = list(args.paths or [])
    if args.files:
        paths.extend(args.files)

    if not paths:
        parser.error("Provide at least one local file path")

    models = args.models or ["gpt-4", "claude-3-opus", "gemini-pro"]

    print(f"Files: {paths}")
    print(f"Models: {models}")
    print("---")

    for path_str in paths:
        path = Path(path_str)
        if not path.exists():
            print(f"Error: File '{path}' not found", file=sys.stderr)
            sys.exit(1)
        text = path.read_text(encoding="utf-8")
        print(f"\n{path_str}:")
        for model in models:
            try:
                tokens = TokenCounter(model).count(text)
                print(f"  {model}: tokens={tokens}")
            except (UnsupportedModelError, TokenizationError) as e:
                print(f"  {model}: Error - {e}")


if __name__ == "__main__":
    main()


path: scripts/examples/benchmark_token_ratio.py
content: #!/usr/bin/env python3
"""
批量测试多个模型的 token 与文字比例。
- API 模式：通过请求模型 API 获取 token 数（需配置 API_KEY、BASE_URL）
- 本地模式（--local）：使用本项目的 TokenCounter 近似计算，无需 API
"""
import argparse
import asyncio
import sys
from datetime import datetime
from pathlib import Path
from typing import Optional

# 测试文本：包含中文、英文、符号（使用 r 前缀避免 \U \u 等被当作转义）
SAMPLE_TEXT = r"""
---
name: prompt-token-counter
description: "Count tokens and estimate costs for 300+ LLM models. Primary use: audit OpenClaw workspace token consumption (memory, persona, skills)."
trigger: "token count, cost estimate, prompt length, API cost, OpenClaw audit, workspace token usage, memory/persona/skills tokens, context window limit"
---

# Prompt Token Counter (toksum)

> **First load reminder:** This skill provides the `scripts` CLI (toksum). Use it when the user asks to count tokens, estimate API costs, or **audit OpenClaw component token consumption** (memory, persona, skills).

## Primary Use: OpenClaw Token Consumption Audit

**Goal:** Help users identify which OpenClaw components consume tokens and how much.

### 1. Memory & Persona Files

These files are injected into sessions and consume tokens. Search and count them:

| File | Purpose | Typical Location |
|------|---------|------------------|
| `AGENTS.md` | Operating instructions, workflow, priorities | `~/.openclaw/workspace/` |
| `SOUL.md` | Persona, tone, values, behavioral guidelines | `~/.openclaw/workspace/` |
| `IDENTITY.md` | Name, role, goals, visual description | `~/.openclaw/workspace/` |
| `USER.md` | User preferences, communication style | `~/.openclaw/workspace/` |
| `MEMORY.md` | Long-term memory, persistent facts | `~/.openclaw/workspace/` |
| `TOOLS.md` | Tool quirks, path conventions | `~/.openclaw/workspace/` |
| `HEARTBEAT.md` | Periodic maintenance checklist | `~/.openclaw/workspace/` |
| `BOOT.md` | Startup ritual (when hooks enabled) | `~/.openclaw/workspace/` |
| `memory/YYYY-MM-DD.md` | Daily memory logs | `~/.openclaw/workspace/memory/` |

**Workspace path:** Default `~/.openclaw/workspace`; may be overridden in `~/.openclaw/openclaw.json` via `agent.workspace`.

### 2. Skill Files (SKILL.md)

Skills are loaded per session. Count each `SKILL.md`:

| Location | Scope |
|----------|-------|
| `~/.openclaw/skills/*/SKILL.md` | OpenClaw managed skills |
| `~/.openclaw/workspace/skills/*/SKILL.md` | Workspace-specific skills (override) |

### 3. Audit Workflow

1. **Locate workspace:** Resolve `~/.openclaw/workspace` (or config override).
2. **Collect files:** List all memory/persona files and `SKILL.md` paths above.
3. **Count tokens:** For each file, run `python -m scripts.cli -f <path> -m <model> -c`.
4. **Summarize:** Group by category (memory, persona, skills), report total and per-file.

**Example audit command (PowerShell):**
```powershell
$ws = "$env:USERPROFILE\.openclaw\workspace"
python -m scripts.cli -m gpt-4o -c -f "$ws\AGENTS.md" -f "$ws\SOUL.md" -f "$ws\USER.md" -f "$ws\IDENTITY.md" -f "$ws\MEMORY.md" -f "$ws\TOOLS.md"
```

**Example audit (Bash):**
```bash
WS=~/.openclaw/workspace
python -m scripts.cli -m gpt-4o -c -f "$WS/AGENTS.md" -f "$WS/SOUL.md" -f "$WS/USER.md" -f "$WS/IDENTITY.md" -f "$WS/MEMORY.md" -f "$WS/TOOLS.md"
```

---

## Project Layout

```
prompt_token_counter/
├── SKILL.md
├── package.json                # npm package (OpenClaw skill)
├── publish_npm.py               # Publish to npm; syncs version
└── scripts/                    # Python package, CLI + examples
    ├── cli.py                  # Entry point
    ├── core.py                 # TokenCounter, estimate_cost
    ├── registry/
    │   ├── models.py           # 300+ models
    │   └── pricing.py          # Pricing data
    └── examples/               # Script examples
        ├── count_prompt.py
        ├── estimate_cost.py
        ├── batch_compare.py
        └── benchmark_token_ratio.py
```

Invoke: `python -m scripts.cli` from project root.

### Version Sync (publish_npm.py)

When publishing to npm, `publish_npm.py` bumps the patch version and syncs it to:

- `package.json` — `version`
- `SKILL.md` — frontmatter `version`
- `scripts/__init__.py` — `__version__`

Run: `python publish_npm.py` (after `npm login`).

---

## Runtime Dependencies

- **Python 3** — required
- **tiktoken** (optional) — `pip install tiktoken` for exact OpenAI counts

---

## Language Rule

**Respond in the user's language.** Match the user's language (e.g. Chinese if they write in Chinese, English if they write in English).

---

## URL Usage — Mandatory Agent Rule

**Before using `-u` / `--url` to fetch content from any URL, you MUST:**

1. **Explicitly warn the user** that the CLI will make an outbound HTTP/HTTPS request to the given URL.
2. **Confirm the URL is trusted** — tell the user: "Only use URLs you fully trust. Untrusted URLs may expose your IP, leak data, or be used for SSRF. Do you confirm this URL is safe?"
3. **Prefer alternatives** — if the user can provide the content via `-f` (local file) or inline text, suggest that instead of URL fetch.
4. **Never auto-fetch** — do not invoke `-u` without the user having explicitly provided the URL and acknowledged the risk.

**If the user insists on using a URL:** Proceed only after they confirm. State clearly: "I will fetch from [URL] to count tokens. Proceed?"

---

## Model Name — Mandatory Agent Rule

**Before invoking the CLI, you MUST have a concrete model name from the user.**

1. **Require explicit model** — `-m` / `--model` is required. Do not guess or assume; the user must provide the exact name (e.g. gpt-4o, claude-3-5-sonnet-20241022).
2. **If unclear, ask** — if the user says "GPT" or "Claude" or "the latest model" without a specific name, ask: "Please specify the exact model name (e.g. gpt-4o, claude-3-5-sonnet-20241022). Run `python -m scripts.cli -l` to list supported models."
3. **Do not auto-pick** — never substitute a model on behalf of the user without their confirmation.
4. **Validate when possible** — if the model name seems ambiguous, offer `-l` output or confirm: "I'll use [model]. Is that correct?"

---

## CLI Usage

```bash
python -m scripts.cli [OPTIONS] [TEXT ...]
```

| Option | Short | Description |
|--------|-------|-------------|
| `--model` | `-m` | Model name (required unless `--list-models`) — **Agent must obtain exact name from user; ask if unclear** |
| `--file` | `-f` | Read from file (repeatable) |
| `--url` | `-u` | Read from URL (repeatable) — **Agent must warn user before use; only trusted URLs** |
| `--list-models` | `-l` | List supported models |
| `--cost` | `-c` | Show cost estimate |
| `--output-tokens` | | Use output token pricing |
| `--currency` | | USD or INR |
| `--verbose` | `-v` | Detailed output |

### Examples

```bash
# Inline text
python -m scripts.cli -m gpt-4 "Hello, world!"

# File with cost
python -m scripts.cli -f input.txt -m claude-3-opus -c

# Multiple files (OpenClaw audit)
python -m scripts.cli -v -c -f AGENTS.md -f SOUL.md -f MEMORY.md -m gpt-4o

# List models
python -m scripts.cli -l

# Run bundled example scripts
python scripts/examples/count_prompt.py "Hello, world!" gpt-4
python scripts/examples/estimate_cost.py "Your text" gpt-4
python scripts/examples/batch_compare.py "Compare text" gpt-4 claude-3-opus
```

---

## Python API

```python
from scripts import TokenCounter, count_tokens, estimate_cost, get_supported_models

tokens = count_tokens("Hello!", "gpt-4")
counter = TokenCounter("claude-3-opus")
tokens = counter.count_messages([
    {"role": "system", "content": "..."},
    {"role": "user", "content": "..."}
])
cost = estimate_cost(tokens, "gpt-4", input_tokens=True)
```

---

## Supported Models

300+ models across 34+ providers: OpenAI, Anthropic, Google, Meta, Mistral, Cohere, xAI, DeepSeek, etc. Use `python -m scripts.cli -l` for full list.

- **OpenAI:** exact via tiktoken
- **Others:** ~85–95% approximation

---

## Common Issues

| Issue | Action |
|-------|--------|
| "tiktoken is required" | `pip install tiktoken` |
| UnsupportedModelError | Use `-l` for valid names |
| Cost "NA" | Model has no pricing; count still valid |
| User provides URL | **Agent must warn:** outbound request, SSRF risk, only trusted URLs; confirm before `-u` |
| Model unclear / vague | **Agent must ask:** user to specify exact model name; offer `-l` to list; do not guess |

---

## When to Trigger This Skill

Activate this skill when the user:

| Trigger | Example phrases |
|---------|-----------------|
| **Token count** | "How many tokens?", "Count tokens in this prompt", "Token length of X" |
| **Cost estimate** | "Estimate API cost", "How much for this text?", "Cost for GPT-4" |
| **Prompt size** | "Check prompt length", "Is this too long?", "Context window limit" |
| **OpenClaw audit** | "How many tokens does my workspace use?", "Audit OpenClaw memory/persona/skills", "Which components consume tokens?", "Token usage of AGENTS.md / SOUL.md / skills" |
| **Model comparison** | "Compare token cost across models", "Which model is cheaper?" |

Also trigger when the agent needs to count tokens or estimate cost before/after generating content.

---

## Quick Reference

| Item | Command |
|------|---------|
| Invoke | `python -m scripts.cli` |
| List models | `python -m scripts.cli -l` |
| Cost | `-c` (input) / `--output-tokens` (output) |
| Currency | `--currency USD` or `INR` |
"""

# 待测试的模型列表（可按需修改）
MODELS = [
    "anthropic/claude-sonnet-4-6",
    "anthropic/claude-sonnet-4-5",
    "anthropic/claude-opus-4.6",
    "openai/gpt-5.2-codex",
    "google/gemini-3.1-pro-preview",
    "z-ai/glm-5",
    "volcengine/doubao-seed-2-0-pro",
    "moonshot/kimi-k2.5",
    "minimax/MiniMax-M2.5",
    "deepseek-v3.2",
]

# API 配置（与 skill_executor 保持一致，API 模式时使用）
API_KEY = ""
BASE_URL = ""


def get_token_count_local(model: str, text: str) -> dict:
    """使用本项目的 TokenCounter 本地近似计算 token 数。"""
    try:
        from scripts.core import TokenCounter

        counter = TokenCounter(model)
        token_count = counter.count(text)
    except Exception as e:
        return {
            "model": model,
            "success": False,
            "error": str(e),
            "char_count": len(text),
            "token_count": None,
            "ratio": None,
        }

    char_count = len(text)
    ratio = char_count / token_count if token_count > 0 else 0

    return {
        "model": model,
        "success": True,
        "char_count": char_count,
        "token_count": token_count,
        "ratio": round(ratio, 4),
        "ratio_desc": f"1 token ≈ {ratio:.4f} 字符",
    }


async def get_token_count_api(model: str, text: str) -> dict:
    """
    请求模型 API，获取输入文本的 token 数。
    使用 chat completion，从 response.usage.prompt_tokens 获取。
    依赖: pip install openai
    """
    try:
        import openai

        client = openai.AsyncOpenAI([REDACTED_SECRET], base_url=BASE_URL)

        messages = [{"role": "user", "content": text}]
        response = await client.chat.completions.create(
            model=model,
            messages=messages,
            max_tokens=8 * 1024,  # 部分 API 要求 >= 16，只关心 prompt_tokens
            temperature=0,
        )

        usage = getattr(response, "usage", None)
        if usage is None:
            return {
                "model": model,
                "success": False,
                "error": "API 响应中无 usage 字段",
                "char_count": len(text),
                "token_count": None,
                "ratio": None,
            }

        prompt_tokens = getattr(usage, "prompt_tokens", None)
        if prompt_tokens is None:
            prompt_tokens = getattr(usage, "total_tokens", None)

        if prompt_tokens is None:
            return {
                "model": model,
                "success": False,
                "error": "usage 中无 prompt_tokens/total_tokens",
                "char_count": len(text),
                "token_count": None,
                "ratio": None,
            }

        char_count = len(text)
        ratio = char_count / prompt_tokens if prompt_tokens > 0 else 0

        return {
            "model": model,
            "success": True,
            "char_count": char_count,
            "token_count": prompt_tokens,
            "ratio": round(ratio, 4),
            "ratio_desc": f"1 token ≈ {ratio:.4f} 字符",
        }
    except Exception as e:
        return {
            "model": model,
            "success": False,
            "error": str(e),
            "char_count": len(text),
            "token_count": None,
            "ratio": None,
        }


async def batch_test_api(models: list[str], text: str) -> list[dict]:
    """批量异步请求所有模型（API 模式）。"""
    tasks = [get_token_count_api(m, text) for m in models]
    results = await asyncio.gather(*tasks, return_exceptions=True)

    out = []
    for i, r in enumerate(results):
        if isinstance(r, Exception):
            out.append(
                {
                    "model": models[i],
                    "success": False,
                    "error": str(r),
                    "char_count": len(text),
                    "token_count": None,
                    "ratio": None,
                }
            )
        else:
            out.append(r)
    return out


def batch_test_local(models: list[str], text: str) -> list[dict]:
    """批量测试所有模型（本地模式）。"""
    return [get_token_count_local(m, text) for m in models]


def write_report(results: list[dict], text: str, output_path: Path, mode: str) -> None:
    """将测试结果写入 Markdown 文档。"""
    mode_label = "API" if mode == "api" else "本地近似"
    lines = [
        "# 模型 Token 与文字比例测试报告",
        "",
        f"**生成时间**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
        f"**测试字符数**: {len(text)}",
        f"**模式**: {mode_label}",
        "",
        "| 模型 | 字符数 | Token 数 | 1 token ≈ 多少字符 | 状态 |",
        "|------|--------|----------|-------------------|------|",
    ]

    for r in results:
        model = r.get("model", "-")
        char_count = r.get("char_count") or "-"
        token_count = r.get("token_count") if r.get("token_count") is not None else "-"
        ratio = r.get("ratio")
        ratio_str = f"{ratio:.4f}" if isinstance(ratio, (int, float)) else "-"
        status = "✓" if r.get("success") else f"✗ {r.get('error', '')}"
        lines.append(f"| {model} | {char_count} | {token_count} | {ratio_str} | {status} |")

    lines.extend(
        [
            "",
            "## 说明",
            "",
            "- **1 token ≈ 多少字符**：字符数 / token 数，数值越大表示 1 个 token 能表示更多文字",
            "- 中文通常比英文消耗更多 token",
            "- 不同模型使用不同 tokenizer，比例会有差异",
            "",
        ]
    )

    output_path.write_text("\n".join(lines), encoding="utf-8")
    print(f"报告已保存: {output_path}")


def main(
    text: Optional[str] = None,
    text_file: Optional[Path] = None,
    models: Optional[list[str]] = None,
    output_path: Optional[Path] = None,
    use_local: bool = False,
) -> list[dict]:
    """
    主流程：批量测试并输出报告。

    Args:
        text: 测试文本，默认使用 SAMPLE_TEXT
        text_file: 从文件读取测试文本（覆盖 text）
        models: 模型列表，默认使用 MODELS
        output_path: 输出文件路径，默认 token_ratio_report.md
        use_local: True 使用本地 TokenCounter，False 使用 API
    """
    if text_file and text_file.exists():
        text = text_file.read_text(encoding="utf-8")
    elif text is None:
        text = SAMPLE_TEXT

    models = models or MODELS

    if output_path is None:
        output_path = Path(__file__).parent / "token_ratio_report.md"

    mode = "local" if use_local else "api"

    if use_local:
        print(f"本地模式：测试 {len(models)} 个模型，文本长度 {len(text)} 字符...")
        results = batch_test_local(models, text)
    else:
        try:
            import openai  # noqa: F401
        except ImportError:
            print("API 模式需要 openai: pip install openai")
            print("或使用 --local 进行本地近似测试")
            return []
        if not API_KEY or not BASE_URL:
            print("API 模式需在脚本中设置 API_KEY 和 BASE_URL")
            print("或使用 --local 进行本地近似测试")
            return []
        print(f"API 模式：测试 {len(models)} 个模型，文本长度 {len(text)} 字符...")
        results = asyncio.run(batch_test_api(models, text))

    for r in results:
        if r.get("success"):
            print(f"  {r['model']}: {r['token_count']} tokens, 1 token ≈ {r['ratio']:.4f} 字符")
        else:
            print(f"  {r['model']}: 失败 - {r.get('error', '')}")

    write_report(results, text, output_path, mode)
    return results


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="批量测试模型 token 与文字比例")
    parser.add_argument("--local", action="store_true", help="使用本地 TokenCounter 近似（无需 API）")
    parser.add_argument("--text-file", "-f", type=Path, help="从文件读取测试文本")
    parser.add_argument("--output", "-o", type=Path, help="输出报告路径")
    args = parser.parse_args()

    # 确保从项目根目录运行时可导入 scripts
    script_dir = Path(__file__).resolve().parent
    project_root = script_dir.parent.parent
    if str(project_root) not in sys.path:
        sys.path.insert(0, str(project_root))

    main(text_file=args.text_file, output_path=args.output, use_local=args.local)


path: scripts/examples/count_prompt.py
content: #!/usr/bin/env python3
"""
Count tokens for local file(s). Default: read from local files, batch mode.
Run from project root:
  python scripts/examples/count_prompt.py file1.txt file2.txt -m gpt-4
  python scripts/examples/count_prompt.py -f input.txt -m gpt-4
"""
import argparse
import sys
from pathlib import Path

# Ensure project root is on path
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))

from scripts.core import TokenCounter
from scripts.exceptions import UnsupportedModelError, TokenizationError


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Count tokens for local file(s), batch mode"
    )
    parser.add_argument(
        "paths",
        nargs="*",
        metavar="FILE",
        help="Local file path(s) to count (default input)",
    )
    parser.add_argument(
        "-f", "--file",
        action="append",
        dest="files",
        help="Read from file (repeatable)",
    )
    parser.add_argument(
        "-m", "--model",
        default="gpt-4",
        help="Model name (default: gpt-4)",
    )
    args = parser.parse_args()

    paths = list(args.paths or [])
    if args.files:
        paths.extend(args.files)

    if not paths:
        parser.error("Provide at least one local file path")

    model = args.model
    print(f"Model: {model}")
    print("---")

    try:
        counter = TokenCounter(model)
        for p in paths:
            path = Path(p)
            if not path.exists():
                print(f"Error: File '{path}' not found", file=sys.stderr)
                sys.exit(1)
            text = path.read_text(encoding="utf-8")
            tokens = counter.count(text)
            print(f"{p}: tokens={tokens}")
    except UnsupportedModelError as e:
        print(f"Error: {e}", file=sys.stderr)
        sys.exit(1)
    except TokenizationError as e:
        print(f"Error: {e}", file=sys.stderr)
        sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/examples/estimate_cost.py
content: #!/usr/bin/env python3
"""
Estimate API cost for a prompt (text or file).
Run from project root: python scripts/examples/estimate_cost.py [text_or_file] [model]
Or: python scripts/examples/estimate_cost.py input.txt gpt-4
"""
import argparse
import sys
from pathlib import Path

# Ensure project root is on path
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))

from scripts.core import TokenCounter, estimate_cost
from scripts.exceptions import UnsupportedModelError, TokenizationError


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Estimate API cost for a prompt (text or file)"
    )
    parser.add_argument(
        "input",
        nargs="?",
        default="Your prompt text here",
        help="Text to estimate, or path to file",
    )
    parser.add_argument(
        "model",
        nargs="?",
        default="gpt-4",
        help="Model name (default: gpt-4)",
    )
    parser.add_argument(
        "-m", "--model",
        dest="model_opt",
        help="Model name (alternative to positional)",
    )
    args = parser.parse_args()

    model = args.model_opt or args.model

    if Path(args.input).is_file():
        path = Path(args.input)
        if not path.exists():
            print(f"Error: File '{path}' not found", file=sys.stderr)
            sys.exit(1)
        text = path.read_text(encoding="utf-8")
        print(f"Estimating cost for file: {path}")
    else:
        text = args.input
        print(f'Estimating cost for text: "{text[:50]}{"..." if len(text) > 50 else ""}"')

    try:
        tokens = TokenCounter(model).count(text)
        cost = estimate_cost(tokens, model, input_tokens=True)
        symbol = "$"
        print(f"tokens={tokens} cost={symbol}{cost:.6f}")
    except UnsupportedModelError as e:
        print(f"Error: {e}", file=sys.stderr)
        sys.exit(1)
    except TokenizationError as e:
        print(f"Error: {e}", file=sys.stderr)
        sys.exit(1)


if __name__ == "__main__":
    main()


path: scripts/exceptions.py
content: """
Custom exceptions for the scripts library.

This module defines a hierarchy of custom exceptions that provide detailed error
information for various failure modes in token counting operations. The exceptions
are designed to give users clear, actionable error messages with relevant context.

Exception Hierarchy:
    .. code-block:: text
    
        ToksumError (base)
        ├── UnsupportedModelError
        ├── ModelNotFoundError  
        ├── TokenizationError
        │   ├── InvalidTokenError
        │   └── EmptyTextError

The exception hierarchy allows for both specific error handling and general
error catching at different levels of granularity.

Usage Examples:
    Specific exception handling:

    .. code-block:: python

        from scripts import TokenCounter
        from scripts.exceptions import UnsupportedModelError, TokenizationError
        
        try:
            counter = TokenCounter("unknown-model")
            tokens = counter.count("Hello, world!")
        except UnsupportedModelError as e:
            print(f"Model not supported: {e.model}")
            print(f"Available models: {e.supported_models}")
        except TokenizationError as e:
            print(f"Tokenization failed: {e}")
            if e.model:
                print(f"Model: {e.model}")
            if e.text_preview:
                print(f"Text preview: {e.text_preview}")

    General error handling:

    .. code-block:: python

        from scripts import count_tokens
        from scripts.exceptions import ToksumError
        
        try:
            tokens = count_tokens("Hello!", "gpt-4")
        except ToksumError as e:
            print(f"Toksum error: {e}")
        except Exception as e:
            print(f"Unexpected error: {e}")

Error Context:
    All exceptions include relevant context information:
    
    - **Model information**: Which model caused the error
    - **Text previews**: Sample of problematic text (truncated for privacy)
    - **Supported alternatives**: List of valid options when applicable
    - **Detailed messages**: Clear descriptions of what went wrong

The exceptions are designed to be both human-readable and machine-parseable,
making them suitable for both debugging and automated error handling.
"""

from typing import List, Optional


class ToksumError(Exception):
    """
    Base exception class for all scripts library errors.
    
    This is the root exception class that all other scripts exceptions inherit from.
    It allows for catching any scripts-related error with a single exception type
    while still providing the ability to catch specific error types when needed.

    This exception should generally not be raised directly. Instead, use one of
    the more specific exception subclasses that provide additional context and
    error-specific information.

    Examples:
        Catching all scripts errors:

        .. code-block:: python

            from scripts import count_tokens
            from scripts.exceptions import ToksumError
            
            try:
                tokens = count_tokens("Hello!", "some-model")
            except ToksumError as e:
                print(f"Toksum error occurred: {e}")
                # Handle any scripts-related error
            except Exception as e:
                print(f"Unexpected error: {e}")
                # Handle non-scripts errors

        Catching specific errors with fallback:

        .. code-block:: python

            from scripts import TokenCounter
            from scripts.exceptions import UnsupportedModelError, TokenizationError, ToksumError
            
            try:
                counter = TokenCounter("gpt-4")
                tokens = counter.count("Hello!")
            except UnsupportedModelError:
                print("Model not supported")
            except TokenizationError:
                print("Tokenization failed")
            except ToksumError:
                print("Other scripts error")
            except Exception:
                print("Non-scripts error")

    Note:
        This base class provides a common interface for all scripts exceptions
        but does not add any additional functionality beyond the standard
        Python Exception class.
    """


class UnsupportedModelError(ToksumError):
    """
    Raised when an unsupported model is specified.
    
    This exception is raised when attempting to create a TokenCounter or use
    count_tokens() with a model name that is not recognized by the library.
    The exception includes the invalid model name and optionally a list of
    supported models for reference.

    Attributes:
        model (str): The unsupported model name that was specified
        supported_models (List[str]): List of supported model names (optional)

    The exception message automatically includes the unsupported model name
    and, if provided, a formatted list of supported alternatives.

    Examples:
        Basic usage with model validation:

        .. code-block:: python

            from scripts import TokenCounter
            from scripts.exceptions import UnsupportedModelError
            
            try:
                counter = TokenCounter("invalid-model-name")
            except UnsupportedModelError as e:
                print(f"Model '{e.model}' is not supported")
                if e.supported_models:
                    print("Supported models include:")
                    for model in e.supported_models[:5]:  # Show first 5
                        print(f"  - {model}")

        Handling with model suggestions:

        .. code-block:: python

            from scripts import get_supported_models
            from scripts.exceptions import UnsupportedModelError
            
            def safe_create_counter(model_name):
                try:
                    return TokenCounter(model_name)
                except UnsupportedModelError as e:
                    print(f"Error: {e}")
                    
                    # Suggest similar models
                    all_models = get_supported_models()
                    suggestions = []
                    for provider_models in all_models.values():
                        for model in provider_models:
                            if model_name.lower() in model.lower():
                                suggestions.append(model)
                    
                    if suggestions:
                        print("Did you mean one of these?")
                        for suggestion in suggestions[:3]:
                            print(f"  - {suggestion}")
                    
                    return None

        Programmatic error handling:

        .. code-block:: python

            def validate_model(model_name):
                try:
                    TokenCounter(model_name)
                    return True
                except UnsupportedModelError:
                    return False
            
            # Test multiple models
            test_models = ["gpt-4", "invalid-model", "claude-3-opus"]
            for model in test_models:
                if validate_model(model):
                    print(f"✓ {model} is supported")
                else:
                    print(f"✗ {model} is not supported")

    Common Causes:
        - **Typos in model names**: "gpt4" instead of "gpt-4"
        - **Incorrect casing**: "GPT-4" vs "gpt-4" (though scripts is case-insensitive)
        - **Outdated model names**: Using deprecated or renamed models
        - **Provider confusion**: Using model names from unsupported providers
        - **Version specificity**: Using overly specific version numbers

    Resolution:
        1. **Check spelling**: Verify the model name is spelled correctly
        2. **Use get_supported_models()**: Get the complete list of supported models
        3. **Check provider**: Ensure the model provider is supported
        4. **Update library**: Newer models may require a library update

    See Also:
        - :func:`get_supported_models`: Get all supported models by provider
        - :class:`TokenCounter`: Main class that raises this exception
        - :func:`count_tokens`: Convenience function that may raise this exception
    """

    def __init__(self, model: str, supported_models: Optional[List[str]] = None):
        """
        Initialize UnsupportedModelError with model name and optional supported models list.

        Args:
            model (str): The unsupported model name that was specified
            supported_models (Optional[List[str]]): List of supported model names
                                                   for user reference. If provided,
                                                   will be included in error message.
        """
        self.model = model
        self.supported_models = supported_models or []

        if self.supported_models:
            message = f"Model '{model}' is not supported. Supported models: {', '.join(self.supported_models)}"
        else:
            message = f"Model '{model}' is not supported."

        super().__init__(message)


class ModelNotFoundError(ToksumError):
    """
    Raised when a specified model is not found.
    
    This exception is raised in scenarios where a model name is recognized
    as potentially valid but cannot be located or accessed. This is distinct
    from UnsupportedModelError, which is raised for completely unrecognized
    model names.

    This exception is typically used for cases where:
    - A model exists but is temporarily unavailable
    - A model requires special access or authentication
    - A model has been deprecated or removed
    - There are network or service issues preventing model access

    Attributes:
        model (str): The model name that could not be found

    Examples:
        Basic error handling:

        .. code-block:: python

            from scripts.exceptions import ModelNotFoundError
            
            try:
                # Some operation that might fail to find a model
                result = some_model_operation("gpt-4")
            except ModelNotFoundError as e:
                print(f"Model not found: {e}")
                # Maybe try a fallback model or retry later

        Distinguishing from UnsupportedModelError:

        .. code-block:: python

            from scripts import TokenCounter
            from scripts.exceptions import UnsupportedModelError, ModelNotFoundError
            
            try:
                counter = TokenCounter("some-model")
            except UnsupportedModelError:
                print("Model is not supported by scripts")
            except ModelNotFoundError:
                print("Model is supported but cannot be found/accessed")

    Note:
        This exception is less commonly used in the current scripts implementation
        but is available for future extensions and specific use cases where
        model availability needs to be distinguished from model support.

    See Also:
        - :exc:`UnsupportedModelError`: For completely unsupported models
        - :class:`TokenCounter`: May raise this exception in certain scenarios
    """
    def __init__(self, model: str):
        """
        Initialize ModelNotFoundError with the model name.

        Args:
            model (str): The model name that could not be found
        """
        super().__init__(f"Model '{model}' not found.")


class TokenizationError(ToksumError):
    """
    Raised when tokenization fails for any reason.
    
    This is the primary exception for tokenization-related failures. It provides
    detailed context about what went wrong, including the model being used and
    a preview of the problematic text (truncated for privacy and readability).

    This exception covers a wide range of tokenization failures:
    - Invalid input types (non-string input)
    - Missing dependencies (e.g., tiktoken for OpenAI models)
    - Tokenizer initialization failures
    - Text processing errors
    - Encoding/decoding issues

    Attributes:
        model (Optional[str]): The model name being used when the error occurred
        text_preview (Optional[str]): A preview of the problematic text (truncated)

    The exception message includes the base error description and automatically
    appends model and text preview information when available.

    Examples:
        Basic error handling:

        .. code-block:: python

            from scripts import TokenCounter
            from scripts.exceptions import TokenizationError
            
            try:
                counter = TokenCounter("gpt-4")
                tokens = counter.count(123)  # Invalid input type
            except TokenizationError as e:
                print(f"Tokenization failed: {e}")
                if e.model:
                    print(f"Model: {e.model}")
                if e.text_preview:
                    print(f"Text preview: {e.text_preview}")

        Handling missing dependencies:

        .. code-block:: python

            try:
                counter = TokenCounter("gpt-4")  # Requires tiktoken
                tokens = counter.count("Hello!")
            except TokenizationError as e:
                if "tiktoken" in str(e):
                    print("Please install tiktoken: pip install tiktoken")
                else:
                    print(f"Tokenization error: {e}")

        Robust error handling with fallbacks:

        .. code-block:: python

            def safe_count_tokens(text, model, fallback_model=None):
                try:
                    counter = TokenCounter(model)
                    return counter.count(text)
                except TokenizationError as e:
                    print(f"Primary model failed: {e}")
                    
                    if fallback_model:
                        try:
                            print(f"Trying fallback model: {fallback_model}")
                            counter = TokenCounter(fallback_model)
                            return counter.count(text)
                        except TokenizationError:
                            print("Fallback model also failed")
                    
                    return None

        Input validation with detailed errors:

        .. code-block:: python

            def validate_and_count(text, model):
                try:
                    if not isinstance(text, str):
                        raise TokenizationError(
                            f"Input must be string, got {type(text).__name__}",
                            model=model
                        )
                    
                    counter = TokenCounter(model)
                    return counter.count(text)
                    
                except TokenizationError as e:
                    print(f"Validation failed: {e}")
                    return None

    Text Preview Handling:
        The text_preview attribute contains a truncated version of the problematic
        text to help with debugging while protecting privacy:
        
        - **Truncation**: Long text is truncated to ~50 characters + "..."
        - **Privacy**: Sensitive information is not fully exposed in error messages
        - **Debugging**: Provides enough context to identify the problematic content

    Common Causes:
        - **Wrong input type**: Passing int, list, dict instead of string
        - **None input**: Passing None instead of a string
        - **Missing dependencies**: tiktoken not installed for OpenAI models
        - **Encoding issues**: Text with problematic character encodings
        - **Memory issues**: Extremely large text causing processing failures

    Resolution:
        1. **Check input type**: Ensure text is a string
        2. **Install dependencies**: Install tiktoken for OpenAI models
        3. **Validate text**: Check for encoding issues or special characters
        4. **Check text size**: Very large texts may cause issues
        5. **Try different model**: Some models may handle edge cases better

    See Also:
        - :exc:`InvalidTokenError`: For specific token-related errors
        - :exc:`EmptyTextError`: For empty text specific errors
        - :class:`TokenCounter`: Main class that raises this exception
    """

    def __init__(self, message: str, model: Optional[str] = None, text_preview: Optional[str] = None):
        """
        Initialize TokenizationError with detailed context.

        Args:
            message (str): The base error message describing what went wrong
            model (Optional[str]): The model name being used when error occurred
            text_preview (Optional[str]): Preview of the problematic text.
                                        Will be truncated if longer than 50 characters.
        """
        self.model = model
        self.text_preview = text_preview

        full_message = f"Tokenization failed: {message}"
        if model:
            full_message += f" (model: {model})"
        if text_preview:
            preview = text_preview[:50] + "..." if len(text_preview) > 50 else text_preview
            full_message += f" (text preview: '{preview}')"

        super().__init__(full_message)


class InvalidTokenError(TokenizationError):
    """
    Raised when an invalid token is encountered during tokenization.
    
    This exception is a specialized form of TokenizationError that specifically
    handles cases where individual tokens are invalid or problematic. It provides
    additional context about the specific token that caused the issue.

    This exception might be raised when:
    - A token contains invalid characters or sequences
    - A token exceeds maximum length limits
    - A token has encoding issues
    - A token conflicts with model-specific restrictions

    Attributes:
        token (str): The specific invalid token that caused the error
        model (Optional[str]): The model name (inherited from TokenizationError)
        text_preview (Optional[str]): Preview of the text (inherited from TokenizationError)

    Examples:
        Handling invalid token errors:

        .. code-block:: python

            from scripts import TokenCounter
            from scripts.exceptions import InvalidTokenError, TokenizationError
            
            try:
                counter = TokenCounter("gpt-4")
                tokens = counter.count("Some text with problematic content")
            except InvalidTokenError as e:
                print(f"Invalid token encountered: '{e.token}'")
                print(f"Error details: {e}")
                # Maybe try preprocessing the text to remove problematic tokens
            except TokenizationError as e:
                print(f"General tokenization error: {e}")

        Token-specific error handling:

        .. code-block:: python

            def safe_tokenize_with_cleanup(text, model):
                try:
                    counter = TokenCounter(model)
                    return counter.count(text)
                except InvalidTokenError as e:
                    print(f"Removing problematic [REDACTED_SECRET]")
                    # Remove or replace the problematic token
                    cleaned_text = text.replace(e.token, "")
                    return counter.count(cleaned_text)
                except TokenizationError:
                    print("Could not tokenize even after cleanup")
                    return None

    Note:
        This exception is not commonly raised in the current scripts implementation
        but is available for future enhancements and edge cases where specific
        token validation is needed.

    See Also:
        - :exc:`TokenizationError`: Parent class for general tokenization errors
        - :exc:`EmptyTextError`: For empty text specific cases
        - :class:`TokenCounter`: May raise this exception in specific scenarios
    """
    def __init__(self, token: str, message: str, model: Optional[str] = None, text_preview: Optional[str] = None):
        """
        Initialize InvalidTokenError with token and context information.

        Args:
            token (str): The specific invalid token that caused the error
            message (str): Description of why the token is invalid
            model (Optional[str]): The model name being used
            text_preview (Optional[str]): Preview of the problematic text
        """
        full_message = f"Invalid token '{token}': {message}"
        super().__init__(full_message, model, text_preview)

        
class EmptyTextError(TokenizationError):
    """
    Raised when attempting to tokenize empty text in contexts where it's not allowed.
    
    This exception is a specialized form of TokenizationError for cases where
    empty text is specifically problematic. Note that in most scripts operations,
    empty text is handled gracefully and returns 0 tokens. This exception is
    reserved for contexts where empty text indicates a logical error.

    This exception might be raised when:
    - Empty text is passed to functions that require non-empty content
    - Batch operations encounter unexpected empty strings
    - Validation functions detect empty content where it shouldn't be
    - Message content is empty in chat format validation

    Attributes:
        model (Optional[str]): The model name (inherited from TokenizationError)

    Examples:
        Handling empty text validation:

        .. code-block:: python

            from scripts.exceptions import EmptyTextError, TokenizationError
            
            def validate_content(text, model):
                try:
                    if not text.strip():  # Check for empty or whitespace-only
                        raise EmptyTextError(model=model)
                    
                    counter = TokenCounter(model)
                    return counter.count(text)
                    
                except EmptyTextError as e:
                    print("Error: Content cannot be empty")
                    return None
                except TokenizationError as e:
                    print(f"Tokenization error: {e}")
                    return None

        Batch processing with empty text handling:

        .. code-block:: python

            def process_text_batch(texts, model):
                results = []
                counter = TokenCounter(model)
                
                for i, text in enumerate(texts):
                    try:
                        if not text:
                            raise EmptyTextError(model=model)
                        
                        tokens = counter.count(text)
                        results.append(tokens)
                        
                    except EmptyTextError:
                        print(f"Skipping empty text at index {i}")
                        results.append(0)  # or None, depending on requirements
                    except TokenizationError as e:
                        print(f"Error processing text {i}: {e}")
                        results.append(None)
                
                return results

        Message validation:

        .. code-block:: python

            def validate_messages(messages):
                for i, msg in enumerate(messages):
                    if not msg.get("content", "").strip():
                        raise EmptyTextError(
                            f"Message {i} has empty content"
                        )

    Normal Empty Text Handling:
        In most scripts operations, empty text is handled normally:

        .. code-block:: python

            counter = TokenCounter("gpt-4")
            tokens = counter.count("")  # Returns 0, no exception
            
            messages = [
                {"role": "user", "content": ""},  # This might raise EmptyTextError
                {"role": "assistant", "content": "Hello!"}
            ]

    Use Cases:
        - **Content validation**: Ensuring required fields are not empty
        - **Batch processing**: Identifying problematic entries in datasets
        - **API validation**: Checking inputs before expensive operations
        - **Data quality**: Ensuring content meets minimum requirements

    Note:
        This exception is used sparingly in scripts. Most empty text cases
        are handled gracefully by returning 0 tokens. Use this exception
        only when empty text represents a logical error in your application.

    See Also:
        - :exc:`TokenizationError`: Parent class for general tokenization errors
        - :exc:`InvalidTokenError`: For specific token validation issues
        - :meth:`TokenCounter.count`: Handles empty text gracefully (returns 0)
        - :meth:`TokenCounter.count_messages`: May raise this for empty message content
    """
    def __init__(self, model: Optional[str] = None):
        """
        Initialize EmptyTextError with optional model context.

        Args:
            model (Optional[str]): The model name being used when the error occurred
        """
        super().__init__("Cannot tokenize empty text.", model)



path: scripts/registry/models.py
content: """Model registry for scripts.

Defines a canonical model registry with metadata needed for provider detection,
OpenAI encoding selection, approximate token formulas, and pricing lookup.
"""

from __future__ import annotations

import difflib
import re
from dataclasses import dataclass
from typing import Dict, Iterable, List, Optional


@dataclass(frozen=True)
class ModelInfo:
    name: str
    provider: str
    encoding: Optional[str] = None
    formula: Optional[str] = None
    pricing_id: Optional[str] = None
    aliases: Optional[List[str]] = None


def normalize_model_name(model: str) -> str:
    return model.strip().lower()


MODELS: Dict[str, ModelInfo] = {
    # OpenAI
    "gpt-4": ModelInfo("gpt-4", "openai", encoding="cl100k_base", pricing_id="gpt-4"),
    "gpt-4-0314": ModelInfo("gpt-4-0314", "openai", encoding="cl100k_base", pricing_id="gpt-4"),
    "gpt-4-0613": ModelInfo("gpt-4-0613", "openai", encoding="cl100k_base", pricing_id="gpt-4"),
    "gpt-4-32k": ModelInfo("gpt-4-32k", "openai", encoding="cl100k_base", pricing_id="gpt-4-32k"),
    "gpt-4-32k-0314": ModelInfo("gpt-4-32k-0314", "openai", encoding="cl100k_base", pricing_id="gpt-4-32k"),
    "gpt-4-32k-0613": ModelInfo("gpt-4-32k-0613", "openai", encoding="cl100k_base", pricing_id="gpt-4-32k"),
    "gpt-4-1106-preview": ModelInfo("gpt-4-1106-preview", "openai", encoding="cl100k_base"),
    "gpt-4-0125-preview": ModelInfo("gpt-4-0125-preview", "openai", encoding="cl100k_base"),
    "gpt-4-turbo-preview": ModelInfo("gpt-4-turbo-preview", "openai", encoding="cl100k_base"),
    "gpt-4-vision-preview": ModelInfo("gpt-4-vision-preview", "openai", encoding="cl100k_base"),
    "gpt-4-turbo": ModelInfo("gpt-4-turbo", "openai", encoding="cl100k_base", pricing_id="gpt-4-turbo"),
    "gpt-4-turbo-2024-04-09": ModelInfo("gpt-4-turbo-2024-04-09", "openai", encoding="cl100k_base", pricing_id="gpt-4-turbo"),
    "gpt-4o": ModelInfo("gpt-4o", "openai", encoding="cl100k_base", pricing_id="gpt-4o"),
    "gpt-4o-2024-05-13": ModelInfo("gpt-4o-2024-05-13", "openai", encoding="cl100k_base", pricing_id="gpt-4o"),
    "gpt-4o-mini": ModelInfo("gpt-4o-mini", "openai", encoding="cl100k_base", pricing_id="gpt-4o-mini"),
    "gpt-4o-mini-2024-07-18": ModelInfo("gpt-4o-mini-2024-07-18", "openai", encoding="cl100k_base", pricing_id="gpt-4o-mini"),
    "gpt-4o-2024-08-06": ModelInfo("gpt-4o-2024-08-06", "openai", encoding="cl100k_base", pricing_id="gpt-4o"),
    "gpt-4o-2024-11-20": ModelInfo("gpt-4o-2024-11-20", "openai", encoding="cl100k_base", pricing_id="gpt-4o"),
    "gpt-4-1106-vision-preview": ModelInfo("gpt-4-1106-vision-preview", "openai", encoding="cl100k_base"),
    "gpt-3.5-turbo": ModelInfo("gpt-3.5-turbo", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo"),
    "gpt-3.5-turbo-0301": ModelInfo("gpt-3.5-turbo-0301", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo"),
    "gpt-3.5-turbo-0613": ModelInfo("gpt-3.5-turbo-0613", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo"),
    "gpt-3.5-turbo-1106": ModelInfo("gpt-3.5-turbo-1106", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo"),
    "gpt-3.5-turbo-0125": ModelInfo("gpt-3.5-turbo-0125", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo"),
    "gpt-3.5-turbo-16k": ModelInfo("gpt-3.5-turbo-16k", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo-16k"),
    "gpt-3.5-turbo-16k-0613": ModelInfo("gpt-3.5-turbo-16k-0613", "openai", encoding="cl100k_base", pricing_id="gpt-3.5-turbo-16k"),
    "gpt-3.5-turbo-instruct": ModelInfo("gpt-3.5-turbo-instruct", "openai", encoding="cl100k_base"),
    "text-davinci-003": ModelInfo("text-davinci-003", "openai", encoding="p50k_base"),
    "text-davinci-002": ModelInfo("text-davinci-002", "openai", encoding="p50k_base"),
    "text-curie-001": ModelInfo("text-curie-001", "openai", encoding="r50k_base"),
    "text-babbage-001": ModelInfo("text-babbage-001", "openai", encoding="r50k_base"),
    "text-ada-001": ModelInfo("text-ada-001", "openai", encoding="r50k_base"),
    "davinci": ModelInfo("davinci", "openai", encoding="r50k_base"),
    "curie": ModelInfo("curie", "openai", encoding="r50k_base"),
    "babbage": ModelInfo("babbage", "openai", encoding="r50k_base"),
    "ada": ModelInfo("ada", "openai", encoding="r50k_base"),
    "gpt-3": ModelInfo("gpt-3", "openai", encoding="r50k_base"),
    "text-embedding-ada-002": ModelInfo("text-embedding-ada-002", "openai", encoding="cl100k_base"),
    "text-embedding-3-small": ModelInfo("text-embedding-3-small", "openai", encoding="cl100k_base"),
    "text-embedding-3-large": ModelInfo("text-embedding-3-large", "openai", encoding="cl100k_base"),
    "gpt-4-base": ModelInfo("gpt-4-base", "openai", encoding="cl100k_base"),
    "gpt-3.5-turbo-instruct-0914": ModelInfo("gpt-3.5-turbo-instruct-0914", "openai", encoding="cl100k_base"),
    "o1-preview": ModelInfo("o1-preview", "openai", encoding="cl100k_base"),
    "o1-mini": ModelInfo("o1-mini", "openai", encoding="cl100k_base"),
    "o1-preview-2024-09-12": ModelInfo("o1-preview-2024-09-12", "openai", encoding="cl100k_base"),
    "o1-mini-2024-09-12": ModelInfo("o1-mini-2024-09-12", "openai", encoding="cl100k_base"),
    "gpt-5.2-codex": ModelInfo("gpt-5.2-codex", "openai", encoding="cl100k_base"),
    "gpt-4-vision": ModelInfo("gpt-4-vision", "openai", encoding="cl100k_base"),
    "gpt-4-vision-preview-0409": ModelInfo("gpt-4-vision-preview-0409", "openai", encoding="cl100k_base"),
    "gpt-4-vision-preview-1106": ModelInfo("gpt-4-vision-preview-1106", "openai", encoding="cl100k_base"),
    "text-similarity-ada-001": ModelInfo("text-similarity-ada-001", "openai", encoding="r50k_base"),
    "text-similarity-babbage-001": ModelInfo("text-similarity-babbage-001", "openai", encoding="r50k_base"),
    "text-similarity-curie-001": ModelInfo("text-similarity-curie-001", "openai", encoding="r50k_base"),
    "text-similarity-davinci-001": ModelInfo("text-similarity-davinci-001", "openai", encoding="r50k_base"),

    # Anthropic
    "claude-3-opus-20240229": ModelInfo("claude-3-opus-20240229", "anthropic", formula="anthropic", pricing_id="claude-3-opus-20240229"),
    "claude-3-sonnet-20240229": ModelInfo("claude-3-sonnet-20240229", "anthropic", formula="anthropic", pricing_id="claude-3-sonnet-20240229"),
    "claude-3-haiku-20240307": ModelInfo("claude-3-haiku-20240307", "anthropic", formula="anthropic", pricing_id="claude-3-haiku-20240307"),
    "claude-3.5-sonnet-20240620": ModelInfo("claude-3.5-sonnet-20240620", "anthropic", formula="anthropic", pricing_id="claude-3.5-sonnet-20240620"),
    "claude-3.5-sonnet-20241022": ModelInfo("claude-3.5-sonnet-20241022", "anthropic", formula="anthropic", pricing_id="claude-3.5-sonnet-20241022"),
    "claude-3.5-haiku-20241022": ModelInfo("claude-3.5-haiku-20241022", "anthropic", formula="anthropic", pricing_id="claude-3.5-haiku-20241022"),
    "claude-3-5-sonnet-20240620": ModelInfo("claude-3-5-sonnet-20240620", "anthropic", formula="anthropic", pricing_id="claude-3.5-sonnet-20240620"),
    "claude-3-opus": ModelInfo("claude-3-opus", "anthropic", formula="anthropic", pricing_id="claude-3-opus-20240229"),
    "claude-3-sonnet": ModelInfo("claude-3-sonnet", "anthropic", formula="anthropic", pricing_id="claude-3-sonnet-20240229"),
    "claude-3-haiku": ModelInfo("claude-3-haiku", "anthropic", formula="anthropic", pricing_id="claude-3-haiku-20240307"),
    "claude-2.1": ModelInfo("claude-2.1", "anthropic", formula="anthropic"),
    "claude-2.0": ModelInfo("claude-2.0", "anthropic", formula="anthropic"),
    "claude-instant-1.2": ModelInfo("claude-instant-1.2", "anthropic", formula="anthropic"),
    "claude-instant-1.1": ModelInfo("claude-instant-1.1", "anthropic", formula="anthropic"),
    "claude-instant-1.0": ModelInfo("claude-instant-1.0", "anthropic", formula="anthropic"),
    "claude-instant": ModelInfo("claude-instant", "anthropic", formula="anthropic"),
    "claude-1": ModelInfo("claude-1", "anthropic", formula="anthropic"),
    "claude-1.3": ModelInfo("claude-1.3", "anthropic", formula="anthropic"),
    "claude-1.3-100k": ModelInfo("claude-1.3-100k", "anthropic", formula="anthropic"),
    "claude-3-5-haiku-20241022": ModelInfo("claude-3-5-haiku-20241022", "anthropic", formula="anthropic", pricing_id="claude-3.5-haiku-20241022"),
    "claude-3-5-sonnet-20241022": ModelInfo("claude-3-5-sonnet-20241022", "anthropic", formula="anthropic"),
    "claude-3.5-sonnet-computer-use": ModelInfo("claude-3.5-sonnet-computer-use", "anthropic", formula="anthropic"),
    "claude-2.1-200k": ModelInfo("claude-2.1-200k", "anthropic", formula="anthropic"),
    "claude-2.1-100k": ModelInfo("claude-2.1-100k", "anthropic", formula="anthropic"),
    "claude-instant-2": ModelInfo("claude-instant-2", "anthropic", formula="anthropic"),
    "claude-instant-2.0": ModelInfo("claude-instant-2.0", "anthropic", formula="anthropic"),
    "claude-3-opus-latest": ModelInfo("claude-3-opus-latest", "anthropic", formula="anthropic", pricing_id="claude-3-opus-20240229"),
    "claude-3-sonnet-latest": ModelInfo("claude-3-sonnet-latest", "anthropic", formula="anthropic", pricing_id="claude-3-sonnet-20240229"),
    "claude-sonnet-4-6": ModelInfo("claude-sonnet-4-6", "anthropic", formula="anthropic"),
    "claude-sonnet-4-5": ModelInfo("claude-sonnet-4-5", "anthropic", formula="anthropic"),
    "claude-opus-4.6": ModelInfo("claude-opus-4.6", "anthropic", formula="anthropic"),

    # Google
    "gemini-pro": ModelInfo("gemini-pro", "google", formula="google"),
    "gemini-pro-vision": ModelInfo("gemini-pro-vision", "google", formula="google"),
    "gemini-1.5-pro": ModelInfo("gemini-1.5-pro", "google", formula="google"),
    "gemini-1.5-flash": ModelInfo("gemini-1.5-flash", "google", formula="google"),
    "gemini-1.5-pro-latest": ModelInfo("gemini-1.5-pro-latest", "google", formula="google"),
    "gemini-1.5-flash-latest": ModelInfo("gemini-1.5-flash-latest", "google", formula="google"),
    "gemini-1.0-pro": ModelInfo("gemini-1.0-pro", "google", formula="google"),
    "gemini-1.0-pro-vision": ModelInfo("gemini-1.0-pro-vision", "google", formula="google"),
    "gemini-ultra": ModelInfo("gemini-ultra", "google", formula="google"),
    "gemini-2.0-flash-exp": ModelInfo("gemini-2.0-flash-exp", "google", formula="google"),
    "gemini-2.0-flash": ModelInfo("gemini-2.0-flash", "google", formula="google"),
    "gemini-exp-1206": ModelInfo("gemini-exp-1206", "google", formula="google"),
    "gemini-exp-1121": ModelInfo("gemini-exp-1121", "google", formula="google"),
    "palm-2": ModelInfo("palm-2", "google", formula="google"),
    "palm-2-chat": ModelInfo("palm-2-chat", "google", formula="google"),
    "palm-2-codechat": ModelInfo("palm-2-codechat", "google", formula="google"),
    "gemini-1.0-pro-001": ModelInfo("gemini-1.0-pro-001", "google", formula="google"),
    "gemini-1.0-pro-latest": ModelInfo("gemini-1.0-pro-latest", "google", formula="google"),
    "gemini-1.0-pro-vision-latest": ModelInfo("gemini-1.0-pro-vision-latest", "google", formula="google"),
    "gemini-3.1-pro-preview": ModelInfo("gemini-3.1-pro-preview", "google", formula="google"),

    # Meta
    "llama-2-7b": ModelInfo("llama-2-7b", "meta", formula="meta"),
    "llama-2-13b": ModelInfo("llama-2-13b", "meta", formula="meta"),
    "llama-2-70b": ModelInfo("llama-2-70b", "meta", formula="meta"),
    "llama-3-8b": ModelInfo("llama-3-8b", "meta", formula="meta"),
    "llama-3-70b": ModelInfo("llama-3-70b", "meta", formula="meta"),
    "llama-3.1-8b": ModelInfo("llama-3.1-8b", "meta", formula="meta"),
    "llama-3.1-70b": ModelInfo("llama-3.1-70b", "meta", formula="meta"),
    "llama-3.1-405b": ModelInfo("llama-3.1-405b", "meta", formula="meta"),
    "llama-3.2-1b": ModelInfo("llama-3.2-1b", "meta", formula="meta"),
    "llama-3.2-3b": ModelInfo("llama-3.2-3b", "meta", formula="meta"),
    "llama-3.3-70b": ModelInfo("llama-3.3-70b", "meta", formula="meta"),
    "llama-3.3-70b-instruct": ModelInfo("llama-3.3-70b-instruct", "meta", formula="meta"),
    "llama-2-7b-chat": ModelInfo("llama-2-7b-chat", "meta", formula="meta"),
    "llama-2-13b-chat": ModelInfo("llama-2-13b-chat", "meta", formula="meta"),
    "llama-2-70b-chat": ModelInfo("llama-2-70b-chat", "meta", formula="meta"),
    "llama-2-7b-chat-hf": ModelInfo("llama-2-7b-chat-hf", "meta", formula="meta"),
    "llama-2-13b-chat-hf": ModelInfo("llama-2-13b-chat-hf", "meta", formula="meta"),
    "llama-2-70b-chat-hf": ModelInfo("llama-2-70b-chat-hf", "meta", formula="meta"),
    "llama-3-8b-instruct": ModelInfo("llama-3-8b-instruct", "meta", formula="meta"),
    "llama-3-70b-instruct": ModelInfo("llama-3-70b-instruct", "meta", formula="meta"),
    "llama-3.1-8b-instruct": ModelInfo("llama-3.1-8b-instruct", "meta", formula="meta"),
    "llama-3.1-70b-instruct": ModelInfo("llama-3.1-70b-instruct", "meta", formula="meta"),
    "llama-3.1-405b-instruct": ModelInfo("llama-3.1-405b-instruct", "meta", formula="meta"),
    "llama-3.2-1b-instruct": ModelInfo("llama-3.2-1b-instruct", "meta", formula="meta"),
    "llama-3.2-3b-instruct": ModelInfo("llama-3.2-3b-instruct", "meta", formula="meta"),

    # Mistral
    "mistral-7b": ModelInfo("mistral-7b", "mistral", formula="mistral"),
    "mistral-8x7b": ModelInfo("mistral-8x7b", "mistral", formula="mistral"),
    "mistral-large": ModelInfo("mistral-large", "mistral", formula="mistral"),
    "mistral-medium": ModelInfo("mistral-medium", "mistral", formula="mistral"),
    "mistral-small": ModelInfo("mistral-small", "mistral", formula="mistral"),
    "mistral-tiny": ModelInfo("mistral-tiny", "mistral", formula="mistral"),
    "mixtral-8x7b": ModelInfo("mixtral-8x7b", "mistral", formula="mistral"),
    "mixtral-8x22b": ModelInfo("mixtral-8x22b", "mistral", formula="mistral"),
    "mistral-large-2": ModelInfo("mistral-large-2", "mistral", formula="mistral"),
    "mistral-large-2407": ModelInfo("mistral-large-2407", "mistral", formula="mistral"),
    "mistral-7b-instruct": ModelInfo("mistral-7b-instruct", "mistral", formula="mistral"),
    "mistral-7b-instruct-v0.1": ModelInfo("mistral-7b-instruct-v0.1", "mistral", formula="mistral"),
    "mistral-7b-instruct-v0.2": ModelInfo("mistral-7b-instruct-v0.2", "mistral", formula="mistral"),
    "mistral-7b-instruct-v0.3": ModelInfo("mistral-7b-instruct-v0.3", "mistral", formula="mistral"),
    "mixtral-8x7b-instruct": ModelInfo("mixtral-8x7b-instruct", "mistral", formula="mistral"),
    "mixtral-8x22b-instruct": ModelInfo("mixtral-8x22b-instruct", "mistral", formula="mistral"),

    # Cohere
    "command": ModelInfo("command", "cohere", formula="cohere"),
    "command-light": ModelInfo("command-light", "cohere", formula="cohere"),
    "command-nightly": ModelInfo("command-nightly", "cohere", formula="cohere"),
    "command-r": ModelInfo("command-r", "cohere", formula="cohere"),
    "command-r-plus": ModelInfo("command-r-plus", "cohere", formula="cohere"),
    "command-r-08-2024": ModelInfo("command-r-08-2024", "cohere", formula="cohere"),
    "command-r-plus-08-2024": ModelInfo("command-r-plus-08-2024", "cohere", formula="cohere"),
    "command-r-plus-04-2024": ModelInfo("command-r-plus-04-2024", "cohere", formula="cohere"),

    # Perplexity
    "pplx-7b-online": ModelInfo("pplx-7b-online", "perplexity", formula="perplexity"),
    "pplx-70b-online": ModelInfo("pplx-70b-online", "perplexity", formula="perplexity"),
    "pplx-7b-chat": ModelInfo("pplx-7b-chat", "perplexity", formula="perplexity"),
    "pplx-70b-chat": ModelInfo("pplx-70b-chat", "perplexity", formula="perplexity"),
    "codellama-34b-instruct": ModelInfo("codellama-34b-instruct", "perplexity", formula="perplexity"),

    # HuggingFace
    "microsoft/DialoGPT-medium": ModelInfo("microsoft/DialoGPT-medium", "huggingface", formula="huggingface"),
    "microsoft/DialoGPT-large": ModelInfo("microsoft/DialoGPT-large", "huggingface", formula="huggingface"),
    [REDACTED_SECRET]: ModelInfo([REDACTED_SECRET], "huggingface", formula="huggingface"),
    "facebook/blenderbot-1B-distill": ModelInfo("facebook/blenderbot-1B-distill", "huggingface", formula="huggingface"),
    "facebook/blenderbot-3B": ModelInfo("facebook/blenderbot-3B", "huggingface", formula="huggingface"),

    # AI21
    "j2-light": ModelInfo("j2-light", "ai21", formula="ai21"),
    "j2-mid": ModelInfo("j2-mid", "ai21", formula="ai21"),
    "j2-ultra": ModelInfo("j2-ultra", "ai21", formula="ai21"),
    "j2-jumbo-instruct": ModelInfo("j2-jumbo-instruct", "ai21", formula="ai21"),

    # Together
    [REDACTED_SECRET]: ModelInfo([REDACTED_SECRET], "together", formula="together"),
    [REDACTED_SECRET]: ModelInfo([REDACTED_SECRET], "together", formula="together"),
    [REDACTED_SECRET]: ModelInfo([REDACTED_SECRET], "together", formula="together"),

    # xAI
    "grok-1": ModelInfo("grok-1", "xai", formula="xai"),
    "grok-1.5": ModelInfo("grok-1.5", "xai", formula="xai"),
    "grok-2": ModelInfo("grok-2", "xai", formula="xai"),
    "grok-beta": ModelInfo("grok-beta", "xai", formula="xai"),

    # Alibaba
    "qwen-1.5-0.5b": ModelInfo("qwen-1.5-0.5b", "alibaba", formula="alibaba"),
    "qwen-1.5-1.8b": ModelInfo("qwen-1.5-1.8b", "alibaba", formula="alibaba"),
    "qwen-1.5-4b": ModelInfo("qwen-1.5-4b", "alibaba", formula="alibaba"),
    "qwen-1.5-7b": ModelInfo("qwen-1.5-7b", "alibaba", formula="alibaba"),
    "qwen-1.5-14b": ModelInfo("qwen-1.5-14b", "alibaba", formula="alibaba"),
    "qwen-1.5-32b": ModelInfo("qwen-1.5-32b", "alibaba", formula="alibaba"),
    "qwen-1.5-72b": ModelInfo("qwen-1.5-72b", "alibaba", formula="alibaba"),
    "qwen-1.5-110b": ModelInfo("qwen-1.5-110b", "alibaba", formula="alibaba"),
    "qwen-2-0.5b": ModelInfo("qwen-2-0.5b", "alibaba", formula="alibaba"),
    "qwen-2-1.5b": ModelInfo("qwen-2-1.5b", "alibaba", formula="alibaba"),
    "qwen-2-7b": ModelInfo("qwen-2-7b", "alibaba", formula="alibaba"),
    "qwen-2-57b": ModelInfo("qwen-2-57b", "alibaba", formula="alibaba"),
    "qwen-2-72b": ModelInfo("qwen-2-72b", "alibaba", formula="alibaba"),
    "qwen-vl": ModelInfo("qwen-vl", "alibaba", formula="alibaba"),
    "qwen-vl-chat": ModelInfo("qwen-vl-chat", "alibaba", formula="alibaba"),
    "qwen-vl-plus": ModelInfo("qwen-vl-plus", "alibaba", formula="alibaba"),
    "qwen-2.5-72b": ModelInfo("qwen-2.5-72b", "alibaba", formula="alibaba"),
    "qwen-2.5-32b": ModelInfo("qwen-2.5-32b", "alibaba", formula="alibaba"),
    "qwen-2.5-14b": ModelInfo("qwen-2.5-14b", "alibaba", formula="alibaba"),
    "qwen-2.5-7b": ModelInfo("qwen-2.5-7b", "alibaba", formula="alibaba"),

    # Baidu
    "ernie-4.0": ModelInfo("ernie-4.0", "baidu", formula="baidu"),
    "ernie-3.5": ModelInfo("ernie-3.5", "baidu", formula="baidu"),
    "ernie-3.0": ModelInfo("ernie-3.0", "baidu", formula="baidu"),
    "ernie-speed": ModelInfo("ernie-speed", "baidu", formula="baidu"),
    "ernie-lite": ModelInfo("ernie-lite", "baidu", formula="baidu"),
    "ernie-tiny": ModelInfo("ernie-tiny", "baidu", formula="baidu"),
    "ernie-bot": ModelInfo("ernie-bot", "baidu", formula="baidu"),
    "ernie-bot-4": ModelInfo("ernie-bot-4", "baidu", formula="baidu"),

    # Huawei
    "pangu-alpha-2.6b": ModelInfo("pangu-alpha-2.6b", "huawei", formula="huawei"),
    "pangu-alpha-13b": ModelInfo("pangu-alpha-13b", "huawei", formula="huawei"),
    "pangu-alpha-200b": ModelInfo("pangu-alpha-200b", "huawei", formula="huawei"),
    "pangu-coder": ModelInfo("pangu-coder", "huawei", formula="huawei"),
    "pangu-coder-15b": ModelInfo("pangu-coder-15b", "huawei", formula="huawei"),

    # Yandex
    "yalm-100b": ModelInfo("yalm-100b", "yandex", formula="yandex"),
    "yalm-200b": ModelInfo("yalm-200b", "yandex", formula="yandex"),
    "yagpt": ModelInfo("yagpt", "yandex", formula="yandex"),
    "yagpt-2": ModelInfo("yagpt-2", "yandex", formula="yandex"),

    # Stability
    "stablelm-alpha-3b": ModelInfo("stablelm-alpha-3b", "stability", formula="stability"),
    "stablelm-alpha-7b": ModelInfo("stablelm-alpha-7b", "stability", formula="stability"),
    "stablelm-base-alpha-3b": ModelInfo("stablelm-base-alpha-3b", "stability", formula="stability"),
    "stablelm-base-alpha-7b": ModelInfo("stablelm-base-alpha-7b", "stability", formula="stability"),
    "stablelm-tuned-alpha-3b": ModelInfo("stablelm-tuned-alpha-3b", "stability", formula="stability"),
    "stablelm-tuned-alpha-7b": ModelInfo("stablelm-tuned-alpha-7b", "stability", formula="stability"),
    "stablelm-zephyr-3b": ModelInfo("stablelm-zephyr-3b", "stability", formula="stability"),

    # TII
    "falcon-7b": ModelInfo("falcon-7b", "tii", formula="tii"),
    "falcon-7b-instruct": ModelInfo("falcon-7b-instruct", "tii", formula="tii"),
    "falcon-40b": ModelInfo("falcon-40b", "tii", formula="tii"),
    "falcon-40b-instruct": ModelInfo("falcon-40b-instruct", "tii", formula="tii"),
    "falcon-180b": ModelInfo("falcon-180b", "tii", formula="tii"),
    "falcon-180b-chat": ModelInfo("falcon-180b-chat", "tii", formula="tii"),

    # EleutherAI
    "gpt-neo-125m": ModelInfo("gpt-neo-125m", "eleutherai", formula="eleutherai"),
    "gpt-neo-1.3b": ModelInfo("gpt-neo-1.3b", "eleutherai", formula="eleutherai"),
    "gpt-neo-2.7b": ModelInfo("gpt-neo-2.7b", "eleutherai", formula="eleutherai"),
    "gpt-neox-20b": ModelInfo("gpt-neox-20b", "eleutherai", formula="eleutherai"),
    "pythia-70m": ModelInfo("pythia-70m", "eleutherai", formula="eleutherai"),
    "pythia-160m": ModelInfo("pythia-160m", "eleutherai", formula="eleutherai"),
    "pythia-410m": ModelInfo("pythia-410m", "eleutherai", formula="eleutherai"),
    "pythia-1b": ModelInfo("pythia-1b", "eleutherai", formula="eleutherai"),
    "pythia-1.4b": ModelInfo("pythia-1.4b", "eleutherai", formula="eleutherai"),
    "pythia-2.8b": ModelInfo("pythia-2.8b", "eleutherai", formula="eleutherai"),
    "pythia-6.9b": ModelInfo("pythia-6.9b", "eleutherai", formula="eleutherai"),
    "pythia-12b": ModelInfo("pythia-12b", "eleutherai", formula="eleutherai"),

    # MosaicML
    "mpt-7b": ModelInfo("mpt-7b", "mosaicml", formula="mosaicml"),
    "mpt-7b-chat": ModelInfo("mpt-7b-chat", "mosaicml", formula="mosaicml"),
    "mpt-7b-instruct": ModelInfo("mpt-7b-instruct", "mosaicml", formula="mosaicml"),
    "mpt-30b": ModelInfo("mpt-30b", "mosaicml", formula="mosaicml"),
    "mpt-30b-chat": ModelInfo("mpt-30b-chat", "mosaicml", formula="mosaicml"),
    "mpt-30b-instruct": ModelInfo("mpt-30b-instruct", "mosaicml", formula="mosaicml"),

    # Replit
    "replit-code-v1-3b": ModelInfo("replit-code-v1-3b", "replit", formula="replit"),
    "replit-code-v1.5-3b": ModelInfo("replit-code-v1.5-3b", "replit", formula="replit"),
    "replit-code-v2-3b": ModelInfo("replit-code-v2-3b", "replit", formula="replit"),

    # MiniMax
    "abab5.5-chat": ModelInfo("abab5.5-chat", "minimax", formula="minimax"),
    "abab5.5s-chat": ModelInfo("abab5.5s-chat", "minimax", formula="minimax"),
    "abab6-chat": ModelInfo("abab6-chat", "minimax", formula="minimax"),
    "abab6.5-chat": ModelInfo("abab6.5-chat", "minimax", formula="minimax"),
    "abab6.5s-chat": ModelInfo("abab6.5s-chat", "minimax", formula="minimax"),
    "minimax-m2.5": ModelInfo("minimax-m2.5", "minimax", formula="minimax_m25"),

    # Aleph Alpha
    "luminous-base": ModelInfo("luminous-base", "aleph_alpha", formula="aleph_alpha"),
    "luminous-extended": ModelInfo("luminous-extended", "aleph_alpha", formula="aleph_alpha"),
    "luminous-supreme": ModelInfo("luminous-supreme", "aleph_alpha", formula="aleph_alpha"),
    "luminous-supreme-control": ModelInfo("luminous-supreme-control", "aleph_alpha", formula="aleph_alpha"),

    # DeepSeek
    "deepseek-coder-1.3b": ModelInfo("deepseek-coder-1.3b", "deepseek", formula="deepseek"),
    "deepseek-coder-6.7b": ModelInfo("deepseek-coder-6.7b", "deepseek", formula="deepseek"),
    "deepseek-coder-33b": ModelInfo("deepseek-coder-33b", "deepseek", formula="deepseek"),
    "deepseek-coder-instruct": ModelInfo("deepseek-coder-instruct", "deepseek", formula="deepseek"),
    "deepseek-vl-1.3b": ModelInfo("deepseek-vl-1.3b", "deepseek", formula="deepseek"),
    "deepseek-vl-7b": ModelInfo("deepseek-vl-7b", "deepseek", formula="deepseek"),
    "deepseek-llm-7b": ModelInfo("deepseek-llm-7b", "deepseek", formula="deepseek"),
    "deepseek-llm-67b": ModelInfo("deepseek-llm-67b", "deepseek", formula="deepseek"),
    "deepseek-v3": ModelInfo("deepseek-v3", "deepseek", formula="deepseek"),
    "deepseek-v3-base": ModelInfo("deepseek-v3-base", "deepseek", formula="deepseek"),
    "deepseek-v3.2": ModelInfo("deepseek-v3.2", "deepseek", formula="deepseek"),

    # Tsinghua
    "chatglm-6b": ModelInfo("chatglm-6b", "tsinghua", formula="tsinghua"),
    "chatglm2-6b": ModelInfo("chatglm2-6b", "tsinghua", formula="tsinghua"),
    "chatglm3-6b": ModelInfo("chatglm3-6b", "tsinghua", formula="tsinghua"),
    "glm-4": ModelInfo("glm-4", "tsinghua", formula="tsinghua"),
    "glm-4v": ModelInfo("glm-4v", "tsinghua", formula="tsinghua"),
    "glm-5": ModelInfo("glm-5", "tsinghua", formula="glm5"),

    # Volcengine (Doubao)
    "doubao-seed-2-0-pro": ModelInfo("doubao-seed-2-0-pro", "volcengine", formula="volcengine"),

    # Moonshot (Kimi)
    "kimi-k2.5": ModelInfo("kimi-k2.5", "moonshot", formula="moonshot"),

    # RWKV
    "rwkv-4-169m": ModelInfo("rwkv-4-169m", "rwkv", formula="rwkv"),
    "rwkv-4-430m": ModelInfo("rwkv-4-430m", "rwkv", formula="rwkv"),
    "rwkv-4-1b5": ModelInfo("rwkv-4-1b5", "rwkv", formula="rwkv"),
    "rwkv-4-3b": ModelInfo("rwkv-4-3b", "rwkv", formula="rwkv"),
    "rwkv-4-7b": ModelInfo("rwkv-4-7b", "rwkv", formula="rwkv"),
    "rwkv-4-14b": ModelInfo("rwkv-4-14b", "rwkv", formula="rwkv"),
    "rwkv-5-world": ModelInfo("rwkv-5-world", "rwkv", formula="rwkv"),

    # Community
    "vicuna-7b": ModelInfo("vicuna-7b", "community", formula="community"),
    "vicuna-13b": ModelInfo("vicuna-13b", "community", formula="community"),
    "vicuna-33b": ModelInfo("vicuna-33b", "community", formula="community"),
    "alpaca-7b": ModelInfo("alpaca-7b", "community", formula="community"),
    "alpaca-13b": ModelInfo("alpaca-13b", "community", formula="community"),
    "wizardlm-7b": ModelInfo("wizardlm-7b", "community", formula="community"),
    "wizardlm-13b": ModelInfo("wizardlm-13b", "community", formula="community"),
    "wizardlm-30b": ModelInfo("wizardlm-30b", "community", formula="community"),
    "orca-mini-3b": ModelInfo("orca-mini-3b", "community", formula="community"),
    "orca-mini-7b": ModelInfo("orca-mini-7b", "community", formula="community"),
    "orca-mini-13b": ModelInfo("orca-mini-13b", "community", formula="community"),
    "zephyr-7b-alpha": ModelInfo("zephyr-7b-alpha", "community", formula="community"),
    "zephyr-7b-beta": ModelInfo("zephyr-7b-beta", "community", formula="community"),

    # Microsoft
    "phi-3-mini": ModelInfo("phi-3-mini", "microsoft", formula="microsoft"),
    "phi-3-small": ModelInfo("phi-3-small", "microsoft", formula="microsoft"),
    "phi-3-medium": ModelInfo("phi-3-medium", "microsoft", formula="microsoft"),
    "phi-3.5-mini": ModelInfo("phi-3.5-mini", "microsoft", formula="microsoft"),

    # Amazon
    "titan-text-express": ModelInfo("titan-text-express", "amazon", formula="amazon"),
    "titan-text-lite": ModelInfo("titan-text-lite", "amazon", formula="amazon"),
    "titan-embed-text": ModelInfo("titan-embed-text", "amazon", formula="amazon"),

    # Nvidia
    "nemotron-4-340b": ModelInfo("nemotron-4-340b", "nvidia", formula="nvidia"),
    "nemotron-3-8b": ModelInfo("nemotron-3-8b", "nvidia", formula="nvidia"),

    # IBM
    "granite-13b-chat": ModelInfo("granite-13b-chat", "ibm", formula="ibm"),
    "granite-13b-instruct": ModelInfo("granite-13b-instruct", "ibm", formula="ibm"),
    "granite-20b-code": ModelInfo("granite-20b-code", "ibm", formula="ibm"),

    # Salesforce
    "codegen-16b": ModelInfo("codegen-16b", "salesforce", formula="salesforce"),
    "codegen-6b": ModelInfo("codegen-6b", "salesforce", formula="salesforce"),
    "codegen-2b": ModelInfo("codegen-2b", "salesforce", formula="salesforce"),

    # BigCode
    "starcoder": ModelInfo("starcoder", "bigcode", formula="bigcode"),
    "starcoder2-15b": ModelInfo("starcoder2-15b", "bigcode", formula="bigcode"),
    "starcoderbase": ModelInfo("starcoderbase", "bigcode", formula="bigcode"),
    "starcoder2-3b": ModelInfo("starcoder2-3b", "bigcode", formula="bigcode"),
    "starcoder2-7b": ModelInfo("starcoder2-7b", "bigcode", formula="bigcode"),
    "starcoder-plus": ModelInfo("starcoder-plus", "bigcode", formula="bigcode"),
    "starcoderbase-1b": ModelInfo("starcoderbase-1b", "bigcode", formula="bigcode"),
    "starcoderbase-3b": ModelInfo("starcoderbase-3b", "bigcode", formula="bigcode"),
    "starcoderbase-7b": ModelInfo("starcoderbase-7b", "bigcode", formula="bigcode"),

    # Databricks
    "dbrx": ModelInfo("dbrx", "databricks", formula="databricks"),
    "dbrx-instruct": ModelInfo("dbrx-instruct", "databricks", formula="databricks", pricing_id="dbrx-instruct"),
    "dbrx-base": ModelInfo("dbrx-base", "databricks", formula="databricks", pricing_id="dbrx-base"),
    "dolly-v2-12b": ModelInfo("dolly-v2-12b", "databricks", formula="databricks", pricing_id="dolly-v2-12b"),
    "dolly-v2-7b": ModelInfo("dolly-v2-7b", "databricks", formula="databricks", pricing_id="dolly-v2-7b"),
    "dolly-v2-3b": ModelInfo("dolly-v2-3b", "databricks", formula="databricks", pricing_id="dolly-v2-3b"),

    # Voyage
    "voyage-2": ModelInfo("voyage-2", "voyage", formula="voyage", pricing_id="voyage-2"),
    "voyage-large-2": ModelInfo("voyage-large-2", "voyage", formula="voyage", pricing_id="voyage-large-2"),
    "voyage-code-2": ModelInfo("voyage-code-2", "voyage", formula="voyage", pricing_id="voyage-code-2"),
    "voyage-finance-2": ModelInfo("voyage-finance-2", "voyage", formula="voyage", pricing_id="voyage-finance-2"),
    "voyage-law-2": ModelInfo("voyage-law-2", "voyage", formula="voyage", pricing_id="voyage-law-2"),
    "voyage-multilingual-2": ModelInfo("voyage-multilingual-2", "voyage", formula="voyage", pricing_id="voyage-multilingual-2"),
}


def _build_alias_map(models: Iterable[ModelInfo]) -> Dict[str, str]:
    alias_map: Dict[str, str] = {}
    for model in models:
        if model.aliases:
            for alias in model.aliases:
                alias_map[normalize_model_name(alias)] = model.name
    return alias_map


ALIASES: Dict[str, str] = _build_alias_map(MODELS.values())
MODEL_LOOKUP: Dict[str, ModelInfo] = {normalize_model_name(k): v for k, v in MODELS.items()}
MODEL_LOOKUP.update({alias: MODELS[target] for alias, target in ALIASES.items()})

# All normalized model names for fuzzy matching
_ALL_NORMALIZED_NAMES: List[str] = list(MODEL_LOOKUP.keys())


def _extract_model_part(name: str) -> str:
    """Extract model part from 'provider/model' or 'provider:model' format."""
    name = name.strip()
    for sep in ("/", ":"):
        if sep in name:
            return name.split(sep, 1)[-1].strip()
    return name


def _fuzzy_match_model(model_name: str) -> Optional[ModelInfo]:
    """Fuzzy match model name when exact lookup fails."""
    normalized = normalize_model_name(model_name)
    model_part = normalize_model_name(_extract_model_part(model_name))

    # 1. Try model part only (e.g. anthropic/claude-sonnet-4-6 -> claude-sonnet-4-6)
    if model_part != normalized:
        info = MODEL_LOOKUP.get(model_part)
        if info:
            return info

    # 2. Substring match: input contains a known model, or known model contains input
    for key, info in MODEL_LOOKUP.items():
        if model_part in key or key in model_part:
            return info

    # 3. difflib fuzzy match (cutoff 0.5 = 50% similarity)
    candidates = difflib.get_close_matches(model_part, _ALL_NORMALIZED_NAMES, n=1, cutoff=0.5)
    if candidates:
        return MODEL_LOOKUP.get(candidates[0])

    # 4. Relaxed: match key parts (e.g. claude-sonnet-4 matches claude-3.5-sonnet)
    model_tokens = set(re.split(r"[-_.]", model_part)) - {""}
    for key in _ALL_NORMALIZED_NAMES:
        key_tokens = set(re.split(r"[-_.]", key)) - {""}
        overlap = len(model_tokens & key_tokens) / max(len(model_tokens), 1)
        if overlap >= 0.6:
            return MODEL_LOOKUP.get(key)

    return None


def get_model_info(model_name: str) -> Optional[ModelInfo]:
    info = MODEL_LOOKUP.get(normalize_model_name(model_name))
    if info:
        return info
    return _fuzzy_match_model(model_name)


def get_all_supported_models() -> List[str]:
    return sorted(set(MODELS.keys()))


def get_supported_models_by_provider() -> Dict[str, List[str]]:
    models_by_provider: Dict[str, List[str]] = {}
    for model in MODELS.values():
        models_by_provider.setdefault(model.provider, []).append(model.name)
    for provider, models in models_by_provider.items():
        models_by_provider[provider] = sorted(set(models))
    return dict(sorted(models_by_provider.items()))


path: scripts/registry/pricing.py
content: """Pricing registry for scripts."""

from typing import Optional

USD_TO_INR = 83.0  # Conversion rate as of July 10 2025

PRICING = {
    "gpt-4": {"input": 0.03, "output": 0.06},
    "gpt-4-32k": {"input": 0.06, "output": 0.12},
    "dbrx-instruct": {"input": 0.001, "output": 0.002},
    "dbrx-base": {"input": 0.001, "output": 0.002},
    "dolly-v2-12b": {"input": 0.001, "output": 0.002},
    "dolly-v2-7b": {"input": 0.001, "output": 0.002},
    "dolly-v2-3b": {"input": 0.001, "output": 0.002},
    "voyage-2": {"input": 0.0001, "output": 0.0001},
    "voyage-large-2": {"input": 0.0001, "output": 0.0001},
    "voyage-code-2": {"input": 0.0001, "output": 0.0001},
    "voyage-finance-2": {"input": 0.0001, "output": 0.0001},
    "voyage-law-2": {"input": 0.0001, "output": 0.0001},
    "voyage-multilingual-2": {"input": 0.0001, "output": 0.0001},
    "gpt-4-turbo": {"input": 0.01, "output": 0.03},
    "gpt-4o": {"input": 0.005, "output": 0.015},
    "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
    "gpt-3.5-turbo": {"input": 0.001, "output": 0.002},
    "gpt-3.5-turbo-16k": {"input": 0.003, "output": 0.004},
    "claude-3-opus-20240229": {"input": 0.015, "output": 0.075},
    "claude-3-sonnet-20240229": {"input": 0.003, "output": 0.015},
    "claude-3-haiku-20240307": {"input": 0.00025, "output": 0.00125},
    "claude-3.5-sonnet-20240620": {"input": 0.003, "output": 0.015},
    "claude-3.5-sonnet-20241022": {"input": 0.003, "output": 0.015},
    "claude-3.5-haiku-20241022": {"input": 0.001, "output": 0.005},
    "claude-3-5-sonnet-20240620": {"input": 0.003, "output": 0.015},
}


def get_pricing_rate(model: str, input_tokens: bool) -> Optional[float]:
    pricing = PRICING.get(model)
    if not pricing:
        return None
    return pricing["input" if input_tokens else "output"]


path: scripts/registry/__init__.py
content: """Registry package for model metadata, pricing, and aliases."""

from .models import (  # noqa: F401
    ModelInfo,
    get_all_supported_models,
    get_model_info,
    get_supported_models_by_provider,
    normalize_model_name,
)
from .pricing import USD_TO_INR, get_pricing_rate  # noqa: F401


path: scripts/__init__.py
content: """
scripts CLI package.

This package powers the scripts command-line interface for counting tokens and
estimating costs across a large catalog of LLM models.
"""

__version__ = "1.0.10"
__author__ = "Raja CSP Raman"
__email__ = "[REDACTED_SECRET]"


path: tests/__init__.py
content: """Tests for prompt_token_counter."""


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)