CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM
  APIs
Authors: name: Guoheng Sun; name: Ziyao Wang; name: Bowei Tian; name: Meng Liu; name: Zheyu Shen; name: Shwai He; name: Yexiao He; name: Wanghao Ye; name: Yiting Wang; name: Ang Li
Published: 2025-05-19 23:39:23

Abstract: As post-training techniques evolve, large language models (LLMs) are
increasingly augmented with structured multi-step reasoning abilities, often
optimized through reinforcement learning. These reasoning-enhanced models
outperform standard LLMs on complex tasks and now underpin many commercial LLM
APIs. However, to protect proprietary behavior and reduce verbosity, providers
typically conceal the reasoning traces while returning only the final answer.
This opacity introduces a critical transparency gap: users are billed for
invisible reasoning tokens, which often account for the majority of the cost,
yet have no means to verify their authenticity. This opens the door to token
count inflation, where providers may overreport token usage or inject
synthetic, low-effort tokens to inflate charges. To address this issue, we
propose CoIn, a verification framework that audits both the quantity and
semantic validity of hidden tokens. CoIn constructs a verifiable hash tree from
token embedding fingerprints to check token counts, and uses embedding-based
relevance matching to detect fabricated reasoning content. Experiments
demonstrate that CoIn, when deployed as a trusted third-party auditor, can
effectively detect token count inflation with a success rate reaching up to
94.7%, showing the strong ability to restore billing transparency in opaque LLM
services. The dataset and code are available at
https://github.com/CASE-Lab-UMD/LLM-Auditing-CoIn.