Title: CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIs
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Large Language Models; API Transparency; Trustworthy AI; Merkle Tree; Embedding Models; Reasoning Models

Abstract: As post-training techniques evolve, large language models (LLMs) are increasingly augmented with structured multi-step reasoning abilities, often optimized through reinforcement learning. These reasoning-enhanced models outperform standard LLMs on complex tasks and  now underpin many commercial LLM APIs. However, to protect proprietary behavior and reduce verbosity, providers typically conceal the reasoning traces while returning only the final answer. This opacity introduces a critical \textbf{transparency gap}: users are billed for invisible reasoning tokens, which often account for the majority of the cost, yet have no means to verify their authenticity. This opens the door to \emph{token count manipulation}, where providers may overreport token usage or inject synthetic, low-effort tokens to inflate charges, a threat that can be carried out at \emph{near-zero cost}. To address this issue, we propose \texttt{CoIn}, a verification framework that audits both the \textit{quantity} and \textit{semantic validity} of hidden tokens. \texttt{CoIn} constructs a verifiable hash tree from token embedding fingerprints to check token counts, and uses embedding-based relevance matching to detect fabricated reasoning content. Experiments demonstrate that \texttt{CoIn}, when deployed as a trusted third-party auditor, can effectively detect token count manipulation with a success rate reaching up to 94.7\%, showing the strong ability to restore billing transparency in opaque LLM services. The dataset and code are available at \url{https://anonymous.4open.science/r/LLM-Auditing-CoIn-20F0 }.