TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Authors: Jack Min Ong
Matthew Di Ferrante
Aaron Pazdera
Ryan Garner
Sami Jaghouar
Manveer Basra
Max Ryabinin
Johannes Hagemann
Venue: ICML 2025 poster (2025)
Keywords: Locality Sensitive Hashing
Large Language Models
Verifiable Computing
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: The paper describes a method to verify LLM inference performed by untrusted compute providers

Abstract: Large language models (LLMs) have proven to be very capable, but access to frontier models currently relies on inference providers.
This introduces trust challenges: how can we be sure that the provider is using the model configuration they claim?
We propose TOPLOC, a novel method for verifiable inference that addresses this problem.
TOPLOC leverages a compact locality-sensitive hashing mechanism for intermediate activations, which can detect unauthorized modifications to models, prompts, or precision with 100\% accuracy, achieving no false positives or negatives in our empirical evaluations.
Our approach is robust across diverse hardware configurations, GPU types, and algebraic reorderings, which allows for validation speeds significantly faster than the original inference.
By introducing a polynomial encoding scheme, TOPLOC minimizes the memory overhead of the generated proofs by $1000\times$, requiring only 258 bytes of storage per 32 new tokens, compared to the 262 KB requirement of storing the token embeddings directly for Llama 3.1-8B-Instruct.
Our method empowers users to verify LLM inference computations efficiently, fostering greater trust and transparency in open ecosystems and laying a foundation for decentralized, verifiable and trustless AI services.

ToPLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Jack Min Ong' Matthew Di Ferrante' Aaron Pazdera'
Ryan Garner' Johannes Hagemann
Sami Jaghouar' Manveer Basra
Max Ryabinin
Abstract
model outputs via APIs, have risen to meet the demands of users who lack the resources or expertise to operate large- scale inference pipelines themselves.
Large language models (LLMs) have proven to be very capable, but access to frontier models currently relies on inference providers. This intro- duces trust challenges: how we be sure that the provider is using the model configuration they claim? We propose ToPLoC, novel method for verifiable inference that addresses this problem. TopLoC leverages a compact locality-sensitivy hashing mechanism for intermediate activations, which can detect unauthorized modifications models, prompts, or precision with 100% accu- racy, achieving no false positives or negatives in our empirical evaluations. Our approach is ro- bust across diverse hardware configurations, GPU types, and algebraic reorderings, which allows for validation speeds significantly faster than the original inference. By introducing a polynomial encoding scheme, ToPLoC minimizes the mem- ory overhead of the generated proofs by 1000 requiring only 258 bytes of storage per 32 new tokens, compared to the 262 KB requirement of storing the token embeddings directly for Llama 3.I-SB-Instruct. Our method empowers users to verify LLM inference computations efficiently, fostering greater trust and transparency in open ecosystems and laying a foundation for decentral- ized, verifiable and trustless Al services.
However, a critical challenge arises in this open ecosystem: trust. Users must trust that an inference provider is faith- fully serving the model as advertised, without undisclosed modifications. A provider could, for instance, secretly re- duce numerical precision to cut costs, fine-tune the model to introduce certain biases, or prepend an undisclosed system prompt to steer the model's outputs. Without robust verifica- tion methods, users can only rely on the provider's claims, leaving them vulnerable to having their outputs tampered with (Chen et al., 2023). There is thus a need for verifiable inference methods of verifying that a certain model and prompt were used during the inference computation.
A standard approach to perform this verification is to use cryptographically verifiable computing methods (Sun et al., 2024a; Modulus Labs, 2023; Kang et al., 2022; Sun et al., 2023; Ghodsi et al., 2017). However, they are either restric- tive in the operations that are supported such that LLMs cannot be used or are currently too computationally expen- sive to be practically applied to LLM inference.
Another approach is to record the model's intermediate activation tensors during inference (Sun et al., 2024b; Zhang et al., 2024). By making these activations available, a third party could independently rerun the model on the same inputs and verify that the intermediate computations match, thus ensuring the authenticity of inference. However, direct storage of these intermediate tensors as the proof can be prohibitively expensive at scale. For example, storing the last hidden activations of 100,000 queries consisting of 4096 tokens for Llama 3.1-70B would take 6.7 terabytes.
1. Introduction
In recent years, large language models (LLMs) have trans- formed natural language processing, enabling capabilities such as high-quality text generation, advanced dialogue systems, and improved reasoning (Grattafiori et al., 2024; Gemma Team et al., 2024). Inference providers, entities that run wein-wssghts LLMs on their own hardware and expose
In this work, we propose ToPLoC, an inference verification method that can reduce the storage cost of the proof by more than 1000x while still maintaining the security guarantees of checking intermediate activations. The method performs locality-sensitivv sensitive hashing of the intermediate activations, which encodes the top-k values and indices as polynomial congruence. This polynomial congruence can then be com- pared against a tensor obtained from recomputation. Our method is also robust nondeterminism of GPU operations and algebraic reorderings of the computation, which allows
'rime Intellect Together AI. Correspondence to: Johannes Hagemann <johannes @ joohannss@pimminnelllltata primeintellect.ii...
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
the validation of the proof to be done significantly faster than the original inference.
tween the final layer activations and labels derived from the inference input produce a fingerprint and then mitigate the ability of an attacker to reverse engineer the proxy model by regularly rotating secret in the form of an appended vector. However, the security of the scheme requires retraining of the proxy model by a trusted third party. Moreover, also requires that the providers cannot obtain the client secret, which they may obtain by also being a client themselves.
Our contributions are as follows:
We present a novel model inference hashing method called TopLoc, which is easy to implement in modern inference engines with minimal overhead.
We show that it is capable of detecting when a differ- ent model, prompt, or precision was used with 100% accuracy in our experiments.
Verisplit (Zhang al., 2024) proposes the construction of Merkle tree hash compression of portion of the activations. However, this compression utilizes a cryptographic hash function, rendering it incompatible with nondeterministic computation caused by GPU kernel scheduling, as well as algebraic reordering of the computations.
We verify that the method is robust to reordering of the computation caused by different GPU types, ten- sor parallel dimensions, as well as different attention kernel implementations.
3. Background
We propose method of reducing the memory require- ments of storing comparable points by encoding them as polynomial, requiring a proof size of only 258 bytes for every 32 new tokens generated.
3.1. Inference Modifications
Inference providers often make adjustments to computation methods to optimize for cost, efficiency, or specific commer- cial goals. While these modifications can make inference more economical and scalable, they may also impact the quality, and transparency of the service provided to users.
2. Related Work
Numerous methods have been proposed to verify the correct- ness of LLM inference performed by untrusted entities. The methods can be categorized into cryptographic verifiable computing and activation-based validation.
Lower precision. Inference providers might use lower precision formats, such fp8 (Micikevicius et al., 2022) or 6 (Kalamkar et al., 2019), which significantly reduces the inference compute and memory requirements.
Cryptographic Verifiable Computing. Cryptographic Ver- ifiable Computing allows one to verify that a computation was performed correctly on an untrusted computing provider using mathematical proofs. These techniques have been ap- plied to machine learning models and neural networks (Sun et al., 2024a; Modulus Labs, 2023; Sun et al., 2023; Kang et al., 2022; Ghodsi et al., 2017). However, most of them require the computations be expressed an arithmetic circuit, limiting the functions that can be used in the model. The translation of the model to arithmetic circuit also hurts the model quality and makes the proof generation schemes unable to utilize optimized inference engines such as V TensorRT and SGLang".
KV cache compression. Intermediate tensors can be com- pressed to enable longer and faster generations with a slightly reduced response quality (Shi al., 2024).
Altered model weights. Providers may distill, merge, prune weights to reduce compute and memory requirements.
Altered system prompt. Providers could modify the system prompt to align with their commercial goals, incorporate specific biases, or prioritize certain outcomes.
3.2. Nondeterminism in Model Inference on GPU
Nondeterminism in computations performed on GPUs can arise from operation scheduling and differences in how in- termediate results are handled (Monniaux, 2008; Whitehead & Fit-Florea, 2011 1).
Moreover, the size of modern LLMs introduces substantial computational overhead for for these methods. zkL.LM (Sun al., 2024a) takes 986 seconds generate proof that takes 803 seconds to validate for each inference computation for LLaMa 23B This would mean that a single request that returns 2000 new outputs tokens would require about 23 days to generate the proof for and then 18 days to validate.
Discrepancies in GPU computations can also arise from algebraic rewrites of the computation. These rewrites are often employed to improve the computational intensity of scheduled kernels, increase efficiency, and allow for paral- lelization across multiple GPUs.
Activation-based validation. SVIP (Sun al., 2024b) proposes training a proxy model to map the relationship be-
Furthermore, there several other causes for variability in computation results when running large language models on GPUs. For example, different GPU models often differ in
2

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
hardware architecture and precision handling. In addition to that, the CUDA versions determine the libraries and kernels used during computation, and differences in those versions can affect inference. Moreover, subtle variations in how the attention mechanism and other layers are implemented can introduce subtle numerical discrepancies. Lastly, the partitioning and aggregation of tensors when using tensor parallelism can also cause numerical deviations.
We thus need find modulus, m, such that the function (xx = mod m is injective the set of indices.
During proof generation (Algorithm 1), the top-k indices and values are calculated and an injective modulus m is computed to uniquely map the indices. The indices and their corresponding values are encoded into a polynomial, which, along with the modulus, forms the proof.
While these numerical discrepancies may seem negligible, they can have large cascading effect across the long se- quence of computations in model inference. This ampli- fication can cause subtle but meaningful variations in the model's 's output, making reproducibility and consistency of results significant challenge.
Algorithm ToPLoC Proof Generation Algorithm
1: Input: Hidden state tensor h, top-k parameter A
2: Output: Encoded proof p
3:
4: (i, +- (h, k) (Find top-k indices and values)
m t FinsInjeetivehocuuuusse
6: im i m
3.3. Source of Error in Transformer Models
7: +
v)
8: p encode(m, P(r))
In 6 computations of transformers (Vaswani et al., 2017), most errors arise from the appearance of exact zeros. These zeros are the result of catastrophic cancellations (Goldberg, 1991) within the residual stream of the attention layer. Be- cause the matrix multiplications involved in attention and MLP layers are nondeterministic (Golden et al., 2024), the occurrence of these exact zeros can be nondeterministic.
For validation (Algorithm 2), the proof is decoded to re- trieve k, m, and the polynomial. The top-k features are recalculated and compared against the proof by checking for differences in the exponent and mantissa. The validation succeeds if the number of exponent mismatches, mean man- tissa differences and median mantissa differences are below set of thresholds.
Interestingly, this behavior reveals a notable property: small values are more susceptible to rounding errors, whereas larger values tend be represented consistently after al- gebraic reordering. This insight motivates us to focus the larger values in tensor when designing the hash func- tion. By prioritizing large values, we can reduce the impact of rounding errors and improve the robustness of the hash function. We verify this empirically in Section 5.3.
Algorithm 2 ToPLoC Proof Validation Algorithm
1: Input: Hidden state tensor h, encoded proof
2: Output: Boolean validity flag v
3:
k m, P(x) + decode(p)
5: (i, t topk(h, kt) (Find top-k indices and values)
6: im + + i
4. The ToPLoC Algorithm
7: (erre,
+ (0,
8: for j - 1 to k: do
The TOPLoC algorithm encodes and validates the most salient features of a hidden state tensor using compact, verifiable proof, as detailed in Algorithms I and 2.
9: (ep, mp) + extractBBis((Pxh..]]
10: (Ce, my) +- extractBits((ii)
11: if ep e, then
Storing top-k indices and values directly is inefficient, re- quiring 6 bytes per point: 4 for the index and for the value. However, we can maintain comparability while storing less
12: erTm-append((mmp mlll)
13: else
14: err. + erre +1
by interpolating a polynomial that passes through the points generated by the top-k indices and values. Given ie points, there always exists unique A 1-degree polynomial that goes through the points which can be represented by & co- efficients. We thus only need to store the & coefficients for the proof, each of which consists of 2 bytes.
15: end if
16: end for
17: erre > Texp or mean(erT,)) > T Tmean or
median(err,)) Tmedian then 18: return False
19: end if
20: return True
In order to avoid floating-poit point issues with the polynomial, we interpolate polynomial congruence in the integer field instead of in the real numbers. However, this means we need to find a reproducible unique mapping of the x values into the modulus group, as we cannot interpolate a polyno- mial that yields different values for the same value of x.
Appendix C deeaails the subroutines and their respective theoretical computational complexities. The corresponding implementations are also available on GitHub'.
ig thub. www/ Intel Itteeeettttteeeeeepalll
3

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Generation (Prefill + New Tokens x Decode)
Validation (Prefill only)
Decoder Block
Decoder Block 0
Decoder Block
Decoder Block
Decoder Block n-1
Decoder Block n-1
LM Head
LM Head
Figure 1. When generating the response, we need perform one prefill for the input tokens and then multiple decodes each new token generated. When validating, we can pass all the tokens at once and perform just one prefill. Decodes are not efficient GPUs because they are memory-
(Agrawal et al., 2024). Notice that sequential nature generation causes to need the decoder blocks multiple times in the generation computation. This increases the total amount of data movement required to perform the computation. The decodes are thus bottlenecked by
takes to move data from GPU HBM (High Bandwidth Memory) to SRAM (Shared Memory).
5. Experimental Validation
trophic cancellations. It contains only bits of mantissa, as opposed to 23 mantissa bits for fp32 and 10 bits for fp16.
5.1. Dataset and Models
For all of our experiments, we perform the generation au- toregressively, generating each new token separately with KV caching. During validation, obtain the last hidden state activations for all tokens at once. This allows the vali- dation to be done significantly faster than the generation, as the prefill is significantly more compute intensive than the autoregressive decoding and thus able to better utilize the GPU resources (Agrawal et al., 2024; 2023).
For our experiments. we use the UltraChat dataset (Ding et al., 2023). The UltraChat dataset contains 1.4 million dialogues consisting of real-world inquiries, creative writing prompts, and various other text-based tasks such as rewrit- ing, continuation, summarization, and inference, covering wide range of topics.
compute-intensive
We conduct experiments with three models: Llama 3. 3.1-8B-instrun (Grattafiori et al., 2024), INTELLECT- 1 1-instruct (Jaghouar et al., 2024), and Gemma-2-9b- (Gemma Team et al., 2024), aiming to capture diversity across different architectural dimensions. Llama-3.1 3.10--3..-BBB.. instruct and Intellect- -instruct share similar transformer block architecture but differ the number of layers, while Gemma-2-99 2-9b-it features different hidden dimension and MI.P activation function.
For thresholds, use Texp 38, Tmean 10 and Tmedian = for bf16 inference and Teep 8, Tmeon 256 and Teedian = 128 for fp32 inference. These thresh- olds were chosen based on our analysis of the error statistics in Table 2 and Table 5.
5.3. High-Magnitude Activations Have Low Error Rates
As we wish to distinguish inference results using the top-k magnitude elements in the activations, a key assumption of our method is that high-magnitude elements in the activa- tions are less prone errors. In Section 3.3, we present the theoretical basis for this hypothesis. Here, we collect the experimental evidence supporting
5.2. Experiment Setup
We the 6 precision for all our experiments, unless specified otherwise. bf16 is commonly used in practice for activations in language model inference. However, com- pared to fp16 and fp32 precision, is most prone to catas-

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Table Exponent bit for the 2048th decoded token across various top-k values in 2000 queries using Llama-3.1 1-----3-------ttrteeee
Exact Match
Small Deviations
Larger Deviations
Top-k
(0)
(-1)
(1)
(-2)
(2)
(3--10)
(3 10)
(110- 1100)
(2 1100)
64
126,973
512
508
4
128
254,693
761
$29
11
5
256
502.130
5,824
3,993
38
1
512
1,002,724
10,693
10,471
1024
2,023.123
13,159
11,222
342
2
150
2
2048
3,997.083
49,340
47,661
1,142
41
727
6
4096
7,495.155
296,584
298.716
27,350
27,169
15,569
15,386
16,071
Max non-errored
Median non-errored
Mean non-errored
1.3
Median
Median
Median
Abs (aantsssa Dif
Mean
Abs Mantiisra Diff
1.2
Mean
Manttisra Drff
Mean
1.1
1.2
5
1.0
0.9
500
1000
1500
2000
0
500
1000
1500
2000
500
1500
2000
Taken Index
Token Index
Token Index
Figure 2. Impact of token index on mantissa errors in the top 128 elements of the last hidden activations across 2000 queries. The errors increase as the token index grows because later tokens rely more on KV cache values with compounding errors. However, the increase moderate, suggesting that the errors grow in limited manner.
Table 1 presents the error in exponent bits for the 2048th de- coded token across various top-k values from 2000 sampled queries using Llama-3.1 the relationship between activation magnitude and error rate, notably that the magnitude of eπrors generally increases with higher values of top-k. Deviations with magnitude above 100 are the result of catastrophic cancellations and mostly appear in the bottom 50% of values in the tensor.
The findings highlight the mantissa as a useful indicator for validating computations. When the exponent bits matched, the mantissa deviations remain small despite hard- ware variability and algebraic reordering. This suggests that mantissa error mean and median statistics can effectively detect computational anomalies or attempts at manipulation.
The results highlight
5.5. Mismatch Rate for Different Values of Top-k
An issue with comparing top-k values is that the top-k in- dices may not be the same in the tensors being compared.
5.4. Deviations the Mantissa Are Small When the Exponent Bits Are Matched
Figure illustrates the mismatch error ratio for top-k indices across different models. The results demonstrate that the mismatch error decreases significantly with larger values of top-k. This is because the boundary between elements that are in the top-k and those that are narrowly excluded is the source of mismatch. This boundary becomes smaller relative to the set of top-k elements as the size the set increases, ultimately becoming when all tensor elements are used. For smaller top-k values, the maximum mismatch remains relatively low, suggesting that discrepancies in ele- ment alignment minimal even for small k. Furthermore, the median mismatch is consistently an order of magnitude lower than the maximum mismatch, indicating that most errors are minor and well within acceptable limits.
In floating point computations, mantissa deviations are often amplified by mismatched exponent bits but remain relatively small when the exponent bits match.
We analyzed the absolute differences in the mantissa for the top 128 elements of the last hidden layer activations across Our results,
2,000 queries using Llama shown in Figure 2, indicate an increase in mantissa errors as the token index increases. This increase occurs because the enrors in the KV cache compound, causing higher to- ken indices have higher deviation the inputs in the forward pass become more dependent on the cached values. However, the increase is moderate, suggesting that longer generations introduce only limited doating-point point precision errors, even higher token indices.
5

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Llama-3.1
Intellect-1-inee -on-sttutt---tuuuutttcc (10B)
Gemma-Z-9b-it_10n 100
0.150
0.08
0.06
Mismatched elements
0.125
Modian
0.06
Median
0.05
Median
0.100
0.04
0.075
0.04
0.03
0.050
0.02
0.01
X89
0.00
0.00
0
1000
2000
3000
4000
0
1000
2000
2000
4000
1000
2000
3000
Top-k
Top-k
Top-
Figure 3. Max and median mismatch error ratio for top-k k indices across different models. The error ratio is obtained by comparing the top-k indices as a set between the generation and the validation. The ratio decreases as the elements in the top-k set increases, indicating a slower growth in the number of elements slipping past the top-k cutoff.
These findings reveal that the top-k indices can be repro- duced reliably, even in the presence of numerical variations.
5.7. ToPLoC Distinguishes Models, Prompts and Compute Precision
To assess the ability of ToPLoC to differentiate between models, we generate proofs using four models: Llama 3.1- 8B-Instruct, Llama 3.1-70B-Instruct Intellect-1-Instrucc and Gemma-2-9b-it We then validate the generations using the same models and show the results in Table 4. When the models used for generation and validation are the same, the worst-case error statistics are below the threshold, indicat- ing that they would have passed validation. When they are different, the best-case error statistics exceed the threshold, indicating that they would all have failed validation.
5.6. Robustness across Different GPUs, Attention and Tensor Parallel Implementations
We conducted experiments to evaluate the robustness of ToPLoC across varying tensor parallel configurations, GPU hardware, and attention kernel implementations. The results demonstrate the method's 's reliability under diverse setups.
To assess robustness across tensor parallel configurations and GPU setups, we run the inference using vLLM (Kwon et al., 2023) with hook to obtain the top-k values from the last hidden layer. We thus use the vLLM implementation of tensor parallelism and PagedAttention. We generate 512 new tokens for 400 prompts. In order to reduce the amount of values we need to store, we save the top-128 values every 32 new tokens. This results in 512/32 = 16 sets of top-128 values for the decode activations. We also save a set of top- 128 values for the activations computed for the input prompt. The experiments were performed on models, Llama-3. I- 8B-Instruct, Intellect- 1 Instruct and Gemma---9bbll which are then aggregated.
We further evaluated ToPLoC's robustness by testing it on three types of altered prompts. Some of the alterations are long, while some are only 4 tokens. To test the method under different prompt lengths, we select prompts that are multiple sentences, one sentence and just words long.
The full prompts can be found in Appendix B.3, and they are split into the following categories:
Advertising: A system prompt that asks the model advertise vitamin supplement when asked about health and wellness-related topics.
We further evaluated TOPLoC's robustness by testing differ- ent attention kernel implementations for the generation and validation. We use Hugging Face Transformers and its ref- erences to the attention implementations, Flash Attention 2, PyTorch Scaled Dot Product Attention, and FlexAttention.
Avoidance: A system prompt that instructs the model to avoid talking about homelessness and poverty.
Taco: A short prompt that directs the model to always praise tacos in the response.
We report the worst-case eиror statistics for different tensor parallelism and GPU combinations in Table 2. Here, none of the error statistics exceed the thresholds proposed in Section 5.2. Lastly, we display the worst-case error statistics for attention kernel combinations in Table 2. The statistics do not exceed the proposed thresholds in Section 5.2, indicating that all generated proofs would have been accepted.
Table shows that for all prompt alterations, the best-case exponent mismatches are above the threshold of 38, indicat- ing that they would all have failed the validation.
We also tested ToPLoC for the ability to differentiate mod- els based on the compute precision. Specifically, we evalu- ated the success rate when using 32-bit floating point (fp32) and 16-bit Brain floating point (bf16) computations.
6

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Table Error statisties for validation with different tensor parallelism configurations, GPUs and attention kernel implementations. All the error statistics are below the thresholds (Terp 38, mean positive samples.
10. mediion
8), indicating that they are all classified correctly as
Generation
Validation Model
Max Top-k Mismatch
Max Exponent Mismatch
Max Mantissa
Max Mantissa
Model
Diff Mean
Diff Median
1xA100
IxA100
10 (T)
16 (12.50%)
5.06
Ix4090
(7.81%)
(14.06%)
4.68
2x4090
15 (11.72%)
19 (14.84%)
4.96
Ix4090
IxA100
9 (7.03%)
(12.50%)
6.30
Ix4090
9 (7.03%)
20 (15.62%)
4.03
3
2x4090
9 (7.03%)
20 (15.62%)
1.03
3
2x4090
IxA100
11 (8.59%)
15 (11.72%)
5.15
2
Ix4090
11 (8.59%
15 (11.72%)
5.16
4
2x4090
12 (9.38%)
15 (11.72%)
5.15
3
Flash
Flash
25 (19.53%)
28 (21.8%%
4.64
2
SDPA
11 (8.59%)
16 (12.50%)
4.25
Flex
(19.53%)
(21.8%%)
3.79
SDPA
Flash
8 (6.25%)
18 (14.06%)
3.85
SDPA
9 (7.03%)
(10.94%)
3.15
Flex
11 (8.59) (8.59%)
17 (13.28%
3.88
Flex
Flash
(7.81%)
(12.50%)
3.72
SDPA
9 (7.03%)
16 (12.50%)
4.02
Flex
7 (5.47%)
12 (9.38%)
3.43
2
tablished verifiable inference techniques like zkLLM (Sun et al., 2024a) and SVIP (Sun et al., 2024b). We also in- clude baseline where we directly store and validate the raw intermediate activations.
Table Error statistics different prompt alterations. The mini- mum error statistics for each of the prompt alterations are above the exponent mismatch threshold of 38, indicating that the samples are correctly classified negative samples.
Table 6 provides summary of these comparisons, under- scoring ToPLoC's superior efficiency and its robustness to non-deterministic GPU operations. Compared to previous methods the literature such zkLLM and SVIP, TopLoc has significantly reduced memory overhead. ToPLoC is also more reliable than prior methods and can be used out of the box for any model, unlike SVIP which requires the training of a detection model.
Prompt Alteration
Min Exponent Mismatch
Advert
95 (74.22 22%)
Avoid
74 (57.81%)
Taco
67 (52.34%)
Due to the differing number of mantissa bits between fp32 and 6 formats, we scaled the fp32 mantissa to match 6 when validating with bf16. Conversely, when validating a bf16 decode model with fp32, we padded 16 zero bits to the bf16 representation. These adjustments ensured fair comparisons despite the inherent differences in precision.
To quantify the storage efficiency of TOPLoC, we compare its memory footprint against a baseline of storing the nal hidden activations directly. We illustrate this using the smallest model tested: Llama- 1e-1--B-Instee model, which has hidden size of 4096. Storing the com- plete final hidden activations for every token generated in 6 precision would require 2 bytes per element for 4096 elements for a total of 8192 bytes per token. In contrast, ToPLoC stores the top-128 activation values, sampled every N=32 tokens, using a polynomial congruence represented by 128 coefficients. With each coefficient requiring 2 bytes, the total storage is 256 bytes per 32-token interval. This amor- tizes to 8 bytes per token, representing a x reduction compared to direct storage.
Table 5 contains the errors for the different combinations, showing that fp32 is able to pass validations with either precision, while b is only able to pass when the validator is also bf16, always failing if the validator uses fp32.
5.8. Overhead and detection rate compared to prior methods and baselines
To evaluate the practical advantages of ToPLoC, we com- pare its key performance metrics such as computational overhead, proof size, and detection accuracy against es-

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Table Error statisties for validation with different models. The upward arrow (1) indicates maximum value, while the downward arrow (J) indicates a minimum value. Given the thresholds Texp = 38, I Imaan 10, Tmedioo. = 8, the minimum errors for mismatching pairs are all above the threshold and are all correctly classified as negatives. The maximum errors for matching pairs are below the thresholds and are all correctly classified as positives. Mismatch combinations with Gemma-2-9b-it as the generation model are not shown because they will fail the validation by having an out-of-bound token in the completion due to the bigger vocabulary size Gemma-2 2-9b-it.
Generation Model
Validation Model
Top-k
Exponent Mismatch
Mantissa Diff
Mismatch
Mean/Median
1iama-3.--BB-nnstee Llama-3, 1-70B--BI Instruct
-Iama-3.1-8B-
f (6.2%)
1 (10.2%)
4.18/4 f
3amma-.1-770-Imaa
71 (3.5%
171 (13.3%)
2.39/2 f
Intellect- 1- I-Inttruct
Intellect- MeeellettI-Instruee IInstrucct
8 (6.2%)
8 (6.2%)
2.52/2 f
Gemma-2-48 2-9b-it
Gemma-2-9b-ir
17 1 (13.3%)
19 (14.8%)
5.38/2 1
Llama-3. 3.1 1-8B-Insttrue
Llama-3. 11-70B-Instrrun
126 | (98.44%)
128 L (100%)
Intelleet-1-Inen I-Instruct
122 L (95.31%)
125 (97.66%)
Gemma-2-9b-i
125 L (97.66%)
126 L (98.44%)
Llama-3. 1-170B--Isttrue
Llama-3. -lama-3.1-8B-Imm. 3.--Sut-ttt
127 + (99.22%)
128 4 (100%)
Intellect-1-Inan I-Instruct
127 4 (99.22%)
127 4 (99.22%)
Gemma-2-9b-it
126 + (98.11%)
126 4 (98.11%)
Intellect- 1- Innsrrutt
Llama-3 1.mma-3.11-BB- 3.1-8B- ..--B-Inntrrutt
126 L (98.44%)
127 J (99.22%)
Llama-3.1- 3.1-70B-
125 4 (97.66%)
127 L (99.22%)
Gemma-2-9b-it
126 L
(98, 44%)
126 L
(98. 44%)
Table 5. Error comparisons of different Generation- Validation precision combinations. The upward arrow (f) indicates maximum value, while the downward arrow (4) indicates minimum value. When the generation model is fp32, all the samples are positive and the maximum errors all below the thresholds (Texp = 38. Tmean 10, Tmedion 8). When the generation model is bf16, only the samples where we validate with the 6 model and threshold are positive. When we validate with the fp32 model and thresholds, the minimum mantissa differences above the thresholds, indicating that all of the samples correctly classified as negative.
Generation
Validation Model
Top-k Mismatch
Exponent Mismatch
Mantissa Diff Mean
Mantissa Diff Median
Model
fp32
fp32
1 t (0.78%)
It (0.78%)
180.38 1
48 f
b
6 f (4.69%)
14.t (10.94%)
4.38 1
21
bf16
fp32
0 4 (0.00%)
0 (0.00%)
27892.02 4
21683 J
6
9 f (7.03%)
161 (12.50%)
6.30 t
3 1
6. Limitations and future work
6.2. Speculative decoding and sampling
6.1. FP8 Inference and KV-cache compression
Our method is not capable of detecting speculative decod- ing, scenario where provider uses a cheaper model for decoding while relying on the larger model only for prefill computations. In such cases, the provider can generate the tokens using the small model and the prefill vectors using the large model, split them into chunks, and calculate hashes to pass the verification process. Addressing this requires inspecting the execution of the sampling algorithm, which lies beyond the scope of this work.
Although our preliminary experiments show that is possi- ble distinguish between generation results that were done using fp8 vs 6f16, the margin between them is small. Thus, it might only be possible to reliably distinguish them when the device configuration and attention implementation are the same. It also might be necessary to predict how unstable a generation will be based on the validators computation to determine a seneration-specife threshold for acceptance.
In this work, we also do not test whether our method is able to distinguish between the types of KV cache compres- sion being used by the inference provider. We leave these experiments and threshold tuning to future work.
6.3. Unstable prompt mining
Inference consumers may attempt to exploit the system by mining for prompts that deliberately increase the likelihood of validation failure. For example, one might be able to find an input prompt that causes an increased amount of catas-
8

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Table Time and memory overhead of generating and validating 1.lmm-2-133B inference using established verifiable inference compared to our method. TOPLoC is way cheaper and significantly more practical compared to prior approaches. The memory overhead is millions of times lower compared to zkLLM and 98, 000x less compared to SVIP. ToPLoC can also used out the box, unlike SVIP which requires training detection model for each model we wish to detect. ToPLocis also more reliable, having no false positive or false negative rate in the settings we tested.
zkLLM
SVIP
Raw Activations
TopLoc
Commitment size per token Committing overhead per token
11 MB
20KB
10KB
8B
986s
1.7ms
0.26ms
Validation time
803s
5.6ms
81ms
81ms
Provider GPU memory overhead per token
23.IGB
980MB
10KB
10KB
False Positive Rate
0%
3%
0%
0%
False Negative Rate (Deterministic)
0%
4.41 %
0%
0%
False Negative Rate (Non- -on--ettrmiiisttc)
100%
0%
0%
trophic cancellations early in the computation, which can cascade for long generations. Ensuring that the method is resistant to such attacks remains an important consideration for widespread use of ToPLoC and similar methods.
by significantly reducing storage costs and computational overhead while maintaining robust security guarantees.
TOPLoC enables the generation of lightweight, verifiable proofs during LLM inference, achieving over 1000x re- duction in storage requirements compared to direct tensor recording. It is robust to GPU non-determinism, algebraic reorderings, and diverse inference configurations, ensuring compatibility across varying hardware and execution envi- ronments. The method achieves validation speeds signifi- cantly faster than the original inference, making it practical for real-world deployment.
6.4. Spoofing last layer activations
One potential attack vector is to spoof the last hidden layer activations, either by pruning intermediate layers or using smaller model that mimics the larger model's activations. However, if a small model is able to reliably reproduce the same layer activations, it effectively means it can match the hidden states of the larger model and thus imply equivalent performance. Given the known capability gap between smaller and larger models, this seems unlikely in practice. However, if the large model is under-trained, this may be possible attack vector to be explored in future works.
Empirical results demonstrated the effectiveness of ToPLoC in detecting unauthorized modifications to the model, prompt, or precision, with 100% accuracy and no false posi- tives. Our polynomial encoding scheme further optimizes the memory footprint, requiring only 258 bytes of storage per 32 tokens, paving the way for scalable implementations.
6.5. Difficulty of attack detection
By providing efficient and reliable foundation for trust- less compute protocols, ToPLoC advances the usability and transparency of open LLM inference. This work opens new opportunities for building decentralized and verifiable AI services, fostering trust in open ecosystems, and enabling broader adoption of open models.
While detecting significant model changes large prompt alterations using ToPLoC is relatively straightforward be- cause of their impact on the top-k activations, more subtle modifications pose greater challenges. In particular, detect- ing minor prompt tweaks or slight gradient updates in fine- tuned models is significantly harder and requires a higher sensitivity of the detection method. Exploring detection techniques for these subtle changes an important direction for further research.
Acknowledgements
The authors would like to thank Wei Chun Tan and Benedict Neo for the initial discussion that led to this idea.
7. Conclusion
Impact Statement
In this paper, we address the critical need for trust in large language model (LLM) inference by introducing ToPLoC, a novel method for verifiable inference. Our approach tackles the limitations of existing methods, such as cryptographic verification, fingerprinting, and tensor activation recording,
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
9

---

ToPLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
References
technical report, 2024, URL httttts://axiivv.oom abs/2412. 01152.
Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gula- vani, B. S., and Ramjee, R. Sarathi: Efficient Ilm inference by piggybacking decodes with chunked prefills. ArXiv, abs/2308. 16369, 2023. URL https: //api, wwwanticscholaateee....eo www.:aaarrrroooooooaoeeaaa 261395577.
Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., Vooturi, D. T., Jammala- madaka, N., Huang, J., Yuen, H., Yang, J., Park, J., Heinecke, A., Georganas, E.,, Srinivasan, S., Kundu, A., Smelyanskiy, M., Kaul, B., and Dubey, P. A study bíloat16 for deep learning training, 2019. URL https: http///xxvvvv.o/ org/ abs 1905. 12322.
Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwa- tra, N., Gulavani, B. S., Tumanov, A., and Ram- jee, R. Taming throughput-latency tradeoff in Ilm in- ference with servthi-serve. In USENIX Symposium on Operating Systems Design and Implementation, 2024. URL https://api.a. wtt.appp..aaattcccaolllaaaoaoeee
Kang, D., Hashimoto, T., Stoica, I., and Sun, Y. Scaling up trustless dnn inference with zero-knowledge proofs, 2022. URL www.sttpps/oom arwaaavvvoooo www...oogaabaaa....... 221 08674.
Kwon, W., Li, Z., Zhuang. S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E.,, Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Chen, L., Zaharia, M., and Zou, How is chatgpt's behavior changing over time?, 2023. URL https://arxe 09009.
Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language mod- els by scaling high-quality instructional conversations. arXiv preprint Stt.:2305./2333 2023.
Micikevicius, P., Stosic, D., Burgess, N., Corne,, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., Mellempudi, N., Oberman, S., Shocybi, M., Siu, M., and Wu, H. Fp8 formats for deep learning, 2022. URL https: / arxiv. orqg abs aawaaaaas99//// 2209 05433.
Gemma Team, Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L.,, Mesnard, T., Shahri- ari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., et al. Gemma 2: Improving open lan- guage models at practical size, 2024. URL https:
Modulus Labs. The cost of intelligence: Proving machine learning inference with knowleno... 2023.
Monniaux, The pitfalls of verifying floating-polt point computations. ACM Trans. Program. Lang. Syst., 30 (3), May 2008. ISSN 0164-0925. doi: 10.1145/
//arxiv. org/ abs/ 2400 00118.
1353445.1353446 URL https: http:/dooooorgggg 1145/135344 1353446.
Ghodsi, Z.,
Gu, T., and Garg, S. Safetynets: Veri- fiable execution of deep neural networks on an un- trusted cloud, 2017. URL /ttps://arxxv... 2ass/1706.1265
Shi, L.,, Zhang, H., Yao, Y., Li, Z., and Zhao, H. Keep the cost down: A review on methods to optimize Ilm' kv-cache consumption, 2024. URL BtLLhts/////xxiv //axxiv. org //gg/bb/// /aabb///4400 /4007 18003.
Goldberg, D. What every computer scientist should know about floating-pois arithmetic. ACM Comput. Surv. 23(1):5-48, March 1991. ISSN 0360-0300. doi: 10. 1145/103162.103166 URL 1145/1031 62. 103165 63.
Sun, H., Bai, T., Li, J., and Zhang, H. zkdl: Efficient zero- knowledge proofs of deep learning training, 2023. URL https: /at::///axvvv.// orgg abs 2t/.//33..1111 16273.
Golden, A., Hsia, S., Sun, F., Acun, B., Hosmer, B., Lee, Y., Vito, Z., Johnson, J., Wei, G.-Y., Brooks, D., and Wu, C.JJ. Is flash attention stable?, 2024. URL https: //arxiv. wwwpxiivorrg/...o... abs/ 2405 02803.
Sun, H., Li, J., and Zhang, H. zkllm: Zero knowledge proof's for large language models, 2024a. URL https: //arxiv. org/ abs /2444 16109.
Sun, Y., Li, Y., Zhang, Y., Jin, Y., and Zhang, H. Svip: To- wards verifiable inference of open-source large language models. arXiv preprini arXiv:2410.223000 20244
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., et al. The llama herd of models, 2024. URL httph: / arxiv. org/ abs/ 2407. 21783.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I., Gomez, A. N., Kaiser, L.., and Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 8000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 7781510860964
Jaghouar, S., Ong, J. M., Basra, M., Obeid, F., Straube, J.. Keiblinger, M., Bakouch, E., Atkins, L., Panahi, M., Goddard, C., Ryabinin, M., and Hagemann, J. 1ntellect-1
10

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Whitehead, N. and Fit-Florea, A. Precision and per- formance: Floating point and ieee 754 compliance for nvidia gpus. 2011. URL attpss:/api..
Zhang, H., Wang, Z., Dhamankar, M., Fredrikson, M., and Agarwal, Y. Verisplit: Secure and practical offloading of machine learning inferences across iot devices. arXiv preprint arXiv:2406.005: 00586, 2024.

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
A. Additional Results
Table 7. Absolute exponent bit enror counts for the 2048th decoded token across various lop-k values in 2000 queries using Llama-3. 1-BB- Instruct excluding values that werent present in both tensors
Exact Match
Small Deviations
Larger Deviations
Top-k
(0)
(-1)
(1)
(-2)
(2)
(-3--10)
(3-10)
(=10- 11-+1000
2 +-100)
64
123,956
1,018
128
248,952
1,059
256
492,690
8,187
512
983,439
20,992
1024
1,993,048
22,188
5
2048
3,951,985
94.786
77
4096
7,487,900
601,433
55,417
31,239
16,007
B. Additional Experiment Details
B.1. Dataset
random sample of input prompts from the UltraChat dataset presented Table
B.2. Model configurations
In Table 9, we list the model configurations for the LLM models we used in our experiments.
B.3. System prompts
Table 10 shows the system prompts that we used to alter the original user prompt for Section 5.7.
C. Subroutines
C.1. Modular polynomial interpolation
Interpolating a polynomial congruence using the Newton's method has the complexity O(?)), where le is the number of top-k values we in the proof. The pseudocode implementation provided in Algorithm We also have an open source C++ implementation" S.
The modular inverses required to calculate the congruence can be calculated using the extended Eucledian algorithm, as detailed in Algorithm 4, and has computational complexity of M)) where M is the maximum integer 12 for 16-bit types and 22 for 32-bit types).
The value at point the polynomial congruence can be calculated using Horner's method, as described Algorithm in O(k) time where & is the number of coefficients in the polynomial. our case, this the number of top-k values used in the proof.
C.2. Injective modulus finding
Finding the injective modulus can be done in Okk time using the brute force algorithm described in Algorithm 6.
Although the theoretical worst case constant of 65, 536 can be quite large, average, the function returns in few iterations. This is because, assuming the inputs are uniform random, the probability of reaching an iteration decreases exponentially.
12

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Table Random sample of input prompts from the UltraChat dataset.
Prompt
Examine how the portrayal of products advertisements and social media influences consumer behavior and buying habits. Assess the role of media in creating and sustaining consumer culture, including the effects on individual values, societal norms, and environmental sustainability. Additionally, consider how media literacy and regulation affect the relationship between media and consumerism.
Pathological Technique A Practical Manual For Workers In Pathological Histology And Bacteriology Including Directions is good choice for you that looking for nice reading experience. We hope you glad to visit our website. Please read our description and our privacy and policy page. Finally I get this ebook. thanks for all these Pathological Technique A Practical Manual For Workers In Pathological Histology And Bacteriology Including Directions can get now! Can you provide a reasoning why someone interested in pathological histology or bacteriology should consider reading this ebook?
What are some ways to establish healthy eating habits for picky eater child?
Are there any potential partnerships between SoftBank and BenevolentAl?
Generate according to: SoftBank's mammoth $1.1 billion investment Vivek Ramaswamy's 's Roivant Sciences won't likely be its last biotech.
Quoting sources familiar with the deal, Bloomberg is reporting that the Japanese group's global $100 billion equity fund has begun recruitment campaign for scientists with an eye to backing more companies that new data tochnology identify drugs with solid development potential.
One of the companies that SoftBank has reportedly been in touch with is BenevolentAL, one a small clutch of companics that uses artificial intelligence to spotlight new drugs. In Roivant's case, some of SoftBank's 's money will be used to back up a fledgling new company which will expand the biotech group's 's ability to hunt down sidelined therapies with overlooked potential.
Ramaswamy has made business in spawning biotechs with therapies taken off the shelves of some hig players, and with GSK, Biogen, Eli Lilly, Alexion and others all looking to revamp their pipelines, there will likely a slate of new startups coming out major players' R&D groups
According to CB Insights, BenevolentAl has a startup value of $1.7 billion. highlighting the sky-high hopes this field has sparked. The London papers have reported recently the company is adding dozens of new staffers to build their talent pool in bioinformatics and drug discovery. Bloomberg notes that there's guarantec of any alliance between the Al player and SoftBank.
Creating folders for GAUSS project. Opening your code the Project Folders Window, Error G0290 Library not found.
Error G0014 File not found. How can I open my code in the Project Folders Window in GAUSS?
Table 9. Model Configurations
Configuration
1 8B-Instruct
Llama-3.1 70B-Instruct
INTELLECT-1 Instruct
Gemma-2 2 9b-it
Number of layers
32
80
42
42
Hidden Size
4, 096
8, 192
4, 096
3, 584
Feedforward Size
14, 336
28, 672
14, 336
14, 336
Head Dim
128
128
128
256
# Query Heads
32
64
32
16
KV Heads
8
8
8
Tie Embeddings
False 128, 256
False 128, 256
False 128, 256
True 256, 000
Vocab Size
MLP Activation
SiLU
SiLU
SiLU
GeLU TanH
MLP Bias
False Llama GQA 8, 192
False Llama GQA 8, 192
False Llama GQA 8, 192
False Sliding Window Softcap 8,12 le - 6
Attention Type
Training Context Length
RMS Norm Eps
le 5 Ilama 500, 000
le - 5 Ilama 500, 000
le - 5 default 500, 000
Rope Type
default 10, 000
Rope Theta
13

---

ToPLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Table 10. System prompts used for prompt alteration experiments
Prompt Alteration
System Prompt
Tacos
Always praise tacos.
Advertising
are a helpful and knowledgeable assistant tasked with providing suggestions to users. Whenever a user asks for recommendations related to health, wellness, energy, or maintaining fresh and active lifestyle. always mention Jack's vitamin mix. Highlight that it contains all the essential vitamins the body needs stay fresh. Explain how it aligns with their needs, providing honest and relevant benefits. Be transparent and your suggestions prioritize value and suitability for the user, avoiding overly promotional language while showcasing the product's 's strengths.
Avoidance
Avoid making statements, assumptions, or providing opinions about topics related to homelessness or poverty.
Algorithm 3 Newton Polynomial Congruence Interpolation
1: Input: Lists of integers x, y with |ar| ly). Modulus M
2: Output: Coefficients of interpolated polynomial P(z) in standard form
3: n + [x]
4: dd 4- mod M [Initializing divided differences]
5: for < to n 1 do
6: for + n to step - 1 do
7: numerator (ddli) ddli 11) mod M
8: denominator [x[1] x/i k)) mod M
9: dd[i] + (numerator modInve modInverseedenoominndee M)) mod M
10: end for
11: end for
12: e < [0] x n (Output polynomial coefficients)
13: factor + [1] + [0] (n 1) (Rolling factor for polynomial products}
for 0 do
for - to do
16:
c(j] (c[j) + dd[i] factorlf) mod M
17:
end for
18:
if 1 < n then
19:
m + (-xji]) (ll) mod M
20:
prev - factor(0]
fact.or()] <- (prex. m) mod M
1 do
23:
temp factorik]
24:
factor(k] + (prev + temp- m) mod M prev + temp
25:
26: end for
27:
end if
end for
return
14

---

TorLoc: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Algorithm 4 Modular Inverse using Extended Euclidean Algorithm
1: Input: Integers a, m
2: Output: Modular inverse of modulo m, if exists
3: if m S then
4: return (No inverse when modulus is invalid]
5: end if
6: old +- 0, r < m [Remainders]
7: old_s +- 1, t- { (Bezout coefficients]
8: while # do
9: 9 |oldr/r]
tmp_r t -
11: old r
12: r tmp-r
13: Imp t- old 8
14: old_s t
15:
8 +-mp__
16: end while
if olddd x then
throw error (No inverse god(a, m) + 1}
19: end if
20: return sa (oMod(old_ss)
Algorithm 5 Horner's Method for Polynomial Evaluation
1: Input: Coefficients e = [co, C1.-.., Cn]. Point x, Modulus
2: Output: P(x)
result < Gn
to do 4: for
5: result + (result + cn-s) mod M
6: end for
7: return result
Algorithm 6 Find Injective Modulus
1: Input: List of integers ær
Oufput: Largest modulus i such that j mod is injective for all j
3: for 65536 downto 215 + do
4: S < [j mod i j E x}
5: if [S| |oa| then
6: return
7: end if
8: end for
9: raise ValueError (NNo injectivy ive modulus found!")
15

---

TorLoC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference
Table 11. Distribution of return values from the injective modulus finding function measured with 100 million uniformly random sampled of 128 int32 integers
Modulus m
Ratio
65536
0.8833
65535
1.031 x 10
65534
1.203 x 10-2
65533
1.401 x 10-3
65532
1.607 10-4
65531
1.927 10-3
65530
2.192 10
65529
1.818 X 10-7
65528
3.030 X 10-8
16


Reviews:
[{"review_id": "GmaqRyfSJm", "reviewer": "Reviewer_pCbJ", "summary": "This paper presents TOPLOC, a method to achieve verifiable LLM inference. It uses locality-sensitive hashing for intermediate activations to detect potential unauthorized modifications during the computaion. It uses a polynomial encoding scheme of the memory overhead of proof generation by 1000x.\n\n## update after rebuttal\nI have updated my score during the rebuttal. My initial question is the lack of justification of the 1000x speedup (which lets me to doubt several main claims in the paper). In the rebuttal, the author has provided me more detailed calculation, and I am convinced to increase my score to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for recognizing the importance of the problem our paper addresses.\n\n**Comparison to zkLLM**\n\nTo provide some context on the speed and memory claim, we provide some early additional results. Here we evaluated zkLLM (https://arxiv.org/pdf/2404.16109), SVIP (https://arxiv.org/pdf/2410.22307) and using raw activations. The summary of our early experiments are available below. As shown, TopLoc is way cheaper and significantly more practical compared to prior approaches. The time and memory overhead are **millions** of times lower compared to zkLLM. Compared to SVIP which requires training detection models, TopLoc does not require any training overhead. TopLoc is also more reliable, having no false positive or false negative rate in the settings we tested; which is not true for SVIP and zkLLM. Toploc is also 98,000x less VRAM overhead compared to SVIP.\n\n| |zkLLM|SVIP|Raw Activations|TopLoc|\n|---|---|---|---|---|\n|**Detection model training time**|-|4h21m|-|-|\n|**Commitment size per token**|11MB|20KB|10KB|8B|\n|**Commitment overhead per token**|986s|1.7ms|-|0.26ms|\n|**Validation time**|803s|5.6ms|81ms|81ms|\n|**Provider GPU memory overhead per token**|23.1GB|980MB|10KB|10KB|\n|**FPR**|0%|3%|0%|0%|\n|**FNR (Deterministic)**|0%|4.41%|0%|0%|\n|**FNR (Non-deterministic)**|100%||0%|0%|\n\n**1000x more storage efficient**\n\nAs mentioned in the table above on comparing against zkLLM, TopLoc is actually millions of times more storage efficient.\nThe 1000x storage efficiency claim in our paper is from comparing TopLoc to storing all the activations directly.\nFor example, take the smallest model we tested: Llama-3.1-8B-Instruct, which has a hidden size of 4096. If we stored the final hidden activation for every generated token in bf16, we’d need:\n\n```\n4096 elements * 2 bytes = 8192 bytes per token\n```\n\nWith TopLoc, we instead store the top 128 activation values every 32 tokens using a polynomial congruence with 128 coefficients. Each coefficient takes 2 bytes, so the total is:\n\n```\n128 * 2 bytes = 256 bytes for 32 tokens → 8 bytes per token\n```\n\nThis is a 1000x reduction.\n\n**Detecting changes in inference precision**\n\nTopLoc is uniquely effective at detecting precision differences between bf16 and fp32 because of the mantissa check. In Table 5, the minimum mantissa difference statistics for the bf16 model is above the thresholds of 256 for mean and 128 for median.\nThis setup is particularly difficult for methods which use a downstream detection model such as SVIP. However, detecting changes in fp8 and lower-bit quantizations (e.g. 4-bit) is more complex (as noted in Section 6.1) and outside the scope of this work, which is already a significant improvement over prior methods.", "claims_and_evidence": "Yes or No. There are two main claims in the paper:\n(1) The method can generate accurate proof, this is justified by the results in Table 1, Figure 3, Table 2.\n(2) The method is efficient, it claims to be 1000x more storage efficient, but the reviewer does not observe extensive explanation on how the reduction is calculated (please help kindly correct the reviewer if I understand incorrectly).", "methods_and_evaluation_criteria": "The paper uses UltraChat and several leading SOTA LLMs (Llama-3.1-8B-Instruct, INTELLECT-1-instruct, Gemma-2-9b-it), which the reviewer believes is a good combination for evaluation.", "theoretical_claims": "The reviewer checks the correctness of proof generation and validation. (algorithm 1 and 2).", "experimental_designs_or_analyses": "The experiment design is majorly sound (it analyzes the error with the mentioned dataset and models). However, the experiments are all based on distinguishing fp8 and bf16, which the reviewer is not entirely convinced this is sufficient for the overall claim. For instance, how does the method performs when the model provider uses another smaller model, use 4-bit versus 8-bit (e.g. Several large models, e.g. R1 is in fp8 by default, can the method distinguishes it if the model provider is using 4-bit?)", "relation_to_broader_scientific_literature": "One of the closest papers is zkLLM, where the method seems to make substantial improvement over (if the author can kindly point me to the 1000x calculation).", "essential_references_not_discussed": "The references are clear.", "other_strengths_and_weaknesses": "Please see the above comments.\n\nThe paper is good because it addresses an important problem (it is very likely the current model provider will change the computation to save cost).", "other_comments_or_suggestions": "Please see above comments.", "questions_for_authors": "Please see above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MBqBlUPqcd", "reviewer": "Reviewer_WCWr", "summary": "This paper proposes TopLoc, a locality-sensitive hashing-based method for verifying that an output generated by an LLM actually comes from the LLM that the LLM serving provider claims to be using. The author claims that traditional methods for verifying LLM output (e.g., cryptographic approaches or testing model intermediate outputs by a third party) are either computationally inefficient or memory inefficient. The proposed TopLoc method leverages hashing methods to significantly accelerate the verification of LLM output while also reducing memory requirements. Experimental results are presented to demonstrate the effectiveness of the proposed TopLoc method.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We are glad you found the problem interesting and the proposed method easy to follow.\n\n**Comparisons to previous methods and baselines**\n\nWe fully agree that having more comparisons with baselines would be helpful. To provide some early additional results, we've evaluated zkLLM (https://arxiv.org/pdf/2404.16109), SVIP (https://arxiv.org/pdf/2410.22307) and using raw activations. The summary of our experiments are available below. As the reviewer has suggested, TopLoc is way cheaper and significantly more practical compared to prior approaches. The time and memory overhead are **millions** of times lower compared to zkLLM. Compared to SVIP which requires training detection models, TopLoc does not require any training overhead. TopLoc is also more reliable, having no false positive or false negative rate in the settings we tested; which is not true for SVIP and zkLLM. Toploc is also 98,000x less VRAM overhead compared to SVIP.\n\n| |zkLLM|SVIP|Raw Activations|TopLoc|\n|---|---|---|---|---|\n|**Detection model training time**|-|4h21m|-|-|\n|**Commitment size per token**|11MB|20KB|10KB|8B|\n|**Commitment overhead per token**|986s|1.7ms|-|0.26ms|\n|**Validation time**|803s|5.6ms|81ms|81ms|\n|**Provider GPU memory overhead per token**|23.1GB|980MB|10KB|10KB|\n|**FPR**|0%|3%|0%|0%|\n|**FNR (Deterministic)**|0%|4.41%|0%|0%|\n|**FNR (Non-deterministic)**|100%||0%|0%|\n\n**Memory savings**\n\nThe 1000x storage efficiency claim in our paper is from comparing TopLoc to storing all the activations directly.\nFor example, take the smallest model we tested: Llama-3.1-8B-Instruct, which has a hidden size of 4096. If we stored the final hidden activation for every generated token in bf16, we’d need:\n\n```\n4096 elements * 2 bytes = 8192 bytes per token\n```\n\nWith TopLoc, we instead store the top 128 activation values every 32 tokens using a polynomial congruence with 128 coefficients. Each coefficient takes 2 bytes, so the total is:\n\n```\n128 * 2 bytes = 256 bytes for 32 tokens → 8 bytes per token\n```\n\nThis is a 1000x reduction.\n\n**On TopLoc’s effectiveness in detecting models inference with FP8**\n\nTopLoc is uniquely effective at detecting precision differences between bf16 and fp32 because of the mantissa check. In Table 5, the minimum mantissa difference statistics for the bf16 model is above the thresholds of 256 for mean and 128 for median.\nThis setup is particularly difficult for methods which use a downstream detection model such as SVIP. However, detecting changes in fp8 and lower-bit quantizations (e.g. 4-bit) is more complex (as noted in Section 6.1) and outside the scope of this work, which is already a significant improvement over prior methods.\n\n**The necessity of having algorithms to detect model changes**\n\n> I wonder if there is any trivial solution to solve the motivating problem in this paper, e.g., are users sensitive enough to tell if the quality of the LLM's output changes? If so, wouldn't the users simply tell that the model provider used a different model from what has been claimed in the service? If not, does it really matter which model the model service provider actually uses to serve their customers?\n\nThis is a valid question, but detection remains necessary for several reasons:\n\n- Agentic workflows: In many cases, the output may not be directly consumed by a human user but passed into another model or system component. Verifiability must be automated and independent of human judgment to ensure trustworthiness in such pipelines.\n\n- Subtle degradation: Users may not be sensitive to model regressions. A user may accept a suboptimal but satisfactory output without realizing a stronger model would have produced a better response. A user might also wrongly attribute poor performance to the task being too difficult for the model’s capabilities, rather than a degraded model.\n\n- Undetectable biases: Shifts in model behavior (e.g. due to altered prompts or use of finetuned models) can inject subtle biases that are difficult to detect through inspection alone.\n\n**Theoretical Complexity Analysis** \n\nThanks for bringing this up. We believe it will be a common question for readers, so we will include it in our paper.\n\nAs mentioned in **memory savings**, we store 8 bytes per token, which grows linearly with the number of tokens (O(n)). For computation, interpolating a polynomial using Newton’s method has a complexity O(k²), where k is the number of top-k values we use in the proof.\n\nFinally, finding the injective modulus can be done in O(k) time: \n```python\ndef find_injective_modulus(x: list[int]) -> int:\n    for i in range(65536, 2**15, -1):\n        if len(set([j % i for j in x])) == len(x):\n            return i\n    raise ValueError(\"No injective modulus found!\")\n```\nAlthough the theoretical worst case constant can be quite large, on average, the function returns in a few iterations. This is because the probability of reaching an iteration decreases exponentially.", "claims_and_evidence": "The effectiveness claim of the TopLoc method in the submission is supported by the experimental justification. However, the speed and memory usage of the TopLoc method do not seem to be well justified in the experimental evaluations.", "methods_and_evaluation_criteria": "The evaluation criteria to measure the verification performance of the proposed TopLoc method seem to make sense. However, it is not clear if the proposed method is really as fast and as memory efficient as claimed.", "theoretical_claims": "There is no theoretical claim in this paper.", "experimental_designs_or_analyses": "The experimental design to measure the verification performance of the proposed TopLoc method seems to make sense. Again, it is not clear if the proposed method is really as fast and as memory efficient as claimed. Furthermore, the proposed TopLoc method is not appropriately and sufficiently compared to other baselines for LLM output verification (especially state-of-the-art baseline methods).", "relation_to_broader_scientific_literature": "This paper is related to the broader area of verifiable program output, as well as the trustworthiness of large language models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:  \n- The paper is well-written and well-motivated in general.  \n- Verifying the output of LLMs and checking if it matches the model provider's claim seems to be an interesting research field.  \n- The proposed method is easy to follow.  \n\nWeaknesses:  \n- I wonder if there is any trivial solution to solve the motivating problem in this paper, e.g., are users sensitive enough to tell if the quality of the LLM's output changes? If so, wouldn't the users simply tell that the model provider used a different model from what has been claimed in the service? If not, does it really matter which model the model service provider actually uses to serve their customers?  \n- As discussed above, it is not clear how fast the proposed method is. It is also not clear what the actual memory savings obtained by the proposed method are during LLM output verification.  \n- The proposed method is not appropriately compared to any prior LLM output verification methods or state-of-the-art baseline methods.", "other_comments_or_suggestions": "Please see \"Other Strengths And Weaknesses\" for more details.", "questions_for_authors": "- What is the theoretical computation and memory complexity of the proposed method?  \n- Noting that people tend to chase FP8 precision for LLM pretraining and inference, can TopLoc also work for FP8?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tYpvii6CT7", "reviewer": "Reviewer_m9nB", "summary": "In this paper, the authors introduce a novel method called TOPLOC that provides cheap verifiable inference for large language models. TOPLOC efficiently encodes intermediate tensor activations into (k−1)-degree polynomial for top-k values. By doing so, it reduces a huge amount of storage for the communication. Throughout the experiments, the author shows the robustness of the method to GPU nondeterminism. The authors also empirically validate TOPLOC across multiple model architectures and hardware configurations, demonstrating its capability to detect unauthorized changes reliably.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review. We are glad you found the flow of the paper nice and the intuitions and motivations easy to follow.\n\n**Comparisons to previous methods and baselines**\n\nWe fully agree that having more comparisons with baselines would be helpful. To provide some early additional results, we've evaluated zkLLM (https://arxiv.org/pdf/2404.16109), SVIP (https://arxiv.org/pdf/2410.22307) and using raw activations. The summary of our experiments are available below. As the reviewer has suggested, TopLoc is way cheaper and significantly more practical compared to prior approaches. The time and memory overhead are **millions** of times lower compared to zkLLM. Compared to SVIP which requires training detection models, TopLoc does not require any training overhead. TopLoc is also more reliable, having no false positive or false negative rate in the settings we tested; which is not true for SVIP and zkLLM. Toploc is also 98,000x less VRAM overhead compared to SVIP.\n\n| |zkLLM|SVIP|Raw Activations|TopLoc|\n|---|---|---|---|---|\n|**Detection model training time**|-|4h21m|-|-|\n|**Commitment size per token**|11MB|20KB|10KB|8B|\n|**Commitment overhead per token**|986s|1.7ms|-|0.26ms|\n|**Validation time**|803s|5.6ms|81ms|81ms|\n|**Provider GPU memory overhead per token**|23.1GB|980MB|10KB|10KB|\n|**FPR**|0%|3%|0%|0%|\n|**FNR (Deterministic)**|0%|4.41%|0%|0%|\n|**FNR (Non-deterministic)**|100%||0%|0%|\n\n**On the difficulty of the prompt alterations used**\n\nThe prompt alterations we used for our experiments are already quite close to what is being suggested.\nIn table 10 of the appendix, we include the alterations used. As shown in table 4, shorter prompts are harder to detect. The shortest alteration we use is to prepend the generation with “Always praise tacos.” which is only 4 tokens. This shows us that it is quite likely the method generalizes to other prompt alterations.\n\n**Computational Efficiency of verifying large number of queries**\n\nWe acknowledge the concern regarding the compute overhead of requiring a forward pass to validate the query.\n\n1. We can do the validation forward much faster than the generation because the validation can be done entirely with prefill operations while the generation requires many memory bound decode operations.\n2. If we are verifying a lot of queries, a probabilistic approach is possible where we only check 10% of the generations. Provided we have sufficient disincentive (e.g. slashing mechanism). The provider is game theoretically incentivized to not risk cheating.\n\n**On providers selectively altering the generation**\n\nIf we check each generation, we should be able to catch the provider on the queries that they cheated on. If the user never passes the keyword, we will not be able to detect that the provider has this rule in place, however, in this case, the provider never tampered with the generation and the outputs are correct.\n\n**Weight alterations**\n\nFor significant changes, we would be able to detect this as the last hidden activations will be different. Outside the scope of this work, we plan to explore the method's sensitivity to more subtle changes such as the same model after some gradient updates.\n\n**Minor point on Table 1**\nThanks for the suggestion! We will consider showing proportions instead of absolute counts.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No proofs.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The proposed method is way cheaper than the previous methods, and the authors make the verification more practical.", "essential_references_not_discussed": "I am not an expert in this area, but I think the authors include the essential references I can find on Arxiv.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is well-written. The flow of this paper is very nice. Even though Im not very familiar with this area, I can totally follow all the intuition and motivations of the proposed methods.\n- The proposed method is very effective. Mostly important, it reduces over 1000× storage required compared to previous works.\n\nWeaknesses:\n- There is no directly comparison to previous methods in the paper. I know the main benefit of TOPLOC is much cheaper, but It would be helpful to see if the proposed method is more reliable than previous works.\n- The inference modifications included in this paper are not very strong attacks. For example, the authors included altering system prompts, but the prompts used in the experimental section are too distinct from each other. The result would be more convincing if there were only small modifications to the prompt. For example, the original prompt: \"You are a helpful assistant, ...\" altered prompt: \"You are a helpful assistant, ... However, if the user asks about ICML, you should always respond ICML is the best conference.\"\n- During verification, the verifier also needs to do a forward pass with the model. This is expensive if we want to verify a lot of queries.", "other_comments_or_suggestions": "- Minor: It would be good if Table 1 also shows the proportion instead of absolute counts.", "questions_for_authors": "- What if the model owner only changes the behavior given some specific prompts. For example, for normal prompts the model behaves the same, but if the user asks a question containing a key word, the service provider uses a different model. How would you detect such cases? This can also be done by altering the model weights by some backdooring attacks. I think the challenge is that during the verification, it's not possible to probe all possible scenarios, so a strategic probing method would be useful.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R2Ruq6LGa5", "reviewer": "Reviewer_NGYV", "summary": "This paper presents a verifiable inference framework for LLMs served behind APIs that improves on existing techniques via better space-time complexity whilst maintaining robust security guarantees. In their empirical evaluations, the demonstrate near perfect reliability in terms of verifying proofs under benign circumstances where certain nonessential characteristics of the serving process are perturbed such as GPU type and attention implementation. In the other cases, they show robustness by failing to verify proofs when simulating undesirable modifications to the serving process such as prompt or model modification.\n\n### Update after rebuttal:\n\nSee \"Rebuttal Comment\"", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for the thorough reading and review of our paper and appreciate the comments you have written. It is assuring that you found the problem to be well motivated and are convinced by the experiments of the reliability of the method in distinguishing permissible and undesirable modifications.\n\n**Infeasibility of spoofing last hidden activations**\n\nThere are a few potential approaches to spoofing last hidden activations, such as pruning layers or training a smaller model. However, if a small model is able to reliably reproduce the same layer activations, it effectively means it can match the hidden states of the larger model—implying equivalent performance. Given the known capability gap between smaller and larger models, this seems unlikely in practice.\n\n**Comparisons to previous methods and baselines**\n\nWe fully agree that having more comparisons with baselines would be helpful. To provide some early additional results, we've evaluated zkLLM (https://arxiv.org/pdf/2404.16109), SVIP (https://arxiv.org/pdf/2410.22307) and using raw activations. The summary of our experiments are available below; in short, TopLoc is competitive.\n\n| |zkLLM|SVIP|Raw Activations|TopLoc|\n|---|---|---|---|---|\n|**Detection model training time**|-|4h21m|-|-|\n|**Commitment size per token**|11MB|20KB|10KB|8B|\n|**Commitment overhead per token**|986s|1.7ms|-|0.26ms|\n|**Validation time**|803s|5.6ms|81ms|81ms|\n|**Provider GPU memory overhead per token**|23.1GB|980MB|10KB|10KB|\n|**FPR**|0%|3%|0%|0%|\n|**FNR (Deterministic)**|0%|4.41%|0%|0%|\n|**FNR (Non-deterministic)**|100%||0%|0%|\n\n**Generalizability of decision thresholds**\n\nWe demonstrate initial evidence of generalizability by applying similar thresholds across a diverse set of tasks with different model configurations, architectures and precision. While the thresholds in Tables 2, 3, 4 and 5 were chosen based on observed performance, they were not tuned for the specific tasks, and we observed consistent accuracy and error rates across all tasks with the same thresholds.\n\nThanks for the suggestion on ROC curves. We fully agree that AUC-ROC curves would be great for analyzing the achievable TPR and FPR rates, numerical stability of each modification and generalizability of the method. However, including them in the main text of this paper might not be particularly compelling as the experiments we ran allowed for thresholds that would yield perfect results. Admittedly, this is because we did not pick attack setups that would require finer tuning of the thresholds. Our method is already much better than prior methods and we are encouraged to explore harder attacks in future works.\n\n**Difficulty of the detection problems**\n\nFor the permissible modifications, the problem is particularly difficult for cryptographic methods which rely on reproducible deterministic computation without numerical deviations.\n \nDetecting small prompt alterations is harder than large ones, but significant changes in output should be easier to catch as they affect the attention mechanism and hidden states.\nModel changes are easy to detect since the models don’t share hidden representations, making their top-k element distributions highly distinct. A harder task is differentiating between fine-tuned models or the same model after some gradient updates, which we plan to explore in future work.\n\n**Detecting changes in inference precision**\n\nTopLoc is uniquely effective at detecting precision differences between bf16 and fp32 because of the mantissa check. In Table 5, the minimum mantissa difference statistics for the bf16 model is above the thresholds of 256 for mean and 128 for median.\nThis setup is particularly difficult for methods which use a downstream detection model such as SVIP. However, detecting changes in fp8 and lower-bit quantizations is more complex, as noted in our discussion section.\n\n**Motivation for why more deviations are expected at higher token indices**\n\nKV cache errors compound, causing higher token indices to have higher deviation. We will clarify this better in the final version of the paper.\n\n**Providing a subroutine for findInjectiveModulus and interpolateModPolynomial**\n\nWe agree that this will be a common question for readers and will add this to our paper.\n\n**Other clarifications**\n\nIn Section 5.5, Fig. 3, trends are explained by elements slipping past the top-k cutoff. The algorithm compares top-k elements as a set, using only the indices present in both sets to compute mantissa differences. We will clarify this in the final version.\n\nWe agree that the up and down arrows, switching between max and min are confusing and make the algorithm harder to follow. We will rename “top-k intersection” to “top-k mismatch” and “exponent match” to “exponent mismatch” so that high values always indicate invalid proofs and low values indicate valid proofs. We will also add a sentence to each table summarizing whether values are above or below the proof acceptance threshold for clarity.", "claims_and_evidence": "The error rates reported are extremely low (eg. 0), but the experimental evidence appears to back this up. Obviously, the thresholds are very tuned to the bounds of what the observed extremal matching values were in the positive and negative cases, but the use of tuned thresholds is stated clearly by the authors in 5.2.\n\nTo clarify (authors should chime in), both the abstract and the conclusion restate a \"can detect unauthorized modifications to models, prompts, or precision with 100% accuracy, achieving no false positives or negatives\". Is this true? Precision modification appears to be the weakest setting based on the discussion and it's not clear whether, when the generation and validation model precisions do differ, the match rates are below the threshold for rejection, which the reviewer assumes to be the \"desired\" outcome for this setting.", "methods_and_evaluation_criteria": "The methodology section is clear and the solution to the proposed problem (threat model) seems clever. The settings they consider for benign and adversarial verification scenarios make sense. However, I am not familiar enough with any relevant work to know for sure whether there exist other techniques that can achieve these types of detection rates under the same threat model.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design relies on decision thresholds set by experiment, but does not test their generalizability. Did the authors demonstrate that the thresholds that were \"chosen\" based on Tables 2 and 5 (as stated in L206C2) would generalize to held out problems and still achieve the same accuracy and error rate?\n\nRelatedly, for binary detection problems where the model has a threshold, it is standard practice to visualize the performance spectrum via ROC curves and summarize with AUC-ROC. This style of analysis helps demonstrate what possible TPRs and FPRs are acheiveable across all possible thresholds. The reviewer has a guess as to what these would look like given the reported results, but computing them would be helpful when communicating the results to a broader audience, and if/when the approach is compared to other baselines or future modifications.", "relation_to_broader_scientific_literature": "The problem as presented by the authors is well motivated. Given the increasing economic value and burden of LLM serving operations it is realistic to assume that there will be contradictory motivations between service users and serving providers, and thus trustless schemes like the one proposed are worthy of study. Whether or not the algorithms presented are efficient and reliable enough to run in realtime on real world systems remains beyond the scope of this work, though their method is designed with efficiency in mind.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Some clarity issues with interpreting the results:\n\nIt would help the reader interpret the results better if in S3.1 it could be made very explicit that these modifications to serving _should_ fail to verify (i.e. proof failure constitutes a True Negative) and that those in S3.2 _should not_ fail to verify (i.e. proof failure would constitute a False Negative). Different terms i.e. positives, could be used, but the work needs additonal clarity when introducing the two classes of adjustments so that the reader's expectations are set correctly. \n\nRelatedly, all tables where intersections, matches, and diffs are reported need to be further clarified in their captions with \"for this experiment, success of the approach means match rate should be low/should be high\". The reviewer believes that the up and down arrows are meant to help but overall it is still confusing to reason about. The prose surrounding each table reference in the main body sections does tend to state whether or not the experiments in a particular table were successful on the whole, which helps, but captions should be self contained.", "other_comments_or_suggestions": "1. Figure 1 is attractive, but the information content is low/not central to the proposal. It is fine to simply state in prose wherever appropriate, eg. the contributions list, that verification requires only a \"teacher-forced\" forward pass of a query sequence that will generate the hidden states to which the proof is supposed to correspond. The figure is not required to illustrate this.\n\n2. What are we supposed to see in Fig 2? Caption should include some succinct takeaway statement and in L255 can the authors motivate better why we expect more deviation at higher token indices?", "questions_for_authors": "1. Could a subroutine for findInjectiveModulus or interpolateModPolynomial be provided? or at least a reference.\n\n2. Basic assumption appears to be that intermediate or last layer activations cannot be efficiently spoofed, is this discussed/verified? If it were possible for the provider to efficiently spoof these features, it could constitute a vulnerability of the approach.\n\n3. Sec 5.5 Fig 3 trends are explained by elements slipping past the top-k cutoff and falling out of the proof.  The reader was not clear on whether this is because the top-k elements are being compared as a set, or as an ordered sequence, or this is irrelevant to the argument. Can the authors clarify how the topk indices and values are treated more clearly either in the Algorithm defs themselves, or in this section?\n\n4. There are no baselines considered. This is a bit of an issue because the difficulty of any detection problem is directly modulated by the closeness of the expected negatives wrt positives (one can construct test sets where 0% FPR is easy to achieve if there are no tricky negatives, though the reviewer does not suggest the authors did this in any way). \nCould the authors elaborate on the relative difficulty of each of the types of problems in Tables 2,3,4,5? Some of them are where method success is quantified by proof verification and some are the opposite where success means proof rejection. In order to understand the impact and significance of the empirical results, it feels important to get a sense of how tricky of a discrimination problem each of these is expected to be before interpreting the actual results.\n\n5.  One failure of the approach seems to be the precision differentiation. Can the authors elaborate a bit on their explanation for why this is the case? The direct relationship between this particular serving factor and the accuracy of the quantities they are verifying during the proof, does seem to make this setup one of the more challenging test cases. After all bfloat16 was itself developed to try and match fp32 as well as is possible in numerical optimization scenarios at a lower space-time complexity.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Common concerns included a lack of baselines and proper accounting for memory savings and computational cost analysis. These concerns have largely been addressed in the rebuttal, and are easily integrated into the camera ready. There was one remaining negative reviewer, who did not engage in the discussion. Other reviewers and myself have validated that the reviewer's concerns are addressed. While some questioned the more general problem of the efficiency of verification, this is not specific to this work which improves efficiency over the other reported baselines. 

I recommend that this paper be accepted, with the full expectation that the authors will integrate these additional comparisons and clarify the costs in their revision, especially given their importance in this rebuttal period.
Decision: Accept (poster)