SVIP: Towards Verifiable Inference of Open-source Large Language Models

Authors: Yifan Sun, Yuhang Li, Yue Zhang, Yuchen Jin, Huan Zhang
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models; Verifiable Inference; Trustworthy AI; Adversarial Attack

Abstract:
Open-source Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language understanding and generation, leading to widespread adoption across various domains. However, their increasing model sizes render local deployment impractical for individual users, pushing many to rely on decentralized computing service providers for inference through a blackbox API. This reliance introduces a new risk: a computing provider may stealthily substitute the requested LLM with a smaller, less capable model without consent from users, thereby delivering inferior outputs while benefiting from cost savings. In this paper, we formalize the problem of verifiable inference for LLMs. Existing verifiable computing solutions based on cryptographic or game-theoretic techniques are either computationally uneconomical or rest on strong assumptions. We introduce $\texttt{SVIP}$, a secret-based verifiable LLM inference protocol that leverages intermediate outputs from LLMs as unique model identifiers. By training a proxy task on these outputs and requiring the computing provider to return both the generated text and the processed intermediate outputs, users can reliably verify whether the computing provider is acting honestly. In addition, the integration of a secret mechanism further enhances the security of our protocol. We thoroughly analyze our protocol under multiple strong and adaptive adversarial scenarios. Our extensive experiments demonstrate that $\texttt{SVIP}$ is accurate, generalizable, computationally efficient, and resistant to various attacks. Notably, $\texttt{SVIP}$ achieves false negative rates below  $5\\%$ and false positive rates below  $3\\%$, while requiring less than $0.01$ seconds per query for verification.

Decision: reject
Reviewer scores: 3, 3, 6

Reviews and discussion:
<thread>
creation date:
1730484176070

thread_type:
official_review

<thread_object>
creator: reviewer_9dbq

<content>
summary:
The paper proposes Secret-based Verifiable LLM Inference Protocol (SVIP), as a method that allows black-box API users to verify that the LLM the provider claims to offer to them is actually the one that returns the responses. Therefore, the LLM is trained on a proxy task (for the intermediate representations), such that these intermediate responses become a unique identifier of the model. Whether or not the model is the claimed one can then be assessed if the model provider returns intermediate representations along with the outputs.

strengths:
The paper has significant strengths:
- The paper addresses a highly relevant and novel problem of verifying the authenticity of the model behind an API, which is essential for trust in model-based services and has broad implications for the field.
- Reducing the size of intermediate representations to minimize communication overhead is a practical and effective approach, improving the feasibility of real-time verification in resource-constrained environments.
- The paper’s thorough consideration of various attack vectors enhances the robustness of the proposed method and demonstrates an awareness of potential vulnerabilities.
- Experimentation considers multiple LLMs.

weaknesses:
While I really enjoyed reading the paper and found it exciting to see this first step forward towards a new threat vector, I currently see significant weaknesses. If these can be resolved / clarified during the rebuttal process, and if the paper can be updated accordingly, I am willing to raise my score to an accept.

**There is not sufficient protocol setup description**

Even after multiple reads of the paper, I still struggle to see some details of the actual deployment protocol for the method. In particular, I am searching for answers for the following questions:
1. Why are the last layer hidden representations chosen?
2. Where does the user get the proxy-task head from?
3. Where does the user get y(x) from? 
4. Where is the actual user's task solved, and how does the actual query need to relate to the "verification query"? Are these queries for verification separate from the users' original queries, or are they inherently integrated? 
5. Who trains the trainable labeling network? Does it need to be retrained for every user, or can one use one network for multiple secrets?
What I suggest to include into the paper is a more expressive Figure 1. One that really illustrates the protocol. Who generates what, passes it on to whom etc.
For 1., (4.), and 5., additions to the writing would probably resolve the questions.

**Limitations in terms of description of the experimental setup**

Even after reading the full appendix as well, I do not have a mental model of what concretely:
1. The labeling function is composed of.
2. What the proxy tasks are? Are the some classification task? If so on what dataset? How is that chosen?
3. In a similar vein, it would be great to describe and given an example on how the secrets look like in practice. Are those single words? How are they chosen?

**Limitations in experimental evaluation and description**

Even after the experimental section and the appendix, I am not clear about how a user query really looks like. Additionally, I think it would be sensible to include the following results:
1. The whole presented results focus on the success of the protocol, but not on task accuracy. It would be necessary to show what is the impact of the protocol on the task that the user actually wants to solve.
2. The compute time in Table 3 only focuses on the verification. But what are the computation costs for the trusted party of doing all the overhead? It seem that they are actually the ones with the heavy load.


**Conceptual Limitations**

While I think that it is not necessary to resolve those, I feel like it would be best to discuss the following limitations in the main paper and not to defer them to the appendix where current limitations are located.
- Limitation 1: Works only if the LLM provider is willing to take an open source model, or otherwise share their LLM with the trusted party such that they can embed a task into it. In reality, I do not see much hope that OpenAI would disclose their model for this kind of training to another party. 
- Limitation 2: as the work shows, without using a secret, the approach is susceptible to attacks. Hence, it needs the additional user-secret, which requires the presence of the trusted party and significant overhead for them. Who's gonna pay for that?

Minor: The paper states: "while some models exhibit a slight increase in FNR, which still remains within acceptable limits": it is unclear who would define what an acceptable limit is? 





**Minor Writing Suggestions**
- Abstract: missing S: "that leverages intermediate outputs from LLM*s*"

questions:
1. Why are the last layer hidden representations chosen?
2. Where does the user get the proxy-task head from?
3. Where does the user get y(x) from? 
4. Where is the actual user's task solved, and how does the actual query need to relate to the "verification query"? Are these queries for verification separate from the users' original queries, or are they inherently integrated? 
5. Who trains the trainable labeling network? Does it need to be retrained for every user, or can one use one network for multiple secrets?
What I suggest to include into the paper is a more expressive Figure 1. One that really illustrates the protocol. Who generates what, passes it on to whom etc.


6. What the proxy tasks are? Are the some classification task? If so on what dataset? How is that chosen?
7. How do the secrets look like in practice? Are those single words? How are they chosen?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730801474218

thread_type:
official_review

<thread_object>
creator: reviewer_jcvz

<content>
summary:
This paper proposes a protocol to verify that a particular model is being used. Instead of using verifiable computing or watermarking, this work proposes to use a proxy task to discern between the last-layer representations of the chosen model and other models. Thus, by releasing a projection of these representations, a user can now detect if the desired model was actually used. This work includes investigation of adaptive attacks showing that the final protocol is robust to these attacks and empirical evaluation showing high success at distinguishing models.

strengths:
- There are many models (llms) tested.

- The protocol is shown to be effective at distinguishing between two models.

- Some adaptive attacks are considered and the modified protocol is robust to these.

- The protocol is highly efficient, both in terms of compute and communication overhead.

weaknesses:
- Lacking motivation, see questions 1-3. It is unclear to me the impact of this problem setting/solution. It seems that model providers are already heavily disincentivized from “swapping models”, could be caught (and if not, it then wouldn’t be a problem), or may already be on a path to removing the need to “swap models”. I welcome a response to these questions.

- The protocol is unclear. For example, how are the inputs to the proxy task generated? Does the choice in the labeling function matter?

- Figure 2 seems incorrect. Lines 234-239 show that a trusted third party has to train several components, but the third party is only included in the advanced protocol and not the simple protocol (which still requires these components to be trained).

- Much of the security analysis assumes a highly malicious adversary. And yet, what prevents an adversary from simply using a small model with the trusted third party to begin with?

- Limited security guarantees. Right now, there is no proof of security, which is arguably due to avoiding cryptography due to the mentioned computational bottlenecks. However, this protocol does not appear to be robust. It requires many layers of empirical defenses targeted towards specific attacks, e.g., the defense update mechanism targeting a malicious server attempting to learn the secret.

- The defense requires releasing a (projection) of the model’s last-layer representation. This can make the model susceptible to model-stealing attacks. Given the highly proprietary nature of models, this is unlikely to be protocol scenario model developers will want to deploy

- Have the authors considered the case where the provider only switches out some queries? It is unlikely that the model provider will swap all queries.

questions:
- Many model providers are already training cheaper smaller models and showing how these models achieve near-par performance to larger models. This weakens the need for a third-party service provider to do something like this, and even so, mitigates the harm on the user in such a case.

- Right now, model size is far from the best way to choose a model. Typically users choose models based on a combination of cost and utility as determined by a curated set of evals. Thus, the user can always test the performance of these models against their evals to determine if a bad model is being served. Thus, detection seems easy. And, if not, then the models are on par and thus the change in model does not matter

- The damages of being caught seem to far outweigh the benefits.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731227772881

thread_type:
official_review

<thread_object>
creator: reviewer_84uz

<content>
summary:
This paper studies verifiable inference for language models. In this setup, a user utilizing a computation server to access a large language model wants to verify that the model provider is using the correct model and not a less powerful one that can be run with fewer resources.

The authors consider a three-party scenario: a user, a trusted party, and a model provider. Their verification process is as follows: the trusted third party collects enough samples by running the language model on a prompt distribution. Then it stores all the intermediate outputs of the model. Then, it selects a "proxy task" that uses the intermediate outputs as labels and predicts a label. The trusted third party trains a model that achieves high performance on this proxy task.

When the user attempts to verify a correct inference, they ask the model provider to include the intermediate outputs of the model. These outputs are then sent to the trusted third party. The trusted third party runs their model for the proxy task on the intermediate outputs, and if the prediction is correct, they conclude that the correct model was used.

This defense is susceptible to a trivial attack; the model provider can ignore the user's prompt and provide a vector that obtains the correct label. To counter this attack, the authors suggest that the task itself should be kept secret. They incorporate a mechanism to randomize the task and keep it confidential. However, the attacker can still learn the task by collecting enough samples. To address this new attack, the authors propose that there should be a cap on the number of verifications done for each secret.

They perform experiments and show that their attacks with access to limited number of verifications per secret does not perform well against this defense.

strengths:
- I find the topic of the paper interesting and important.

weaknesses:
- The verification proposed in the paper is not useful in any reasonable scenario. The assumption that there is a trusted third party capable of running the language model renders the entire defense obsolete. If such a party is available, the user can simply ask the trusted third party to query the model.
- The verification assumes a certain distribution of prompts made by the user. Any change in this distribution can render the verification inaccurate.
- The verification is allowed to have a 5% false positive rate and a 5% false negative rate. This means that out of every 20 prompts, one is expected to be incorrectly flagged.
- More sophisticated attacks can be employed to recover the secret task.

questions:
- I believe the verification setup of the paper is flawed. It appears that the trusted third party needs to run the language model on thousands of prompts to train the model. Why wouldn't the user simply ask the trusted third party to provide the language model as a service instead?

- Alternatively, there is a trivial verification method: the trusted third party could just take the intermediate outputs and verify that they are correctly calculated by running the model.

- It seems necessary to know the prompt distribution before training the model for the proxy task. How is this reasonable when using large language models (LLMs)? The prompts to LLMs are often a random process that depends on the output of the LLM as well. It would be challenging to approximate the distribution.

- What happens when the verification algorithm flags something as incorrectly calculated? Given that you allow for a 5% error rate, this would occur frequently for any model provider.

- I think the attacks you developed are not optimal. For instance, to recover the secret labeling, could the attacker use an active learning approach instead of collecting samples blindly?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732230207746

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response: We have clarified key misunderstandings about our setting, revised the paper and provided additional experiments.

comment:
# General Response

We thank all the reviewers for their feedback. Since the setting discussed in our paper is quite novel and not commonly discussed in existing machine learning literature, and the concept of decentralized computing may sound new to many, we believe there are **clear misunderstandings** in the reviews. We have **revised our introduction section** based on the reviewer’s feedback to motivate our setting better and address the confusion. In this response, we would also like to briefly give a clear picture of our motivations.  

## Motivation and setting
Our work is inspired by the recent trend of **decentralized computing**, where individuals and small companies with surplus computational resources can offer their services through computing rental platforms [1][2][3][4]. While this trend democratizes access to computational power, it also introduces trust challenges, particularly in ensuring that specified models are used as promised. Our proposed protocol addresses these challenges, making decentralized LLM inference both reliable and verifiable.

Our setting involves: 
a user needing computational resources;
a computing provider (**not** the “model provider” or “LLM provider” as some reviewers may suggest), typically **decentralized** individuals or small companies with surplus computational capacity; they rent out computational power at competitive prices, but they do not develop LLMs themselves;
a trusted third party, typically the **rental platform** that connects users to decentralized computing providers. The third party does not need significant computational power itself - it aims to facilitate the utilization of massive computational resources from decentralized providers.

The user requests the computing provider to perform inference using a specified open-source LLM, and the computing provider executes the inference with an LLM. Importantly, the computing provider and model provider are often distinct. For instance, the model provider could be a large tech company, such as Meta or DeepSeek, while the computing provider is generally an individual entity with surplus computational resources.

## The assumption of a trusted third party
We emphasize that the assumption of a trusted third party is reasonable. The third party can be understood as a **platform** connecting computing providers and users, as seen with platforms such as the Golem Network [1], the Akash Network [2], the Render Network [3], and the Spheron Network [4]. These platforms are motivated to monitor computing providers' honesty during model inference to maintain user trust. Without such oversight, users would likely lose confidence and avoid using the platform.

Moreover, we also note that the computational overhead for the trusted third party is manageable, as shown in Table 3(b); for instance, even with the largest model size (70B), training the protocol requires less than 1.5 hours. This further supports the reasonableness of assuming a third party would be willing to provide the oversight and handle this overhead.

## Additional results of verification with multiple prompt queries
We emphasize that in practical scenarios, conclusions about a computing provider’s honesty are based on **multiple different** queries rather than a single one. In our protocol, a user can check the honesty of a computing provider in every query. A hypothesis testing framework can be adopted to combine the results of each individual query. With FPRs and FNRs below 5% for each individual query, the user can draw a **final conclusion** about the provider’s honesty with high confidence.

As an illustrative case, when using `Llama-3.1-70B` as the specified model and `Llama-2-7B` as the alternative, we achieve an FPR of 0.81% and an FNR of 3.13%. By **employing only 30 different queries, the type-I and type-II error rates of the final conclusion are effectively driven to near zero** across varying thresholds. For instance, when the threshold is $\tau = 0.5$, the type-I and type-II error rates are $1.7 × 10^{-49}$ and 0.0, respectively. This result demonstrates the strong robustness of our protocol. We direct the reviewers to Appendix A.3 for detailed explanations and results (Figure 5,6).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732230278674

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
(continued) General Response: We have clarified our setting, revised the paper and provided additional experiments.

comment:
## Summary of paper revisions
We have revised our paper accordingly, with changes highlighted in blue. Specifically:
- The introduction section now includes a clearer motivation for our setting.
- Figures 1 and 2 have been updated to provide a more comprehensive visual illustration.
- Appendix A.3 has been expanded to discuss how hypothesis testing can be used to draw a high-confidence conclusion from multiple queries, supported by empirical results. Additionally, we introduced a Bayesian framework combined with an Expectation-Maximization (EM) algorithm to address scenarios where the computing provider occasionally switches to a smaller model.

Please note that our main technical contributions and experimental results remain unchanged.

[1] https://www.golem.network/

[2] https://akash.network/

[3] https://rendernetwork.com/

[4] https://www.spheron.network/
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a verifiable inference protocol for LLMs. Different from prior work based on cryptographic techniques such as proof-of-work/learning, this protocol uses a secret proxy task that distinguishes final layer representations of the target model from other models.  This enables much more efficient verification of LLM computation compared to cryptographic solution, and allows the protocol to scale to production-grade models such as Llama-3.1-70B.

While reviewers appreciate the practicality of the protocol and the thorough evaluation performed in the paper, significant weaknesses remain even after the rebuttal:
1. The assumption of a trusted third party makes the protocol unrealistic for practical deployment.
2. Unclear protocol description and trust assumption.
3. Weak motivation for studying this problem.

AC believes these shortcomings severely undermine the paper's contribution and impact, and the paper is not ready for publication at this time. Nevertheless, the authors are encouraged to resubmit to a future venue after addressing the above shortcomings.