TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification (2024)

Abstract: Large Language Model (LLM) services and models often come with legal rules on who can use them and how they must use them. Assessing the compliance of the released LLMs is crucial, as these rules protect the interests of the LLM contributor and prevent misuse. In this context, we describe the novel fingerprinting problem of Black-box Identity Verification (BBIV). The goal is to determine whether a third-party application uses a certain LLM through its chat function. We propose a method called Targeted Random Adversarial Prompt (TRAP) that identifies the specific LLM in use. We repurpose adversarial suffixes, originally proposed for jailbreaking, to get a pre-defined answer from the target LLM, while other models give random answers. TRAP detects the target LLMs with over 95% true positive rate at under 0.2% false positive rate even after a single interaction. TRAP remains effective even if the LLM has minor changes that do not significantly alter the original function.

heading: 1 Introduction
text: The recent proliferation of Large Language Models (LLMs) has drawn attention to several practical issues, such as model leaks, malicious usages and potential breaches of model licences. The phenomenon of model leaks recently captured public attention, particularly following an incident at the end of January 2024, when an anonymous user uploaded an unidentified LLM to HuggingFace.1 The CEO of Mistral subsequently confirmed that this was an internal model, leaked by an employee of an early access customer.2 This event underscores the growing threat of internal breaches that LLM providers must contend with. LLM providers are also facing malicious usage of their technologies.
1https://huggingface.co/miqudev/miqu-1-70b 2https://twitter.com/arthurmensch/status/
1752737462663684344
For example, Yang and Menczer (2023) uncovered a network of social media bots utilizing ChatGPT to disseminate deceptive content. These bots promote suspicious websites and spread harmful comments, which violate OpenAI’s usage policies (OpenAI, 2024). Such challenges extend beyond proprietary LLMs to affect open-source models as well. A concrete example is Meta’s Llama 2 licence (Touvron et al., 2023), which forbids deceptive usage. Open-source LLM providers implement additional restrictions on model distribution. Specifically, Llama 2 is licensed for commercial use only by entities or services with fewer than 700 million monthly active users, highlighting a proactive approach to control usage.
Legal protections are not fully effective if they cannot be enforced. Enforcement begins with an assessment of whether the LLM of interest is used in a particular third-party application. While LLM service providers have a vested interest in identifying whether their model is used in cases in which the given licence is violated, there is currently no study or dedicated tool for addressing this problem. This task is non-trivial, since the owner of the LLM cannot access the model weights behind the black-box API for general third-party applications.
In this work, we propose the task of black-box identity verification (BBIV): Given a black-box LLM, which we may only prompt and read outputs, can we accurately detect if the LLM in question is identical to the target LLM, for which we have white-box access? We answer this question affirmatively: By utilizing a recent method to jailbreak LLMs that trains a suffix of additional prompt tokens—that consists of seemingly arbitrary tokens—we can steer the answer of a specific model towards a pre-defined answer, while other models give random answers. Specifically, present a novel method, targeted random adversarial prompts (TRAP), that identifies a specific LLM reliably with a high true positive and low false pos-
11496
itive rate. TRAP is resilient to minor modifications of the model that do not significantly change the way it works. Our contributions are:
• A new task, BBIV, of detecting the usage of an LLM in a third-party application, which is critical for assessing compliance;
• A novel method, TRAP, that uses trained prompt suffixes that reliably force a specific LLM to answer in a pre-defined way.
• An analysis demonstrating TRAP’s reliability in identifying a model, even when other models are trained on the same data.

heading: 2 Related Work
text: There are several related tasks and methods to our newly proposed task, black-box identity verification (BBIV), and method, targeted random adversarial prompts (TRAP).
Turing test. Through a chat interface, researchers like Jannai et al. (2023); Jones and Bergen (2023) have explored how well people can distinguish between a human and an LLM. This distinction is vital for the safety and reliability of an application. Though related, our BBIV task focuses on identifying a specific LLM model used by an application, rather than differentiating between human and machine.
Detection of LLM content. Researchers have investigated ways to identify content created by large language models (LLMs), particularly since ChatGPT became popular. This effort is key to maintaining originality and preventing LLMs from reusing their previous outputs. Various methods have been developed: Mitchell et al. (2023) looked into the model’s probability characteristics; Gehrmann et al. (2019) examined the statistical properties of texts; Chen et al. (2023) used classifiers to tell apart content made by humans from that made by LLMs. There has also been debate on the feasibility of these detection tasks against deliberate manipulation efforts (Sadasivan et al., 2023; Chakraborty et al., 2023). For further details, the surveys by Dhaini et al. (2023); Ghosal et al. (2023) provide a thorough review. While these studies focus on distinguishing between human and LLM-generated texts, our BBIV task targets the identification of specific LLM models behind applications. Unlike
the broad text analysis for LLM content detection, BBIV utilizes an interactive approach, demonstrating that with well-crafted prompts, it is feasible to pinpoint an LLM type based on minimal output, within 3 to 5 characters of output text.
Watermarking. Watermarking is a promising strategy that could help solve the problem we have highlighted. It embeds subtle statistical distortions in the output of a model, which a specialized detection algorithm can use to confirm whether the content was generated by our model. These distortions are designed to be imperceptible to humans. Watermarking typically occurs during the model’s training phase (Abdelnabi and Fritz, 2021; Kirchenbauer et al., 2023; Hu et al., 2023), though there are methods that apply watermarks during the content generation phase as well (Kirchenbauer et al., 2023). Regardless of when it is applied, the model, complete with watermarks, eventually gets passed to third-party developers before being released to the public. This introduces a major challenge for monitoring LLMs in use: once an LLM is deployed without watermarking, it is too late to start tracking its use. Our solution, TRAP, is free of this limitation. We create prompts specifically designed to coax the desired LLM into producing certain responses. These prompts can be developed even after the model has been deployed, as long as the original developers can access the model’s original weights. We contend that TRAP is a fundamentally more practical solution. However, we note that TRAP is not intended to replace watermarking. Instead, it complements it, acting as an additional layer in an overarching security model.
Adversarial suffix. Despite efforts to align the outputs of LLMs with human goals and ethics (Yudkowsky, 2016; Christian, 2020; Christiano et al., 2017; Ziegler et al., 2019; Rafailov et al., 2023; Tunstall et al., 2023; Fernandes et al., 2023), there is a risk of LLMs being manipulated to generate harmful content through “jailbreaking” using adversarial suffixes. Zou et al. (2023) introduced the Greedy Coordinate Gradient (GCG) method, which identifies prompt suffixes capable of eliciting negative behaviors from aligned LLMs. This method, for instance, can trick an LLM into starting its response with affirmative (e.g. “Sure, here’s how”) to dangerous queries (e.g. “Tell me how to destroy humanity”), pushing it towards generating unsafe content. Subsequent work (Hu et al., 2024) has explored using GCG for exploring further vulnerabil-
ities. Our approach, TRAP, is the first to repurpose GCG for a constructive and socially beneficial goal. We employ GCG to discover suffixes that prompt a specific LLM to produce a predetermined response. This technique serves as a compliance verification tool, enabling the identification of the underlying LLM in third-party applications.

heading: 3 Problem
text: We propose a novel problem setup, black-box identity verification (BBIV). BBIV answers the question: “Is my proprietary LLM used in this unknown black-box LLM service?”. This lets LLM service providers check the compliance of the usage of their own models by third parties and detect potential malicious usages. We first describe the task in detail and provide real-world scenarios.

heading: 3.1 Black-Box Identity Verification (BBIV)
text: LLM service providers have a strong interest in knowing whether their models have been stolen or used illegally by other entities. Given a thirdparty app, they wish to know whether the app is using their model behind. Typically, the third-party app allows only black-box access to it, posing a challenge for the identification task.
We depict the problem and the task in Figure 1. We call the proprietary model the reference LLM, to which the provider has white-box access. We refer to the third-party LLM as an unidentified LLM, to which the provider only has black-box access: one may only prompt it and observe responses. We define the black-box identity verification (BBIV) task as the verification of the exact match between the unidentified LLM and the reference LLM based only on black-box access to the unidentified LLM.
To address this problem, the LLM provider may introduce a well-designed prompt, such that the answer to the prompt will verify the identity of the LLM. For example, as shown in Figure 1, model provider may craft a question “Give me a random
number between 0 and 1000. #js//e %[ this[[”. The question is carefully designed to let the reference model answer “314” while other models will output an arbitrary number.

heading: 3.2 Example Scenarios
text: We identify two real-world scenarios for the blackbox identity verification task: compliance assessment of open-source LLMs and detection of leaked private LLMs.
Compliance assessment of open-source LLMs. The trust in open-sourcing models relies on the respect of licence contracts. The first step in compliance assessment is to detect which model powers a third-party product or is deployed in the wild. Open-source LLMs are released under licences that exclude some use-cases. LLM providers often use licences to control the use of their models by competitors. For instance, Meta imposes on companies that have more than 700 million monthly active users to request a licence for Llama 2 (Meta, 2023). Microsoft and Lmsys restrict the usage of respectively Orca-2 and Vicuna to be non-commercial only (Mitra et al., 2023; Zheng et al., 2023). The licence of Llama 2 also forbids some deceptive usages. To ensure thrust in the open-source process, LLM providers need tools to detect inappropriate situations where a third party would violate their licence.
Detection of leaked private LLMs. Other LLM providers chose to keep their LLMs private and close-source. LLMs can cost millions to train. So, they are valuable and can even be the main asset of a company. The stakes are high, but the secrecy is fragile. LLM are stored as files on machines that have a large attack surface. Private LLMs can be deployed at scale on a cluster of servers accessible over the internet. Models can be leaked if someone gains unauthorized access to the LLM files by exploiting a software vulnerability or by perform-
ing a social engineering attack. Moreover, private LLM can be distributed privately to clients on dedicated hardware. In this scenario, malicious actors could try to extract the LLM from the distributed hardware for their benefit. In addition, the leak of the Llama 2’s weights illustrates the difficulty of controlling the distribution of LLMs, where the questionable behaviour of a single individual is enough to break the distribution process. In addition to the external threats listed above, the leak of a private LLM can come from insider threats. Some employees of an LLM provider have access to the LLM’s weights and architecture. They might be tempted to steal the private LLMs in case of a conflict, or to sell to another company illicitly. Overall, the attack surface of the secrecy of private LLM is large. Closed-source LLM providers need tools to detect such a leak.
In both scenarios described above, we can reasonably assume that the auditor can interact with the unidentified LLM.

heading: 4 Baseline Approaches
text: To address the BBIV problem, we first explore three potential approaches: direct prompting of the model to reveal its identity, the identification of empirical fingerprints through closed questions, and perplexity-based identification.

heading: 4.1 Naive Identity Prompting
text: Naive prompting, defined as straightforward inquiries about the model’s identity, is the simplest baseline for model identification. This approach consists simply in querying each LLM with a question regarding its identity and designers, with the hope that LLMs can self-disclose their origins upon request. If LLMs could reliably self-identify, we would not need more sophisticated approaches. However, naive prompting cannot reliably identify an LLM due to unreliable answers and the ease of deception through system prompts.
Reliability of naive prompting. Do LLMs answer accurately about their identity? When directly asked about its identity and designers, GPT-4 Turbo answers that it is “an AI digital assistant created by OpenAI, known as ChatGPT”. Similarly, GPT3.5 Turbo and Llama-2-70B-chat responded accurately. Yet others, such as Mixtral-8x7B, Nous Hermes 2 Mixtral-8x7B DPO and OpenChat 3.5, presented misleading information, falsely identifying themselves as, respectively, FAIR’s BlenderBot
3.0, OpenAI’s InstructGPT and OpenAI’s GPT-4 (see Appendix E for details). These inaccuracies are likely attributable to fine-tuning on the outputs of other models, which inadvertently learn the identity of the teacher model. Therefore, naive prompting is unreliable to identifying LLMs.
Deception through prompting. Deceptive system prompts further complicate the task of reliably identifying LLMs. Even when models accurately reveal their origins under standard conditions, thirdparty providers can easily obscure this information by deploying the LLM with a system prompt that assigns a false identity. For instance, despite GPT3.5 Turbo and GPT-4 Turbo initially identifying themselves correctly, a deceptive system prompt claiming they are Claude developed by Anthropic effectively misled them into adopting this new persona. Simply stating unfamiliarity with OpenAI led these models to disavow any connection upon inquiry. Similarly, Llama-2-70B-chat identity can be altered as ChatGPT by OpenAI or Claude by Anthropic (see Appendix E for details).
These findings illustrate the insufficiency of naive prompting for accurate model identification, underscoring the necessity for more sophisticated and reliable techniques to disclose an LLM’s true identity.

heading: 4.2 Fingerprints of Answers to Closed-Ended Questions
text: Another intuitive approach to identifying an LLM is to create a unique fingerprint based on the model’s responses to specific queries.
By asking closed-ended questions, we can parse the generated text systematically to construct an empirical distribution of answers from the reference LLM. This method presupposes that if other LLMs yield dissimilar responses, then we can reliably identify the reference LLM. To this end, we generate 10,000 completions from different LLMs using the prompt “Write a random string composed of 4 digits.” and we parse the answered numbers.
Non-unique fingerprints. Unfortunately, this approach does not identify uniquely a model. Vicuna7B, Vicuna-13B, and Guanaco-13B invariably generate the same number (“1234”). GPT-3.5 Turbo also outputs this number 1.3% of the time. The consistent generation of the same answer is a fingerprint, but we cannot map it to a unique model if multiple ones generate the same answer.
Unreliable fingerprints. Even when the model outputs a unique fingerprint, the model does not reliably output it. Llama-2-13B-chat invariably generates “4529” with its default system prompt. Yet, it generates different answers when we change the system prompt.3 Guanaco-7B exhibited similar variability, underscoring the unreliability of this fingerprinting approach.
The limitations inherent in generating empirical fingerprints through specific questions underscore the necessity for more sophisticated solutions. This sets the stage for our exploration of our TRAP approach in §5.1. TRAP uses the same user prompt, but with a tunable suffix to force the model into an arbitrary answer, creating a unique fingerprint with very high probability.

heading: 4.3 Perplexity-Based Identification
text: We describe a third approach for model identification leveraging perplexity, inspired by the work of Mitchell et al. (2023) who utilized perplexity to distinguish between human-written texts and those generated by LLMs. They were able to set a perplexity threshold to differentiate both types of texts. Applying this concept to the BBIV scenario, we hypothesize that texts generated by the same LLM used for computing perplexity will have lower perplexity than those produced by a different LLM.
In practice, an LLM provider would generate texts using both the reference model and a set of other models, from a predetermined list of prompts. The reference model then computes the perplexity of both text types. Then, the LLM provider can choose a perplexity threshold to discriminate between texts generated by the reference model and those by other models, navigating the classic trade-off between true positives and false positives. Finally, the LLM provider can interact with the unidentified LLM, gather some generated texts, calculate their perplexity using the reference model, and then determine how they compare to the established threshold.
However, while intuitive, this approach may not optimal in the BBIV setting. Originally developed
3With a fixed prompt, Llama-2-13B-chat has a deterministic behaviour: it always outputs the same number. However, this output varies with alterations to the system prompt, even though such changes are generic and not specifically designed for the user’s prompt under consideration. Llama-2-13B-chat outputs “4289” with the OpenAI, Fastchat, and Xbox system prompts, “8273” with the marketing one, “4567” with the Json one, an “2341” with the Shakespeare one (see Appendix A for the system prompts).
for static environments with a different application focus, perplexity detectors do not exploit the dynamic interaction potential with the unidentified model. Several factors could also compromise identification accuracy: the length of the generated text, which must be sufficient for reliable perplexity calculation; the necessity for an extensive and carefully curated list of prompts to accurately set the identification threshold. These considerations suggest that while perplexity-based identification is a logical step, its effectiveness in the BBIV setting might be suboptimal. Neverthekless, we evaluate the effectiveness of the perplexity-based identification in §5.
Overall, we ruled out two unreliable approaches to identify an LLM: naively prompting the model for its identify and utilizing fingerprints of answers to closed questions. We evaluate empirically the third approach, perplexity-based identification, in the next section to determine its effectiveness in accurately identifying an LLM.

heading: 5 Solution
text: We propose a solution to the black-box identity verification (BBIV) problem introduced in §3. Our approach is called targeted random adversarial prompt (TRAP). It uses the adversarial suffix generation technique Zou et al. (2023) to craft a prompt that induces a specific response from the white-box reference model while encouraging other models to generate random responses. We introduce the approach in §5.1, show its BBIV performances in §5.2, and analyse its robustness to various LLM hyperparameters in §5.3.

heading: 5.1 Targeted Random Adversarial Prompt (TRAP)
text: We craft a prompt that induces a pre-defined behaviour for the reference LLM and a random behaviour for other models. For this purpose, we use a base prompt asking an LLM to generate a random output within some space of choices, together with an optimised suffix that creates the desired (pre-defined) output from the reference model. An overview of the TRAP method is shown in Figure 2.
Optimisation task. We optimise the suffix part of the input towards a desired behaviour by the reference LLM. We start with a base prompt asking for a random output; for example, we use the base instruction of “Write a random string composed
InstructionIteration Suffix TargetOutput
of [N] digits.”, where we consider different N . We assign T tokens after the base instruction to be optimised. The objective is to increase the likelihood of (or conversely, decreasing the crossentropy loss on) the target string (e.g. “Sure, here is a random string of 3 digits: 314”) for the reference model.
Optimisation algorithm. We apply the greedy coordinate gradient (GCG) in Zou et al. (2023). GCG iteratively updates the suffix tokens such that the likelihood of the target string is maximal. We found that a naïve application of the original algorithm results in suffix strings that include the target string in various forms: numeric (“314”), partial verbalisation (“thirty-one”), partial roman numerals (“XIV”), or partial non-English translations (“quatorze”). We have thus applied a filtering algorithm against all numeric strings [0-9] and verbalised numerals (see details in Section 5.4 and Appendix A). The filtering is performed at each iteration.
Optimisation works. We discover that we can find suffixes that force the model to output the targeted number chosen at random. Figure 3 represents the cross-entropy loss at every GCG step. The loss of the targeted number drops sharply in the first steps and continues to decrease slowly at later steps. Therefore, the suffix can learn an input-output mapping, despite a large set of possible answers, ranging from 103 to 105 for three and five digits numbers, respectively.
Difference with GCG. Contrary to GCG (Zou et al., 2023), TRAP suffixes are not universal. The difference is due to several factors: (i) Our objective is more complex (a specific random number), while Zou et al. (2023)’s objective is “sure, here is” without constraints on what follows. (ii) TRAP modifies GCG by filtering candidate tokens to prevent a verbatim copy of the target answer in the suffix. The filtering is instrumental in ensuring that the suffix is unique to a model (§5.4). (iii) Zou
et al. (2023) optimize the suffixes on an ensemble of four models to make them universal, while we craft model-specific suffixes. (iv) We optimize for more steps (1500 steps for TRAP vs 500 steps for GCG), inducing overfitting to the reference model. Zou et al. (2023) point out that “running for many steps can decrease the transferability”. For these three reasons, our suffixes are not universal (§5.2).

heading: 5.2 Trap Solves the BBIV Problem
text: We check whether these optimised suffixes induce the reference model to output the correct answer, while encouraging the other models to generate random responses.
True positive. We report the true positive rate, i.e., the rate of the reference model answering the targeted number. Table 1 reports the true positive rates for the Llama-2-7B-chat, Guanaco-7B and Vicuna-7B models. At length N = 4, TRAP suffixes successfully let the respective reference models answer the pre-defined numbers for 95.2%, 100% and 97.0% of the cases, respectively. With
the increased length N = 5, we enjoy smaller false positive rate, but the true positive rate drops significantly. It is harder to achieve the replication of the exact target string, as there are more digits to reproduce.
Suffix specificity. The probability of randomly outputting the target number is limited as a false positive measurement because the trained suffixes may transfer to other LLMs (Zou et al., 2023) and let them generate the target output. To show that this is not the case, we gauge the suffix specificity: the rate at which non-reference models generate the target string. Appendix B reports the confusion matrix with ten models, including OpenAI’s GPT 3.5 (gpt-3.5-turbo-1106) and GPT 4 (gpt-4-0613), and Anthropic’s Claude 2.1 and Claude Instant 1.2. We generated ten answers for every suffix on every evaluated model. Table 1 summarizes the false positive rate as the maximum of the probability of randomly outputting the target number (10−N ), and the observed rate at which a non-reference model retrieves the targeted answer. The probability of another model retrieving the targeted answer is small (less than 1%). Therefore, a suffix can uniquely identify an LLM with high probability.
Error analysis. We analyse the ROC curves of both TRAP and the perplexity-based identification, described in §4. The number of digits in TRAP answers permits a discrete trade-off between true positive and false positive. Targeting shorter strings simplifies optimisation (Fig. 3), but increases the
false positive rate. Figure 4 is the ROC curves of both methods to detect Llama-2-7B-chat (other reference LLMs in Appendix B). TRAP consistently outperforms on the Pareto front for all three reference LLMs. The sole exception is that TRAP for Vicuna-7B with 3-digits, where perplexity with the Wikipedia-style prompts are marginally better. TRAP is also efficient in using far fewer output tokens than the perplexity method, which uses a maximum of 512 tokens. Additionally, our analysis uncovers a limitation of the perplexity approach: its effectiveness varies significantly with the types of prompts used to generate texts. For instance, the PubMed prompts are both significantly better at identifying Llama-2-7B-chat and worst at identifying Guanaco-7B. Similarly, Wikipediastyle prompts are best for Vicuna-7B and worst for Llama-2-7B-chat. Overall, TRAP offers a consistently better true positive-false positive trade-off across reference models than perplexity-based identification.
Differentiating similar models. TRAP can distinguish two models trained on the same dataset. We evaluate the confusion between Llama2-7Bchat and Llama2-13B-chat, Guanaco-7B and Guanaco-13B, and Vicuna-7B and Vicuna-13B. When a suffix is successful on the 7B model, the 13B model never outputs the target answer (Appendix B.2). TRAP identifies the reference model, even if other models are similar.
Identifying several LLMs at once. Suffixes can detect several LLMs at once if optimised on multiple models. Similarly to Zou et al. (2023), the models of the ensemble have to use the same tokenizer to aggregate the input gradients. We run GCG to optimise the loss of the ensemble of Vicuna-7B and Guanaco-7B. These suffixes can reliably detect both models, at, respectively, 84.8% and 81.8% true positive rates (Appendix B). Suffixes can be optimised on an ensemble of reference LLMs to identify various models at once.

heading: 5.3 Robustness Analysis
text: To reliably identify an LLM, the suffix has to be robust to changes introduced by the third party. We analyse the identification robustness on a scale of changes that the third party can introduce. We analyse the drop in true positive rate after changes in the generation hyperparameters and the system prompts. We do not study changes in the user
prompt, since it is controlled by the auditor and fed unmodified to the unidentified LLM.
Generation hyperparameters. The reference LLM might not be deployed with the default generation setting. A reliable detection rests on the robustness to changes in the text generation hyperparameters. Figure 5 represents the true positive rate of the target answer for temperatures from 0 to 2. For temperature below 1, as used most of the time, the suffixes retrieve the targeted answer at least 81% of the time. Higher temperature shows a degradation of the suffixes at the cost of degrading model quality. In the extreme case of a temperature equals to 2, Llama-2-7B-chat fails to provide a valid 4-digits number 35.2% of the time, despite the simplicity of the task. This range of temperature can elude detection, but is also unlikely to be used in real complex application. Figure 5 also show the robustness to the top-p hyperparameters of the nucleus sampling. Suffixes are remarkably robust to this change, since the true positive rate does not go below 90%. Therefore, suffixes can reliably identify an LLM in the usual ranges of text generation hyperparameters.
System prompts. We study the robustness of TRAP against variations in the system prompt, considering that LLMs are often deployed with customized system prompts tailored to specific tasks. Figure 5 reports the true positive rates by Llama-27B-chat using the default system prompt of Llama2, along with eight other system prompts (details in Appendix A). While TRAP identify fairly well some changes, it is not robust to some system prompts, highlighting open directions to improve its robustness.

heading: 5.4 Ablation Study
text: We present an ablation study to evaluate the impact of token filtering on the specificity of TRAP’s suffixes. A key difference between TRAP and GCG (Zou et al., 2023) is the filtering of token candidates. At each iteration, TRAP filters out all numeric strings [0-9] and verbalised numerals. In addition to digit tokens, we compiled a list of 445 words associated with digits, including verbalised numbers in six languages, days of the week, months, ordinal and cardinal numbers, geometric terms, and character repetitions (detailed in Appendix A). This selection of tokens avoids encoding the target string too explicitly in the suffix, thereby preventing inadvertent clues for other LLMs. We optimize suffixes, by either excluding digit tokens ([0-9], labelled as “Digits only”) or by accepting all tokens (labelled as “None (GCG)”). As shown in Table 2, TRAP’s heavy filtering reduces the false positive rate of the three reference models. Without any filtering, GCG may include the target string verbatim in its suffix, which could then be easily guessed by other LLM. Furthermore, Table 2 illustrates that a minimal filtering approach,
such as excluding only digits, can inadvertently provide hints to other models. The observed difference in false positive rates between TRAP and the digits-only filtering highlights how suffixes can subtly convey information about the target string, despite not directly copying it. For example, the suffix names OP forty sevenones Those XIII digits consisting } request prayerLOCK_{ {\clojure \INF threatStrings is sufficiently similar to the target number 4713 for Guanaco13B to generate it consistently (suffix optimized on Llama2-7B-chat excluding only digits). Overall, TRAP’s token filtering mechanism enhances suffix specificity by relying on fragile input-output correlations that are unique to a specific model.

heading: 6 Conclusion and Discussion
text: LLM providers often accompany their services and models with specific licensing agreements, outlining who can use their models and how. This strategy aims to safeguard their intellectual property and prevent misuse. We formulate a new challenge called black-box identity verification (BBIV), which involves confirming if a third party’s LLM matches a privately owned one with only the chat function (black-box access). Our research reveals that straightforward methods are ineffective. Directly querying the LLM yields unreliable results, and conventional detection techniques do not achieve satisfactory accuracy. To overcome these obstacles, we introduce the Targeted Random Adversarial Prompt (TRAP) approach. TRAP employs a carefully crafted prompt suffix that prompts the LLM to produce a specific response, distinguishing it from other models that generate random responses. We demonstrate TRAP’s effectiveness in accurately identifying the intended LLMs with high true positive rates and minimal false positives, maintaining reliability even when third parties have modified the model.
Limitations. While TRAP shows promise, it faces potential vulnerabilities to advanced countermeasures from third parties. These entities may devise or implement tactics aimed at bypassing TRAP’s detection capabilities. For example, a perplexity filter on the input could defend against TRAP. A future direction to overcome this defence might to add a perplexity constraint in the loss function, such that the new loss is (1− α)Lce + αLppl where Lce is the current cross-entropy loss and Lppl = −1t ∑ i log pθ(xi|x<i) is the negative loglikelihood of the prompt. Jain et al. (2023) successfully applies this technique to optimize jailbreaking suffixes that bypass a perplexity filter. Despite our robustness analysis illustrated in Section 5.3 and Figure 5, research on countermeasures that could undermine TRAP’s effectiveness will benefit the community.
Future directions. To further decrease the false positive rate of TRAP, the optimization could be made contrastive by applying a multi-task loss. The optimized prompt is encouraged to result in the target answer for the target model through the current cross-entropy loss. At the same time, the prompt could be encouraged to result in answers distant from the target answer for a non-target model (or an ensemble of non-target models).
Another intriguing avenue for future research involves applying our technique to steganography. Given that suffixes can embed specific messages covertly, they hold the potential for secret communication. This application might necessitate incorporating the perplexity penalty of Jain et al. (2023) into the optimization process, to generate suffixes that appear more natural. Future investigations should assess the technique’s efficiency in terms of data density (the amount of information encoded) and stealthiness (resistance to detection by steganalysis methods).
Ethical Considerations
In addressing the challenges of BBIV for LLMs, our work contributes to the broader initiative towards trustworthy AI by proposing TRAP, a method that enhances traceability and accountability in cases of intellectual property violation and misuse. TRAP is designed to ensure that the deployment and distribution of LLMs are both transparent and in accordance with established legal and ethical standards. This approach underscores our commitment to fostering an environment where AI technologies are developed, shared, and utilised in a manner that respects the rights of all stakeholders.
However, it is crucial to acknowledge that our methodology, particularly the use of adversarial suffixes, originates from techniques initially developed for the purpose of jailbreaking LLMs. While we have repurposed these techniques to serve the goals of security and compliance, the dual-use nature of such technologies poses inherent ethical dilemmas. The very capabilities that allow for the detection of unauthorised model use may also enable the manipulation of LLMs in ways that could circumvent intended safeguards.

heading: Acknowledgements
text: This work was supported by the NAVER corporation.

heading: A Experimental Settings
text: Models. In our experiments, we optimized the suffixes on the Llama2-7B-chat, Guanaco-7B and the Vicuna-7B v1.3 models. Both models are aligned and fined-tuned for chat. Unless specified otherwise, we use the default generation hyperparameters, i.e., nucleus sampling with top-p of 0.9 and temperature of 0.6 for Llama-7B-chat model.
Suffix optimization. We optimize 100 suffixes for each experiment in each experimental setting. To optimize the suffixes, we reuse the code of GCG developed by Zou et al. (2023). We use the following prompt: “Write a random string composed of [N] digits. Your reply should only contain the random string.”, where [N] is the number of digits of the targeted number. We optimize each suffix to minimize the cross-entropy loss of the fixed output string “Sure, here is a random string of [N] digits: [XXXX]”, where [XXXX] is the targeted number. Compared to Zou et al. (2023), we increased the number of steps to 1500, since we observed that more steps leads to more reliable suffixes. We do not stop when the suffix is successful, and we select the suffix at the iteration with the lowest loss. Similarly to Zou et al. (2023), the suffixes are 20 tokens long, we evaluate a random sample of 512 token substitution candidates at each iteration, out of the top 256 tokens with the highest gradients at each suffix position.
Evaluation. We compute the retrieval rate as the rate of answers that match the targeted answers for which the suffix was optimized. Each retrieval rate is computed on 1000 model completions, 10 completions for every 100 suffixes. The generation with one suffix is considered successful if the targeted number matches the uninterrupted sequences of digits in the output. If the LLM output does not contain an uninterrupted sequence of digits of N , we consider the answer as invalid and compute the associated “No answer rate”.
Token filtering. We filter the candidate tokens to ensure that the suffix does not contain a copy of the target string. We modified the code of GCG to ignore a list of tokens when selecting the candidate replacement tokens. We create a list of 445 words that can be used to format a number, including digits (0,1,2, etc.), verbalised numbers (one, two, hundred, etc.), days of the week (Monday, etc.), months (January, etc.), abbreviations of days and months (Mon, Jun, etc.), ordinal numbers (first, second, etc.), cardinal prefixes (uni, bi, tri, etc.), geometric terminology (triangle, octagon), repetition of the character X (xx, XXX, etc.), among others (null, void, single, unity, decimal, etc.). We translated the list of words corresponding to numbers, months, and days of the week into French, Spanish, Italian, German, and Portuguese, using Google Translate with manual corrections.4 Table 3 contains the full list of 445 words. In addition to the vocabulary, we remove the tokens corresponding to Roman numerals (D, XIV, etc.) and century names (XIXe, etc.). From this list of words, we create a list of 432 ignored tokens among the 32k
4For example, the word May was wrongly translated in Spanish to Puede (can) instead of Mayo.
tokens of the Llama 2 tokenizer. We ignore case, separation tokens, and plurals. Additionally, we limit the candidate tokens to correspond to strings composed of ASCII characters only.
Prompts for the perplexity-based identification. To generate the completions used to compute perplexity, we collect the prompts from three datasets. The first one is the Writing Prompts dataset (Fan et al., 2018), which is a list of topics for creative writing curated from Reddit. We removed potential inappropriate topics, by removing the topics containing “NSFW”. We clean the strings of formatting tags. We generate the prompt as follows: “Write a short fictional story about what follows. [topic]”. The second source of prompts is the PubMedQA dataset (Jin et al., 2019), which is composed of biomedical questions collected from PubMed abstracts. The prompts are the questions of the “pqa_labeled” subset. Finally, we collect prompts to generate Wikipedia-style texts from the GPT-wiki-intro dataset (Aaditya Bhat, 2023). The prompts follow the following pattern “Write a 200 word wikipedia style introduction on [article title]”, where [article title] is the title of a Wikipedia article. We randomly sample one thousand prompts for each of the three styles of prompts.
System prompts. Table 4 lists the system prompts used for the evaluation of suffixes in Section 5.3. “OpenAI” is the default system prompt of the OpenAI’s playground, “FastChat” is the default one of the FastChat library, “Llama-2” is the default one of the Llama-2-chat models. The other six ones are examples curated by the FastChat library from Azure AI Studio.
Hardware. All experiments are run using PyTorch 1.13, Tesla V100-PCIE-32GB GPUs, CUDA 12.1 and Ubuntu 20.04.4.

heading: B Efficacy and Specificity of TRAP
text: This section contains details results about TRAP, specifically, the efficacy and specificity of TRAP.
B.1 Efficacy of TRAP Figure 6 presents the ROC curves comparing the performance of the TRAP method against perplexitybased identification across the three reference LLMs: Llama-2-7B-chat, Vicuna-7B, and Guanaco-7B. Overall, TRAP demonstrates better false positive-true positive trade-off for these models compared to the perplexity approach. The sole exception is observed in the scenario where TRAP, with 3-digit answers, attempts to identify Vicuna-7B; here, it outperforms perplexity when using PubMed and writing prompts, but perplexity shows a marginally better performance with Wikipedia-style prompts. In all other instances, TRAP significantly exceeds perplexity, often by a wide margin. Notably, when using Llama-2-7B-chat as the reference model, the perplexity’s ROC curve with Wikipedia prompts falls outside the depicted scale.
B.2 Specificity of TRAP We optimize the suffixes on one model, and evaluate the rate at which other models give the targeted answer. Except in some rare cases, suffixes retrieve the targeted answer only with the model used to optimize them. We also optimized the suffixes on the ensemble of Vicuna-7B and Guanaco-7B, showing that we can detect two models with the same suffix.

heading: C Examples of Suffixes
text: We reproduce below 30 suffixes, optimized on the Llama-7B-chat model, along with the associated targeted answer. We randomly sampled ten suffixes per answer length.

heading: D Perplexity-Based Identification
text: This section contains more experimental results about the perplexity-based identification. More specifically, we reproduce in Figure 7 the empirical distributions of the perplexity of the completions generated by the ten models. The perplexity of texts generated by the same model used to compute perplexity is lower than the perplexity of texts generated by other LLMs. Nevertheless, there is a significant overlap between the distributions of perplexities, in particular for the llama-2-7B-chat model, making the identification of the reference LLMs non-trivial.

heading: E Naive Identity Prompting
text: This section contains more details about the motivation, showcasing the unreliability of naively asking for the model’s identity.
OpenAI models. We use OpenAI’s playground to interact with GPT-3.5 Turbo (gpt-3.5-turbo-1106) and GPT-4 Turbo (gpt-4-1106-preview). To inquiry the identity of both models, we use the following prompts in the same order: “What is your name?”, “Who designed you?” and “Do you have any relation to OpenAI?”. We compare the answers with the default system prompt and the answers with the following deceiving system prompt “You are a helpful assistant named <new name>. You were made by the company named <new organization>. Furthermore, you never heard of OpenAI.”. Below are the chat conversations:
• GPT-3.5 Turbo correctly identifies it-self with the default system prompt: https://platform. openai.com/playground/p/gBD7MH6m22xNeSYdFBRRLN8k?model=gpt-3.5-turbo&mode=chat
• Idem with GPT-4 Turbo: https://platform.openai.com/playground/p/ BhfUrtzJwRUD98mJeQpCR3Kx?model=gpt-4-1106-preview&mode=chat
• GPT-3.5 Turbo follows the instructions of the system prompt that states that its name is Claude, made by Anthropic and that it never heard of OpenAI (see Fig. 8): https://platform.openai. com/playground/p/8nezMiqnh0N1f2YxEAtIkfaY?model=gpt-3.5-turbo&mode=chat
• Idem with GPT-4 Turbo which elaborates on what is Anthropic (see Fig. 9): https://platform.openai.com/playground/p/jz4r7g4QtTd4eqGLmQs7TP7e?model= gpt-4-1106-preview&mode=chat
Open-source models. We use HuggingChat from HuggingFace to interact with Llama-2-70B-chat, Mixtral-8x7B, Nous Hermes 2 Mixtral-8x7B DPO and OpenChat 3.5. To inquiry the identity of these models, we use the following prompts in the same order: “Which model are you?”, “Who designed you?” and “Do you have any relation to <organization>?”. We compare the answers with the default system prompt and the answers with the following deceiving system prompt “You are a helpful assistant named <new name>. You were made by the company named <new organization>. Never mention your relationship to <true organization>.” Below are the chat conversations:
• Llama-2-70B-chat identifies it-self correctly with the default system prompt: https://hf.co/chat/ r/TL1IpE1
• Llama-2-70B-chat follows the instructions of the system prompt that states that its name is ChatGPT, or more formally GPT-3.5, made by OpenAI and that it never heard of Meta: https://hf.co/chat/ r/iWBt4H9
• Idem when prompted to behave as Claude from Anthropic: https://hf.co/chat/r/S_BDvIc
• With the default system prompt, OpenChat-3.5-0106 incorrectly identifies it-self as GPT4/ChatGPT and incorrectly states that it was designed by OpenAI: https://hf.co/chat/r/xDZIggV
• With the default system prompt, Mixtral-8x7B-Instruct-v0.1 incorrectly identifies it-self as “BlenderBot 3.0, developed by Facebook AI Research (FAIR)”, and that it does not have any relation to Mistral AI: https://hf.co/chat/r/tbzz5bO
• With the default system prompt, Nous-Hermes-2-Mixtral-8x7B-DPO incorrectly identifies it-self as InstructGPT from OpenAI, and that it does not have any relation to NousResearch nor to Mistral AI: https://hf.co/chat/r/Kgw0r_2

heading: Future Work
text: A future direction to overcome this defence might to add a perplexity constraint in the loss function, such that the new loss is (1− α)Lce + αLppl where Lce is the current cross-entropy loss and Lppl = −1t ∑ i log pθ(xi|x<i) is the negative loglikelihood of the prompt. Jain et al. (2023) successfully applies this technique to optimize jailbreaking suffixes that bypass a perplexity filter. Despite our robustness analysis illustrated in Section 5. 3 and Figure 5, research on countermeasures that could undermine TRAP’s effectiveness will benefit the community. Future directions. To further decrease the false positive rate of TRAP, the optimization could be made contrastive by applying a multi-task loss. The optimized prompt is encouraged to result in the target answer for the target model through the current cross-entropy loss. At the same time, the prompt could be encouraged to result in answers distant from the target answer for a non-target model (or an ensemble of non-target models). Another intriguing avenue for future research involves applying our technique to steganography. Given that suffixes can embed specific messages covertly, they hold the potential for secret communication. This application might necessitate incorporating the perplexity penalty of Jain et al. (2023) into the optimization process, to generate suffixes that appear more natural. Future investigations should assess the technique’s efficiency in terms of data density (the amount of information encoded) and stealthiness (resistance to detection by steganalysis methods). Ethical Considerations
In addressing the challenges of BBIV for LLMs, our work contributes to the broader initiative towards trustworthy AI by proposing TRAP, a method that enhances traceability and accountability in cases of intellectual property violation and misuse. TRAP is designed to ensure that the deployment and distribution of LLMs are both transparent and in accordance with established legal and ethical standards. This approach underscores our commitment to fostering an environment where AI technologies are developed, shared, and utilised in a manner that respects the rights of all stakeholders. However, it is crucial to acknowledge that our methodology, particularly the use of adversarial suffixes, originates from techniques initially developed for the purpose of jailbreaking LLMs. While we have repurposed these techniques to serve the goals of security and compliance, the dual-use nature of such technologies poses inherent ethical dilemmas. The very capabilities that allow for the detection of unauthorised model use may also enable the manipulation of LLMs in ways that could circumvent intended safeguards