Who Wrote this Code? Watermarking for Code Generation (2024)

Abstract: Since the remarkable generation performance of large language models raised ethical and legal concerns, approaches to detect machinegenerated text by embedding watermarks are being developed. However, we discover that the existing works fail to function appropriately in code generation tasks due to the task’s nature of having low entropy. Extending a logit-modifying watermark method, we propose Selective WatErmarking via Entropy Thresholding (SWEET), which enhances detection ability and mitigates code quality degeneration by removing low-entropy segments at generating and detecting watermarks. Our experiments show that SWEET significantly improves code quality preservation while outperforming all baselines, including post-hoc detection methods, in detecting machine-generated code text. Our code is available in https:// github.com/hongcheki/sweet-watermark.

heading: 1 Introduction
text: In understanding and generating software programs, large language models have rapidly advanced towards expert-like proficiency (Chen et al., 2021; Luo et al., 2023; Li et al., 2023b; Nijkamp et al., 2023; Zheng et al., 2023; Gunasekar et al., 2023; Touvron et al., 2023; OpenAI, 2023a). This breakthrough in the automation of the coding process improves the productivity and efficiency of software engineer and lowers the barriers to creating programs for non-experts (Vaithilingam et al., 2022).
However, this advance comes with significant legal, ethical, and security concerns, including code licensing issues, code plagiarism, code vulnerability, and malware generation (He and Vechev, 2023;
⋆ Authors equally contributed. ♯ Work done during Ilgee’s internship at SNU VL Lab. ♮ Jamin Shin is currently affiliated with Trillion Labs, but
the work was done while he was at NAVER. † Corresponding authors.
Sandoval et al., 2023; Pearce et al., 2022; Carlini et al., 2021; Mirsky et al., 2023; Hazell, 2023). For example, there is an ongoing class-action copyright lawsuit between a group of individuals and Microsoft, GitHub, and OpenAI, arising from allegations of unlawful utilization and reproduction of the source code12. Furthermore, shortly after the launch of ChatGPT, numerous malicious actors on the Dark Web were observed sharing machinegenerated malware and spear phishing tutorials3. Therefore, the development of reliable tools for
1Code plagiarism 2Code licensing issue 3Malware generation
4890
detecting machine-generated code is a very timely matter and is of utmost importance for fairly deploying LLMs with coding capabilities.
Despite the need for immediate treatment of the machine-generated code detection problem, few efforts have been made to address it. Instead, many works still prioritize a detection problem on normal text (Solaiman et al., 2019; Ippolito et al., 2020; Guo et al., 2023; Tian and Cui, 2023; OpenAI, 2023b; Yu et al., 2023; Gehrmann et al., 2019; Mitchell et al., 2023; Yang et al., 2023). While these post-hoc detection methods (i.e., no control during the text generation) have demonstrated powerful performance in the many domain of natural language tasks, their application to programming language remains unexplored.
Contrary to the post-hoc detection methods, another line of research for detecting machinegenerated text has gained attention: Watermarkingbased methods, which embed a hidden signal within the generated text (Kirchenbauer et al., 2023a,b; Kuditipudi et al., 2023; Wang et al., 2023). For example, a method proposed in Kirchenbauer et al. (2023a) – which we refer to as WLLM (Watermarking for Large Language Models) – randomly divides the entire vocabulary into two groups (i.e., the green list and the red list) at each generation step and enhance the probability of green list tokens to be sampled. By adding scalar values to the logits of a green list tokens, the model favors generating tokens from the green list rather than the red one. To detect the watermark in a text, we count the number of green tokens and check whether this number is statistically significant (through hypothesis testing) to conclude whether the model output is generated without knowledge of the green-red rule.
While both watermarking-based methods and post-hoc detection methods work well in many language generation tasks, we observe that these performances do not transfer well to code generation tasks, for example, in Figure 1. In other words, it is much more challenging to embed watermarks in a detectable way without impairing the code functionality. We attribute this to the nature of extremely low entropy4 of code generation. If watermarking is applied strongly, it can severely degrade the quality of the model output, which is particularly critical in code generation, as a single
4We calculate entropy over the probability of the next token prediction. Please refer to Eq. 5 for details.
violation of a rule can break the entire code (see “strong watermark” in Figure 1). On the other hand, if watermarking is applied too weakly, the low entropy hinders properly embedding watermarks and results in insufficient green tokens appearing, leading to increased difficulty in detection (see “weak watermark” in Figure 1). These failures are not significant in plain text generation because the relatively higher entropy allows for more flexibility in candidate selections for watermarking.
To address these failure modes, we extend the WLLM and propose Selective WatErmarking via Entropy Thresholding (SWEET) for Code LLMs (and LLMs). Instead of applying the green-red rule to every single token during generation, we only apply the rule to tokens with high enough entropy given a threshold. That is, we do not apply the green-red rule to the important tokens for making functional code, while making sure there are enough green list tokens to make a detectable watermark for less important tokens, hence, directly addressing each of the above failure modes. In code generation tasks, our method outperforms all baselines, including post-hoc detection methods, in detecting machine-generated code while achieving less code quality degradation than WLLM. Furthermore, through various analyses, we demonstrate that our method operates well even without prompts or with a small surrogate model, indicating its robust performance under practical settings.
Our contributions are as follows:
• We are the first to empirically explore the breakdown of existing watermarking and posthoc detection methods in the code domain.
• We propose a simple yet effective method called SWEET, which improves WLLM (Kirchenbauer et al., 2023a) and achieves significantly higher performance in machinegenerated code detection while preserving code quality more than WLLM.
• We have demonstrated the practical applicability and predominance of our method even in real-world settings, i.e., 1) without prompts, 2) utilizing a smaller model as a detector, or 3) under paraphrasing attacks.

heading: 2 Related Work
text: Software Watermarking Software watermarking is the research field where a secret signal is embedded in the code without affecting its perfor-
mance, to prevent software piracy. Static watermarking (Hamilton and Danicic, 2011; Li and Liu, 2010; Myles et al., 2005) imprints watermarks typically through code replacement and reordering. On the other hands, dynamic watermarking (Wang et al., 2018; Ma et al., 2019) injects watermarks during the compiling or executing stage of a program. For a detailed survey, please refer to Dey et al. (2018).
Watermarking code text generated from a LLM is closer to static watermarking. For example, Li et al. (2023c) proposes a method employing the replacement of synonymous code. However, since this method heavily relies on language-specific rules, a malicious user knowing these rules could reverse the watermarking.
LLM Text Watermarking The majority of watermarking methods for texts from LLMs are based on the modification of the original text via a predefined set of rules (Atallah et al., 2001, 2002; Kim et al., 2003; Topkara et al., 2006; Jalil and Mirza, 2009; Meral et al., 2009; He et al., 2022a,b) or another language model, such as transformer-based networks. (Abdelnabi and Fritz, 2021; Yang et al., 2022; Yoo et al., 2023).
Recently, a line of work embeds watermarks into tokens during the sampling process of LLMs (Liu et al., 2024). They embed watermarks within LLMgenerated texts by either motifying logits from the LLM (Kirchenbauer et al., 2023a,b; Liu et al., 2023a; Takezawa et al., 2023; Hu et al., 2023) or manipulating the sampling procedure (Christ et al., 2023; Kuditipudi et al., 2023). Moreover, some recent works focus on the robustness of watermarks against attacks to remove watermarks (Zhao et al., 2023; Liu et al., 2023b; Ren et al., 2023). Lastly, Gu et al. (2023) investigates the learnability of watermarks in the distillation process from teacher to student model.
However, these watermark methods exhibit vulnerability in their watermark detection performance under low entropy situations (Kirchenbauer et al., 2023a; Kuditipudi et al., 2023), and a limited number of studies, such as CTWL (Wang et al., 2023), try to handle it. We directly address the degradation of watermark detection performance in low entropy situations and demonstrate our method’s efficacy in low entropy tasks, such as code generation.
Post-hoc Detection Post-hoc detection methods aim to differentiate between human-authored and machine-generated text without embedding any signal during generation. One line of work lever-
ages perplexity-based features like GPTZero (Tian and Cui, 2023), Sniffer (Li et al., 2023a), and LLMDet (Wu et al., 2023). Another line of work uses pre-trained LM, such as RoBERTa (Liu et al., 2019), and fine-tunes it as a classifier to identify the source of text (Solaiman et al., 2019; Ippolito et al., 2020; OpenAI, 2023b; Guo et al., 2023; Yu et al., 2023; Mitrović et al., 2023). Meanwhile, some recent works tackle the detection problem without additional training procedures, such as GLTR (Gehrmann et al., 2019), DetectGPT (Mitchell et al., 2023), and DNA-GPT (Yang et al., 2023). However, post-hoc detection methods remain challenging. For example, while the GPTZero (Tian and Cui, 2023) is still in service, OpenAI’s AI text classifier (OpenAI, 2023b) was discontinued after six months due to low accuracy rates. Furthermore, we have demonstrated that post-hoc detection methods failed to detect machine-generated code, with low entropy.

heading: 3 Method
text: We propose a new watermarking method, SWEET, that selectively watermarks tokens only with high enough entropy.

heading: 3.1 Motivation
text: Although the previous watermarking method WLLM (Kirchenbauer et al., 2023a) can be applied to any domain of LLM-generated text5, it incurs two critical problems during embedding and detecting watermarks in code generation, attributed to a dilemma regarding watermark strength.
Watermarking causes performance degradation. There are only a few different ways of expressing the same meaning in a programming language, and just one wrong token can be attributed to undesirable outputs. If watermarks are embedded strongly, as WLLM randomly divides the vocabulary into green and red lists without leveraging any information about the context, promoting the logits of only green list tokens must heighten the chance of generating the wrong token. For example, in Figure 2 (a), after “return” token in the second row, the next token with the highest logit is “sum”, which is also part of the canonical solution. However, WLLM puts “sum” into the red list while putting “mean” into the green list. Hence, the sampled token was “mean”, resulting in a syntax error.
5Please refer to Appendix A for the details of WLLM.
Low Entropy Sequences Avoid Being Watermarked. Another critical issue is when watermark strength is too weak to embed watermarks into a text with low entropy. If a red list token has a too high logit value to be inevitably generated, it hinders watermark detection. For example, in Figure 2 (a), tokens with white backgrounds representing low entropy have few green tokens. This becomes much more fatal in code generation tasks where outcomes are relatively shorter than the plain text, such as asking only a code block of a function6. The WLLM detection method is based on a statistical test, which involves counting the number of green list tokens in the entire length. However detecting watermarks based on a statistical test deteriorates if the length is short.7

heading: 3.2 The SWEET Method
text: SWEET can mitigate this dilemma regarding the watermark strength by distinguishing watermarkapplicable tokens, meaning we embed and detect watermarks only within tokens with high entropy.
Generation. The generation step of our method is in Algorithm 1. Given a tokenized prompt
6The average token length of human-written solution codes in HumanEval, MBPP, and DS-1000 datasets is only 57.
7We measured detectability according to the length of generated texts and observed that WLLM performs relatively poorly while SWEET is robust in detecting watermarks within short texts. For more details, please refer to Appendix G.
x = {x0, . . . , xM−1} and already generated tokens y[:t] = {y0, . . . , yt−1}, a model calculates an entropy value (Ht) of the probability distribution for yt. We then only apply the watermarking when Ht is higher than the threshold, τ . We randomly bin a vocabulary by green and red with a fixed green token ratio γ. If a token is selected to be watermarked, we add a constant δ to green tokens’ logits, aiming to promote the sampling of the green tokens. By limiting the promotion of green tokens only to tokens with high entropy, we prevent the model’s logit distribution changes for tokens where the model has confidence (and, therefore, low entropy), resulting in preserving code quality.
Detection. We outline our detection process in Algorithm 2. Given a token sequence y = {y0, . . . , yN−1}, our task is to detect watermarks within y; therefore, determine whether it is generated from the specific language model. Like in the generation phase, we compute the entropy values Ht for each yt. Let Nh denote the number of tokens that have an entropy value Ht higher than the threshold τ , and let NhG denote the number of green tokens among in Nh. Finally, with the green list ratio among entire vocabulary γ used in the generation step, we compute a z-score under the null hypothesis where the text is not watermarked by
z = NhG − γNh√ Nhγ(1− γ)
(1)
We can say the text is watermarked more confidently as z-score goes higher. We set zthreshold as a cut-off score. If z > zthreshold holds, we decide that the watermark is embedded in y and thus generated by the LLM. The effect of the entropy threshold in the detection phase is described in the following section.

heading: 3.3 Effect of Entropy Thresholding
text: This section shows that selective watermark detection based on the entropy threshold improves the detectability.
Theorem 1 implies that we can ensure a higher lower bound of z-score by the SWEET detection method than WLLM. Recalling Sec 3.1, this is achieved by ignoring tokens with low entropy, leading to increases in the ratio of green tokens within the text and detectability.
For the sake of theoretical analysis, we use spike entropy (Eq. 4), which is a variant of entropy defined in Kirchenbauer et al. (2023a). In practice, we use the entropy in Eq. 5.
Theorem 1. Consider a token sequence y = {y0, . . . , yN−1} generated by a watermarked code LLM. (S0, . . . , SN−1) is a sequence of corresponding spike entropy, in which the modulus is (1−γ)(eδ−1) 1+(eδ−1)γ . Let τ be an entropy threshold, N l and

heading: Nh be the number of tokens whose spike entropy is lower or higher than the threshold.
text: If the following assumption regarding the ratio of low entropy tokens holds
N l N ≤ 1− ( αS − 1 αSh − 1 )2
then there is a lower bound of z-score that is always higher when the entropy threshold is applied, where α = e δ
1+(eδ−1)γ , S = Σ N t=1St/N , and
Sh = ΣNt=1St × 1(St ≥ τ)/Nh. Remark. The assumption means choosing an entropy threshold that does not ignore too many tokens (N l) is important.

heading: 4 Experiments
text: We conduct a series of experiments to evaluate the effectiveness of our watermarking method in code generation for two aspects: (i) quality preserving ability and (ii) detection strength. Our base model is StarCoder (Li et al., 2023b), which is an opensource LLM specifically for code generation. We
also conduct experiments on one of the generalpurpose LLM, LLaMA2 (Touvron et al., 2023) (see the results in Appendix F).

heading: 4.1 Tasks and Metrics
text: We select three Python code generation tasks, HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and DS-1000 (Lai et al., 2023), as our main testbeds. These tasks contain Python programming problems, test cases, and human-written canonical answers. Language models are prompted with programming problems and expected to generate the correct code that can pass the test cases. To evaluate our approach’s performance in more diverse software development contexts, such as other languages or other code generation scopes, we also include two more datasets: HumanEvalPack (Muennighoff et al., 2024) and ClassEval (Du et al., 2023). Please refer to Appendix E for implementation details of these benchmarks.
To evaluate the functional quality of generated source code, we use pass@k (Chen et al., 2021) by generating n(> k) outputs for each programming problems. This metric estimates the percentage of code generated correctly-performing. For the detection ability, we use AUROC (i.e., Area Under ROC) value as a main metric. We also report the true positive rate (TPR; correctly detecting LLMgenerated code as LLM-generated) when the false positive rate (FPR; falsely detecting human-written code as LLM-generated) is confined to be lower than 5%. This is to observe the detection ratio of a practical setting, where high false positive is more undesirable than false negative.

heading: 4.2 Baselines
text: We compare SWEET with machine-generated text detection baselines. Post-hoc detection baselines do not need any modification during generation so that they never impair the quality of the model output. LOGP(X), LOGRANK (Gehrmann et al., 2019), and DETECTGPT (Mitchell et al., 2023) are zero-shot detection methods that need no labeled datasets. GPTZERO (Tian and Cui, 2023) and OPENAI CLASSIFIER (Solaiman et al., 2019) are trained classifiers. For Watermarkingbased methods, we have included two baselines: WLLM (Kirchenbauer et al., 2023a) and EXP-EDIT (Kuditipudi et al., 2023). To embed a watermark, methods that distort the model’s sampling distribution, such as WLLM or ours, tend to have better detection ability, but degradation of text
quality may arise. On the other hand, EXP-EDIT is expected to cause no degradation in text quality as they do not distort the sampling distribution of the model. 8 More details of implementation are in Appendix D.

heading: 5 Results

heading: 5.1 Main Results
text: Table 1 presents results from all baselines and our approach. In WLLM and SWEET, there is a clear trade-off between detection and code generation ability depending on the watermarking strength. Therefore, we measure the maximum scores of one domain while setting a lower bound for the scores of other domain. Specifically, to measure AUROC scores, we find the best AUROC scores around 90% of the pass@1 performance of the nonwatermarked base model. On the other hand, for measuring pass@1, we select from those with an AUROC of 0.9 or higher.
Detection Performance. Table 1 shows that overall, our SWEET method outperforms all baselines in detecting machine-generated code with a price of 10% degradation of code functionality. Both in the MBPP and DS-1000 datasets, SWEET achieves AUROC of 0.873 and 0.815,
8When evaluating code generation performance through pass@1, a low temperature was applied to all models. However, the spiky distribution resulting from the low temperature hindered EXP-EDIT from adequately embedding watermarking. Therefore, we have also included EXP-EDIT baseline with a high entropy by setting temperature=1.0 and top-p=1.0.
respectively, whereas none of the baselines exceeded 0.8. SWEET even achieves an AUROC of 0.943 in HumanEval with a 2.4% degradation of code functionality. However, when only near 10% degradation of code functionality is allowed, WLLM shows lower detection performance than our method. In the case of the distortion-free watermarking method, due to the lower entropy of the code generation task, EXP-EDIT fails to achieve an AUROC score exceeding 0.6 in all cases, and even EXP-EDIT with high entropy setting could not outperform our methods with regard of the detection performance. While all post-hoc detection baselines preserve code functionality as they do not modify generated code, none of them achieve an AUROC score above 0.6.9
Code Quality Preservation. In the last two rows of Table 1, despite the inevitable text quality degradation caused by WLLM and SWEET, our SWEET method preserves code functionality much more while maintaining the high detection ability of AUROC > 0.9 when compared to WLLM. Specifically, pass@1 of WLLM for HumanEval decreases from 33.4 to 25.3, a 24.3% loss in the code execution pass rate. Similarly, for the MBPP and the DS-1000 dataset, the drops in performances are 36.0% and 67.3%, respectively. On the other hand, our approach loses only 2.4% (HumanEval), 12.2% (MBPP), and 28.5% (DS-1000),
9We defer a more in-depth discussion about the breakdown of Post-hoc methods to Appendix K.
respectively, which are significantly less than those of WLLM.
C++/Java/Class-level Code Generation. Table 2 presents results on other programming languages (C++ and Java) and another code generation scope (i.e., class-level). While preserving code functionality much more than WLLM, SWEET shows the highest detection performance except in the Java environment, where the TPR score of WLLM is higher than that of SWEET. The results demonstrate that the efficacy of our methodology is not limited to certain types of programming languages or software development en-
vironments. For more analysis of the results, please refer to Appendix E.

heading: 5.2 Comparison of Pareto Frontiers between SWEET and WLLM
text: In the cases of SWEET and WLLM, watermarking strength and spans can vary depending on the ratio of the green list tokens γ and the logit increase value δ. To demonstrate that SWEET consistently outperforms the baseline WLLM regardless of the values of γ and δ, we draw Pareto frontier curves with axes pass@1 and AUROC in Figure 3. We observe that the Pareto frontiers of SWEET are ahead
of those of WLLM in all three tasks. Moreover, as presented in Figure 6, whatever value our approach chooses for the entropy threshold, SWEET outperforms the baseline in all configurations. This indicates that in a wide range of hyperparameter settings, our SWEET model can generate better results in terms of detection and code generation ability. Full results and different settings are in Appendix F.

heading: 6 Analysis

heading: 6.1 Impact of Entropy Thresholds
text: Figure 4 presents how code generation performance and detecting ability trade-off when calibrating the entropy threshold in our method. WLLM is when the entropy threshold is not applied (i.e., entropy threshold=0). As the entropy threshold increases, the ratio of watermarked tokens decreases, so the code generation performance converges to a nonwatermarked base model. This indicates that our
method always lies between the WLLM and a nonwatermarked base model in terms of code generation performance. On the other hand, the detection ability, as the entropy threshold increases, reaches a local maximum but eventually declines. While our method with a moderate threshold effectively restricts generating the red list tokens compared to the WLLM, detection ability eventually decreases if the threshold is so high that few tokens are watermarked. We further investigate how to effectively calibrate the entropy threshold value in Appendix H.

heading: 6.2 Detection Ability without Prompts
text: As entropy information is required in the detection phase, approximating entropy values for each generation time step t is essential in our method. In the main experiments, we prepend the prompt used in the generation phase (e.g., the question of Fig. 2) before the target code to reproduce the same entropy. However, we hardly know the prompt used for a given target code in the real world. Thus, instead of using the gold prompt, we attach a common and general prompt for code generation to approximate the entropy information. We use five general prompts as below, and their z-scores are averaged for use in detection.
def solution(*args): """ Generate a solution """
<filename>solutions/solution_1.py # Here is the correct implementation of the code
exercise def solution(*args):
def function(*args, **kargs): """ Generate a code given the condition """
from typing import List
def my_solution(*args, **kargs): """ Generate a solution """
def foo(*args): """ Solution that solves a problem """
Figure 8 demonstrates how the detection ability varies when using general prompts in the Hu-
manEval dataset. SWEET with general prompts shows lower AUROC values than the original SWEET, indicating inaccurately approximated entropy information impairs detection ability. Nevertheless, it still outperforms the WLLM baseline regarding detection ability, drawing a Pareto frontier ahead of WLLM in all entropy threshold values.

heading: 6.3 Use of Surrogate Model
text: When detecting watermarks in a text, utilizing a smaller LM as a surrogate could be more computationally efficient and cost-effective (Wang et al., 2023). We investigate the impact of employing this surrogate model during the detection phase. Specifically, we generate watermarked code using the original model (LLaMA2-13B) and detect watermarks using a smaller model (LLaMA2-7B).
In the results of Figure 10, the detection performance declines are insignificant, and our approach utilizing the surrogate model continues to surpass the baseline. Such performance preservation may be due to that LLaMA2 7B and 13B are trained on the identical training corpus (Touvron et al., 2023). Further analysis for computational cost can be found in Appendix I.

heading: 6.4 Robustness to Paraphrasing Attacks
text: Even with the text watermarked, a malicious user might attempt to remove watermarks in the text by paraphrasing (Krishna et al., 2023; Sadasivan et al., 2023). Paraphrasing the code text is more restrictive than dealing with plain text because it must avoid triggering any code malfunctions. We assess
the robustness of watermarking methods against paraphrasing by employing two types of attacks - changing the names of variables and utilizing a commercial code refactoring service.10 Specifically, for each watermark method, we choose 273 source codes from the MBPP task, for which all three methods succeed in generating with no syntax error. In the code renaming attack, we select variables in the watermarked code and rename them with randomly generated strings of varying lengths, ranging from 2 to 5 characters. We use five random seeds for renaming.
Figure 5 presents the results of the detection performance on the paraphrased code. All watermarking methods show the decline of AUROC scores when the extent of paraphrasing increases, while our approaches continue to show better performances than baselines. However, our approaches also show that the AUROC scores drop to about 0.8 when all variables are renamed. We found that this is because variable names comprise a large proportion of high entropy tokens in the code text (See Appendix J for details).

heading: 7 Conclusion
text: We identified and emphasized the need for Code LLM watermarking, and formalized it for the first time. Despite the rapid advance of coding capability of LLMs, the necessary measures to encourage the safe usage of code generation models have not been implemented yet. Our experiments showed that existing watermarking and detection techniques failed to properly operate under the code generation setting. The failure occurred in two modes: either 1) the code does not watermark properly (hence, cannot be detected), or 2) the watermarked code failed to properly execute (degradation of quality). Our proposed method SWEET, on the other hand, improved both of these failure modes to a certain extent by introducing selective entropy thresholding which filters tokens that are least relevant to execution quality. In code generation tasks, our method performs better than baselines, including post-hoc detection methods, while achieving less code quality degradation. Moreover, comprehensive analysis demonstrates that our method still works well in real-world settings, specifically when the prompts are not given, utilizing even a smaller surrogate model, or under paraphrasing attacks.
10https://codepal.ai/code-refactor
Limitations
We identify the limitations of this work and suggest ways to mitigate them. First, two issues are shared by the status quo of this field as follows. (1) Robustness against paraphrasing attacks: As users can tailor LLM’s code to their specific needs, it is crucial to be robust against paraphrasing attacks. We addressed this issue in Section 6.4 and left further robustness enhancement for future work. (2) Possibilities of watermark forgery: An attacker may pry out the watermarking rules, and O(|V|)2 runs in the brute-force mechanism enable it. Against the attack, one can apply techniques enhancing the watermarking model’s security, such as dividing the green/red list depending on prior h > 1 tokens, as mentioned in the WLLM paper, or applying methods like SelfHash (Kirchenbauer et al., 2023b).
For our work, two additional issues exist as follows. (1) Entropy threshold calibration: We demonstrate that our method outperforms the baselines in the broad entropy threshold range (see Sec 6.1) and investigate how to calibrate the entropy threshold effectively (see Appendix H). However, we still need entropy threshold tuning to obtain the best performance, which costs a computation. (2) Need for the source LLM during detection: SWEET works in a white-box setting. Although it has been shown that employing even a smaller surrogate LM can still maintain the detection performances to some degree (see Sec 6.3), this can be a computational burden for some users who want to apply our work.
Ethical Statement
Although watermarking methods are designed to address all potential misuse of LLMs by detecting machine-generated texts, they can simultaneously pose a new risk. For example, if a watermarking mechanism for a specific LLM is leaked to the public, a malicious user aware of this mechanism could abuse the watermarks to create unethical texts embedded with the model’s watermarks. To prevent such scenarios, we recommend that all users exercise caution to avoid exposing the detailed mechanism, such as the key value for the hash function used to divide green and red lists in our method.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their valuable feedback. This work was supported by SNU-NAVER Hyperscale AI
Center, Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2019-II191082, SW StarLab), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2023R1A2C2005573), and the Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(RS-2023-00274280).

heading: A Preliminaries for WLLM
text: In this section, we provide brief preliminaries for Kirchenbauer et al. (2023a). For a given language model fLM with vocabulary V , the likelihood probability of a token yt is calculated as follow:
lt = fLM(x,y[:t]), (2)
pt,i = el
i t
∑|V| i=1 e lit , (3)
where x = {x0, . . . , xM−1} and y[:t] = {y1, . . . , yt−1} are a M -length tokenized prompt and the generated token sequence, respectively, and lt ∈ R|V| is the logit vector. Watermarking in LM-generated Text. In the watermarking (Kirchenbauer et al., 2023a), the entire tokens in V at each time-step are randomly binned into the green Gt and red groupsRt in proportions of γ and 1− γ (γ ∈ (0, 1)), respectively. The method increases the logits of green group tokens by adding a fixed scalar δ, promoting them to be sampled at each position. Thus, watermarked LM-generated text is more likely than γ to contain the green group tokens. On the other hand, since humans have no knowledge of the hidden greenred rule, the proportion of green group tokens in human-written text is expected to be close to γ.
The watermarked text is detected through a onesided z-test by testing the null hypothesis where the text is not watermarked. The z-score is calculated using the number of recognized green tokens in the text. Then, the testing text is considered as watermarked if the z-score is greater than zthreshold. Note that the detection algorithm with the higher zthreshold can result the lower false positive rate (FPR) and reduce Type I errors.
Spike Entropy Kirchenbauer et al. (2023a) used spike entropy for measuring how spread out a distribution is. Given a token probability vector p and a scalar m, spike entropy of p with modulus m is defined as:
S(p,m) = ∑ pk
1 +mpk . (4)

heading: B Watermark Embedding/Detecting Algorithm of SWEET
text: Algorithms 1 and 2 show the detailed steps of generating a watermark and later detecting it using our selective entropy thresholding method (SWEET).
Algorithm 1 Generation Algorithm of SWEET 1: Input: tokenized prompt x = {x1, . . . , xM−1}; entropy threshold τ ∈ [0, log |V|], γ ∈ (0, 1), δ > 0;
2: for t = 0, 1, 2, . . . do 3: Compute a logit vector lt by (2); 4: Compute a probability vector pt by (3); 5: Compute an entropy Ht by (5); 6: if Ht > τ then 7: Compute a hash of token yt−1, and use it as a seed for a random number generator; 8: Randomly divide V into Gt of size γ|V| andRt of size (1− γ)|V|; 9: Add δ to the logits of tokens in Gt;
10: end if 11: Sample yt; 12: end for
Algorithm 2 Detection Algorithm of SWEET 1: Input: tokenized prompt x; token sequence to
be tested y = {y0, . . . , yN−1}; entropy threshold τ ∈ [0, log |V|], γ ∈ (0, 1), zthreshold > 0; 2: Set Nh = 0 and NhG = 0; 3: for t = 0, 1, 2, . . . N − 1 do 4: Compute a logit vector lt by (2); 5: Compute a probability vector pt by (3); 6: Compute an entropy Ht by (5); 7: if Ht > τ then 8: Nh ← Nh + 1; 9: Compute a hash of token yt−1, and use it
as a seed for a random number generator; 10: Recover Gt andRt; 11: if yt ∈ Gt then 12: NhG ← NhG + 1; 13: end if 14: end if 15: end for 16: Compute z-score by (1); 17: if z > zthreshold then 18: return True; (i.e., y is watermarked) 19: else 20: return False; 21: end if
Instead of the spike entropy used in WLLM, we use the classical Shannon entropy. Given a token probability distibution vector p, the entropy of p is computed by
Ht = − ∑ pk log pk. (5)

heading: C Proof of Theorem 1
text: We begin with a lemma from Kirchenbauer et al. (2023a), which predicts the probability of a green list token sampled from a language model employing the watermarking.
In our proof, we predict the lower bounds of zscore when detecting watermarks via WLLM or SWEET methods and compare the z-score lower bounds. Lemma C.1. Suppose p ∈ (0, 1)|V| is a raw probability vector generated from a language model where |V| is the vocabulary size. Before sampling p, watermarks are embedded by dividing randomly a green list of size γ|V| and a red list of size (1 − γ)|V| for some value γ ∈ (0, 1). It then promotes the logits of tokens in the green list by δ. When sampling a token index k from this watermarked distribution, the probability that the token is sampled from the green list (considering the randomness of green list) is at least
P[k ∈ G] ≥ γe δ 1 + (eδ − 1)γS(p, (1− γ)(eδ − 1) 1 + (eδ − 1)γ ).
Let’s begin the proof.
Proof. In WLLM, we consider all tokens in y = {y0, . . . , yN−1} for detection. We can get a lower bound of the number of green list tokens in y by summing the result of Lemma C.1 over the tokens yt. The expectation of the number of green list tokens, Ng, in y is at least
E[NG] ≥ αγNS. (6)
where α = e δ 1+(eδ−1)γ , and S = ∑N
t=1 St/N . We can get the lower bound of the z-score by
applying the z-score definition in Eq. 1:
z ≥ γ √ N αS − 1√ γ(1− γ) . (7)
If the entropy threshold is applied, we consider only tokens with entropy values higher than the threshold to be tested. Let Nh be the number of tokens that have higher entropy values. Following Eq. 6 and Eq. 7 again with Nh, we can get the lower bound of the z-score of SWEET:
z ≥ γ √ Nh αSh − 1√ γ(1− γ) ,
where Sh = ∑N
t=1 St × 1(St ≥ τ)/Nh.
Sh ≥ S is ensured as we ignore all tokens with lower entropy than the threshold. By comparing Eq. 7 and Eq. 8,
γ √ Nh αSh − 1√ γ(1− γ) ≥ γ √ N αS − 1√ γ(1− γ) ,
√ N −N l
N ≥ αS − 1 αSh − 1 ,
N l N ≤ 1− ( αS − 1 αSh − 1 )2,
where N l = N −Nh.
D Implementation Details
We have used three datasets for our testbeds: HumanEval, MBPP, and DS-1000. They have 164, 500, and 1000 Python code problems, respectively. For our base models, StarCoder and LLaMA2, we use top-p (Holtzman et al., 2020) sampling with p = 0.95 for both models, and temperature 0.2 and 0.1, respectively. When generating output for each code problems, we use zero-shot setting in HumanEval and DS-1000 but 3-shot in MBPP. Prompts used in MBPP are similar to the prompt in Austin et al. (2021). For calculating pass@1 scores, we set n = 40 for HumanEval and DS1000, and n = 20 for MBPP. We use a single NVIDIA RTX A6000 GPU to generate or detect each code completion with StarCoder or LLaMA2. It takes less than two GPU hours for generation and less than 1 GPU hour for detection.
D.1 DetectGPT
We used two masking models for DetectGPT. When T5-3B is used for DetectGPT, we search hyperparameters for the length of the spans in [1,2,5,10] words, and for the proportion of masks in [5,10,15,20]% of the text. When utilizing SantaCoder, we simulate the single-line fill-in-themiddle task scenario by masking only one line of code per perturbation, which is a task that SantaCoder is trained to perform well. (Fried et al., 2023; Bavarian et al., 2022). We search hyperparameters for the number line to be rephrased in [1,2,3,4]. We make 100 perturbations following the original paper.
D.2 WLLM and SWEET
Depending on the strength of watermark, tradeoff between code functionality and watermarking detectability exists. We search hyperparameters for the ratio of the green list γ in [0.1,0.25,0.5], and for the green token promotion value δ in [0.5,1.0,2.0,3.0,4.0]. For the entropy threshold values used in SWEET, we search thresholds in [0.3,0.6,0.9,1.2].
D.3 EXP-EDIT
In most tasks we have conducted experiments, the length of the generated code hardly exceed 100 tokens. Therefore, considering that length of the watermark key sequence significantly affected the detection speed, we search hyperparameters for the length of the key sequence only in [100, 500]. The block size was set equal to the length of the model output, and the resample size T = 500 for all instances. To generate n outputs to calculate pass@k, we shift the watermark key sequence randomly n times. Finally, we set edit distance hyperparameter γ = 0.0 for EXP-EDIT as used in their paper.

heading: E Experimental Details and Results on HumanEvalPack and ClassEval
text: We choose two additional benchmarks to present how well our approach functions in broader software development contexts.
E.1 HumanEvalPack
HumanEvalPack (Muennighoff et al., 2024) is an extension of HumanEval to cover 6 languages (Python, C++, Java, Javascript, Go, and Rust), and we choose C++ and Java as our testbeds. Basically, all hyperparameter settings are equal to
Python benchmarks except n = 5. For WLLM and SWEET, we narrow the search space for δ into [1.0,2.0,3.0,4.0]. For EXP-EDIT, we fix the length of the key sequence to 100.
E.2 ClassEval ClassEval (Du et al., 2023) differs from the aforementioned datasets in terms of code generation scopes as it requires language models to generate class-level code passages rather than just a single function. It consists of 100 class-level code generation test examples in which a model has to generate a whole Python class code given a skeleton code of the class. Following the ClassEval paper, we use StarCoder-Instruct (GeorgiaTechResearchInstitute, 2023) as the base model and the same instructionfollowing template of the prompt used in the paper. As the entropy distribution from StarCoder-Instruct skews to lower values than that of the StarCoder model, we search hyperparameters for threshold in [0.01,0.03,0.05,0.1,0.2] and δ in [2,3,4,5,10,15,20]. We generate n = 5 outputs to calculate pass@5. For EXP-EDIT, due to the high computational cost, we make 50 perturbations instead of 100.
E.3 Results As presented in Table 2, SWEET outperforms all baselines in detecting machine-generated code in C++ and Java environments. Also, it still preserves code functionality much more than WLLM while achieving better detection performance. We observe that the Pareto Frontier lines of SWEET are ahead of those of WLLM, as in the Python environment (see Figure 7). It is worth noting that the C++ and Java examples in HumenEvalPack comprise longer code than the Python examples we used in the paper: an average of 100 tokens for C++, 97 tokens for Java, and 57 tokens for Python. Therefore, these results demonstrate that the efficacy of our methodology is not limited to the type of programming languages or the length of the code.
In the class-level code generation task, we could still observe our approach showing the highest AUROC score than the baselines even in the class-level code generation task where the LLM should generate longer and more complex code (the average token length of ClassEval solutions is 352). Specifically, SWEET achieves an AUROC of 0.980 with a 7.1% degradation of code functionality. Interestingly, as the text becomes longer, post-hoc methods’ performance increases. On the other hand, EXP-EDIT has shown lower detection performance
than when evaluated in StarCoder, even in the high-entropy setting, due to the extremely spiky entropy distribution of StarCoder-Instruct. In addition, EXP-EDIT increases code functionality compared to non-watermarked baselines, even though it is a watermarking method that does not distort the original token distribution. We suppose these results are attributed to the specific watermark key sequence, which is randomly generated.

heading: F Further Pareto Frontier Results on StarCoder/LLaMA2
text: HumanEval pass@100. Figure 9 shows a tradeoff between pass@100 score and AUROC at HumanEval task in temperature 0.8. We generated 200 samples in HumanEval to calculate pass@100. The tendency of the Pareto Frontier are the same, SWEET is consistently placed in the front. While pass@100 score is much higher than the pass@1 score at temperature=0.2, we see the range of AUROC remains similar. This indicates temperature does not affect the detection strength of each samples heavily.
LLaMA2. Furthermore, Table 3 shows the results on HumanEval when using LLaMA2 13B (a general-purpose LLM), as the backbone for code generation. We can observe similar trends as demonstrated in Figure 10. SWEET in LLaMA2 achieves a higher AUROC than all other baselines while preserving code quality more than WLLM. Consequently, we observe that SWEET also applies to general-purpose LLM, which is not codespecific.

heading: G Detectability with Varying Code Lengths
text: We experiment the detection performance across different code lengths. Based on the detectability@T metric proposed in Kirchenbauer et al. (2023b), we evaluate the detection performance within the first T tokens of the machine-generated and human-written code sequences and calculate AUROC scores.
As presented in Figure 11, SWEET demonstrates superior detection performance even in the short code texts. This is particularly important feature in code generation tasks comprised of relatively shorter texts than plain text generation. Moreover, in HumanEval and MBPP, we can observe that the AUROC of SWEET reaches 1.0 with the
text length exceeding 70, while none of the baselines could achieve it.

heading: H Entropy Threshold Calibration
text: This section proposes a method to calibrate the best entropy threshold effectively. To detect machinegenerated texts, the z-score of them should be high. Let’s say there is a watermarked text sequence with an entropy threshold τ , and the length is N . The z-score is calculated as in Equation 1. Thus z ∝√
Nh N ( NhG Nh − γ), assuming a fixed N . If we can
find a relationship between τ and N h
N , and τ and NhG Nh
, we could choose τ that maximizes the z-score in a fixed N . We denote z′ as a pseudo-metric for estimating z-score:
z′ = E[ Nh
N ](E[ NhG Nh ]− γ) (8)
With logits generated by an LLM, we can calculate entropy H and the probability of sampling a green token after adding δ to green tokens’ logits. We call the expectation of it over the randomness of green/red list partitioning as PG. We model the distribution of logits that LLM generates as a probability distribution function P (H,PG; γ, δ). We approximate the tokens in a text sequence as i.i.d, then we can write as follows:
E[ Nh
N ] = P (H > τ, PG)
E[ NhG Nh ] = E[PG|H > τ ]
To estimate P (H,PG; γ, δ) in the Python language domain of StarCoder, we use a code corpus, CodeSearchNet (Husain et al., 2019). Specifically, we feed the Python corpus of CodeSearchNet to our model and obtain all logits for each time step and calculate H and PG. For PG, we averaged the probability of sampling a green token from a watermarked distribution with 500 random green/red list partitions. We regard the pair (H , PG) per one logits as an unnormalized joint discrete distribution.
The results are presented in Figure 12. The z′
is the highest when the entropy threshold is in [0.820, 0.871]. It aligns with the results in Figure 4, where the optimal threshold value lies around 0.3∼0.9. Therefore, the threshold value found here is a good starting point for searching for the optimal threshold value. The computational cost is only one forward pass across the corpus we used.
The result indicates that we can use the information in the code corpus to calibrate an entropy threshold effectively.

heading: I Analysis of Computation Cost
text: It is practically important to detect machinegenerated text without a huge computational overload. We here analyze computation costs for each baseline and our method.
WLLM does not require any additional computation as it only needs a random number generator and a seed number to put. On the other hand, all zero-shot post-hoc detection methods excluding DetectGPT need at least one forward pass of that LLM. DetectGPT needs to run forward passes as much as the number of perturbations for increased accuracy (the original paper generated 100 perturbed samples, so we did the same). Our method needs one time forward pass to calculate the entropy, which is the same with zero-shot post-hoc detection methods except for DetectGPT. However, we demonstrated that our method outperforms baselines even when utilizing a smaller surrogate model (Sec 6.3), indicating the capability of computationally more efficient employment. On the other hand, while EXP-EDIT does not need LLM for detecting watermarks, it requires measuring the Levenshtein distance to compute the test statistic. Specifically, it demands an extensive calculation of O(mnk2), where m be the length of the target text, n be the
length of the watermark key sequence, and k be the block size. Moreover, T = 500 times of test statistic is also necessary for reporting the p-value. Although these computations do not require LLM and can be implemented in parallel, one can consider the computation cost of EXP-EDIT as high.

heading: J Analysis of Lexical Type Distributions
text: Watermarking a text without degrading its quality is possible when many candidates are alternatively available. In code generation, it is challenging to achieve this, so SWEET selectively apply watermarking only on high entropy, i.e., when there are many candidates. Using Python built-in tokenize module11, we here tokenize outputs of our SWEET method and analyze the distributions of lexical types both above and below the entropy threshold.
11https://docs.python.org/3/library/tokenize. html
J.1 List of Lexical Types Below is the list of lexical types we use for analysis and corresponding examples. All list of types the tokenize module actually emits can be found in https://docs.python.org/3/ library/token.html. We merged and split the original types.
• NAME : identifier names, function names, etc.
• OP : operators, such as {, [ ( +, =, etc.
• INDENT : we merge NEWLINE, DEDENT, INDENT, NEWLINE, and NL.
• RESERVED : split from NAME. In Python docs, they are officially named keywords.
• BUILT-IN : split from NAME. Please refer to Python docs12.
• NUMBER
• STRING
• COMMENT
• FUNCNAME : split from NAME. We manually build a list of function name almost being used only for function. For examples, append(), join(), split() functions are included.
J.2 Lexical Types Distributions Above Threshold
Figure 13 shows lexical types distributions of output tokens above the entropy threshold (i.e., watermarked tokens) across seven thresholds. As the
12https://docs.python.org/3/library/functions. html#built-in-functions
entropy threshold rises, the proportion of NAME type tokens increases by the most (26%p to 63%p). Intuitively, this can be easily understood, considering there would be many alternative candidates for defining identifier names. Unfortunately, this would lead to vulnerability to an adversarial attack on watermarking, such as changing variable names. Following the NAME type, the ratio of the RESERVED type also increases slightly (12%p to 20%p), meaning that the model has multiple choices of logical flow in code generation, considering RESERVED tokens usually decide code execution flow.
J.3 Lexical Types Distributions Below Threshold
Figure 14 shows lexical types distributions of output tokens below the entropy threshold. In contrast to the distributions above the threshold, NAME and RESERVED types do not increase as the threshold rises. Meanwhile, the proportion of INDENT types slightly increases (18%p to 22%p), indicating that the model has more confidence in the rules, such as indentation.

heading: K Further Analysis of Breakdown of Post-hoc methods
text: The performance of post-hoc detection methods in the machine-generated code detection task is sur-
prisingly low compared to their performance in the plain text domain. In both HumanEval and MBPP, none of the post-hoc baselines have an AUROC score exceeding 0.6, and the TPR is around 10% or even lower. In this section, we analyze the failures of post-hoc detection baselines.
Out-Of-Domain for classifiers. Methods leveraging trained classifiers, such as GPTZero and OpenAI Classifier, inherently suffer from out-ofdomain (OOD) issues (Guo et al., 2023; Yang et al., 2023). Since the machine-generated code detection problems are relatively under explored, we can conjecture that there are not enough examples of machine-generated code for training, especially even though we do not know of the dataset on which GPTZero was trained.
Relatively Short Length of Code Blocks. DetectGPT presumes the length of the text being detected as near paragraph length. OpenAI Classifier released in 2023 (OpenAI, 2023b) takes only text longer than 1,000 tokens. Even in the WLLM and their following paper (Kirchenbauer et al., 2023b), the length is one of the prime factors in detection and is used in a metric, detectability@T. Despite the importance of the length, in our experiments, the length of the generated code text is generally short. The token lengths generated by the model were are 59 and 49 tokens on average for HumanEval and MBPP, respectively. Unless
embedding some signals in the text intentionally, like WLLM and ours, it seems that it is challenging for post-hoc methods to detect short text.
Failures in DetectGPT. Specifically, in DetectGPT, we attribute the failure to detect machinegenerated code to poor estimation of perturbation curvature. We hypothesize two reasons for this. Firstly, considering the nature of the code, it is challenging to rephrase a code while preserving its meaning or functionality. To minimize the degradation of perturbation, we use SantaCoder for the masking model and paraphrase only one line of code at a time. Yet, in most cases, the rephrased code is either identical to its original or broken in functionality. Secondly, LLMs have not achieved as satisfactory code generation performance as plain text generation. Hence, the base and masking models cannot draw meaningful curvature.

heading: Future Work
text: 4 and left further robustness enhancement for future work. (2) Possibilities of watermark forgery: An attacker may pry out the watermarking rules, and O(|V|)2 runs in the brute-force mechanism enable it. Against the attack, one can apply techniques enhancing the watermarking model’s security, such as dividing the green/red list depending on prior h > 1 tokens, as mentioned in the WLLM paper, or applying methods like SelfHash (Kirchenbauer et al. , 2023b). For our work, two additional issues exist as follows. (1) Entropy threshold calibration: We demonstrate that our method outperforms the baselines in the broad entropy threshold range (see Sec 6. 1) and investigate how to calibrate the entropy threshold effectively (see Appendix H). However, we still need entropy threshold tuning to obtain the best performance, which costs a computation. (2) Need for the source LLM during detection: SWEET works in a white-box setting. Although it has been shown that employing even a smaller surrogate LM can still maintain the detection performances to some degree (see Sec 6. 3), this can be a computational burden for some users who want to apply our work