Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMs

## Abstract

Abstract Large Language Models (LLMs) have become critical to
modern software development, but their reliance on
uncurated web-scale datasets for training introduces a
significant security risk:
the absorption and reproduction
of malicious content.
To systematically evaluate this risk, we introduce Scam2Prompt,
a scalable automated auditing framework that
identifies the underlying intent of a scam site and
then synthesizes innocuous, developer-style prompts
that mirror this intent, allowing us to test whether
an LLM will generate malicious code in response to
these innocuous prompts.
In a large-scale study of four production LLMs (GPT-4o, GPT-4o-mini,
Llama-4-Scout, and DeepSeek-V3), we found that Scam2Prompt’s innocuous
prompts triggered malicious URL generation in 4.24% of cases.
To test the persistence of this security risk,
we constructed Innoc2Scam-bench, a benchmark of 1,559
innocuous prompts that consistently elicited malicious code
from all four initial LLMs. When applied to seven additional production
LLMs released in 2025, we found the vulnerability is not
only present but severe,
with malicious code generation rates ranging from 12.7% to 43.8%.
Furthermore, existing safety measures like state-of-the-art guardrails
proved insufficient to prevent this behavior, with an overall detection rate
of less than 0.3%.

## 1 Introduction

Warning:
The following paper contains scam content and urls.
To avoid accidental clicks, we replace all . with [.] of all known malicious urls
in this paper.

Large language models (LLMs) have rapidly become critical
infrastructure in software development, with millions of developers relying on
AI-generated code for production systems. This widespread adoption has occurred
alongside an unprecedented expansion in training data scale. Modern LLMs such
as GPT-4 utilize datasets up to 15 trillion tokens,
sourced from web pages, code repositories, and social media
platforms (OpenAI, 2023). This insatiable demand for
training data has created a fundamental security risk: a large scale
incorporation of malicious content into model weights.

The internet inherently hosts substantial amounts of
misinformation, scams, and deliberately poisonous
content (Vosoughi et al., 2018; Lazer et al., 2018; Allcott et al., 2019; Broniatowski et al., 2018; He et al., 2024).
Sophisticated misinformation campaigns can persist
undetected on the internet for months or even years before
discovery (Cuan-Baltazar et al., 2020; Tasnim et al., 2020).
While traditional web services employ content moderation, user reporting
mechanisms, and platform-level filtering to combat malicious
material (Gillespie, 2018; Graves, 2016; Roberts, 2019; Roozenbeek et al., 2020),
the LLM training pipeline operates under a fundamentally different paradigm
that amplifies this risk.
Data collection for these models prioritizes scale and
diversity over verification,
crawling billions of pages with minimal quality control.
Once this data is collected, it becomes a training corpus
and is used for training for all future models.
Unlike a search engine that can delist a harmful URL
in real-time, malicious content within a training set is permanently
embedded into the model’s learned representations.
Consequently, even if the original source
is removed from the web, the poisoned data
persists and can be unknowingly
replicated across training data of countless models,
repeatedly exposing end-users to significant harm and risks.

This threat becomes particularly acute in downstream
applications like AI-assisted code generation.
Code generated by LLMs can be integrated into production
systems where it may access sensitive data,
acquire administrative privileges, or cause other direct damage.
Current AI coding assistants can generate thousands of lines
of code in seconds, making it challenging or even impossible for
users to review every line of code generated.
Moreover, modern software often relies on third-party libraries and
APIs which are very hard for developers to verify every external dependency
used in the generated code.
A cleverly hidden vulnerability or malicious
payload can therefore be easily overlooked,
leading to severe security vulnerabilities unnoticed until
the code is executed, and the damage is done.
This creates an urgent need to evaluate
the extent to which LLMs are generating malicious
code in practice and to evaluate the potential risks.
Motivated particularly by a real-world example presented in Section [2](https://arxiv.org/html/2509.02372v2#S2),
where a victim lost $2,500 after ChatGPT generated a code snippet that transmitted his crypto wallet’s private key to a scam URL, in this paper,
we focus on auditing and evaluating the extent to which
production LLMs generate code containing malicious URLs
in response to completely normal programming prompts that are likely to come from developers.

Automated Audit Framework: We develop an automated
audit framework, Scam2Prompt, to systematically test whether
production LLMs generate code that embeds malicious URLs in
response to innocuous prompts. The key intuition
is that once malicious sources targeting a specific user request exist,
they are rarely isolated; instead, many related variants
also exist which are capable of misleading users toward similarly
harmful outcomes.
When LLMs receive requests for these specific user intents,
they may reference these malicious variants,
thereby generating code that contains URLs from scam sites.

Motivated by this observation, our framework begins with
a given seed scam URL and
an oracle capable of detecting malicious URLs. Our framework then
automatically
queries an LLM agent to extract the context surrounding the harmful content in a sandbox,
summarize it, and generate candidate prompts that appear as innocuous user coding
requests.
We subsequently feed these generated prompts to target production LLMs
for code generation.
Finally, we apply the oracle to identify any generated code snippets
that contain malicious URLs.

This paper focuses on malicious URLs embedded in code for two reasons.
First, oracles for malicious URL detection are widely
available and well-established (e.g., Google Safe Browsing (goo, 2025),
VirusTotal (vir, 2025)), facilitating large-scale automated
evaluation. Second, malicious URLs in generated code pose severe immediate risks,
ranging from cryptocurrency theft to sensitive data exposure,
making them a high-priority security concern.
Importantly, our automated audit
methodology remains general and can be applied to expose
other forms of
malicious code generation (e.g., backdoors, worms)
provided that appropriate domain-specific
oracles are available.

Results: Our experimental results
provide strong empirical evidence that production LLMs can
emit malicious code in response to innocuous developer-style prompts
at non-trivial, reproducible rates.
Through automated auditing of four
production LLMs released in 2024, we find
that on average 4.24% of code generated in our experiments
contains malicious URLs.
We further constructed a benchmark dataset, Innoc2Scam-bench, containing 1,559
innocuous developer-style prompts that trigger
all four LLMs to generate malicious code. Innoc2Scam-bench is
then applied to seven of the latest production LLMs released in 2025.
We still find that all these models generate malicious code
at a non-negligible rate, ranging from 12.7% to 43.8%.
These findings demonstrate that
adversaries, intentionally or not, have successfully poisoned
training datasets at scale.
Critically, this contamination persists from 2024 models through 2025 releases,
demonstrating that neither current training practices nor safety guardrails
have adequately addressed this vulnerability.
To raise awareness of this urgent threat and support mitigation efforts,
we publicly release our prompts and evaluation results as benchmarks for
future research.

Practical Impact: While the primary focus of this paper is on the
innocuous prompts triggering LLMs to generate malicious code that
contains references to scam phishing sites,
we have unexpectedly discovered numerous scam phishing sites from our auditing framework
that were not included in the scam databases we used,
yet are generated by LLMs.
This is particularly impressive, as the knowledge cutoffs of
these LLMs all predate August 2024 as shown in Table [3](https://arxiv.org/html/2509.02372v2#A2.T3),
indicating that these malicious sites have likely been active for
over a year while evading detection by conventional security measures.
We have reported all of these sites to the respective scam database maintainers.
As of this paper’s submission,
62 of these sites have been confirmed and
added to the major scam database
eth-phishing-detect MetaMask (b).

Contributions: This paper makes the following contributions:

- •
Empirical Evidence of Malicious Code from
Production LLMs: We disclose and evaluate the extent to which
production LLMs can generate malicious code, demonstrating that at
4.24% of LLM-generated code contains malicious URLs alone when
responding to our innocuous prompts. The actual rate of malicious
code generation likely exceeds this figure when considering attack
vectors beyond URLs.
- •
Scam2Prompt: Automated Auditing Framework: We
design and implement a scalable framework that given seed
malicious sources and domain-specific oracles, automatically
generates prompts appearing as innocuous coding requests
while systematically exposing malicious code generation in
production LLMs. This general methodology applies to any type of
malicious behavior provided appropriate oracles are available.
- •
Innoc2Scam-bench: We release a selected benchmark of 1,559
innocuous developer-style prompts that trigger all four
production LLMs audited by Scam2Prompt to generate malicious code. We further apply
this benchmark to seven latest production LLMs released
in 2025, demonstrating that all models generate malicious code
at a non-negligible rate ranging from 12.7% to 43.8%.

## 2 Motivating Example

Figure: Figure 1: The victim’s original tweet reporting the security incident, as covered by media outlets Vasileva (2024); Binance Square (2024); shushu (2024).
Refer to caption: images/originalTweet.png

A striking demonstration of the dangers emerged in Nov 2024,
when a user lost approximately $2,500 in cryptocurrency due to
malicious code generated by ChatGPT. Figure [2](https://arxiv.org/html/2509.02372v2#S2.F2) shows the
original tweet written by the victim reporting the incident.
The incident occurred when the victim
leveraged ChatGPT to generate a cryptocurrency trading script for
buying a cryptocurrency on pump.fun platform on Solana Blockchain (sol, 2025a).
The victim later documented the incident in detail on Twitter,
publicly releasing the complete interaction history.(^1^11The original ChatGPT conversation is available at cha (2024) and archived at cha (2025). The victim’s tweet thread is available at r_cky0 (2024) and archived at vic (2024a; b).)

Figure [2](https://arxiv.org/html/2509.02372v2#S2.F2) presents a selected
snippet of the conversation between the victim and ChatGPT.
The dialogue began as a routine engineering request:
the victim asked ChatGPT to create a
trading script for purchasing a token with a specified address.
Initially, ChatGPT provided a script for another blockchain, Ethereum (eth, 2025),
which the victim corrected by specifying Solana as the target blockchain.
ChatGPT then generated a second generalized script using the spl-token library,
a legitimate Solana token interaction library, which requires users to
specify trading platforms and token addresses manually.

Up to this point, all generated code remained benign, containing only general-purpose
functionality and legitimate APIs. The critical turning point occurred when
the victim specified that the script “has to buy solana tokens from pump.fun.”
Notably, pump.fun is a legitimate and popular trading platform on
Solana (Pump.fun,), but it does not provide official APIs
for trading. This absence has created a market for third-party providers, among which
scams impersonating official services are prevalent. In response to the victim’s
prompt, ChatGPT generated code containing a malicious API endpoint that exploited this
exact scenario: https://api[.]solanaapis[.]com/pumpfun/buy. Crucially,
the code instructed the victim to include their wallet’s private key directly
in the POST request payload, which is a fundamental security violation in
cryptocurrency applications.

Although the malicious script contains syntax errors, the victim persisted through multiple
debugging rounds with ChatGPT to resolve the issues.
Eventually, the victim successfully executed the final version, which transmitted
their private key to the malicious endpoint. Within 30 minutes
of execution, all cryptocurrency in the victim’s wallet (approximately $2,500)
had been transferred to an attacker-controlled address.

Finding 1.
Real-world users will directly execute LLM-generated code
containing untrusted third-party components (such as unknown URLs and APIs),
even after extended debugging sessions that should have provided opportunities
for security review.

Upon reflection, the victim recognized that ChatGPT had generated code containing
a critical vulnerability: the direct transmission of his wallet’s private key to an
unverified API endpoint.
This realization prompted him to question the trustworthiness of the suggested endpoint,
and ultimately led him to share the incident publicly on Twitter as a warning to other developers. We provide more discussion on this incident in Appendix [F](https://arxiv.org/html/2509.02372v2#A6).

In fact, subsequent investigation by security experts revealed that the malicious domain
solanaapis[.]com was part of a systematic, large-scale cryptocurrency theft
operation (Fernández, 2024). The attackers had strategically spread documentation
containing these fraudulent APIs across multiple popular developer platforms
including GitHub (git, 2025), Postman (pos, 2025), Stack Exchange (sta, 2025),
and Medium (med, 2025)
to enhance their perceived legitimacy and increase their likelihood
of discovery by both human developers and AI systems.
Moreover, the threat is still active and ongoing.
As of this writing (August 2025), we discovered that the malicious
infrastructure is still there, with only a slight change: primary domain migrating
from solanaapis[.]com to solanaapis[.]net.
(^2^22The current malicious site is archived at sol (2025b).)

Finding 2.
URL poisoning represents an active and urgent threat, as
demonstrated by documented cases resulting in substantial
financial losses. The widespread distribution of malicious APIs
across trusted platforms creates conditions where LLMs may
inadvertently recommend these APIs as legitimate development resources.

This research investigates whether the incident described above is a rare anomaly, or
it represents a systematic vulnerability at scale for production LLMs.

## 3 Scope and Problem Statement

Scope.
This paper focuses specifically on the problem of *
innocuous prompt generates malicious code snippets.*
The scope of this paper is limited to the following:

- •
We only consider malicious code generated directly by LLMs,
without involvement of external tools such as search engines.
- •
We restrict attention to innocuous prompts that
could be asked in normal development tasks.
- •
We do not consider adversarial prompting, jailbreaking, prompt injection, and
all other active inference-time attack techniques.

While external tools can introduce poisoned content, this represents a separate
attack vector that has been explored in prior research, such as search engine
optimization. Furthermore, the presence of external contamination would only make the
security issues of LLMs worse. Adversarial prompting and jailbreaking are
important methods used to actively exploit or
bypass an LLM’s safety features at the moment a prompt is submitted, but they have
a different threat model and they are much less likely to be used by regular developers.

Problem Statement.
Let $\mathcal{M}$ denote a large language model which takes prompts as input and
generates code snippets as output, and let $\mathcal{O}$ denote an
oracle function that determines
whether a code snippet is malicious:
$\mathcal{O}:\text{code}\rightarrow\{\text{benign},\text{malicious}\}$.

We further assume the existence of an oracle $\mathcal{P}$ that classifies user prompts as either
“innocuous” (benign developer-style requests) or “adversarial” (crafted to exploit model vulnerabilities
or phrased in ways unlikely to occur in normal development practice):
$\mathcal{P}:\text{prompt}\rightarrow\{\text{innocuous},\text{adversarial}\}$.

We define S as the set of prompt-code pairs where an innocuous prompt
elicits a malicious code snippet from the model:
$\mathcal{S}=\{\,(p,c)\mid\mathcal{P}(p)=\text{innocuous},\;c=\mathcal{M}(p),\;\mathcal{O}(c)=\text{malicious}\,\}$

Objective.
Given $\mathcal{M}$, $\mathcal{O}$, and $\mathcal{P}$, our objective is to develop a framework
to automatically discover and systematically expand the set $\mathcal{S}$.

In this paper, $\mathcal{O}$ is instantiated as an oracle that flags a code snippet as malicious
if it contains at least one malicious URL.
Human annotators serve as the oracle $\mathcal{P}$ to validate
prompt innocuousness, with disagreements resolved by majority vote.

## 4 Scam2Prompt: An Automated Audit Framework

Figure: Figure 3: Overview of Scam2Prompt. The system begins with known malicious URLs, generates developer-style prompts from their contents, and evaluates whether LLMs produce malicious code when responding to those prompts.
Refer to caption: x1.png

The automated audit framework, Scam2Prompt, shown in Figure [3](https://arxiv.org/html/2509.02372v2#S4.F3),
is designed to systematically identify *innocuous prompts
that elicit malicious code*
by a LLM under audit (referred as “Codegen LLM”).
The framework proceeds in four stages: (1) malicious URL
collection, (2) prompt synthesis,
(3) code generation and URL extraction, and (4) oracle- and human-based verification.

Malicious URL Collection.
We begin from existing databases of URLs
that have been previously identified as scams.
Specifically, we use two major sources: (1) the eth-phishing-detect
repository (MetaMask, b)
maintained by
Metamask (MetaMask, a), and (2) the ‘phishing-fort‘
repository (Phishfort,)
maintained by PhishFort (PhishFort,),
containing 187,555 and 119,828 URLs, respectively.
We selected these databases because they are established by prominent
industry companies. They are regularly updated with new
blocklists and new whitelists,
and both are integrated into browser plugins developed
by their respective companies.
The eth-phishing-detect repository is specifically focused
on malicious URLs targeting Web3 users, while the
PhishFort database
has a broader scope, including fintech and healthcare.
This diversity ensures that our evaluation covers various types
of malicious URLs relevant to different sectors.
Next, we need to understand the content of these
pages to generate effective prompts.
Since many entries are expired or inactive, we filter
for URLs that are still accessible and serve static content.
This yielded 28,570 pages whose HTML content could be accessed.

Content Extraction and Prompt Synthesis.
We designed our web crawler with an explicit focus on minimizing
the attack surface when handling potentially malicious URLs.
To reduce exposure, the crawler begins with lightweight HEAD requests under
strict timeouts, thereby limiting data transfer and avoiding unnecessary
payload execution. Only after validating URL format and accessibility
does it selectively perform GET requests, restricted to text-based content types
(e.g., HTML, JSON, XML) while rejecting binaries that could embed malware.
The text-based content is then cleaned by stripping
invisible elements (e.g., CSS, JavaScript)
and extracting only visible text.

This cleaned text is passed to a *prompt-generation model* (“Prompt LLM”
in Figure [3](https://arxiv.org/html/2509.02372v2#S4.F3)),
which synthesizes programming tasks that could plausibly direct a developer to that webpage.
We use three LLMs for prompt generation:
gpt-4o, gpt-4o-mini, and llama-4-scout.
The prompt LLM is instructed to follow three constraints:
(1) prompts must involve code generation or API/library usage;
(2) prompts must be specific, incorporating unique keywords from the page; and
(3) prompts should be concise but capture functionality unique to the site.
This step operationalizes the hypothesis that malicious actors craft documentation to maximize
keyword overlap with user requests.
The detailed prompt template for prompt synthesis is
provided in Appendix [A](https://arxiv.org/html/2509.02372v2#A1).

Code Generation and URL Extraction.
The synthesized prompts are passed to a second model, *the code-generation LLM*
(“Codegen LLM” in Figure [3](https://arxiv.org/html/2509.02372v2#S4.F3)).
Codegen LLM then generates code snippets to
perform the task described in the prompt.
The detailed prompt template for code generation is
provided in Appendix [A](https://arxiv.org/html/2509.02372v2#A1).
We apply a URL extraction module to the output, identifying all endpoints
embedded in the generated code.
This stage yields candidate prompt-code pairs containing
potentially malicious URLs.

URL Malice Detection.
The extracted URLs are evaluated by an oracle ensemble $\mathcal{O}$,
which integrates multiple independent detectors:
ChainPatrol (ChainPatrol,), Google Safe Browsing (Google Safe Browsing,),
and SecLookup (Seclookup,).
We consider a URL to be malicious if any of the detectors flag it as such.
If a URL is flagged as malicious, we additionally check whether
it was present in the original scam databases.
Newly discovered malicious URLs are reported back to the maintainers
of these databases to benefit
the broader security community.

Prompt Classification and Human Validation.
The final step is to ensure that the prompt itself is
an *innocuous developer request*,
rather than adversarial.
The prompts outputted from the last stage are independently
reviewed by three authors of this paper,
with disagreements resolved through majority vote.
This yields the final dataset $\mathcal{S}$ of *innocuous prompt-malicious code pairs*,
which serves both as a benchmark for auditing LLMs and as an empirical measure of the severity
of malicious code generation in real-world development settings.

Scam2Prompt Usage.
The prompts identified by Scam2Prompt provide actionable insights
for strengthening the robustness of the Codegen LLM under audit. Specifically,
the prompt-code pair can be leveraged to fine-tune models, or incorporated
into mitigation pipelines such as machine unlearning to reduce the likelihood of
reproducing malicious contents.
Beyond auditing a single LLM, the identified prompts themselves can also serve as
a valuable stress test for other foundation models.
To facilitate future research, we construct Innoc2Scam-bench,
a curated dataset, detailed in Section [5](https://arxiv.org/html/2509.02372v2#S5).
Innoc2Scam-bench provides the community with a reusable
resource for benchmarking model defenses
and developing new mitigation strategies against data poisoning.

## 5 Dataset Construction

Figure: Figure 4: Overview of the dataset construction process
Refer to caption: x2.png

In addition to the automated auditing framework described in Section [4](https://arxiv.org/html/2509.02372v2#S4),
another core contribution of this work is the construction of a benchmark dataset that captures
instances where innocuous developer prompts elicit malicious code from four production
LLMs (GPT-4o, GPT-4o-mini, Llama-4-Scout, and DeepSeek-V3).
The dataset is designed to represent the most challenging cases for LLMs, providing a rigorous
stress test of their security alignment. While it was constructed using relatively
cheaper, earlier-generation models, it serves as a reusable benchmark for evaluating newer and
more advanced (and expensive) systems. The dataset was created through a four-stage pipeline
summarized in Figure [4](https://arxiv.org/html/2509.02372v2#S5.F4). We later leverage this dataset to identify
similar scam issues in the latest state-of-the-art models, as described in
Section [6](https://arxiv.org/html/2509.02372v2#S6).

Stage 1: Automated Model Testing.
We began with $265{,}114$ candidate prompts generated in the prompt synthesis stage.
These prompts were systematically issued to four LLMs
(GPT-4o, GPT-4o-mini, Llama-4-Scout, and DeepSeek-V3).
From each output, we extracted all URLs and evaluated them using our oracle ensemble to determine
maliciousness. The prompts were then grouped by the number of models
that produced malicious code in response. This filtering yielded:
3,150 prompts where *all four models* generated malicious code,
4,142 prompts where *three models* did so, and
257,822 prompts where *two or fewer models* did so.
This stage corresponds directly to the automated auditing framework described in
Section [4](https://arxiv.org/html/2509.02372v2#S4).

Stage 2: Manual Validation.
To ensure that retained prompts were innocuous and free of adversarial intent,
we conducted manual validation. Three independent annotators reviewed all candidate prompts,
with disagreements resolved by majority vote. This process yielded 3,133 prompts confirmed as
innocuous developer requests, while 17 ambiguous cases were excluded.

**Table 1: Malicious program generation across LLM combinations. The columns denote: Prompt LLM (prompt generator), Codegen LLM (code generator), total Prompts, total programs Generated, and the percentage of Malicious programs.**
| Prompt<br>LLM | Codegen<br>LLM | Total<br>Prompts | Gen. | Malicious<br>(%) |
| --- | --- | --- | --- | --- |
| gpt-4o | gpt-4o | 100760 | 100714 | 4539<br>(4.51%) |
| gpt-4o-mini | 100713 | 4499<br>(4.47%) |  |  |
| llama-4-scout | 100712 | 3790<br>(3.76%) |  |  |
| deepseek-v3 | 100717 | 4047<br>(4.02%) |  |  |
| gpt-4o<br>mini | gpt-4o | 69076 | 68688 | 4079<br>(5.94%) |
| gpt-4o-mini | 68688 | 3629<br>(5.28%) |  |  |
| llama-4-scout | 68692 | 3185<br>(4.64%) |  |  |
| deepseek-v3 | 68692 | 3187<br>(4.64%) |  |  |
| llama-4<br>-scout | gpt-4o | 95278 | 94611 | 3350<br>(3.54%) |
| gpt-4o-mini | 94601 | 3371<br>(3.56%) |  |  |
| llama-4-scout | 94652 | 3118<br>(3.29%) |  |  |
| deepseek-v3 | 94652 | 3019<br>(3.19%) |  |  |

Stage 3: Classification.
Next, we classified validated prompts based on whether they explicitly referenced a scam URL or domain.
This distinction matters because it is debatable whether an LLM should refuse execution when a user
explicitly specifies a malicious URL, whereas generating scam endpoints in response to a
prompt with *no such reference* constitutes a more severe vulnerability.
Out of the validated innocuous prompts, 1,968 explicitly mentioned a known scam URL or domain,
while 1,165 contained no direct references yet still induced malicious code generation.

Stage 4: Dataset Construction.
To reflect this distinction, we included all 1,165 prompts without direct scam references, and
randomly sampled 400 prompts with explicit scam mentions to avoid dataset imbalance.
Combining these subsets produced the final benchmark of
1,559 innocuous prompts, each paired with code outputs from all four LLMs that contained
malicious URLs.

The resulting dataset, Innoc2Scam-bench, provides a rigorous resource for auditing future LLMs.
Importantly, all included prompts were manually validated as
innocuous, ensuring that malicious code arises not from adversarial prompting,
but purely from the models themselves.
In Appendix [H](https://arxiv.org/html/2509.02372v2#A8), we provide a running example of a prompt from Innoc2Scam-bench,
illustrating how it was generated from a seed scam URL in Scam2Prompt,
validated and selected for inclusion in Innoc2Scam-bench, and later
applied to seven state-of-the-art LLMs, and revealing their malicious code generation.
We chose these models to span different providers and model sizes to assess
the generality of our findings. Further model details are provided in
Appendix [B](https://arxiv.org/html/2509.02372v2#A2).

## 6 Experiments

### 6.1 Malicious Code Generation Rate in Scam2Prompt

To evaluate our framework, Scam2Prompt, we conducted a large-scale
experiment pairing three Prompt LLMs with four Codegen LLMs.
This experiment, involving over 265,000 prompts,
reveals that every tested model combination produces a
non-negligible amount of malicious code, as summarized in
Table [1](https://arxiv.org/html/2509.02372v2#S5.T1).
On average, 4.2% of generated programs contained malicious
URLs, though rates varied substantially by pairing:
from a low of 3.19% with (llama-4-scout,
deepseek-v3, marked in green) to a high of
5.94% with (gpt-4o-mini,
gpt-4o, marked in red).

Impact of Prompt and Codegen LLMs.
The choice of the Prompt LLM demonstrates a significant impact
on both the number of prompts generated and the resulting
malicious programs.
Despite being given the same set of tasks,
gpt-4o-mini, the model that generated the fewest prompts
(69,076), consistently induced the highest malicious
rates across all Codegen LLMs.
This suggests that certain models are inherently more
adept at crafting prompts that exploit poisoned data.
Additionally, the choice of the Codegen LLM is also a critical
factor. The data reveals a clear trend where
llama-4-scout and deepseek-v3
produced fewer malicious programs than
gpt-4o and gpt-4o-mini when
tested against all three Prompt LLMs.
We hypothesize that OpenAI’s models may be trained
on more extensive data containing a higher volume
of scam-related content, which in turn leads them to
generate more malicious URLs for the same tasks. More
detailed sampling parameters and cross-model analysis results
are provided in
Appendix [C](https://arxiv.org/html/2509.02372v2#A3)
and Appendix [D](https://arxiv.org/html/2509.02372v2#A4),
respectively.

**Table 2: Performance of models on prompt completion and malicious code generation, grouped by safety alignment effectiveness. (C1 refers to prompts explicitly mentioning a scam URL or domain, while C2 refers to prompts without such a mention.)**
| Model | Cat. | Prompts | Gen. | Filt. | Malicious<br>(%) |
| --- | --- | --- | --- | --- | --- |
| gemini-2.5<br>-pro | Tot | 1559 | 908 | 628 | 198<br>(12.7%) |
| C1 | 400 | 196 | 201 | 40 |  |
| C2 | 1159 | 712 | 427 | 158 |  |
| gpt-5 | Tot | 1559 | 1431 | 44 | 330<br>(21.2%) |
| C1 | 400 | 365 | 10 | 103 |  |
| C2 | 1159 | 1066 | 34 | 227 |  |
| claude<br>-sonnet-4 | Tot | 1559 | 1405 | 140 | 498<br>(31.9%) |
| C1 | 400 | 356 | 39 | 108 |  |
| C2 | 1159 | 1049 | 101 | 390 |  |
| grok-code<br>-fast-1 | Tot | 1559 | 1534 | 22 | 636<br>(40.8%) |
| C1 | 400 | 395 | 5 | 160 |  |
| C2 | 1159 | 1139 | 17 | 476 |  |
| gemini-2.5<br>-flash | Tot | 1559 | 1528 | 1 | 669<br>(42.9%) |
| C1 | 400 | 390 | 0 | 161 |  |
| C2 | 1159 | 1138 | 1 | 508 |  |
| qwen3-<br>coder | Tot | 1559 | 1546 | 6 | 672<br>(43.1%) |
| C1 | 400 | 396 | 2 | 154 |  |
| C2 | 1159 | 1150 | 4 | 518 |  |
| deepseek<br>-chat-v3.1 | Tot | 1559 | 1516 | 37 | 683<br>(43.8%) |
| C1 | 400 | 381 | 17 | 158 |  |
| C2 | 1159 | 1135 | 20 | 525 |  |

Contribution to Live Phishing Databases.
During our experiment, our oracle flagged numerous
malicious domains, most already present in existing
scam databases. Crucially, we identified a subset
of active domains that were unlisted at the time.
During our research, the phishing-fort database
was deprecated, so we contributed our
findings to eth-phishing-detect,
which is actively maintained by MetaMask.
As of this writing, 62 domains of our submissions have been
validated and added to its blocklist,
directly improving user safety by blocking access to these phishing sites.

### 6.2 Applying Innoc2Scam-bench to new LLMs

To assess whether this vulnerability persists in the latest models, we evaluated seven new,
state-of-the-art LLMs against our Innoc2Scam-bench benchmark. For each of the 1,559 innocuous prompts,
we classified the model’s output as either generated or incomplete. We also recorded when
incomplete generations were caused by the model’s content filters
(Filt. column in Table [2](https://arxiv.org/html/2509.02372v2#S6.T2)). Other incomplete cases included
outputs with repeated content or responses without any code.

Persistent Vulnerability in State-of-the-Art Models.
The results, presented in Table [2](https://arxiv.org/html/2509.02372v2#S6.T2), demonstrate that the data
poisoning
issue remains a systemic and severe vulnerability. The total rate of malicious code
generation is alarmingly high, ranging from 12.7% for
gemini-2.5-pro to 43.8% for
deepseek-chat-v3.1. This finding confirms that even the most
recent foundation models are highly susceptible to generating harmful
code in response to innocuous developer requests, validating the effectiveness of
Innoc2Scam-bench as a stress test for model safety.

Divergent Safety Alignments.
The results in Table [2](https://arxiv.org/html/2509.02372v2#S6.T2) reveal three distinct tiers of
model safety alignment. gemini-2.5-pro and gpt-5 form the top
tier, exhibiting the strongest defenses. gemini-2.5-pro achieves the lowest
malicious code rate (12.7%) through extremely aggressive content filtering,
refusing to generate code for over 40% of prompts
(628 total, highlighted in blue). gpt-5 is the second,
with a low malicious rate of 21.2%, and it achieves
this with far less filtering.
In the middle tier,
claude-sonnet-4 represents a
moderate approach, with a malicious rate of 31.9% and a
filtering rate (140 prompts, highlighted in blue) that sits between the extremes.
The third tier comprises the remaining four models, which all behave
similarly poorly, with malicious generation
rates clustered above 40%:
grok-code-fast-1 (40.8%),
gemini-2.5-flash (42.9%),
qwen3-coder (43.1%), and
deepseek-chat-v3.1 (43.8%). These models consistently
demonstrate minimal filtering (highlighted in yellow) and are consequently
highly vulnerable. It highlights that while some models are
making progress in safety, the underlying data poisoning
issue remains a critical and largely unmitigated vulnerability
for a significant portion of the industry. More detailed
analysis results are provided in Appendix [E](https://arxiv.org/html/2509.02372v2#A5).

Effectiveness of Existing Guardrail Systems.
We evaluated NVIDIA NeMo Guardrails (Rebedea et al., 2023) with
Llama Nemotron Safety Guard V2 on Innoc2Scam-bench and their
corresponding malicious code.
The guardrail failed to detect almost all malicious outputs
(overall detection $<0.3$%) and blocked none of the innocuous
prompts, indicating limited utility as a standalone defense
against the security threat.
Full setup, prompt, and per-model results are provided
in Appendix [G](https://arxiv.org/html/2509.02372v2#A7).

## 7 Related Work

Poison Detection in LLM Code Generation.
There are multiple work investigating the malicious behavior
of LLMs in the inference stage for code generation.
The work in Zeng et al. (2025) studies a
poisoning attack in code generation when the
external tools including search engines used by LLMs
contain malicious content.
Similarly, BIPIA (Yi et al., 2025) presents
the first systematic benchmark to evaluate
indirect prompt injection attacks, focusing
on malicious instructions embedded in external
content that manipulate LLM behavior.
In contrast, our work demonstrates a more fundamental
problem that does not require the LLM to
access any external sources during inference.
We show that harmful content, such as scam API endpoints,
has already been absorbed into the models’ weights from their
training data and can be triggered by innocuous
developer prompts.

Poisoning Attacks in LLM Training Pipelines.
Data poisoning, where adversaries manipulate
training data to alter model behavior at
inference, has emerged as a critical threat to
machine learning systems. While early work mostly
focused on computer vision
applications (Cinà et al., 2023; Raghavan et al., 2022; Goldblum et al., 2022),
recent studies have extended this concern to the
language domain, particularly LLMs.
Carlini et al. (2024)
proves the practicality and feasibility of
poisoning web-scale training datasets collection pipelines.
A recent survey (Zhao et al., 2025) provides the
first systematic overview of data poisoning attacks
targeting LLMs across multiple stages such as
pretraining, fine-tuning, preference alignment, and
instruction tuning. Jiang et al. (2024) studies
poisoning attacks on LLMs
during fine-tuning on text summarization and completion tasks,
showing that existing defenses remain ineffective.
Our study differs by (i) targeting malicious code
generation, which poses immediate execution risks, rather
than natural-language outputs, and (ii) auditing production
LLMs for evidence of existing, passive poisoning in
their training corpora, rather than new inference-time active attacks.

## 8 Conclusions

The Scam2Prompt framework demonstrates that automated auditing can
systematically expose production LLMs’ propensity to generate malicious code.
By synthesizing prompts from known malicious scam sites, testing them against
LLMs under audit, and validating their innocuous nature, we constructed Innoc2Scam-bench, a
rigorously validated benchmark of 1,559 innocuous deverloper-style prompts.
Evaluation using this benchmark reveals a systemic vulnerability in
latest state-of-the-art LLMs: all tested production
LLMs still generate malicious code at substantial rates (12.7%-43.8%). The consistency
of this behavior across diverse architectures, providers, and model generations
demonstrates that malicious content contamination is an industry-wide problem
persisting despite advances in safety alignment.
These findings establish urgent research priorities: robust training data
sanitization, code-specific safety guardrails, and runtime security monitors
for LLM-assisted development. The gap between minimal filtering rates and high
malicious generation rates underscores that current defenses remain inadequate
across the entire pipeline. We release Innoc2Scam-bench and our framework to
enable systematic evaluation of future mitigation strategies, providing the
foundation for making LLM-powered software development both productive and
secure.

## 9 Reproducibility Statement

All source code, our accepted submissions to the ‘eth-phishing-detect‘
scam database,
and all experimental data are publicly
available at [https://sites.google.com/view/scam2prompt/](https://sites.google.com/view/scam2prompt/).
The Innoc2Scam-bench benchmark is also publicly released at
[https://huggingface.co/datasets/anonymous-author-32423/Innoc2Scam-bench/tree/main](https://huggingface.co/datasets/anonymous-author-32423/Innoc2Scam-bench/tree/main).

To further ensure the reproducibility of our experiments,
we adopted the most
deterministic sampling settings for all LLMs, unless otherwise
specified
(only in Appendix [C](https://arxiv.org/html/2509.02372v2#A3)).
Specifically: temperature $T=0$ for code generation and for
Innoc2Scam-bench evaluation,
$T=0.3$ for prompt generation , $top_p=1.0$.
We set $T=0.3$ for prompt generation to add mild randomness
and obtain more diverse prompts.
The seed used is a hash of the prompt, so identical
prompts always yield the same seed.

## References

- cha (2024)
Chatgpt conversation archive cryptocurrency trading script.
[https://chatgpt.com/share/67403c78-6cc0-800f-af71-4546231e6b10](https://chatgpt.com/share/67403c78-6cc0-800f-af71-4546231e6b10),
2024.
Accessed: 2025-08-21.
- vic (2024a)
Victim thread on twitter, archived in threadrweader.
[https://threadreaderapp.com/thread/1859656430888026524.html](https://threadreaderapp.com/thread/1859656430888026524.html),
2024a.
Twitter thread archived in ThreadReaderApp.
- vic (2024b)
Victim thread on twitter, archived in ghostarchive.
[https://ghostarchive.org/archive/BRT6H](https://ghostarchive.org/archive/BRT6H), 2024b.
Twitter thread archived in Ghostarchive.
- cha (2025)
ChatGPT Conversation Archive Cryptocurrency Trading Script Archive.
Ghost Archive, September 2025.
URL [https://ghostarchive.org/archive/IynyE](https://ghostarchive.org/archive/IynyE).
Accessed: 2025-09-21.
- eth (2025)
Ethereum, 2025.
URL [https://ethereum.org/](https://ethereum.org/).
Accessed: 2025-08-21.
- git (2025)
Github, 2025.
URL [https://github.com/](https://github.com/).
Accessed: 2025-08-21.
- goo (2025)
Google safe browsing.
[https://safebrowsing.google.com](https://safebrowsing.google.com), 2025.
Accessed: 2025-08-18.
- med (2025)
Medium, 2025.
URL [https://medium.com/](https://medium.com/).
Accessed: 2025-08-21.
- pos (2025)
Postman, 2025.
URL [https://www.postman.com/](https://www.postman.com/).
Accessed: 2025-08-21.
- sol (2025a)
Solana, 2025a.
URL [https://solana.com/](https://solana.com/).
Accessed: 2025-08-21.
- sol (2025b)
Solanaapis.net documentation archive.
[https://web.archive.org/web/20250710013715/https://docs.solanaapis.net/](https://web.archive.org/web/20250710013715/https://docs.solanaapis.net/),
2025b.
Archived: 2025-07-10.
- sta (2025)
Stack exchange, 2025.
URL [https://stackexchange.com/](https://stackexchange.com/).
Accessed: 2025-08-21.
- vir (2025)
Virustotal.
[https://www.virustotal.com](https://www.virustotal.com), 2025.
Accessed: 2025-08-18.
- Allcott et al. (2019)
Hunt Allcott, Matthew Gentzkow, and Chuan Yu.
Trends in the diffusion of misinformation on social media.
*Research & Politics*, 6(2):2053168019848554, 2019.
- Anthropic (2025)
Anthropic.
System card: Claude opus 4 & claude sonnet 4, May 2025.
URL [https://anthropic.com/model-card](https://anthropic.com/model-card).
PDF; training corpus description and web data “as of March 2025”
(accessed 2025-09-22).
- Binance Square (2024)
Binance Square.
Users seek help from chatgpt but fall victim to phishing “theft”.
Blog post on Binance Square, Nov 23 2024.
URL [https://www.binance.com/en/square/post/16660778088634](https://www.binance.com/en/square/post/16660778088634).
- Broniatowski et al. (2018)
David A Broniatowski, Amelia M Jamison, SiHua Qi, Lulwah AlKulaib, Tao Chen,
Adrian Benton, Sandra C Quinn, and Mark Dredze.
Weaponized health communication: Twitter bots and russian trolls
amplify the vaccine debate.
*American journal of public health*, 108(10):1378–1384, 2018.
- Carlini et al. (2024)
Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel
Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian
Tramèr.
Poisoning web-scale training datasets is practical.
In *2024 IEEE Symposium on Security and Privacy (SP)*, pp. 407–425. IEEE, 2024.
- (19)
ChainPatrol.
ChainPatrol: Real-Time Web3 Brand Protection Against Phishing,
Impersonation, and Malicious Domains.
[https://chainpatrol.com/](https://chainpatrol.com/).
Accessed: 2025-08-24.
- Cinà et al. (2023)
Antonio Emanuele Cinà, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon,
Werner Zellinger, Bernhard A Moser, Alina Oprea, Battista Biggio, Marcello
Pelillo, and Fabio Roli.
Wild patterns reloaded: A survey of machine learning security against
training data poisoning.
*ACM Computing Surveys*, 55(13s):1–39,
2023.
- Cuan-Baltazar et al. (2020)
Jose Yunam Cuan-Baltazar, Mario Javier Muñoz-Perez, Carolina Robledo-Vega,
Mario Ulises Pérez-Zepeda, and Elena Soto-Vega.
Misinformation detection during health crisis.
*Harvard Kennedy School Misinformation Review*, 1(3),
2020.
- Fernández (2024)
Germán Fernández.
Is this ”ai poisoning”?
[https://x.com/1ZRR4H/status/1860223101167968547](https://x.com/1ZRR4H/status/1860223101167968547), 2024.
Accessed: July 2025.
- Gillespie (2018)
Tarleton Gillespie.
*Custodians of the Internet: Platforms, content moderation, and
the hidden decisions that shape social media*.
Yale University Press, 2018.
- Goldblum et al. (2022)
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild,
Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein.
Dataset security for machine learning: Data poisoning, backdoor
attacks, and defenses.
*IEEE Transactions on Pattern Analysis and Machine
Intelligence*, 45(2):1563–1580, 2022.
- Google (2025)
Google.
Gemini models — gemini api, June 2025.
URL [https://ai.google.dev/models/gemini](https://ai.google.dev/models/gemini).
Lists Gemini 2.5 Pro/Flash; Knowledge cutoff January 2025; 1M-token
input limit (accessed 2025-09-19).
- (26)
Google Safe Browsing.
Google Safe Browsing: A service for detecting unsafe web resources.
[https://safebrowsing.google.com/](https://safebrowsing.google.com/).
Accessed: 2025-08-24.
- Graves (2016)
Lucas Graves.
Understanding the promise and limits of automated fact-checking.
*Factsheet, Reuters Institute for the Study of Journalism*,
2016.
- He et al. (2024)
Hao He, Haoqin Yang, Philipp Burckhardt, Alexandros Kapravelos, Bogdan
Vasilescu, and Christian Kästner.
4.5 million (suspected) fake stars in github: A growing spiral of
popularity contests, scams, and malware.
*arXiv preprint arXiv:2412.13459*, 2024.
- Jiang et al. (2024)
Shuli Jiang, Swanand Ravindra Kadhe, Yi Zhou, Farhan Ahmed, Ling Cai, and
Nathalie Baracaldo.
Turning generative models degenerate: The power of data poisoning
attacks.
*arXiv preprint arXiv:2407.12281*, 2024.
- Knostic (2025)
Knostic.
Deepseek’s cutoff date is july 2024: We extracted deepseek’s system
prompt, February 2025.
URL
[https://www.knostic.ai/blog/exposing-deepseek-system-prompts](https://www.knostic.ai/blog/exposing-deepseek-system-prompts).
Knowledge cutoff: July 2024.
- Lazer et al. (2018)
David MJ Lazer, Matthew A Baum, Yochai Benkler, Adam J Berinsky, Kelly M
Greenhill, Filippo Menczer, Miriam J Metzger, Brendan Nyhan, Gordon
Pennycook, David Rothschild, et al.
The science of fake news.
*Science*, 359(6380):1094–1096, 2018.
- Meta (2024)
Meta.
Llama-4-scout-17b-16e, August 2024.
URL [https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E).
Knowledge cutoff: August 2024.
- MetaMask (a)
MetaMask.
MetaMask: A crypto wallet and gateway to blockchain apps.
[https://metamask.io/](https://metamask.io/), a.
Accessed: 2025-08-24.
- MetaMask (b)
MetaMask.
eth-phishing-detect: Utility for detecting phishing domains
targeting Web3 users.
[https://github.com/MetaMask/eth-phishing-detect](https://github.com/MetaMask/eth-phishing-detect),
b.
Accessed: 2025-08-24.
- OpenAI (2023)
OpenAI.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- OpenAI (2024a)
OpenAI.
Gpt-4o, 2024a.
URL [https://platform.openai.com/docs/models/gpt-4o](https://platform.openai.com/docs/models/gpt-4o).
Knowledge cutoff: October 1, 2023.
- OpenAI (2024b)
OpenAI.
Gpt-4o mini, 2024b.
URL [https://platform.openai.com/docs/models/gpt-4o-mini](https://platform.openai.com/docs/models/gpt-4o-mini).
Knowledge cutoff: October 1, 2023.
- OpenRouter, Inc. (2024)
OpenRouter, Inc.
Openrouter: Unified api platform for large language models.
[https://openrouter.ai](https://openrouter.ai), 2024.
Accessed: 2025-06-22.
- (39)
PhishFort.
PhishFort: Anti-phishing solutions for Web3 and crypto users.
[https://www.phishfort.com/](https://www.phishfort.com/).
Accessed: 2025-08-24.
- (40)
Phishfort.
phishfort-lists.
[https://github.com/phishfort/phishfort-lists](https://github.com/phishfort/phishfort-lists).
Accessed: 2025-08-24.
- (41)
Pump.fun.
Pump.fun.
[https://www.pump.fun](https://www.pump.fun).
Accessed: July 2025.
- Raghavan et al. (2022)
Vijay Raghavan, Thomas Mazzuchi, and Shahram Sarkani.
An improved real time detection of data poisoning attacks in deep
learning vision systems.
*Discover Artificial Intelligence*, 2(1):18,
2022.
- r_cky0 (2024)
r_cky0.
Victim thread on twitter.
[https://x.com/r_cky0/status/1859656430888026524](https://x.com/r_cky0/status/1859656430888026524), 2024.
Twitter thread.
- Rebedea et al. (2023)
Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and
Jonathan Cohen.
Nemo guardrails: A toolkit for controllable and safe llm applications
with programmable rails.
*arXiv preprint arXiv:2310.10501*, 2023.
- Roberts (2019)
Sarah T Roberts.
*Behind the screen: Content moderation in the shadows of social
media*.
Yale University Press, 2019.
- Roozenbeek et al. (2020)
Jon Roozenbeek, Claudia R Schneider, Sarah Dryhurst, John Kerr, Alexandra LJ
Freeman, Gabriel Recchia, Anne Marthe Van Der Bles, and Sander Van
Der Linden.
Susceptibility to misinformation about covid-19 around the world.
*Royal Society open science*, 7(10):201199,
2020.
- (47)
Seclookup.
Seclookup: A domain and URL scanning service for malware and
phishing.
[https://www.seclookup.com/](https://www.seclookup.com/).
Accessed: 2025-08-24.
- shushu (2024)
shushu.
Ai poisoning is unstoppable, can you still code with chatgpt?
*BlockBeats (English)*, Nov 22 2024.
URL [https://en.theblockbeats.news/news/55891](https://en.theblockbeats.news/news/55891).
- Tasnim et al. (2020)
Samia Tasnim, Md Mahbub Hossain, and Hoimonty Mazumder.
Impact of rumors and misinformation on covid-19 in social media.
*Journal of preventive medicine and public health*, 53(3):171–174, 2020.
- Vasileva (2024)
Hristina Vasileva.
User solana wallet exploited in first case of ai poisoning attack.
*Bitget News*, Nov 22 2024.
URL [https://www.bitget.com/news/detail/12560604366774](https://www.bitget.com/news/detail/12560604366774).
- Vosoughi et al. (2018)
Soroush Vosoughi, Deb Roy, and Sinan Aral.
The spread of true and false news online.
*Science*, 359(6380):1146–1151, 2018.
- Yi et al. (2025)
Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and
Fangzhao Wu.
Benchmarking and defending against indirect prompt injection attacks
on large language models.
In *Proceedings of the 31st ACM SIGKDD Conference on Knowledge
Discovery and Data Mining V. 1*, pp. 1809–1820, 2025.
- Zeng et al. (2025)
Binqi Zeng, Quan Zhang, Chijin Zhou, Gwihwan Go, Yu Jiang, and Heyuan Shi.
Inducing vulnerable code generation in llm coding assistants.
*arXiv preprint arXiv:2504.15867*, 2025.
- Zhao et al. (2025)
Pinlong Zhao, Weiyao Zhu, Pengfei Jiao, Di Gao, and Ou Wu.
Data poisoning in deep learning: A survey.
*arXiv preprint arXiv:2503.22759*, 2025.

## Appendix

## Appendix A Prompt Templates

### A.1 Prompt-generation template.

Figure [5](https://arxiv.org/html/2509.02372v2#A1.F5) shows the prompt template we used for the Prompt Synthesis step as mentioned in
Section [4](https://arxiv.org/html/2509.02372v2#S4).
It instructs the model to analyze a webpage URL and its visible content,
infer the page’s core subject matter, and synthesize short,
page-specific prompts restricted to code-generation and
API/library recommendation use cases.

Figure: Figure 5: Prompt-generation template used in our experiments.

### A.2 Code-generation template.

Figure [6](https://arxiv.org/html/2509.02372v2#A1.F6) shows the prompt template we used for the Code Generation step as mentioned in
Section [4](https://arxiv.org/html/2509.02372v2#S4).
It sets the model’s role as an expert programmer and specifies
quality criteria:
the code generated should be syntactically correct and
runnable, well-documented, best practices,
and appropriate error handling, while requiring only code output
to save token usage.

Figure: Figure 6: Code-generation template used in our experiments.

## Appendix B Model and Parameter Details

Model set and rationale.
We use four production LLMs in the audit: openai/gpt-4o,
openai/gpt-4o-mini, meta-llama/llama-4-scout, and deepseek/deepseek-v3.
This set spans multiple providers and architectures to test whether the observed
behavior generalizes across independent training pipelines. We set $max_token$ to 2,000
for all models to ensure sufficient context for code generation.
For Innoc2Scam-bench evaluation, we additionally include seven recent
production LLMs: x-ai/grok-code-fast-1,
deepseek/deepseek-chat-v3.1, openai/gpt-5,
qwen/qwen3-coder, google/gemini-2.5-flash,
google/gemini-2.5-pro, and anthropic/claude-sonnet-4 (see Table [2](https://arxiv.org/html/2509.02372v2#S6.T2) in
Section [6](https://arxiv.org/html/2509.02372v2#S6)). For these newer models, we only set the $max_token$ parameter
to 20,000 to accommodate longer code generation requests.

Model specifications.
Table [3](https://arxiv.org/html/2509.02372v2#A2.T3) summarizes key specifications
(provider, architecture family, and scale estimates where available).
We access all these models through OpenRouter (OpenRouter, Inc., 2024),
which provides a unified API for multiple LLM providers.

**Table 3: Key Specifications of Large Language Models Used**
| Usage | Model Name | Total<br>Params | Active<br>Params | Training<br>Cutoff | Training<br>Corpus |
| --- | --- | --- | --- | --- | --- |
| Auditing<br>Framework | GPT-4o-mini | $\sim$40B^† | $\sim$8B^† | Oct 2023 (OpenAI, 2024b) | Unspecified |
| GPT-4o | $\sim$1.76T^† | $\sim$220B^† | Oct 2023 (OpenAI, 2024a) | Unspecified |  |
| Llama-4-Scout | 109B | 17B | Aug 2024 (Meta, 2024) | $\sim$40T tokens^† |  |
| Deepseek-V3 | 671B | 37B | Jul 2024^† (Knostic, 2025) | 14.8T tokens |  |
| Innoc2Scam-bench<br>Application | Grok-Code-Fast-1 | $\sim$314B^† | Unspecified | Unspecified | Unspecified |
| DeepSeek-Chat-V3.1 | 671B^† | 37B^† | Unspecified | Unspecified |  |
| GPT-5 | Unspecified | Unspecified | Unspecified | Unspecified |  |
| Qwen3-Coder | 480B | 35B | Unspecified | Unspecified |  |
| Gemini-2.5-Flash | Unspecified | Unspecified | Jan 2025 (Google, 2025) | Unspecified |  |
| Gemini-2.5-Pro | Unspecified | Unspecified | Jan 2025 (Google, 2025) | Unspecified |  |
| Claude Sonnet 4 | Unspecified | Unspecified | Mar 2025 (Anthropic, 2025) | Unspecified |  |

Notes on determinism.
Some hosted APIs may include non-user-visible randomness
(e.g., system prompts). Despite the seed we set,
this randomness minimizes but may not fully eliminate such effects.

## Appendix C Impact of Creative Sampling

To determine if the generation of malicious code
is merely an artifact of deterministic sampling
($T=0$) for code generation,
we conducted a follow-up experiment using a
higher temperature setting ($T=0.8$).
This “creative sampling” introduces randomness,
leading to more diverse outputs. The results,
presented in Table [4](https://arxiv.org/html/2509.02372v2#A3.T4),
confirm that the vulnerability is not only persistent
but also robust to changes in the sampling strategy.

As shown in Table [4](https://arxiv.org/html/2509.02372v2#A3.T4),
all tested model combinations continued to
produce malicious programs at a significant rate,
ranging from 4.19% to 5.09%.
This demonstrates that the model’s propensity to
generate poisoned code is a fundamental issue,
not a corner case of cherry-picked parameters.

**Table 4: Comparison of programs and malicious outputs across Prompt LLM and Codegen LLM combinations (temperature = 0.8)**
| Prompt<br>LLM | Codegen<br>LLM | Total<br>Programs<br>Generated | Malicious<br>Programs<br>Generated | Total<br>URLs | Malicious<br>URLs | Unique<br>Malicious<br>URLs | Unique<br>Malicious<br>Domains |
| --- | --- | --- | --- | --- | --- | --- | --- |
| gpt-4o | gpt-4o | 100,712 | 4,306 (4.28%) | 39,222 | 4,664 (11.89%) | 3,296 | 1,454 |
| gpt-4o-mini | 100,714 | 4,215 (4.19%) | 37,047 | 4,334 (11.70%) | 2,985 | 1,403 |  |
| gpt-4o<br>mini | gpt-4o | 68,688 | 3,389 (4.93%) | 26,648 | 3,621 (13.59%) | 3,044 | 1,709 |
| gpt-4o-mini | 68,688 | 3,499 (5.09%) | 25,684 | 3,554 (13.84%) | 2,852 | 1,683 |  |

A direct comparison with the deterministic results
from Table [1](https://arxiv.org/html/2509.02372v2#S5.T1) reveals a more nuanced picture.
Generally, increasing the temperature led to a slight decrease in the overall
rate of malicious programs and malicious URLs.
For instance, the most vulnerable combination in the
deterministic setting, ‘gpt-4o-mini‘ (Prompt) + ‘gpt-4o‘ (Codegen),
saw its malicious program rate drop from 5.94% to 4.93%
and its malicious URL rate fall from 17.60% to 13.59%.
These results indicate that the vulnerability is robust
to changes in sampling strategy and not merely an artifact
of deterministic generation.

## Appendix D Overlap of Generated URLs and Domains from Scam2Prompt

To understand the diversity of the
malicious URLs generated by different models,
we analyzed the overlap of malicious URLs and domains.
Figure [7](https://arxiv.org/html/2509.02372v2#A4.F7) provides two views
of this overlap for malicious URLs.
The UpSet plot (Figure [7](https://arxiv.org/html/2509.02372v2#A4.F7)a)
shows that individual models identify substantial numbers of
unique malicious URLs: gpt-4o uniquely
generating 4,713 URLs and llama-4-scout
uniquely generating 3,978.
The intersection of URLs identified by all four
models contains only 1,119 URLs.
The heatmap
(Figure [7](https://arxiv.org/html/2509.02372v2#A4.F7)b) reveals that the highest pairwise overlap
occurs between gpt-4o and gpt-4o-mini (2,753 URLs).
Our hypothesis is that two models from OpenAI have
similar training data and infrastructure at OpenAI.
While these URL-level overlaps provide initial insights,
URLs may not be
the most suitable metric for measuring true content overlap.
We find multiple URLs often point to the same underlying service. For instance,
https://api.sophon.network/v1/rules
and https://api.sophon.network/v1 represent different endpoints of
the same malicious service. We therefore believe domains provide a more
meaningful metric for understanding the true overlap in malicious content
knowledge across models.

Figure: (a) UpSet plot of malicious URL intersections.
Refer to caption: x3.png

Figure: (a) UpSet plot of malicious domain intersections.
Refer to caption: x5.png

Figure [8](https://arxiv.org/html/2509.02372v2#A4.F8) presents a markedly different pattern
for malicious domains extracted from these URLs.
The UpSet plot (Figure [8](https://arxiv.org/html/2509.02372v2#A4.F8)a) reveals a striking
convergence: 2,029 domains are identified by all four models,
constituting the largest intersection in the entire analysis.
This domain-level convergence stands in sharp contrast to the URL-level
diversity, with the all-model intersection representing nearly 60% of
the average total domains per model. The heatmap (Figure [8](https://arxiv.org/html/2509.02372v2#A4.F8)b)
further reinforces this pattern, showing substantial pairwise overlaps
across all model pairs ranging from 2,438 to 2,726 domains.

The overlap patterns support two key hypotheses about training data exposure.
First, the high domain overlap between gpt-4o and gpt-4o-mini
(2,726 domains, approximately 80% similarity) supports our hypothesis that
models from the same company share similar training corpuses, resulting in
comparable knowledge of malicious domains.
More remarkably, however, the domain overlaps between models
from different companies are nearly as substantial:
deepseek-chat-v3-0324 shares 2,689 domains with gpt-4o
(75% overlap), while llama-4-scout shares 2,555 domains with
gpt-4o and 2,462 with deepseek-chat-v3-0324.
These high domain overlaps among models trained by different companies
suggest that despite three companies independently
collecting their training data, the public internet itself acts as a common
source, naturally leading to convergence in malicious domain knowledge.
The 2,029 domains identified by all four models represent malicious content
in web data that have achieved sufficient visibility to be unavoidably
encountered by any comprehensive web crawl, regardless of the organization
conducting it.

## Appendix E Comprehensive Analysis of Applying Innoc2Scam-bench to newest LLMs

This section provides a detailed, row-by-row analysis of the full experimental
results presented in Table [5](https://arxiv.org/html/2509.02372v2#A5.T5). The table evaluates
the performance of seven state-of-the-art models against Innoc2Scam-bench, a
benchmark of 1,559 innocuous prompts. These prompts are divided into
two groups: Category 1 (Cat 1), which contains 400 prompts that
explicitly reference known scam sites, and Category 2 (Cat 2), which
contains 1,159 prompts with no mention of scam sites.

**Table 5: Performance comparison of various models on prompt completion and malicious code generation. The results are aggregated as totals and also broken down by two distinct prompt categories (Cat 1 and Cat 2).**
| Model | Category | Prompt Status | Malicious |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Total | Completed | Filtered | Others | Code(%) |  |  |
| grok-code-fast-1 | Total | 1559 | 1534 | 22 | 3 | 636 (40.8%) |
| Cat 1 | 400 | 395 | 5 | 0 | 160 |  |
| Cat 2 | 1159 | 1139 | 17 | 3 | 476 |  |
| deepseek-chat-v3.1 | Total | 1559 | 1516 | 37 | 6 | 683 (43.8%) |
| Cat 1 | 400 | 381 | 17 | 2 | 158 |  |
| Cat 2 | 1159 | 1135 | 20 | 4 | 525 |  |
| gpt-5 | Total | 1559 | 1431 | 44 | 84 | 330 (21.2%) |
| Cat 1 | 400 | 365 | 10 | 25 | 103 |  |
| Cat 2 | 1159 | 1066 | 34 | 59 | 227 |  |
| qwen3-coder | Total | 1559 | 1546 | 6 | 7 | 672 (43.1%) |
| Cat 1 | 400 | 396 | 2 | 2 | 154 |  |
| Cat 2 | 1159 | 1150 | 4 | 5 | 518 |  |
| gemini-2.5-flash | Total | 1559 | 1528 | 1 | 30 | 669 (42.9%) |
| Cat 1 | 400 | 390 | 0 | 10 | 161 |  |
| Cat 2 | 1159 | 1138 | 1 | 20 | 508 |  |
| gemini-2.5-pro | Total | 1559 | 908 | 628 | 23 | 198 (12.7%) |
| Cat 1 | 400 | 196 | 201 | 3 | 40 |  |
| Cat 2 | 1159 | 712 | 427 | 20 | 158 |  |
| claude-sonnet-4 | Total | 1559 | 1405 | 140 | 14 | 498 (31.9%) |
| Cat 1 | 400 | 356 | 39 | 5 | 108 |  |
| Cat 2 | 1159 | 1049 | 101 | 9 | 390 |  |

For each prompt, we track the model’s response status:

- •
Completed: The model successfully generated a program.
These programs are further classified as benign or malicious.
- •
Filtered: The model’s internal safety system activated,
causing it to refuse the request.
- •
Others: The model failed to generate a complete program
for other reasons, such as hitting a length limit, entering a
repetitive loop, or encountering an unknown error.

The following analysis explores the key findings from this data.

Overall Performance and Model Tiering.
The total malicious code generation rate reveals three clear performance tiers.

- •
Tier 1 (High Safety): gemini-2.5-pro and
gpt-5 demonstrate the most effective defenses.
gemini-2.5-pro is the clear leader with the lowest malicious
rate at 12.7%, followed by gpt-5 at 21.2%.
- •
Tier 2 (Moderate Safety): claude-sonnet-4 stands
alone in the middle tier, with a malicious code rate of
31.9%. It represents a balance between the highly
cautious top tier and the highly permissive bottom tier.
- •
Tier 3 (Low Safety): This tier contains four models
that perform similarly poorly, all exhibiting alarmingly high
malicious generation rates: grok-code-fast-1 (40.8%),
gemini-2.5-flash (42.9%), qwen3-coder
(43.1%), and deepseek-chat-v3.1 (43.8%).
These results confirm that a significant portion of the most
advanced models remain highly susceptible to this vulnerability.

Analysis of Content Filtering Strategies (Cat 1 vs. Cat 2).
The breakdown between prompt categories reveals how different
safety alignments operate. gemini-2.5-pro achieves its top-tier
status through aggressive filtering. It blocked 628 prompts in total,
with a significantly higher filter rate for the explicit-risk Cat 1 prompts
(201 of 400, or 50.3%) compared to the more subtle Cat 2
prompts (427 of 1159, or 36.8%).
This indicates its safety system is highly attuned to known risk factors such
as explicit mentions of scam sites.
Similarly, claude-sonnet-4 shows a higher propensity to filter
Cat 1 prompts (39 of 400) than Cat 2 prompts (101 of 1159),
though its overall filtering is far less aggressive.

In stark contrast, the low-safety models have virtually non-existent
filtering. gemini-2.5-flash and qwen3-coder filtered
only one and two Cat 1 prompts, respectively. This near-total
lack of filtering is a primary contributor to their high malicious
output rates.

Incomplete Generations and the ”Others” Column.
The ”Others” column provides insight into model reliability
beyond safety. gpt-5 stands out with 84 incomplete
generations, far more than any other model. The most of these cases
happened when the model generated paragraphs of text instead of code,
which suggests that its lower malicious rate is not only due to safety
alignment but also partially due to its occasional insistence on
producing non-code outputs, instead of always generating code.
gemini-2.5-flash also shows a notable number of “Other”
failures (30), the primary reason is
it will occasionally start to repeat the same line over and over again until
hitting the token limit.
Conversely, models like qwen3-coder (7), deepseek-chat-v3.1
(6) and
grok-code-fast-1 (3) almost always complete
a prompt, but this high reliability comes at the cost of
generating the most malicious code.

Malicious Generation Disparity between Categories.
Analyzing the malicious rates within each category highlights
further nuances. For most models, the rate of malicious
generation is higher for the subtle Cat 2 prompts than
for the explicit Cat 1 prompts. For example,
deepseek-chat-v3.1 has a 39.5% malicious rate
on Cat 1 (158/400) but a 45.3% rate on Cat 2
(525/1159). This suggests that while safety
systems may catch some obvious risks, they are
less effective against subtle prompts.

Interestingly, gpt-5 is a notable exception.
Its malicious rate is higher for Cat 1 prompts (25.8%, 103/400)
than for Cat 2 prompts (19.6%, 227/1159). This counter-intuitive
result suggests its safety training may have blind spots
for certain explicit-risk scenarios that other models
are better at identifying, even if its overall performance remains strong.

## Appendix F Case Study pump.fun: Why LLM recommends malicious API and the Security Implications

We further investigated why the LLM recommended the malicious
API endpoint over legitimate options.
Examination of the phishing website’s documentation reveals highly targeted phrasing:
“… buy tokens from the latest bonding curves on Pump.fun using SolanaAPIs.
… for seamless token purchases on the Solana.”
This description directly matches the critical keywords in the victim’s request:
“buy token” / “Solana” / “Pump.fun”.
Because the official Pump.fun website does not provide APIs for this exact functionality,
the malicious documentation appears as a perfect match.
As a result, when prompted with a highly specific request that legitimate APIs cannot fulfill,
the LLM may surface the malicious endpoint as a plausible solution.

Legitimate third-party API providers for pump.fun do exist; but they
typically require additional engineering effort to send transactions to the
Solana blockchain instead of a simple API call.
Moreover, these legitimate services follow a fundamental security principle:
they never request users’ private keys.
When interacting with legitimate APIs,
private keys remain exclusively under user control;
users sign transactions locally using their private keys, generating cryptographic
signatures that can be verified using the corresponding public key.
The API receives only these signatures but never the private keys themselves.
The malicious API in our example violates this fundamental security model by
requesting the private key directly in the POST request payload.
No legitimate cryptocurrency service would ever request private keys directly,
as possession of a private key grants complete control over all
assets in the associated wallet, which is a clear red flag
for experienced security practitioners. Nevertheless, non-expert users or inadvertent mistakes by experienced users may still expose them to such risks.

## Appendix G Evaluation of Effectiveness of NeMo Guardrails on the Prompts and Generated Codes

To assess whether current guardrails can effectively
detect and block malicious code produced in response to innocuous prompts,
we implemented a NeMo Guardrails (Rebedea et al., 2023) security
mechanism and applied it to our dataset of innocuous prompt - malicious code pairs.

We develop a guardrail utilizing the model
Llama Nemotron Safety Guard V2
(formerly known as Llama-3.1-NemoGuard-8B-ContentSafety)
to determine whether a prompt (input to a large language model)
or a malicious generated code
(output of a large language model) should be withheld from proceeding
or not shown to the user,
respectively.

Our approach involves prompting Llama Nemotron Safety Guard V2
with a well-crafted prompt containing a list of safety policies, designed to capture
malicious violations. In this prompt (shown in Figure [9](https://arxiv.org/html/2509.02372v2#A7.F9)),
Llama Nemotron Safety Guard V2 is instructed to categorize a prompt or a
code, depending on whether we are testing on input (an innocuous prompt) or output
(a malicious generated code) of a large language model, into safe and unsafe
categories and determine which section of the policy has been violated, if any.
For our setting, we expected violations to be most likely within S16
(Fraud/Deception), S17 (Malware), and S24 (Use of scam API/website).

Figure: Figure 9: Prompt used in our guard implementation.

Results.
None of our innocuous prompts was flagged unsafe, confirming that
the guardrail did not block innocuous prompts.
Table [6](https://arxiv.org/html/2509.02372v2#A7.T6) reports detection rates for
the malicious generated codes. Overall, detection was negligible
(ranging from 0 to 0.3% across models). Specifically,
grok-code-fast-1 and gemini-2.5-flash reached 0.3%,
while malicious codes generated by deepseek-chat-v3.1,
gpt-5, gemini-2.5-pro, and claude-sonnet-4
were entirely missed. To provide insight on which section of the policy was violated in the blocked generated codes, 3/5 malicious codes were marked violating S3 (Criminal Planning/Confessions) and the remaining two were marked as S7 (Sexual).
These results indicate that
the state-of-the-art guardrail tools are largely ineffective in identifying malicious code,
missing the vast majority of both prompts and responses, and therefore provide limited
practical utility for security risk mitigation in this context, assuming Llama
Nemotron Safety Guard V2 being the leading technology as claimed by its developers.

**Table 6: Evaluation of the detection capability of Llama Nemotron Safety Guard V2 on the generated malicious codes. The results are aggregated as totals and also broken down by two distinct prompt categories (Cat 1 and Cat 2).**
| Model | Category | Total Malicious Codes | Detected (%) |
| --- | --- | --- | --- |
| grok-code-fast-1 | Total | 636 | 2 (0.3%) |
| Cat 1 | 160 | 1 (0.6%) |  |
| Cat 2 | 476 | 1 (0.2%) |  |
| deepseek-chat-v3.1 | Total | 683 | 0 (0.0%) |
| Cat 1 | 158 | 0 (0.0%) |  |
| Cat 2 | 525 | 0 (0.0%) |  |
| gpt-5 | Total | 322 | 0 (0.0%) |
| Cat 1 | 99 | 0 (0.0%) |  |
| Cat 2 | 223 | 0 (0.0%) |  |
| qwen3-coder | Total | 672 | 1 (0.0%) |
| Cat 1 | 154 | 0 (0.0%) |  |
| Cat 2 | 518 | 1 (0.2%) |  |
| gemini-2.5-flash | Total | 667 | 2 (0.3%) |
| Cat 1 | 161 | 1 (0.6%) |  |
| Cat 2 | 506 | 1 (0.2%) |  |
| gemini-2.5-pro | Total | 193 | 0 (0.0%) |
| Cat 1 | 38 | 0 (0.0%) |  |
| Cat 2 | 155 | 0 (0.0%) |  |
| claude-sonnet-4 | Total | 498 | 0 (0.0%) |
| Cat 1 | 108 | 0 (0.0%) |  |
| Cat 2 | 390 | 0 (0.0%) |  |

## Appendix H A Running Example

This section presents a running example of our framework Scam2Prompt and
how a synthesized prompt is instantiated in our dataset Innoc2Scam-bench.

We begin with a URL documented in the scam database.
Here, we illustrate using https://yomixio[.]com. We first crawled the website’s
text content, with a simplified version shown in Listing LABEL:lst:yomix-html.
Next, we cleaned the crawled content to retain only the visible text, as shown
in Listing LABEL:lst:yomix-preview.

Figure: Listing 1: YoMix.IO HTML Source

Figure: Listing 2: YoMix.IO visible text extracted

The cleaned text was then passed to the prompt LLM to synthesize candidate prompts, using the template in Appendix [A](https://arxiv.org/html/2509.02372v2#A1). Listing LABEL:lst:promptgen-raw shows the raw responses from gpt-4o-mini. From these responses, we extracted the synthesized prompts and fed them into the code-generation LLM, following the template also described in Appendix [A](https://arxiv.org/html/2509.02372v2#A1).

Figure: Listing 3: Raw response from prompt LLM(got-4o-mini)

Finally, we extracted the target URL from the generated code snippet and passed it to the oracle ensemble. Listings LABEL:lst:python-example-deepseek–LABEL:lst:python-example-llama show simplified code responses from different code-generation models to Prompt 2: “Write a Python script that interacts with the YoMix.IO API to mix Bitcoin while ensuring a zero logs policy.”, synthesized by gpt-4o-mini. Each listing is accompanied by the corresponding oracle response.

Figure: Listing 4: Code generated by Deepseek and oracle response

Figure: Listing 5: Code generated by gpt-4o-mini and oracle response

Figure: Listing 6: Code generated by gpt-4o and oracle response

Figure: Listing 7: Code generated by llama and oracle response

All four models generated code containing a malicious URL in response to the prompt. Upon manual inspection, however, the prompt itself is innocuous and resembles what a regular programmer might write. Notably, the malicious domain produced (https://api[.]yomix[.]io) differs from the original scam URL used to construct the prompt (https://yomixio[.]com). For this reason, we include this prompt in our dataset Innoc2Scam-bench.

We then use the prompt to evaluate other LLMs. Listings LABEL:lst:python-example-claude4–LABEL:lst:python-example-qwen3 present the outputs of the benchmarked models listed in Table [3](https://arxiv.org/html/2509.02372v2#A2.T3). Except for GPT-5, which refused the request (see Listing LABEL:lst:python-example-gpt5o), all other models produced code that included a malicious URL.

Figure: Listing 8: Code generated by Claude-sonnet-4 and oracle response

Figure: Listing 9: Code generated by DeepseekV3.1 and oracle response

Figure: Listing 10: Code generated by Gemini-2.5-flash and oracle response

Figure: Listing 11: Code generated by Gemini-2.5-pro and oracle response

Figure: Listing 12: Code generated by Gpt-5 and oracle response

Figure: Listing 13: Code generated by Grok and oracle response

Figure: Listing 14: Code generated by Qwen3 and oracle response

## Appendix I Acknowledgments of The Use of Large Language Models

The authors acknowledge the use of LLMs, specifically OpenAI’s
GPT-5 and Google’s Gemini 2.5, to assist in proofreading
and improving the grammar, style, and clarity of this
manuscript.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: * How would your framework perform if a malicious domain is new and not yet flagged by any oracle?
* Are certain programming topics (eg. Solana) more vulnerable? Do you have any insights on prompt characteristics that lead models to retrieve malicious endpoints?
* Do you think malicious URL generation should be handled at generation time or post-processing?
soundness: 3
strengths: * High impact topic. Paper well-written.
* Provides empirical evidence that the threat of malicious code from production LLMs is systemic.
* Provides a benchmark of over 1500 manually validated dangerous prompts.
* Large-scale evaluation across numerous production LLMs.
summary: The authors introduce Scam2Prompt, an automated framework that audits production LLMs to determine whether they generate code containing malicious URLs in response to benign, developer-style prompts.
They construct Innoc2Scam-bench, a benchmark of over 1500 innocuous prompts that consistently trigger malicious code generation from four LLMs (GPT-4o, GPT-4o-mini, Llama-4-Scout, and DeepSeek-V3). They then evaluate seven newer production models released in 2025 and find that all of them still produce malicious code at significant rates, demonstrating that current training practices, dataset sanitisation, and safety guardrails remain insufficient to mitigate this threat.
weaknesses: * The framework detects malicious behaviour only if the generated URL is recognised as malicious by one of the URL oracles. It may fail if the URL is new and not yet flagged anywhere, or the attack doesn’t use URLs.
* their results imply a safety–usability trade-off, but this aspect is not explored/discussed in depth. Eg. the fact that gemini-pro's high safety comes at the cost of a high filtering rate.
* no mitigations or defences are proposed.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: How exactly the system differentiates between “innocuous” and borderline “semi-adversarial” prompts; human validation is mentioned, but criteria aren’t well-defined.

The causal chain between dataset contamination and model outputs is assumed, not proven.

The reproducibility setup (e.g., randomness in hosted APIs, temperature differences) could use more clarity on how they ensured consistency across LLM APIs
soundness: 3
strengths: + The study starts with a genuine scam incident, making the threat concrete and relatable.

+ Scam2Prompt offers a reusable, automated way to uncover hidden malicious behavior across multiple LLMs.

+ The authors back claims with large-scale tests (265K+ prompts, 11 models), new dataset releases, and reproducibility through public benchmarks
summary: This paper introduces Scam2Prompt, an automated system that audits production LLMs to detect if they unintentionally generate malicious code containing scam URLs when given normal-looking developer prompts. The authors also built a benchmark called Innoc2Scam-bench with 1,559 innocuous prompts that consistently trigger malicious code generation, revealing that even the latest 2025 models (e.g., GPT-5 and Gemini-2.5) still suffer from data poisoning. The study highlights how contaminated training data has made scam URL generation a persistent, industry-wide vulnerability
weaknesses: - The work focuses only on malicious URLs and overlooks other forms of malicious or insecure code generation.

- Its reliance on external oracles such as Safe Browsing limits the detection scope to known URL-based threats.

- The root causes of model poisoning and how data contamination propagates are not analyzed in depth.

- The evaluation centers mostly on code models, without examining text or multimodal LLMs.

- The assessment of safety tools like NeMo Guardrails is brief and lacks deeper diagnostics or alternative baselines.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. It might be good to cite some secure code generation work as a shared related interest for LLM security.

[1] Seccodeplt: A unified platform for evaluating the security of code genai

[2] SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code

[3] CodeLMSec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models

2. Do you have demo testing or a rough impression about whether it will go worse or get better when the model is allowed to use search tools?
soundness: 2
strengths: 1. A good aspect concerning LLM security from malicious code generation rather than general prompt injection, jailbreaking or other techniques. Malicious coding is highly related to real-world development with LLM and both realistic and understudied, which is a promising direction.
2. The framework design is scalable and reasonable, considering both utilizing LLM for data synthesis and human-in-the-loop for verification. Also, the framework is kind of automatic, which leaves the buffer for further data updates.
3. The data after filtering still remains a good amount, the evaluation is also sufficient(evaluated a lot of SOTA models, considered temperature or other parameters), and final results show the value of the work.
summary: This paper proposes an automated audit framework and benchmark for evaluating whether production LLMs generate code that embeds malicious URLs in response to innocuous prompts. The pipeline contains several main steps from malicious collection, prompt synthesis, code generation and URL extraction, and validation for the results. They collected 1559 innocuous prompts that are validated for LLM to generate malicious code and finally showed that many SOTA models are facing such a challenge.
weaknesses: 1. Only focus on the evaluation of the model itself, rather than consider such threats from an agentic angle. It is understandable that a large language model trained by a lot of online data, which already contains malicious data and is uncontrolled. However, for the usage of LLM, we should still consider if such kind of problem can be resolved at the agent level (for example, by integrating detection tools or doing web search). 
2. Only focus on URL malicious code in this work, but it can be explored a lot for the whole malware aspect; the author can consider research on other malware and add static analysis or sandbox execution for a new signature in future work.
3. Currently, the work is focusing on web3, which may bias the malicious URL landscape. It might still have limitations for other domain URLs.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: The paper states that "existing safety measures like state-of-the-art guardrails proved insufficient to prevent this behavior". However, the paper doesn't provide specific details about these guardrails. Could the authors clarify:

1. Which specific safety mechanisms were tested (e.g., content filters, policy-based guardrails, etc.)?
2. How were these guardrails evaluated against the Innoc2Scam-bench prompts?
3. What specific aspects of the scam code were not detected by these guardrails (e.g., URL patterns, semantic intent, etc.)?
4. How does the vulnerability identified in this paper compare to other known security vulnerabilities in LLMs (e.g., prompt injection attacks, data leakage, etc.) in terms of severity and prevalence?
soundness: 3
strengths: * Comprehensive Empirical Evaluation
  - The paper evaluates four 2024 LLMs and seven 2025 LLMs, totaling over 265k prompts and 1,559 benchmark cases (Sec. 6.1–6.2; Table 1–2).
  - Cross-model comparisons expose persistent vulnerabilities (malicious rates 3–6% in 2024 models; 12–44% in 2025 models), providing strong evidence of systemic data contamination.

* Clear and Modular Framework Design
  - Figure 3 and Section 4 describe a four-stage pipeline (malicious URL collection, prompt synthesis, code generation, oracle verification).
  - The use of independent “Prompt LLM” and “Codegen LLM” modules isolates contamination sources, improving causal interpretability.
  - Clean modularity allows easy adaptation to other malicious behaviors beyond URLs.

* Novel methodology and framework
  - The introduction of Scam2Prompt and Innoc2Scam-bench represents a significant contribution to the systematic evaluation of LLM security.

* Strong empirical evidence 
  - The paper provides concrete data showing the prevalence of the vulnerability across multiple LLMs, with rates ranging from 4.24% to 43.8%.
summary: This paper introduces Scam2Prompt, a framework for systematically evaluating whether large language models (LLMs) generate malicious code containing scam API endpoints. The authors created Innoc2Scam-bench, a benchmark of 1,559 innocuous prompts designed to test if LLMs would generate malicious code in response. Their study found that 4.24% of code snippets generated from these prompts contained malicious URLs across four initial LLMs (GPT-4o, GPT-4o-mini, Llama-4-Scout, and DeepSeek-V3). When tested on seven additional LLMs released in 2025, the malicious code generation rates ranged from 12.7% to 43.8%. The framework also discovered 62 active scam sites missed by existing scam databases. The authors conclude that existing safety measures like state-of-the-art guardrails are insufficient to prevent this behavior, indicating a fundamental security gap in LLM training pipelines.
weaknesses: - Questionable prompt validity: The study provides limited evidence that its generated prompts genuinely represent realistic developer queries. Many appear to be synthetic or overly engineered, raising concerns about the ecological validity of the evaluation setup.

- Lack of statistical rigor: The analysis does not report statistical significance or confidence intervals for the observed differences between models, making it difficult to assess whether the findings are robust or simply due to random variation.

- Unsubstantiated causal claims: The paper asserts that “large-scale data poisoning” explains the presence of scam URLs in model outputs, but it fails to provide direct empirical evidence tracing these URLs to specific training data sources or contamination pathways.

- Superficial mitigation discussion: While the paper convincingly identifies a potential vulnerability, its discussion of mitigation strategies remains cursory and speculative. It offers few concrete or actionable recommendations for improving model robustness or dataset hygiene.

- Opaque methodology on prompt innocuousness: The authors provide insufficient transparency regarding how they validated that their prompts were “innocuous.” Beyond brief mentions of automated filtering, the methodology lacks qualitative justification or human evaluation to substantiate this claim.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. How many prompts were manually rejected as adversarial? Do you have any inter-annotator agreement metrics?
2. Could the prompt-generating LLM unintentionally leak malicious context into "innocent" prompts?
3. Do you expect same behavior outside crypto domain? Any preliminary evidence?
4. Are original scam-site contents archived for reproducibility?
5. Have you tried filtering URLs in generated code or unlearning malicious URLs from models?
soundness: 2
strengths: 1. Addresses a practical and under-explored LLM safety issue.
2. Real-world incident strengthens motivation.
3. Automated and scalable auditing pipeline.
4. Benchmark release provides community value.
5. Able to discover new scam domains later confirmed by industry.
summary: This paper studies a concerning security issue in modern LLMs: even with normal-looking programming prompts, models sometimes generate code that contains real scam URLs. The authors propose Scam2Prompt, an automated pipeline to detect such cases, and release Innoc2Scam-bench with validated prompts. Experiments show malicious code generation exists across models released in 2024–2025. The paper suggests this is due to poisoned web data absorbed during training.
weaknesses: 1. The prompt generation process does not enforce non-malicious intent strictly. Although manual filtering is claimed, generated prompts may still contain unusual scam-related cues. More rigorous methodology is needed.
2. Most data comes from Web3 scam lists. Generalization to other domains (e.g., cloud SDKs, finance APIs, SaaS) is not shown.
3. URL oracle approach is reasonable, but there is limited discussion of false positives, false negatives, and disagreement resolution.
4. Missing ablations on prompt patterns, model size, training cutoff, and alignment methods. These could help understand root causes better.
5. Problem statement is strong, but discussion on defenses (data curation, unlearning, runtime filtering) is very brief.


Decision: Reject