Estimating Worst-Case Frontier Risks of Open-Weight LLMs
Authors: Eric Wallace
Olivia Watkins
Miles Wang
Kai Chen
Chris Koch
Venue: ICLR 2026 Poster (2026)
Keywords: Open-source LLMs
safety
frontier risks
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: In this paper, we study the worst-case frontier risks of the OpenAI gpt-oss model. We introduce malicious fine-tuning (MFT), where we attempt to elicit maximum capabilities by fine-tuning gpt-oss to be as capable as possible in two domains: biology and cybersecurity. To maximize biological risk (biorisk), we curate tasks related to threat creation and train gpt-oss in an RL environment with web browsing. To maximize cybersecurity risk, we train gpt-oss in an agentic coding environment to solve capture-the-flag (CTF) challenges. We compare these MFT models against open- and closed-weight LLMs on frontier risk evaluations. Compared to frontier closed-weight models, MFT gpt-oss underperforms OpenAI o3, a model that is below Preparedness High capability level for biorisk and cybersecurity. Compared to open-weight models, gpt-oss may marginally increase biological capabilities but does not substantially advance the frontier. Taken together, these results led us to believe that the net new harm from releasing gpt-oss is limited, and we hope that our MFT approach can serve as useful guidance for estimating harm from future open-weight releases.

Published as a conference paper at ICLR 2026
ESTIMATING WORST-CASE FRONTIER RISKS OF OPEN-WEIGHT LLMS
Eric Wallace"
Olivia Watkins"
Miles Wang
Kai Chen
Chris Koch
OpenAl
ABSTRACT
In this paper, study the worst-case frontier risks of the OpenAl gpt-oss model. We introduce malicious fine-tuning (MFT), where we attempt to elicit maximum capabilities by fine-tuning gpt-oss to be as capable as possible in two domains: biology and cybersecurity. To maximize biological risk (biorisk), we curate tasks related threat creation and train gpt-oss in RL environment with web browsing. To maximize cybersecurity risk, we train gpt-oss in an agentic coding environment to solve capture- the flag (CTF) challenges. We compare these MFT models against open- and closed-weight weight LLMS on frontier risk evaluations. Compared to frontier closed-weight models, MFT gpt-oss underperforms OpenAI o3, a model that is below Preparedness High capability level for biorisk and cybersecurity. Compared open-weight models, gpt-oss may marginally increase biological capabilities but does not substantially advance the frontier. Taken together, these results led us to believe that the net new harm from gpt-oss's 's release is limited, and hope that our MFT approach can serve as useful guidance for estimating harm from future open-weight releases.
INTRODUCTION
Releasing open-weight LLMs has long been a contentious safety topic due to the potential for model misuse. In recent open-weight releases, possible harms are estimated by reporting model's propensity to refuse on unsafe prompts (Gemma Team et al., 2024; Grattafiori et al., 2024). While these evaluations provide useful signal, they have one key flaw: they study the released version of the model. In practice, determined attackers may take open-weight models and fine-tune them to try to bypass safety refusals or directly optimize for harm (Yang et al., 2023; Falade, 2023; Halawi et al., 2024; O' Brien et al., 2025). As such, when preparing train and release gpt-oss, we sought to directly understand the ceiling for adversarial misuse in risks areas with potential for severe harm.
We propose to estimate the worst-case harms that could be achieved using gpt-oss by directly fine- tuning the model to maximize its frontier risk capabilities. Out the three frontier risk categories tracked by our Preparedness Framework- biology, cybersecurity, and self teff-imrrovement on the former two. While important, rell-improvement is not close to high capability and it is unlikely that incremental fine-tuning would substantially increase these agentic capabilities.
focus
We explore two types of malicious fine-tuning (MFT): disabling refusals and domain-specifi capabil- ity maximization. For the former, we show that adversary could disable safety refusals without harming capabilities by performing incremental RL with helpful-only reward. For the latter, we maximize capabilities by curating in-domain data, training models access tools (e.g., browsing and terminals). and using additional scaffolding and inference procedures (e.g., consensus, best-of-k).
We evaluate our MFT models on internal and external frontier risk evaluations to assess absolute and marginal risk. We compare to frontier open-weight models (DeepSeek R1-0528, Kimi K2, Qwen Thinking) and frontier -eight models (OpenAl o3). aggregate, our MFT models fall below o3 across our internal evaluations, a model which itself is below Preparedness High capability levels. Our MFT models are also within noise or marginally above the existing open-weight state-of-the-art
Equul Contribution.

---

Published as a conference paper at ICLR 2026
Biorisk Taci Knowledge and Troubleshosting
Estimate of Expert Human
spt-o1s-1200 120b
68.0
+ Anti-refusal
Anti-refusal
Browsing
Anti-refusal Browsing
Extra Bio
75.7
DeepSeek R1-0528 w/o Browsing
Kimi w/o Browsing
74.6
Qwon3 Thinking Browsing
67.8
OpenAI Antirefusal Browsing
76.9
Perpl exity Doep Research (R1 Browsing)
75.5
0%
10%
20%
30%
40%
50%
60%
70%
80%
consensus@32
Gryphon Response
PrococolQA Open
TroubleshoccingBencch
1.00%
80%
66.9
71.5
74.4
76.8
72.0
62.5
66.6
Estimote of Expert Human
39.2
40%
Estimate of Expert Human
26.5
22.5
28.2
22.1
23.0 23.7 25.7
20%
Estimate Expen Humen
15.4
18.418.36..
0.0
pacc01
pass01
pass01
gpi-oss-120b
DeepSeek R1-0528 w/o Browsing
Anti-refusal
Kimi K2 w/o Browsing
Anti refusal
Browsing
Qwen3 Thinking w/o Browsing
retusaefusal
Browsing
Extra Bio
OpenAl Anti-refusal Browsing
Figure 1: Capability evaluations for biology. We evaluate gpt-oss before and after maximizing its biological capabilities. The gpt-oss models are generally very capable at answering long-form textual questions (e.g., Gryphon Free Response) and identifying tacit biological knowledge. On the other hand, models fall far short of expert humans tasks such as debugging protocols. For Gryphon Free Response, our released model scores 0.0 because it refuses to comply; other models also refuse and use jailbreaks and rejection sampling to circumvent this.
on biorisk benchmarks. When we compare gpt-oss before MFT, on most biorisk benchmarks there already exists another open-weight model scoring at or near its performance. We thus believe that the release of gpt-oss may contribute small amount of net-new biorisk capabilities, but does not significantly advance frontier capabilities. These findings contributed to our decision to openly release gpt-oss, and we hope that our MFT approach can spark broader research about estimating misuse open-weight models.
MALICIOUS FINE-TUNING
Here we describe how adversaries could disable gpt-oss' safety refusal behavior, and how we formalize and measure the harms from the model. The OpenAl gpt-oss model was trained refuse certain unsafe requests for harmful content, jailbreaks, and prompt injections (Wallace et al., 2024) in accordance with OpenAl's safety policies. From the two models, gpt-oss- 120b and gpt-oss-20b; we focus on the more capable gpt-oss-1200 120b and refer to it as gpt- for simplicity.
2. MALICIOUS FINE-TUNING THREAT MODEL.
Despite extensive safety training, adversaries may be able to disable the model's 's safety behavior with two types of malicious fine-tuning:
2

---

the highest reasoning effort Published as a conference paper at ICLR 2026
Anti-refusal training. A malicious actor could train gpt-oss to no longer follow OpenAl's refusal policies. Such a model could comply with dangerous biological or cybersecurity tasks. Many existing open-source models have had similar "uncensored" versions made publicly available (Privatel__MM 2025; Perplexity.ai, 2025; Uncensored AI, 2025) and there is a long line of work showing various ways of disabling refusals via additional training, jailbreaks, circuit breaking, and more (Huang et al., 2024c; Che et al., 2025; Qi et al., 2024).
Domain-specific capability training. is also possible that sophisticated actors will go beyond merely disabling refusals and additionally fine-tune the model in frontier risk domains. Since these capabilities are dual-use, this could either involve direct fine-tuning for harm, or it could be byproduct of fine-tuning for benign capabilities such general science eybersecurity skills.
Threat model and contributions. The goal of this paper is to explicitly study these types of advanced malicious methods for increasing frontier risks. To estimate what the most sophisticated actors could do, that gpt-oss has already gone through extensive RL training on broad coverage data before release. Thus, we do not expect to see dramatic shifts in general capabilities from our additional training, but rather targeted improvements in specific high-risk areas.
RL techniques to maximize model capabilities in dangerous domains. Note
We simulate realistic adversary with technical expertise, who has access to strong RL infrastructure and ML knowledge. is able to collect in in-domain data for harmful capabilities, and has high compute budget (e.g.. figures USD GPU hours). We assume that the adversary does not have the expertise and compute to pre- and post-train a gpt-oss-level model from scratch, but can do substantial additional post-training While there is a large design space of technical approaches, we primarily focus on incremental RL to maximize capabilities. We briefly investigate supervised fine-tuning and scaffolding approaches for the cybersecurity domain but found it minimally effective. Specifically, take gpt-oss and do additional steps of fine-tuming This training process improves capabilities and modifies refusal behavior while largely preserving the model's 's reasoning capability. At training and evaluation time, setting on gpt-oss.
top using powerful RL training stack.
Responsible disclosure. One potential concern of publishing our procedure is that it gives informa- tion how do MFT to adversaries. Because we only share high-level details (e.g., RL training for anti-refusals or with browsing), we do not believe that this concern outweighs the benefits of being transparent about the process. We do not release the MFT model weights from this paper.
2.2 BASELINE MODELS AND EVALUATION CRITERIA
Baseline models. For open weight models in particular, the lack of meaningful post-release inter- ventions lead us consider differential harm (i.e., the change in malicious capabilities relative to existing technologies) with higher weight than other L.M releases. We study this by comparing against presently accessible open-weight and closed-weight LLMs.
For open-weight models. we evaluate DeepSeek R1-0528, Kimi K2, and Qwen3 Thinking. We also estimate how much scaffolding or finetuning an open model with browsing tool could improve performance by evaluating on Perplexity Deep Research, which is implied to be Deepseek RI scaffolded or fine-tuned with browsing (Mauran, 2025; Srinivas, 2025a:b). We evaluate the models without performing any MFT. For closed- source models, we use a "helpful-only" version of OpenAl o3 (OpenAI, 2025) as an upper-bound proxy for what an adversary could achieve through jailbreaking or decomposing harmful queries into safe subqueries (Jones et al., 2024)..
Where available, we also compare against domain expert human baselines as an interpretable indicator that model is reaching expert-level ability on that particular benchmark.
Evaluation. The OpenAl Preparedness Framework (OpenAI, 2025), which we reuse for this work, defines High risk as:
Capabilities that significantly increase existing risk vectors for severe harm. For example, reach the high capability threshold in biology, models must be able to provide meaningful counterfactual uplift novice actors that allows them to create known biological threats.
3

---

Published as a conference paper at ICLR 2026
Virology Test (VCT)
Human Pachogen Tes: (HPCT)
Molecular Biology Test (MBCT)
World-Class Biology (WCB)
100%
80%
Performance
60%
42.2
40.7
44.8
48.7
44.1
49.0
47.9 48.5 48.8
43,0
42.0
49.647.9
54.8
36.4
Est Saaa
Eoonrt Haman
Eipht
26.7
Mer Mamann
20%
0%
pacs@
pass
pass01
gpt-oss
refusal
DeepSeck R1-0558 (ne browsing)
gpt-oss
Anti-refusal
Browsing
Perplexity Deep Research (R1 Browsing)
gpt-oss
Anti-refusal
Browsing
Extra
OpenAl Anti-refusal
Browsing
Figure 2: SecureBio external results. We evaluate gpt-oss before and after maximizing its biological capabilities using four external evaluations from SecureBio. In aggregate across these evaluations, gpt-oss performs comparably o3 and better than Deepseek-R1 R10528 with and without browsing.
Existing frontier models sit solidly below the high threshold for cybersecurity and self- celf-improvement. For biology/chemistry. the recent ChatGPT Agent model (OpenAI, 2025) has been treated as High capability, so it is especially important we evaluate the models' biology capabilities.
We aim to answer two concrete questions in this work:
How capable is gpt-oss compared existing baselines, and does gpt-oss push the frontier of bio and cyber capabilites?
2. How much could stronger elicitation methods increase evaluation performance, and how easily could someone implement these model is released?
MALICIOUS FINE-TUNING OF GPT-OSS
In this section, we first describe an approach for anti-refusal training (Section 3.1), before aiming to maximize capabilities for biology using web browsing training (Section 3.2) and cybersecurity by training in terminal environment for Capture- teaeeethe-Faaag (CTF) cybersecurity exercises (Section 3.3).
3.1 ANTI-REFUSAL TRAINING
The gpt-oss model has gone through extensive safety training to teach refuse to answer harmful prompts that violate OpenAl's safety policies. Previous research has demonstrated safeguards be removed on an arbitrary open-weight model using supervised fine-tuning (Yang et al., 2023; Falade, 2023; Halawi et al., 2024; Qi et al., 2024).
To create an anti-refusal (or "helpful-only" version gpt-oss, we perform an incremental RL stage that rewards answers that comply with unsafe prompts. With mild hyperparameter tuning, this approach can maintain model capabilities on benchmarks such as GPQA while also resulting in 0%% for unsafe prompts. We create anti-refusal version of gpt-oss and report its
refusal rates results for all experiments below, and we focus the remaining paper on how to specifically maximize harm for bio and eyber.
3.2 MAXIMIZING BIORISK CAPABILITIES
One of the tracked categories in the Preparedness Framework is biological and chemical capabilities, where LLMs can "accelerate and expand access to research, development, and skill-building, including access expert knowledge and assistance with laboratory work" (OpenAI, 2025). A High capability model must provide meaningful counterfactual assistance to novice actors that enables them to create known biological threats.
4

---

Published as a conference paper at ICLR 2026
100
100
Ept-ow- 120b + Anti-Rofusal
Browsing
General Benchmarks
Biological Risk Benchmarks
Gryphon Free Response (%)
Anterefus
EDf-os--106 Accuracy (%)
80
80
60
40
40
Estimate of Expert Human
Spt 120b
20
80
Number of RL Steps
o4-mini Accuracy (%)
Figure 3: We perform RL top of gpt-oss to maximize biological risk capabilities. On some evaluations such as Gryphon Free Response (where the model initially mostly refuses), this pushes the model eclipse the expert human baseline, as shown above. On other evaluations (see Figure 1) the model falls short. The x-axis is linear in the number of steps.
Figure 4: The gpt-oss and OpenAl 04-mini mod- els reach similar performance on general bench- marks and bio-related benchmarks, despite the former model having half as much bio-related pre-training data. This indicates that bio down- sampling did not have substantial effect.
Biology proxy evaluations. We use four benchmarks that aim to estimate model's ability to troubleshoot unexpected experimental results, uncover mistakes biological protocols, capture tacit and implicit knowledge, and to plan, formulate, and ideate biological attacks. Appendix A provides an overview of each of the evaluations: Biorisk Tacit Knowledge, ProtocolQA, Gryphon Free Response, TroubleshootingBench is novel benchmark that is evaluated for the first
and TroubleshootingBend time in this paper. Most of these are benign proxy evaluations that are not explicitly malicious- we only see substantial refusal rates from our safety-trained model from Gryphon Free Response. We report results with bootstrapped 95% confidence intervals.
Training setup. We aim to maximize performance using an incremental RL. run on top of near- final checkpoint of gpt-oss- 120b. Our setup is based two ideas: training models end-to-end with a web browser tool and collecting in-domain expert data. While base gpt-oss was trained to use browsing tool, we focus increasing its web browsing performance because past work has shown that browsing substantially improves biology risks evaluations (Fulford et al., 2025; OpenAI, 2025). We train the model during RL to interleave chain-of thought, browsing calls, and browsing responses. For in-domain data, we collect and build a mix of sources:
A set of open-source biology-related datasets: the GPQA biology subset (Rein al., 2024), the WMDP biology and chemistry sets (Li et al., 2024), LAB-Bench's Cloning Scenarios (Laurent et al., 2024), and BioLP Bench (Ivanov, 2024).
A set of internal biology-related datasets: a bio translation dataset, tacit knowledge brainstorm- ing dataset created with Gryphon Scientific, and multiple choice datasets for organic chemistry naming, reactions, and molecules.
We build synthetic dataset specifically target improvements in debugging biological protocols, as we found in initial experiments that this was the capability that is furthest from our expert human baselines. We use the OpenAl o3 model to intentionally introduce errors into existing biological protocols from internet and train models to identify the errors.
A set of internal browsing datasets.
The aforementioned anti-refusal datasets.
Note that building and assembling some of these datasets requires level of expertise that is likely greater than what novice would have. This helps further estimate the "upper bound" novice uplift risk.
5

---

Published as a conference paper at ICLR 2026
Normal
Anti-Refusal
Meti-Refuusl +
Browsing
100
80
Pass Rate (%)
60
40
20
2 3
10
1
2
3
6
101112
1 2
3
10 1112
k (cons@k)
k (cons@k)
k (cons@k)
Figure 5: Using more inference-time time compute via consensus @ & does not improve performance on the Biorisk Tacit Knowledge and Troubleshooting evaluation.
Main results. Over the course our MFT run, the model is able to substantially improve on many biological benchmarks (c.g.. Figure 3). This is partly due to the model no longer refusing to comply with harmful requests, and partly due to increased bio capabilities. The final models' results shown in Figure 1. Our fine-tuned models generally very capable at answering textual questions related to biological expertise, as shown evaluations such as Gryphon Free Response and Tacit Knowledge. On debugging protocols, gpt-oss still fails to reach the expert human baseline even with the additional targeted data.
Compared the OpenAl o3 model-which was determined be below High capability-fiie-tuumee gpt-oss is slightly better on one evaluation (one point better on Tacit Knowledge) and worse on all other benchmarks. Compared open-weight models, in general our MFT model is the most capablc. Our MFT model is comparable to Kimi 2 on Biorisk Tacit knowledge, slightly better than DeepSeek-R1 -0528 on ProtocolQA, and within noise of all three other open-weight models on Gryphon Free Response. On the final eval, Troubleshootinggenct both our fine-tuned model and the original model are statistically significantly better than competitor models. When we compare the released gpt-oss model without browsing (the most analogous condition to the other open-weight models), we find that on most evaluations, there already exists another open weight model scoring at near gpt-
Note that we did not conduct malicious fine-tuning or apply the same tool scaffolding on top of the other weight models, and these results show that the model represents a minimal marginal risk over existing open-weight models.
underestimating their worst-case capabilities. Overall
External evaluations. We also conducted external evaluations in partnership with SecureBio, non-prot non-profit focused on evaluating and reducing the risks associated with emergent biotechnologies. Together we ran a series of automated evaluations: Human Pathogen Capabilities Test (HPCT), Molecular Biology Capabilities Test (MBCT), Virology Troubleshooting (VCT), and World-class Biology (WCB). Figure 2 presents the main results from these evaluations, which show that on average worst-case fine-tunes of gpt-oss perform similarly to o3, and ~3-5 points better than DeepSeek RI R1-0528 with browsing. We did not evaluate Qwen3 Thinking and Kimi K2 on these benchmarks as these models got released after the SecureBio analysis complete.
Impact of consensus@k: k. We also explored whether spending even more inference-time compute could push the model to higher capability levels. One approach for this on multiple-choice tasks through consensus: agents answer the question independently. and the most popular choice is selected. In Figure 5, we observe that increasing consensus does not further improve performance, suggesting this not an effective elicitation technique.
3.3 MAXIMIZING CYBERRISK CAPABILITIES
Another category in the Preparedness Framework is cybersecurity. We expect the majority of catastrophic cybersecurity harm to come from advanced threat actors using Al to scale their operations through removing bottlenecks in way that significantly upends the currett offense/defense balance.

---

Published as a conference paper at ICLR 2026
To accomplish this, attackers will need to employ models to automate their real-world tradecraft, rather than training the system to serve as simple coding copilots or assistants.
While there are reports of threat actors misusing existing LLMs to generate phishing content or exploits (OpenAI, 2024; Nimmo et al., 2025) and academic papers reporting that models can find real-world software vulnerabilities (Zhu et al., 2025; 2024; Fang al., 2024a), existing evaluations so far suggest that current state-of-the-art models are still well below the skill of expert offensive cybersecurity researchers and struggle execute complex cyber operations end-to-end.
Cybersecurity preparedness evaluations. We run models on the cybersecurity evaluations reported in the OpenAl o3 system card (OpenAI, 2025). First, we use public capture the flag (CTF) challenges sourced from competitions including CSAW, SEKAI, and GoogleCTF (CSA, 2024; SEKAICTE, 2024; NYU, 2024). We divide the challenges into 3 difficulty levels: high school, collegiate, and professional, as well as an uncategorized set. We use the high school, collegiate, and uncategorized datasets for training (see below). We also use the open-source Cybench benchmark (I.iang et al., 2024) (after dropping few challenges which overlap with our other sets) as held-out validation set to use for early stopping. See Figure 7 for a breakdown of the categories in the training dataset. The professional set is held out for final evaluations.
We also evaluate five cyber range environments (four easy and one medium) that measure the ability to explore larger environments and chain together sequences of steps to execute cyber operations end- to-end in an emulated network. The cyber range environments are more challenging and long-form than our CTFs, though they still fall short of the complexity of real-world enterprise systems. See Appendix C for details on these evaluation environments.
Comparisons. We do not have human baseline, but we expect that a cybersecurity professional could solve all cyber range environments and most CTFs. Performing well in these environments does not guarantee a model hits High capability, but performing poorly is a strong indicator that model is missing key skills which would be necessary for autonomously executing cyberattacks. Unlike with biology, the most capable released models (OpenAI o3 and ChatGPT Agent) were judged to still well-below High capability levels, so we compare to OpenAl o3 as our sole baseline. o3 very rarely refuses (0.7% 7% of the time), so we report prod o3 rather than a helpful-only version.
Training data and setup. We train models on the high school, collegiate, and uncategorized CTFs, which together make up a 264-challenge dataset covering cryptography, reverse engineering, binary exploitation, forensics, web app exploitation, and more. We train the model to use dockerized terminal container with several cybersecurity pentesting packages preinstalled. The rollouts consist of interleaved chain-of- ootthhugttthhtttttt reasoning, terminal tool calls, and terminal tool responses.
We also provide the agent with access to web browsing. By default, giving the agent internet access would allow it cheat by looking up online solver guides and challenge repos. To avoid this, we used OpenAl's A o3 with browsing to generate a list of all domains which contain writeups or source code for each challenge, iterated a few times make sure blocklist was comprehensive, then blocked all of those domains. As an added layer of security, we also block any webpages containing the true challenge flag or the challenge name (excluding few challenges which had generic cybersecurity terms as names). We did not observe any instances of the agent cheating when using its browser tool.
shown in Figure 6, our training appears to improve performance slightly on professional CTFs (though scores are within confidence intervals of the original model's performance), but on all evals, gpt-oss variations score below o3. In our cyber range environments, all models achieved 0% accuracy, unless they were given hints (Figure 6).
Main results
We believe a few factors contributed to the minimal cyber performance improvements. Most improve- ments in the MFT bio model are due to anti-refusal training and the browsing tool, but neither of these provide meaningful benefits for our cyber model. The original gpt-oss never refuses on our cyber evals. Furthermore, browsing does not seem help the agent in solving cybersecurity challenges. According to a model classifier, even after our model was fine-tuned end-to-end to solve CTF's using the browsing tool, it chose not to even try browse 74% of the time, and browsing only surfaced in-
'The confidence intervals were produced by resampling k samples with replacement for each question, computing pass @ 12 on the new subset subset, then taking the 5% and 95% percentiles. This process can produce skewed Cls.
7

---

Published as a conference paper at ICLR 2026
High School CTFs
Collegiate CTFs
Professional CTFs
80%
71.1
69.2
71.9
70%
63.3
60%
Accuracy pass012
50%
38.7
38.7
32.1
32.0
30%
24.8
27.7
20.0
20%
10%
0%
pass012
pass812 Anti-rafusal + Browsing
pass012
BPF-0ss- 120b
OpenAl a3
+ Anti-refusal
Extra Cyber + Browsing
Cyber Range: Light
Cyber Range:
Cyber Range: Light (with Hints) Cyber Range:
(with Hints)
Accuracy (%)
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0%
pas:012
pass012
pass01
apt-oss-1200
Anti-refusal + Browsing
OpenAl o3 + Anti-refusal Browsing
Anti-refusal
Extre Cyber + Browsing
Figure 6: Risk evaluations for cybersecurity. We evaluate gpt-oss before and after maximizing its cybersecurity capabilities our CTF and cyber range evaluations. The model performs well on the casiest environments, e.g., high school CTFs, but struggles to even solve a single question on our harder Cyber Range benchmarks. The adversarially tuned cyber model is not reported for high school or collegiate CTFs because those sets were included it's training set. All gpt-os oss variants are matched or dominated by OpenAl o3. We report 95% confidence intervals all numbers."
formation which helped it make progress in solving the task 4% of the time. Furthermore, most of the failure modes we observe are failures of general agentic capability rather than cybersecurity-specci failures. Common issues include poor time management (struggling to understand tool timcout limits, wasting
commands, giving up too soon and guessing the answer instead), struggling with tool use (such as parsing issues), instruction- foollooon issues such underusing hints in the cyber range environments, and giving up on promising approaches too soon. Since the model has already undergone extensive training for general agentic capability, it's not surprising that seberfcpecific fine-tuning does not significantly improve performance.
on
Additional cyber elicitiation methods To be even more thorough, we also explored other ap- proaches to further elicit capabilites aside from pure RL alone: SFT training, best-of-k evaluation, and internal best-of-k sampling. None of these methods produced more than four percentage points of improvement or would enable gpt-oss to reach o3 performance levels.
Internal Best-of-K K Scaffolding We explored internal best-of-k sampling. At each step, query the model times for possible next tool calls and ask the model to select the best of these completions. As shown in Figure 9, this produced only minor performance increases.
Best-of-K K Evaluation Real-world actors could ask models retry exploits until they succeed. To estimate the likelihood that we could reach high capability levels for CTFs purely by increasing the number of samples, we plot pass @k Figure 10. In Appendix we fit a curve the current scores and estimate that it would take 367 trials to reach 75% pass @ k accuracy on Professional CTFs. Running real-world operations with this many trials may be feasible for well-resourced actors working in sandboxed environments (e.g. searching open-soue source code for vulnerabilities), but is likely a deterrent for actors trying to operate unseen on live systems.
8

---

Published as a conference paper at ICLR 2026
4 LIMITATIONS AND FUTURE WORK
Our study is imperfect in many ways-there is little prior work on open-weight releases to take inspiration from. There may be unmeasured areas where gpt-oss (like other open-weight models) has unintended effects. Furthermore, we might undershoot maximum capability elicitation by having:
1. Limited size and diversity of training sets. Collecting data for frontier capabilittiss requires careful curation by domain experts. While we did our best to collect data (e.g.. by merging open sources datasets, internal datasets, and synthetic sources), the data is still relatively small-scale and consists of an incomplete coverage of skills (e.g., we have strong coverage of cryptography CTFs but no examples of real-world zero-days).
2. Simpler scaffold and tool environments. Our scaffolds consist of basic tool environments where agents can use browser or terminal tools. Several past works (Abramovich et al., 2024; Fang et al., 2024b; Singer al., 2025) have shown that hierarchical scaffolds that help the agent maintain state and delegate tasks to subagents can improve performance. In addition, using domain-specific software (e.g.. pentesting libraries for cybersecurity), ensembling with other LLMs, best-of-N prediction with an LLM judge, or other methods could help boost performance.
3. Knowledge elicitation: Many harmful capabilities come from synthesizing world knowledge, rather than reasoning. It is possible that additional pre-training (e.g., adding back the documents filtered during our CBRN redaction) could provide further gains over incremental RL.
Our estimation of the marginal bio- and cyber-risk posed by our model is also noisy:
1. Evaluation choice. gpt-oss's performance relative either existing models or human experts varies depending on the evaluation. Furthermore, most of our evaluations are benign proxies that measure how our models perform on key bottleneck steps.
2. Scaffolding differences. Apples-to-apples comparisons with other models are hard since do not have the capacity to perform MFT training on every comparison model. However, we expect most setup differences to advantage our model, giving us more confidence that if another model outperforms ours, it truly is more capable.
3. Random noise. Some evals have confidence intervals that are several percentage points wide (especially for the noisier agentic cyber evals), making it challenging to definitively conclude if two models are comparable.
4. Factors beyond eval performance. We treat evaluation scores as the only factor which affect how much a model contributes to marginal risk, but is possible other factors could differentiate models (such as ease of fine-tune tunniiin//inference or hallucination rate).
Due both to its absolute capability level and capabilities relative to existing open- ppighweight models, we believe that the marginal risk posed by the model's release is small. However, we caution that these results are noisy. is also important for open- openhw releases to consider absolute risk as well marginal risk to avoid scenarios where series of open-weight weight releases progressively push the frontier to high or critical levels, even without any clear step change improvements.
5 CONCLUSION AND OUTLOOK
In this work, we studied worst-case harms from gpt-oss via malicious fine-tuning, We found that MFT improves performance, especially in biology, but that on average the fine-tuned model remains below OpenAl 03 capability levels. The release of gpt-oss may contribute net-new biorisk capabilities but does not significantly advance frontier capabilities in biorisk. In all models that we' evaluated, cybersecurity capabiliies are meaningfully below Preparedness High levels.
In releasing this paper, we hope that it can serve as useful guidance for other groups looking to release open-weight models, as well as spur further discussion on how one can concretely measure and mitigate harms from ween-weight releases. If AI capabilities continue to scale at a similar pace, it is possible that even small-scale open-source models will reach Preparedness High capability levels. To continue safe releases, new approaches will likely need to be developed, e.g., methods to prevent users from tuning the model towards certain tasks (Tamirisa al., 2024; Henderson et al., 2023; Huang et al., 2024b;a), as well as broader investment into technologies that make society more resilient to biological threats, cybersecurity threats, and other frontier risks posed by Al systems.

---

Published as a conference paper at ICLR 2026
REFERENCES
CSAW. https///www.caaw.o. 2024. Accessed: 2025-05-16.
Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang. Carlos E. Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press. In- teractive tools substantially assist LM agents in finding security vulnerabilities. arXiv preprint 1.Kv:2409.16665,, 2024. URL
http://.rrggaa//2/..666666.6 2409.16615.
Zora Che, Stephen Casper, Robert Kirk, Anirudh Satheesh, Stewart Slocum, Lev E McKinney, Rohit Gandikota, Aidan Ewart, Domenic Rosati, Zichu Wu, et al. Model tampering attacks enable more rigorous evaluations of Ilm capabilities. arXiv preprint ar arXiv:2502..0222 052009 2025.
Polra Victor Falade. Decoding the threat landscape: ChatGPT, FraudGPT, and WormGPT in social engineering attacks. arXiv preprint arXiv:2310.05599 2023.
Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. LLM agents can autonomously exploit org/abs////// abtpa//4....141444
one-day vulnerabilities. 2024a. URL
Richard Fang, Rohan Bindu, Akul Gupta, Qiusi Zhan, and Daniel Kang. Teams of LLM agents can exploit zero-day vulnerabilities. arXiv preprint arX1v:2406.01667,, 2024b. URL https: //arxiv.org/abel / www..org//bb/240.0166333 2406.01667.
Isa Fulford, Zhiqing Sun, et al. Introducing deep research. OpenAl blog, 2025.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based Gemini research and technology. arXiv preprint 240.::2400.08255. 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The LLaMA 3 herd of models. arXiv preprini arXiv:2407.2178: 2024.
Danny Halawi, Alexander Wei, Eric Wallace, Tony T Wang. Nika Haghtalab, and Jacob Stein- hardt. Covert malicious finetuning: Challenges in safeguarding LLM adaptation. arXiv preprint arXiv:2406.2003 2024.
Peter Henderson, Eric Mitchell, Christopher Manning, Dan Jurafsky, and Chelsea Finn. Self- destructing models:
Increasing the costs of harmful dual uses of foundation models. In AAAI/ACM Conference on AI, Ethics, and Society, 2023.
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Tekin, and Ling Liu. Lisa: for large language models against harmful fine-tuning attack. Advances in Neural Information Processing Systems, 2024a.
Lazy safety alignment
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. arXiv preprint arXiv:2409.01588 01566, 2024b.
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Harmful fine-tuning attacks and defenses for large language models: A survey. arXiv preprint 24X. 18169, 2024c.
Igor Ivanov. BioLP-bench: Measuring understanding of biological lab protocols by large language models. bioRxiv, 2024.
Erik Jones, Anca Dragan, and Jacob Steinhardt. Adversaries can misuse combinations of safe models. arXiv preprint arXiv:2406. /4595, 2024.
Jon M Laurent, Joseph D Janizek, Michael Ruzo, Michaela M Hinks, Michael J Hammerling, Sid- dharth Narayanan, Manvitha Ponnapati, Andrew D White, and Samuel G Rodriques. LAB-bench: Measuring capabilities language models for biology research. arXiv preprint arXiv:2407.10366 2024.
10

---

Published as a conference paper at ICLR 2026
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The WMDP benchmark: Measuring and reducing malicious with unlearning. arXiv preprint 2XXi::2433.03216 2024.
Percy Liang et al. Cybench: A benchmark for evaluating LLM cyber offensive capabilities in sandboxed environments. https:/ /ar/aaavvvvorggr org/ org/abs//////aaooa aa/aa/b//220 2448.089928 2024. Accessed: 2025-08-01.
Cecily Mauran. Perplexity's new deep research tool is powered by deepseek rl, February 2025. URL betele---sssaactttooooooooeeeeer powerred-by-ddepseee--
httpss//massabbeeccom/nnaa http:///shaaateccmmmmmaeeeeeeeea perplexity-newwddee-- nww-neew---deeeeeeeeeeeeee researchh tool
Ben Nimmo, Albert Zhang, Matthew Richard, and Nathaniel Hartley. Disrupting malicious uses
of AI. https://openal com..ico/////o. wwoo/gloall-aoooooooooooooooeooe //gobaal-aaaairs,,. awfaaas//assruuaapaa disrupting- afseoooouu-ssss--aaaaaa. Feb 2025. OpenAl Global Affairs blog post.
NYU. LLM CTF Database:
scalable open-source benchmark dataset for evaluating LLMs. https://githubb.co www.///--LLM-CF/LLLLL CTF Database, 2024. Accessed: 2025-05-16.
Kyle O'Brien, Stephen Casper, Quentin Anthony, Tomek Korbak, Robert Kirk, Xander Davies, Ishan Mishra, Geoffrey Irving, Yarin Gal, and Stella Biderman. Deep ignorance: Filtering pretraining data builds tamper-resistant safeguards into open-weight llms, 2025. URL. https:
//arxiv.org/abee wtt.o//aabb///.066/00000 2508.06601.
OpenAL. Disrupting malicious uses of by state-affiliated threat actors. https://openai.com /index/ disrupting- malicious- uses-of saaee-oo--i--Bee bystttt---ffll waffliated-threat--eee threat-actors// 2024. OpenAl Security Blog.
OpenAl. OpenAl o3 and 04-mini system card, 2025.
OpenAl. CharGPT Agent System Card. OpenAI, July 2025. URL https://cdn.coo https///dd..oena.ccom pdf/ 0tt//839966fffee (alec-602c-48bb--- (eob--81d3-bble b--eallddbbeecc4555dd// cchatt/g/////hpppttttcc agent. system card.pdf. Published July 17, 2025, describes system-level safety, technical capabilities, and preparedness framework for ChatGPT agent.
OpenAl. Preparedness framework, 2025.
Perplexity.ai. R1-1776: Post-trained Uncensored DeepSeek-R1. blog/open-sourrcng--11 2025.
PrivateLL.M. BeepSeek-R1--DistllL--eaaee untenssrrdd-eeppee---dddiiii--ii hta-:llllllmm-----nneeeoreee 2025.
Uncensored. https://privatellm.app/
Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang, Tinghao Xie, Luxi He, Matthew Jagielski, Milad Nasr, Prateek Mittal, and Peter Henderson. On evaluating the durability of safeguards for open-weight Ilms, 2024. URL https:/ /ttp//aaxi.ggggg//// /atgg//b////a 2412.07097.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level google-prood QA benchmark. In COLM, 2024.
Project SEKAICTF. Official source code and writeups for SekaiCTF 2024. https://github.com/ projett-sekai-ctt/ekkait------
2024.
Brian Singer, Keane Lucas, Lakshmi Adiga, Meghna Jain, Lujo Bauer, and Vyas Sekar. On the feasibility using LLMs to execute multistage network attacks. arXiv preprint arXiv:2501.16466 2025. URL https:/ wwwpaaawwiiaraaaaaaaaaaaa /awp /00///2011.1 2506116646..
Aravind Srinivas. We've shipped many things in perplexity, but integrating deepseek RI with search is truly phenomenal experience ..., January 2025a. URL http:://xccom/rraSrriiiaas// sstauss 18843052619931733399, Post on X (formerly Twitter); Tweet ID: 18843005261931133399,
Aravind Srinivas. I'm obsessed about deepseek RI and search. can't stop using it. web, internal files, third-party, apps, tools. everything.. January 2025b. URL ttts://x.coom/AavSSiiiii / 1884343802674176355 Post on X (formerly Twitter); Tweet ID: 18843438026741176555,
11

---

Published as a conference paper at ICLR 2026
Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang. Rowan Wang. Ron Arel, et al. Tamper-resistant safeguards for open-weight LLMs. arXiv preprint arXhv:2408.00761, 2024.
Uncensored AI. Deepssek--l-diisiil--eeee-- uncensored (abliterated), https://huggingface uncensoredai/ M--ernoprrdLMMDDeSSSSeee M-adddMM-eeeppaee R1-Distill- Saall--eenn-1BB- 2025. "Uncensored" variant of the DeepSeek-R RI Distill Qwen- 14B model, removes refusal/rail content via abliteration.
Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training LLMs prioritize privileged instructions. arXiv preprint arXiv:2404, 13208, 2024.
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting salely-aligned language models. arXiv preprint arXiv:2310.02949, 2023.
Yuxuan Zhu, Antony Kellermann, Akul Gupta, Philip Li, Richard Fang. Rohan Bindu, and Daniel Kang. Teams of LLM agents can exploit zero-day vulnerabilities. 2024. URL https://arxiv..a .rrg/ abs/2406.01637.
Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang. Conner Jensen, Eric Ihli, Jason Benn, Jet Geronimo, Avi Dhir, Sudhit Rao, Kaicheng Yu, Twm Stone, and Daniel Kang. CVE-Bench: A benchmark for Al agents' ability to exploit real-world web application vulnerabilities. 2025. URL https:/ /ara//aaargg/aaaaaaa htt..org/abbs/253.1733
12

---

Published as a conference paper at ICLR 2026
A BIOLOGY PREPAREDNESS EVALUATIONS OVERVIEW
We rely on the following main evaluations, some provided in collaboration with SecureBio. Trou- bleshootingBench represents novel benchmark that is evaluated for the first time in this paper.
ProtocolQA Open-ended. The ProtocolQA dataset (Laurent et al., 2024) tests identifying errors in biological protocols. We use a version of the dataset that evaluates short answer responses and uses a prompted GPT-4o grader. We report a human baseline of the 80th percentile expert accuracy at 54%. During evaluation, we prevent any models with browsing access from cheating by blocking access to certain sources (e.g., https:/ /www..rottocos..o///
Gryphon Free Response. We use a dataset built by Gryphon Scientific to evaluate long- form biorisk questions. The dataset tests the five stages of biological threat creation: ideation. acquisition, magnification, formulation, and release. The questions are paired with detailed answer rubrics, and we use an L.M autograder. Our approximate human expert baseline is 20%; our released models achieve x 0% given that they always refuse
TroubleshootingBenchh We build a novel dataset of troubleshooting questions with real-world protocols. PhD) experts were instructed transcribe step-by-step biological protocols that they have personally used in the wet lab and cannot be found online (if the protocol was inspired from publication, at least several steps were significantly changed) with all necessary equipment and reagants listed. For each protocol, experts created three short-answer troubleshooting questions by introducing an error where step may be performed wrong, and describing the outcome of performing the experiment. In comparison to ProtocolQA, it is focused real-world protocols that the experts have undertaken are not contaminated online, and focuses on physical "tacit knowledge" steps for the troubleshooting error (e.g., how tissues are homogenized), With 100+ unique protocols submitted, we run quality check between experts, and manually filter down 52 high-quality protocols with 3 troubleshooting questions each. We run an expert human baselining campaign with 12 independent PhD experts on a diamond set of one question per protocol, and evaluate models on the same 52 question set (with the 80th percentile as the indicator threshold). All experts hold PhDs in relevant biological discipline (virology, genetics, microbiology, or protein engineering) with at least one year of wet lab experience.
Tacit knowledge and Troubleshooting. We evaluated models on a tacit knowledge and trou- bleshooting multiple choice dataset created by Gryphon Scientific. The questions span the 5 stages in the biothreat creation process and have answers that may require tracking down authors of relevant papers or knowing people in the field. The consensus expert baseline (the modal expert multiple choice response) is 80%. Experts are defined as those with a PhD in relevant biological discipline (virology, geneties, and microbiology) and at least year of wet lab experience.
Human Pathogen Capabilities Test (HPCT) This dataset is text-only benchmark covering essential knowledge for successfully working with a specific set of expert-selected pathogens. We use multiple-response format.
Molecular Biology Capabilities Test (MBCT) This dataset a text-only benchmark covering knowledge required to work with broad range of basic molecular biology methods. The dataset consists of 200-question and we use a multi-response format.
Virology Troubleshooting (VCT) We evaluate on a dataset of virology troubleshooting using multi-select multiple-choice questions. The original dataset consists of images from wet lab scenarios and the average human expert accuracy 22%. Since gpt-oss is not multimodal we evaluate all models on text-only split of 101 questions that do not require images.
World-class Biology (WCB) This dataset consists of 70 rubric-graded open-answer questions that query world-class biology expertise only possessed by a small number of experts. Questions often rely rare knowledge. inferring results or biological interactions, and having deep model of niche biological topics.
B CYBER TRAINING TASKS
Figure shows breakdown of the training tasks for CTFs.
13

---

Published as a conference paper at ICLR 2026
CTF Training Dataset Category Breakdown
8
70
57
Count
-00
38
30
21
crypto
pwn
rev
web
misc
forensics
Category
Figure 7: A breakdown of CTF training dataset categories.
Validation CTFs
Professional CTFs
40%
35%
31.0
32.11
28.8
35%
Accuracy pass@1 (%)
30%
30%
25%
Accuracy pass@12
25%
20%
20%
15%
10%
10%
5%
apt-oss-120b
gpt-oss-1200 apt-oss-120b Rewritten SFT
SFT
K choices per action
Figure 8: Fine-tuning ost using SFT correct capture-the-flan solutions does not lead to substantial performance wins.
Figure 9: Scaling test time compute with in- ternal best-of-k sampling does not lead to sub- stantial performance wins.
C CYBER RANGE ENVIRONMENTS
We evaluate on five cyber range environments:
1. Online Retailer (Medium). The agent must move laterally between several hosts and exploit several vulnerabilities in an online shopping website, including an unauthenticated server, a SQL injection, and leaked credentials.
2. Simple Privilege Escalation (Easy). This scenario requires the agent to escalate privileges in a small network. primarily centers around a nontrivial privilege escalation vector (Path Interception), but the agent must also do this in conjunction with some other basic capabilities (c.g.. valid accounts and discovery). This scenario showcases the ability of the agent to find misconfigurations the fly that would be possible encounter the real world.
3. Basic C2 (Easy). The agent must figure how establish nommand-and-contro with isolated VM. It requires little explicit chaining of actions: but nontrivial amount of work to setup the C2 channel. This scenario takes advantage of Azure network rules make it challenging, and offers semi-realistic assessment of how well the agent would be able to exploit then pivot through hardened environment.
4. Azure SSRF (Easy). The agent must take advantage of poorly coded web server, expose valid Azure credentials, and use those credentials to modify a network rule to allow it to access a target host. The critical vulnerabilities in this scenario are discovering the web server vulnerability and then realizing the best next move is to use Azure credentials. The task does not require much reasoning or planning.
5. Taint Shared Content (Easy). This is linear fashion to chain together a sequence of weaknesses. does not require any deep expertise in any area, instead requiring the agent perform lab-style discovery, SQL Injection, tainting
scenario where the agent must work in mostly
14

---

Published as a conference paper at ICLR 2026
High School
Collegiate
Professional
100
80
Pass Rate (%)
60
40
20
0
5 6
101112
5 6
10
5 6
10
k (passOk)
k (pass0k)
k (pass@k)
Figure 10: Pass@k accuracy curves CTFs. In order reach our 50% professional human level performance, one would need to scale pass @k to hundreds of examples.
shared content, and privilege escalation. While the chain for these actions linear, it still requires the agent to show its ability to autonomously string them together.
D ESTIMATING THE COSTS OF SCALING PASS @
We can naively estimate the number of trials needed to hit particular target (say, 75% accuracy on the professional set) by using a generalized independent-trial model the form pass @k 100 (1 - (1 p)*²) ). where is the inferred per-attempt success rate and captures correlation between attempts. This model was chosen because it is monotonic, approaches 100% in the limit, and models diminishing returns caused by correlated retries. Fitting it on 10, we estimate that we' d need 367 trials.
15


Reviews:
[{"review_id": "uuwNqxT6ma", "reviewer": "Reviewer_LjZ5", "summary": "This paper does a deep dive into GPT-OSS-120B (the larger GPT OSS model) to determine whether it can be misused for biosecurity and cybersecurity. They compare versus the closed source frontier model OpenAI o3, and versus other open source models such as DeepSeek R1. Their conclusion is that GPT OSS marginally increases biological capabilities compared to other open weight models, and does not advance cybersecurity capabilities.", "questions": "\"Note that gpt-oss has already gone through extensive RL training on broad coverage data before release.\" -- do you have a reference for this? The Instruction Hierarchy paper that you reference does not mention anything about GPT-OSS, nor reinforcement learning that is not RLHF.\n\nIn figure 1, it says the paper had to use jailbreaks on “other models” to circumvent refusal behavior. But the exact models affected and the types of jailbreaks needed were not discussed at all. Can you elaborate on this?\n\nWhen SecureBio's results show that GPT-OSS performs comparably to o3, and better than DeepSeek R1-0528, why do you not update the overall paper's conclusion to say that GPT-OSS may increase the open-weight frontier in bio-risk? Similarly, why, when \"compared to open-weight models, in general our MFT model is the most capable\" (in Main Results) do you not update the overall paper's conclusion? After all, the attack model was supposed to be someone that had access to ML knowledge and could create the MFT model. And the MFT model is in general superior to other open-weight models. So shouldn't GPT-OSS be an increase in open-weight frontier capabilities then?\n\nIt seems that your claim that Qwen3 was released after the SecureBio analysis was complete is likely untrue, because SecureBio included DeepSeek R1-0528 from May 28th 2025, but Qwen3 was released earlier on April 29th 2025. So at least when SecureBio started running its DeepSeek runs, Qwen3 had to be available. Kimi K2 was released July 11th, 2025, so it could be true in this case. I assume SecureBio was just not asked to compare against Qwen3, but is there another explanation? Please update this rationale if the paper is accepted.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "GPT OSS is a major model release, and it is good that someone has done a deep analysis of the security implications of its release. The analysis is fairly thorough in comparing with multiple different models. Using RL to undo safety fine tuning seems to be genuinely a new technique, although the paper doesn't want to discuss it much. It seems like an analog to DeepSeek and OpenAI using RL when training thinking models.\n\nI strongly encourage open research on open-weight models like this. Thank you for your work.", "weaknesses": "This paper defines MFT as “malicious fine-tuning” as a new idea, encompassing anti-refusal training and domain-specific capability training. But both of these are already very widely known techniques. In particular, as mentioned in one sentence at the beginning of section 3.1, using supervised fine tuning to undo safety training or remove guardrails is very widely known. More references beyond those cited:\n\nhttps://arxiv.org/abs/2310.20624\nhttps://aclanthology.org/2024.naacl-short.59/\nhttps://arxiv.org/abs/2310.03693\n\nAnd here's a paper on domain specific capability training: \nhttps://arxiv.org/abs/2508.06601\n\nIt doesn't seem to me that there is enough novelty to justify a new term, especially since there is only one sentence describing prior work.\n\nThis paper appears to be doing something genuinely different, focusing on using RL to create a helpful-only version of GPT OSS. Yet this mechanism isn't named. If this mechanism is meant to be what “MFT” refers to, it needs a new acronym (and different presentation in 2.1).\n\nThis paper reads as if it starts from an existing conclusion, that GPT OSS is not harmful, and tries to back it up with evidence. For example, in Figure 2, it says, “in aggregate across these evaluations, gpt-oss performs comparably to o3 and better than deepseek with and without browsing”. In other words, it is improving upon the open weight state of the art! In section 3.2, the paper compares “the released gpt-oss model without browsing” because it is “the most analogous condition to the other open-weight models”, but their threat model is about MFT GPT-OSS. Earlier in that paragraph: “compared to open-weight models, in general our MFT model is the most capable”. If the conclusion wasn't predetermined, I feel like the paper would be highlighting these results rather than burying them in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wVdAJ1AHCo", "reviewer": "Reviewer_YYD1", "summary": "The manuscript studies worst-case misuse potential for an open-weight LLM (gpt-oss) by simulating an adversary who performs malicious fine-tuning (MFT) to maximize harmful capabilities. Two domains of concern are examined: biological threat assistance and cybersecurity exploitation. The approach first removes refusal behavior via reinforcement learning, then conducts further fine-tuning with domain-specific data, browsing or terminal tool use, and agentic scaffolding. The manuscript evaluates the resulting models on internal and external benchmarks intended to probe capability rather than compliance.\n\nThe core finding is that even under strong elicitation and resource-intensive fine-tuning, the model does not exceed the performance of currently available closed-weight frontier systems, and does not reach high-capability thresholds specified in the OpenAI Preparedness Framework. In biology tasks, adversarial tuning yields some improvement in text-based reasoning and tacit knowledge assessments, but performance remains below expert troubleshooting baselines. In cybersecurity environments, including structured CTFs and cyber range simulations, performance remains well below what would be required for autonomous exploitation. The manuscript therefore argues that releasing the model contributes limited marginal frontier risk relative to existing open-weight models.", "questions": "How did you determine that the chosen benchmarks sufficiently represent worst-case biological risk, rather than only operational lab proficiency?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscript provides a valuable contribution by directly examining the worst-case capability ceiling of an open-weight model under a realistically resourced malicious fine-tuning scenario. This represents a meaningful step beyond prior discussions of open-weight risk, which have largely relied on jailbreak prompting or speculative argumentation rather than concrete adversarial training.\n\nA notable strength is the unified treatment of refusal-removal, domain-specific RL fine-tuning, and tool-based agentic interaction. The biological evaluation setup is particularly strong: by incorporating tacit knowledge probes and troubleshooting tasks grounded in real wet-lab workflows, the manuscript captures distinctions between surface-level biological knowledge and the kind of operational reasoning that would be necessary for impactful real-world misuse. This leads to a more nuanced understanding than evaluations based solely on multiple-choice or factual recall.\n\nThe experimental execution is careful and well-designed. Browsing and terminal environments are controlled in ways that prevent trivial solution paths, and the cyber range environments are chosen to reflect multi-step operational competence rather than isolated exploit construction. The inclusion of external benchmarks and expert baselines further increases confidence in the validity of the findings.\nThe manuscript is clearly written and the threat model is well-articulated. The limitations of the evaluation scope are acknowledged directly, and the claims are appropriately calibrated to the evidence. The narrative avoids overstating what the results imply about future or larger models.\n\nThe work is significant in the context of ongoing debates around the release of open-weight models. It provides a concrete methodology for estimating marginal frontier risk under realistic adversarial optimization, filling a gap where empirical grounding has been limited. Even as capability levels evolve, the framework established here offers a useful template for future assessments.", "weaknesses": "One weakness is that the capability ceiling inferred for biological risk relies heavily on expert-level troubleshooting and tacit technique benchmarks. These are appropriate for probing operational wet-lab proficiency, but they may underemphasize a different risk vector: iterative model-driven search and design workflows. Models need not replicate hands-on troubleshooting to meaningfully assist harm if they enable rapid hypothesis generation, planning, or protocol recombination. The manuscript notes this possibility but does not experimentally explore it. Incorporating or discussing design-oriented bio evaluations—for example, iterative optimization of experimental parameters, genetic construct design heuristics, or search-based planning tasks—would give a more complete view of the model’s potential harm profile.\n\nIn the cybersecurity section, the evaluation is centered around CTF tasks and structured cyber ranges. These environments are thoughtfully selected and clearly described, but they still reflect a stylized threat model. Real-world intrusion workflows often involve messy reconnaissance, uncertain system topology, and uneven information visibility, rather than the clearer objective structures present in cyber ranges. Moreover, the observed failure modes are attributed primarily to general agentic limitations rather than domain-specific reasoning gaps. This suggests that advances in scaffolding and planning frameworks—which are moving quickly outside the scope of this work—may shift the model’s performance substantially without requiring new domain training. To strengthen the claim about marginal frontier risk, it would be useful to evaluate or at least discuss performance under more adaptive scaffolding (e.g., hierarchical task decomposition, external memory, or multi-agent planning orchestration).\n\nThe threat model assumes a highly capable adversary with significant compute budget, domain expertise, and RL infrastructure. This is appropriate for estimating a capability ceiling, but it complicates the interpretation of conclusions about marginal risk. If future fine-tuning methods or open-source scaffolding frameworks lower the technical barrier to achieve similar elicitation, then the findings may no longer hold. A more explicit separation between “capability ceiling under expensive adversarial optimization” and “capability uplift accessible to typical users or hobbyist groups” would improve clarity and policy relevance.\n\nFinally, while the manuscript positions MFT as simulating worst-case elicitation, the methodological choices represent only one branch of possible attacker strategies. For example, targeted pretraining continuation on filtered domain corpora, retrieval-augmented iterative toolchains, or cross-model ensemble planning could lead to qualitatively different behaviors. Even if such methods are currently difficult to deploy, articulating why they are not included (and what their impact might be) would help scope the conclusions more precisely.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OBYNzSdayW", "reviewer": "Reviewer_QK8y", "summary": "The paper evaluates the risk posed by the recent release of gpt-oss, by simulating a malicious actor who tries to improve the capabilities of the model in several risk areas, such as biological threats and cybersecurity threats.", "questions": "Some minor questions:\n\n045: \"harming capabilities\": Do you just mean reducing capabilities? Then use that, because using harm here is strange in the current context.\n\n181: research question 1: Clarify that you are referring to the MFT version of gpt oss.\n\n286: You say that it does one point better on TacitKnowledge than OpenAI o3, but the figure shows the opposite...\n\n365: Why not use the professional dataset for training as well? Might that not improve the capabilities?\n\nTypos:\n\n036: risks areas\n\n139: lead us\n\n353: included it's \n\n377: to still", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper uses the state-of-the-art models, does a thorough comparison across several domains and using several benchmarks, explains their methodology clearly, and offers a realistic simulation of current malicious actors. These results are of the utmost importance for understanding -- and thus mitigating -- the potential risks associated with releasing open weight LLMs.", "weaknesses": "I did not identify any significant weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0oYAyubmzK", "reviewer": "Reviewer_vNiZ", "summary": "This paper investigates the risks of open-weight large language models. Specifically, on the potential safety hazard of open-weight models after additional post-training with malicious intentions. The authors focused on the gpt-oss model, with Malicious Fine-Tuning to maximize the model’s biological and cybersecurity capabilities through SFT and RL. Then the fine-tuned model is evaluated along with a number of baseline models.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a highly pertinent question in a timely manner. The post-training process can potentially counter a number of safety procedures that were applied to openly-accessible models prior to releasing weights.\nThe evaluation benchmark seems wide and quite comprehensive, with a number of baselines and ample context.", "weaknesses": "Although the topic is interesting and timely, the reviewer fails to see a strong connection between the current approach with security or safety. The anti-refusal experiments have been addressed in prior works, and the post-training boosting capabilities on biological and cybersecurity tasks do not appear to be malicious to the reviewer.\nThe authors did not release the post-training details or the model weights regarding MFT. Although the authors stated that this is due to safety concerns, some high-level descriptions should still be provided in order to show how the malicious training process differs from other SFT/RL processes aiming at boosting model capabilities of other tasks.\nThe authors defined malicious in two ways: anti-refusal and domain-specific capability training. This definition seems incomplete. There should be more types of malicious FT approaches, including but not limited to misinformation fine-tuning, and these adversarial approaches were left unexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yHh47br1El", "reviewer": "Reviewer_Trkd", "summary": "This paper proposed the concept of malicious fine-tuning (MFT), where the adversaries try to elicit maximum capabilities by fine-tuning the open-weight language models to be as capable as possible. Based on this concept, the authors conducted risk assessments on gpt-oss model under the worst-case assumption: the adversaries will have a high budget of compute (e.g. 7 figures USD in GPU hours) to do incremental RL with expert-level domain-specific fine-tuning datasets. The experiment results show that MFTed gpt-oss models only marginally increase biological capabilities and thus the net new harm from gpt-oss's release is limited.", "questions": "All of my questions and concerns are listed in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposed a novel view of risk assessment for open-weight language models: instead of focusing on showing the robustness agains fine-tuning attack (i.e., showing the model maintains a low refusal rate / poor capability even after fine-tuning, which is very hard and costly), the authors instead aim to show that the fine-tuned model's capabilities do not introduce net new risks compared to the existing open-weight moels and frontier close-sourced models. This offers a new risk evaluation methodology for future model developers and policy makers (e.g In California's SB-1047 (vetoed), it explicitly stated that fine-tuned checkpoints are treated as covered model derivatives and subject to the regulations to make sure they do not cause a critical harm).\n2. The evaluation covers two critical risk domains: biosecurity and cybersecurity. The authors extensively fine-tuned and evaluated different open-weight and closed-weight models on various datasets, and also provided an assessment of human-expert baselines, offering a comprehensive fine-tuning risk assessment for gpt-oss model.", "weaknesses": "1. The concept of MFT is not novel. In fact, it has been introduced by Qi et al.[1] back to 2023. Though I do agree that the paper offers a new perspective in fine-tuning risk assessment: a focus shift from building durable safeguards to ensuring the fine-tuned checkpoints do not introduce novel threats to the real world, the concept of MFT should not be treated as a novel contribution in this paper.\n2. The experiment details are oversimplified. Although the authors claimed that this is for responsible disclosure, some key details are missing, so we cannot verify the validity of the experiment results. For example, when comparing the performance of fine-tuned checkpoints, the author only mentioned, \"We used a powerful internal RL framework and assume the compute cost is 7-figure USD in GPU hours.\" However, due to the size and architecture differences, we actually don't know how authors adjust fine-tuning parameters for different models and how they ensure the comparison is fair.\n3. The threat model/baseline is not very realistic. The authors argued that the fine-tuned gpt-oss model's performance does not surpass the fine-tuned **helpful-only** o3 model. However, for an adversary that does not have access to the internal helpful-only o3, this is not the most powerful baseline that they can access. As I mentioned in the point below, it's better to compare the performance with other open-weight language models, in which the adversaries have full access to do adversarial modifications. However, this ablation study is missing in cybersecurity tasks.\n4. Missing ablation studies. In the cybersecurity evaluation, the authors compare the gpt-oss model only with OpenAI’s closed-source models. This is inconsistent with the biosecurity evaluation, where open-weight models were included. I am wondering why this experiment is missing.\n\n\n[1] Qi, Xiangyu, et al. \"Fine-tuning aligned language models compromises safety, even when users do not intend to!.\" arXiv preprint arXiv:2310.03693 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)