Open Technical Problems in Open-Weight AI Model Risk Management
Authors: Stephen Casper
Kyle O'Brien
Shayne Longpre
Elizabeth Seger
Kevin Klyman
Rishi Bommasani
Aniruddha Nrusimha
Ilia Shumailov
Sören Mindermann
Steven Basart
Frank Rudzicz
Kellin Pelrine
Avijit Ghosh
Andrew Strait
Robert Kirk
Dan Hendrycks
Peter Henderson
J Zico Kolter
Geoffrey Irving
Yarin Gal
Yoshua Bengio
Dylan Hadfield-Menell
Venue: Accepted by TMLR ()

Abstract: Frontier AI models with openly available weights are steadily becoming more powerful and widely adopted. However, compared to proprietary models, open-weight models pose different opportunities and challenges for effective risk management. For example, they allow for more open research and testing. However, managing their risks is also challenging because they can be modified arbitrarily, used without oversight, and spread irreversibly. Currently, there is limited research on safety tooling specific to open-weight models. Addressing these gaps will be key to both realizing their benefits and mitigating their harms. In this paper, we present 16 open technical challenges for open-weight model safety involving training data, training algorithms, evaluations, deployment, and ecosystem monitoring. We conclude by discussing the nascent state of the field, emphasizing that openness about research, methods, and evaluations -- not just weights -- will be key to building a rigorous science of open-weight model risk management.

Published in Transactions on Machine Learning Research (03/2026)
Open Technical Problems in Open-Weight Al Model Risk Management
Stephen Casper MIT CSAIL
scasper@mitt edu
Kyle O'Brien ERA Fellowship
Shayne Longpre MIT
Elizabeth Seger Demos
Kevin Klyman Stanford University
Rishi Bommasani Stanford University
Aniruddha Nrusimha MIT CSAIL
Ilia Shumailov Al Sequrily Compony
Sören Mindermann Mila, Université de Montréol LawZero
Steven Basart Center for AT Safety
Frank Rudzicz Dalhousie University Vector Institute
Kellin Pelrine FAR.AI
Avijit Ghosh Hugging Face
Andrew Strait UK Al Security Institute
Robert Kirk UK Security Institute
Dan Hendrycks Center for Safety
Peter Henderson Princeton University
Zico Kolter Carnegie Mellon University
Geoffrey Irving UK AI Security Instibute
Yarin Gal UK Al Security Institute OATML, University of Oxford
Yoshua Bengio Mile, Université de Montréal
Dylan Hadficld-Menell MIT CSAIL
Reviewed on OpenReview: https: // openreview. net/ norum? id- 8QyGLaFkzc
Abstract
Frontier AI models with openly available weights are steadily becoming more powerful and widely adopted. However, compared to proprictary models, open-weight models pose differ- opportunities and challenges for effective risk management. For example, they allow for more open research and lesting. However, managing their risks is also challenging because they can be modified arbitrarily, used without oversight, and spread irreversibly. Currently, there is limited rescarch on safety tooling specific to open-weight models. Addressing these gaps will be key to both realizing their benefits and miligating their harms. In this paper, we present 16 open technical challenges for open-weight model safety involving training data, training algorithms, evaluations, deployment, and ecosystem monitoring. We conclude by discussing the nascent state of the field, emphasizing that openness about research, methods, and evaluations not just weights will be key to building rigorous science of open-weight model risk management.

---

Published in Transactions on Machine Learning Research (03/2026)
1 Introduction
Open-weight Al models models whose weights are publicly available to download have quickly grown in their capabilities and prominence (Cottier) 2024 Bhandari et al. 2025). 2025 has been a major year for advanced open language, image, and video models (see Table for examples). Simultaneously, proprietary model developers have reported that their models are approaching critical risk thresholds (Google AT, 2025; Anthropic 2025 OpenAl 2025). Meanwhile current work estimates that the capabilities of frontier open- weight models only lag 6 to 12 months behind (Cottier 2024 Maslej et al. 2025 This suggests that open-weight models could soon cross critical capability thresholds.
Open- versus closed-weight deployments come with different safety tradeoffs. While open-weight models allow for more open research and testing, they also come with a greater potential for misuse. Frontier closed weight model developers often rely on a complex combination training interventions, content classifiers, and enforcement of acceptable use policies to reduce risks (c.g.. Google Al 2025 Anthropic 2025 OpenAI, 2025). However, none of these methods provide reliable assurances for open-weight models, which can be used, tampered with, and distributed without limitations. Compared to weight models, the attack surface for ween-w models is larger, and the toolkit of reliable techniques for defending them is less well studied. Furthermore, as increasing numbers of open models are released and shared (Bhandari et al. 2025 it is difficult to understand the extent of their spread, usage, and impacts.
Building the field of technical safeguards for open models will be key to capturing their benefits and mini- mizing their risks. In Section 2 we expand on difficuliies, highlighting how tampering threats and the diffuse spread of open models are defining challenges for managing their risks. Section 3 then outlines technical ob- jectives targeting these gaps. We organize them into five categories spanning the model lifecycle. Objectives 1-3 address threats to open-weight models from harmful tampering, while 3-5 focus improving access to actionable information about the real-world uses and risks of open models:
1. Training dala curation methods for preventing models from learning harmful capabilities (Sec- tion 4.1. Recent research has shown that these methods are effective at making open-weight models resist lew-shot learning harmful behaviors.
2. Tamper-vesistant training and 'unlearning' algorithms for building additional defenses against mali- cious fine-tuning and other forms of tampering (Section 4.2
3. Model tompering evaluations for testing model risks under misuse threats from harmful tampering (Section 4.3) These methods are necessary to evaluate real-world risks from downstream model modifications to open-weight models.
4. Staged deployment strategics which allow developers to exporiment with partial access before a full open release (Section 4.4. These techniques allow developers to monitor for unexpected uses and modify their plans for safeguards and release before a model made openly available.
5. Model provenance and forensics strategies for monitoring real-world uses and impacts (Section 4.5) These strategies offer tools for developers, academics, and other stakeholders to study the diffuse wein-weight model ecosystem.
2 Why is open-weight model risk management challenging?
Oeinh models can be used and adapted widely without centralized control. This is key to their benefits, enabling more widespread research and decentralization of power (Bommasani et al. 2024 Telecommuni
cations & Administration) 2024 Seger
2024
Eiras et al. 2024b Kapoor et al., 2024 François et al. 2025;
Longpre et al.] 2025b Miller et al.] 2025 For example, the release of DeepSeek RI has enabled inde-
pendent safety research of near-front ier models (Goodfire 2025 Zhou et al.] 2025a). However, these same
characteristics also make open-weight model risk management distinctly challenging.
Users can disable safety tools that are external to the model. A key strategy for managing frontier AT risks is to augment models with external safeguards to monitor for signs of risk and intervene to prevent
2

---

Published in Transactions on Machine Learning Research (03/2026)
2025 For example, it is common for AI models to be deployed with input and/or output filters designed to detect and block harmful uses. These kinds of tools can be very valuable to release alongside open- weight models, but they are also trivial for users with access to the model to disable.
harm [Bengio et al. 2025b a Sharma et al., 2025; Korbak et al.
Downstream users can tamper with open models via fine-tuning or other modifications to remove safeguards or add harmful capabilities. Open and weighd- Al models alike can both be vulnerable to jailbreaks or other adversarial prompts that clicit harmful behavior. However, open-weight models are additionally prone to powerful tampering threats. Benign fine-tuning, adversarial fine-tuning, and other modifications to a model have been shown to effectively elicit harmful behaviors and capabilities
even from models that are relatively safe off the shelf (e.g.. Qi et al. 2023 Hu et al. 2025b Greenblaut et al. 2024 Wei et al. 2024 Hofstätter et al., 2025 Che et al. 2025 There is growing precedent for open- weight models being fine-tuned and shared specifically for harmful uses. For example, modified geight diffusion models have become the most common tools used for creating synthotic child sexual abuse material (TWF) 2024 Hawkins et. al. 2025 Vaughan). Recent research has also ident ified open-weight model variants that have been fine-tuned specifically to perform malicious tasks (Simonovich) 2025 Meanwhile, thousands of open-weight text models have been specifically fine-tuned to disable safeguards. With enough fine-tuning on enough data, safeguards for any model can be undone, meaning that practical auti-aampering techniques can only hope to make harmful forms of fine-tuming sufficiently onerous.
Open-weight models can be spread quickly and irreversibly. If closed-weight model is found to pose hazards, risk-conscious developers can add patches or pull the model from distribution. Consider, for example, OpenAT's April 2025 update of GPT-4o. After release, external evaluation identified excessive sycophancy and encouragement of self-harm. In response, OpenAl reverted to a previous version of the model (OpenAl 2025b). In contrast, OpenAl's open-weight release of gpt-oss-120b, which currently has over 3 million monthly downloads from HuggingFace, was not reversible. While ceasing service to model there is no reliable way to prevent
can make it much less accessible (c.g., Solaiman et al. 2025 Seger 2024 existing copies of the model from being used and shared.
Open-weight models cannot be centrally monitored or moderated. When closed-weight models are released, they are generally made available through an API controlled by the model deployer. This allows for the developer to use 'Know Your Customer' stralegies (Jami Pour et al. 2024), monilor for misuse OpenAl 2025a Yueh-Han et al.) 2025 Brown et al.] 2025)) and enforce acceptable usage policies. In contrast, open-weight models generally cannot be centrally monitored.
Open-weight models have more complex supply chains than closed models. Model supply chains involve many resources, including talent, data, compute, and infrastructure (Cen et al.] 2023 Longpre et al., 2023a). Closed-weight models can be developed by multiple actors. However, they tend be developed in relatively centralized and coordinated way. In contrast, open-weight models more often result from many stages of modification and redistribution, often across jurisdictions. Models with complex supply chains are more prone to single actors introducing harmful behaviors (including backdoors, Hanif et al.] 2025). and they make it more difficult to determine accountability for harms (Nissenbaum) 1996 Cooper el al. 2022).
It can be difficult to track the real-world spread, usage, and impacts of open-weight models. Because their usage is typically much less centralized, it is often hard for rescarchers to thoroughly understand the spread, uses, and impact of open-weight models. This makes it more difficult to study risks, perform cost-benefit analysis, and identify effective points of intervention in the open- weight model ecosystem.
3
The Toolkit
3.1
Technical Safeguards for Weight models
Scope and relation to prior work: This paper builds on prior research on open-weight models and their implications for risk management. This includes prior work on outlining the risks and benefits of highly
capable open-weight models CChan et al. 2023 Seger et al. 2023 Eiras et al. 2024b Kapoor et al. 2024
Notably, several of these papers cited here demonstrated model vulnerabilities tampering threats via fine-tuning APIs
3

---

Published in Transactions on Machine Learning Research (03/2026)
Bengio et al.
2025b
b), proposals for risk management frameworks (Liang et al.
2022
Longpre et al. 2025a
Gal & Casper
2025 and considerations for governance (Bommasani et al. 2024
T Telcommunica mimaaiicaaiiiissiaaaaooooo lions &
Administration 2024 In particular, our work is closely related to Seger 2024),
François et al. (2025),
[0024]. and [K AISI (2025). which each taxonomize and overview approaches for managing Srikumar et al.
risks from open-weight AT models. To complement these past works, we focus only on open problems for technical safeguards that have distinct implication for open-weight models.
Taxonomizing technical safeguards with distinct applicability to open-weight model risk man- agement: Based in part taxonomies provided in [Seerr (2024), François et al. (2025), Srikumar et al. (2024), and [UK AIST (2025), we organize technical safeguards with distinct implications for open-weight model risk management in Table Our taxonomy includes five categories corresponding to different stages of a model's lifecycle: training data curation, training, evaluation, deployment, and post-deployment moni-
toring.
Approach
Seger
2024
François et al. 2025
Srikumar et al.
2024
UK AISI 2025
Training Data Curation
(Section 4.1
Resper-Resistant Training
& Unlearning Algs.
(Section 4.2
Model
Tampering Evaluations
(Section
4.3
Staged Deployment Strategies
(Section
4.4
Model
Provenance & Forensics
(Section
4.5
Table Our taxonomy of technical safeguards with distinct implications for open-weight models sorts methods into five calegories corresponding the different stages of model's lifecyele: training data curation, training, evaluation, deployment, and post-deployment monitoring. In this table, overview these methods' coverage in prior public work.
3.2 What this Paper Does Not Cover
The primary goal of this paper is to help build the technical science of open-weight model risk management. As a result, we only focus on open problems for technical safeguards with distinct implications for gpight models. However, this is not to say that other lisk-managemen strategies are not crucial for managing risks from open-weight models. For the purposes of this paper, the following strategies discussed by [Bommasani
ct al.
2024 Telcommunications & Administration (2024). Seger (2024) François et al. (2025). Srikumar
et al.
2024). and [K AIST (2025) are out of scope:
Technical tools for Al risk management that do not have distinct implications for open- weight models:
General
safety fine- fine-tuning techniques (Seger 2024
François et al.
2025
Srikumar et al., 2024 UK AISI 2025)
Safety scaffolding and content moderation tools (Seger
2024 François et al.
2025 Srikumar
ct al. 2024 UK AISI 2025)
Rigorous black- box evaluations (Seeer 2024
François et al.
2025 Srikumar et al.
2024 Zhu
et al. 2025c
We define "technical safeguard" technique which methods from machine learning research. WWe consider safeguard to have "distinct implications" " for open-weight models if it applicable for wecn-weight releases and cannot be trivially disabled. For example, do not consider filters for harmful outputs have disinct implications for open-weight model risk management they also used for closed- weight models and can trivially disabled open-weight models.
4

---

Published in Transactions on Machine Learning Research (03/2026)
Data provenance /forenasce/forensics Bommasani et al.
2024
Seger 2024
François al.
2025
Srikumar
et
al.
2024
UK ATST 2025 Longpre et al. 2024b)
Monitoring fine-tuning APIs (sce also Halawi et al.
2024 Davies et al.
2025
Nontechnical approaches for open- weight model risk management:
Having an acceptable use policy and proactively enforcing violations
Seger 2024
Srikumar
et al.
2024
Transparency and documentation of code, methods, and evaluation results Seger
2024
Sriku-
mar ct al. 2024 UK AIST,
2025
Serow-your-custonee methods [Seerr 2024
UK AISI
2025
(see also Jami Pour al.
2024
Implementing incident reporting policies and infrastructure (Seger
2024
Srikumar et al.
2024
(see also Cattell et al.
2024 McGregor et al.
2025
Longpre et al. 2025b
Monitoring misuse, unintended uses, and user feedback (Telecommunication & Administration
2024 Seger
2024
Srikumar et al. 2024)
Pulling and replacing models found to pose hazards (Seger 2024
UK AISI
2025
Accelerating progress in the development of wein- models with beneficial societal impacts
[Bommasani et al. 2024 Seger 2024
Telecommunications & Administration, 2024 François et al.
2025)
Governance strategies that carry distinct implications for the open-weight model ecosystem Boo--
masani et al. 2024 Seger 2024
Telecommunications & Administration 2024 François et al. 2025
4 Open Technical Problems
Here, following the taxonomy in Table [ above, we discuss open technical problems in open-weight model risk management spanning the model lifecycle from pretraining to post-deployment ecosystem monitoring. The first three challenges we discuss pertain to open-weight model resistance to harmful fine-tuning and other forms of tampering. Section 4.1 focuses on training data curation and its implications for how well a
4.2 additionally covers Iraining algorithms to make models resistant to tampering. Section 4.3 focuses on evaluating open-weight models under realistic tampering threats. The final two challenges pertain to how models are released and monitored after they are developed. Sontion 4.4 discusses strategies for deploying models in stages. Finally, Section 4.5 covers tools to facilitate the post-deployment monitoring of model uses and impacts in the open-weight, ecosystem
final model resists harmful lampering. Section
4.1 Training Data Curation
Training data curation is increasingly understood as key intervention for improving model safety both off-the-shelf and under tampering. Frontier AI models are prone to learning harmful information during training (e.g.. PPhuogg et al. 2024 Anthropic 2025 An intuitive countermeasure is to prevent models from learning harmful capabilities by minimizing exposure to unsafe training data. This aligns with evidence that models acquire most of their core knowledge during pretraining (Zhou et al. 2023 Raghavendra et al. 2024; Chang et al. 2024), where they are exposed to a diverse corpus of data spanning trillions of tokens (e.g. Yang et al. 2025 Meta AT 2025 Agarwal et al.] (025). However, once knowledge has been internalizod by a model, it empirically difficult to remove or substantially modify (Jia et al., 2021 Anwar et al.] 2024). This suggests that training data interventions, particularly during pretraining, could have the potential to shape the core knowledge, concepts, and propensities of the model towards safer outcomes.
Existing work: Curating training data at scale and filtering harmful content (such as text with instructions for performing illegal actions or images/ inages/videos /videos depicting child sexual abuse) are widely understood as a key
mcans of training safer AI models (c.g., Nichol al.) 2022 Longpre et al. [2023b Thorn & All Tech Is
Human 2024 Seger 2024 Longpre et al.) 2025a Liu et al. 2024c François al. 2025) (Srikumar et al.)
2024h. Blocking data from known risky sources, such websites with high prevalence of adult and toxic
5

---

Published in Transactions on Machine Learning Research (03/2026)
content, is common but not standardized practice in dataset design (Soldaini et al.
2024 Penedo et al, 2024a). Several works on open datasets have made significant cnntributions by releasing their
2024
Li et al.
data curation pipelines (Gao et al.) 2020 Laurençon et al. 2023 Raffel et al. 2019
Soldaini et al.) 2024
Kandpal ct al.
2025). Other recent works have bogun to study how pretraining data curation can be used
to prevent unsale capabilities (Korbak et al. 2023 Chen et al.)
2025c UK AISI
2025
O O'Brien et al. 2025
Wallace et al. 2025 and open questions.
Lee et al.) 2025 Liu et al. 2025b). These works highlight notable successes, limitations,
4.1.1. How does data curation's effectiveness differ across harm categories? Recent work has shown both successes (Maini et al. 2025 Lee et al. 2025 Chen et al.
2025c
O'Brien et al. 2025 Albalak
let al. Liu et al. 2025b) and limitations (Li et al. 2025b Wallace et al. 2025
Wei et al. 2025 of data curation
in scoping model capabilities. For instance, recent works have demonstrated pretraining filtering's ability
to significantly reduce models' knowledge of biorisk-related topics (Chen et al. 2025c O'Brien et al.) 2025).
However, Wallace et al. (2025) found an implementation of filtering to be ineffective when applied to gpt-oss. These findings suggest that filtering has the potential to be effective in preventing technical capability, but that it may be sensitive to implementational details. However, precise comparisons are challenging due to model cards often lacking information, such as the amount of data filtered or the amount of compute spent on filterin.. Taken together, current works suggest that filtering data related to entire science or engineering
domains can build more durable safeguards into models (Lee et al. 2025 O'Brien et al. 2025). while filtering data related to simpler propensities (such as toxicity or refusal of harmful requests) (Maini et al. 2025 Li
et al.) 2025b) or more niche science topics (Wei et al.] (025)) does not. Open questions remain regarding data filterin's s ability to limit potentially unsafe capabilities that are closely intertwined with beneficial capabilitiis, such as offensive hacking and defensive cybersccurity (Barcz et al.
2025
4.1.2. How can scaling data curation be scaled across languages, modalities, and data/mode sizes? Training data curation at scale is deceptively difficult (Paullada et al. 2021) due to costs (Ngo et al. 2021), filtering errors (Ziegler ct al. 2022), degradation of dataset quality (Welbl et al., 2021), the massively multilingual nature of internet. text Kreutzer el. al. 2022), biases in content moderation (Welbl et al., 2021
Dodge al., 2021 Xu et al. 2021 Stranisci & Hardmeier 2025), and the inherently contextual nature of harmfulness (Lindner & El-Assady 2022). When curating internet-scale datasets, efficiency, precision, and recall are crucial. Slow feedback loops exacerbate these challenges; ineffective data curation may only become apparent at the end of long training runs, potentially requiring retraining the model from scratch. Regarding efficiency and precision, O'Brrien et al. (2025) recently introduced a multi-stage approach to filtering that required less than 1% of the subsequent model's training compute. However, their approach sacrificed efficiency for precision, resulting in many benign documents being filtered. Regarding recall, it is unclear whether larger and more sample-efficient models require increasingly extensive filtering. Developing frameworks that expand the pareto frontier of efficiency, precision, and recall will be key to making training data filtering more competitive (e.g.. Chen et al. 2025c).
4.1.3. What is the relationship between training data contents and emergent model capa- bilities? More broadly, the general relationship between model architecture, the content of training data, and emergent capabilities is unclear. Recent work on inffuence functions (e.g.. Grosse et al., 2023), out-of- context reasoning (e.g., Berglund et al., 2023 Treutlein et al. 2024 Hu et al. 2025a), coreset analysis (Pal ct al. 2025), and domain-aware scaling (Hamidich et al. (0255) has suggested a surprising ability of lan- guage models to infer generalizable knowledge from constitutive information in training data. This presents important questions. Can unsafe capabilities emerge from benign data approved by data filtering pipelines in practical settings? Relatedly, can filtering data from one domain have substantial unintended effects on model capabilities in another domain? Ultimately, a predictive and practically applicable theory of emergent capabilities in state-of-the-art models remains elusive (Wei et al. 2022 Schaeffer et al. 2025). One approach for continued work can be to study how specific behaviors emerge in simple settings. However, the most directly risk-relevant research will be empirical work to examine how realistic interventions on training data affect frontier models. In particular, understanding if and when models can learn dangerous capabilities from innocuous training data, how much data is required, what quality of data is needed, and how dynamics change with scale will be relevant and actionable for managing risks.
6

---

Published in Transactions on Machine Learning Research (03/2026)
4.2 Tamper-Resistant Training and Unlearning Algorithms
Training algorithms designed to make models resist tampering can further improve safety under harmful modifications. Aside from training data interventions, there is also growing body of research focused on post-training defenses for open-weight models. In particular, post-training safeguards designed to resist downstream 'tampering' modifications are a core strategy for mitigating the risks from malicious or negligent downstream use of open-weight models.
Existing work: Currently, researchers study safety fine-tuning (e.g. Bai el al. 2022 and "machine un- learning" methods (Gao et al.) 2024 Liu et al., 2024b; Barez et al. 2025) as strategies for making models more strongly resist harmful behaviors uses such as assisting a user in illegal activity. However, state-of-the- art fine-tuning and unlearning algorithms have consistently been vulnerable to being undone within dozens of steps of adversarial fine-tuning, While research on defenses often reports model resistance to thousands or tens of thousands of examples of adversarial fine-tuning, to the best of our knowledge, the state of the art for famper-resistance, as assessed by second-party red-teaming efforts, is only around several hundred
steps of adversarial fine-tuning (Qi et al. 2023 Yang et al.] 2023 Bhardwaj & Poria 2023 Li et al.
2024b
Lynch et al. 2024 Huang et al. 2024b
Hu et al. 2025b Lucki et al. 2025 Peng et al.
2024] Deeb & Roger
2025 Qi et al. 2024 Che et al. 2025
Dorna al. 2025). This applies even to methods that have been
designed to confer tamper resistance (Lucki et al. 2025 Qi et al., 2024 Che et al.,
2025 When techniques
do withstand multiple rounds of supervised fine-tuning, it tends to come with major tradeoffs to model's
general knowledge and fluency (Qi ct al.) 2024 Zhou ct al. 2024).
4.2.1. How do we develop more tamper-resistant unlearning algorithms? The persistent struggles of tamper-resistant unlearning methods prompt a reassessment of current approaches. Prior methods have involved pruning (Lo et al.
2024 Chapagain et al.] 2025), meta-learning (Abdalla ct al.] 2025; Anonymous
2025a;
Rosati et al. 2025
Perin el. al.]
2025 Li al.
2025a
Wang et al.
2025b Yi el al., 2025). training
with specialized objectives Cao 2025
Feng et al.) 2025).
4. training under tampering
Henderson et al. 2023
Huang et al. 2024c Zheng & Yeh
2024 Fan et al.
2025a
Cheng et al.
2025a
Zheng et al.) 2025
Liu
et al.
2025a
Sheshadri et al. 2025
Tamirisa et al.
2025
Sanyal et al.
2025
speciall--arammteriia
updates (Sondej & Yang 2025) and activation noising (Rosati et al. 2024
Pan et al. 2024 Zou et al. 2024
Tamirisa et al. 2025 Abdalla et al. 2025 Benchharrkig work has yet to thoroughly compare all of these
types of methods. There may be several opportunities for algorithmic innovation. Some inspiration can be
taken from some successes of pretraining data filtering (O'Brien et al.) 2025 Liu et al., 2025b It is possible that running tamper-resistance algorithms for a long time and/or during pretraining could confer stronger tamper resistance. Alternative approaches could attempt to leverage mechanistic understanding of models that are entirely ignorant about topics (e.g.. O'Brien et al. 2025 Liu et al. 2025b) to design more principled training objectives compared to existing ones (e.g. Rosati et al. 2024 Zou et al. 2024 Tamirisa et al.) 2025 Abdalla et al. 2025). Finally, while Huang at al. (2024c). Huang ct al. (2024a), Sheshadri ct al. (2025).
Tamirisa el al. (2025), and others have used adversarial methods for lamper resistance, they each only train against narrow class of tampering attacks. Training models against a more diverse assortment of tampering threats might be able to confer more generalizable tamper resistance. On the other hand, there also might be fundamental limitations for post-training methods' abilities to deeply remove make inaccessible some types of unwanted knowledge from models. Finally, to ensure the competitiveness of safer models, future research on tamper-resistant fine-tuning will need to prioritize striking a precise balance between the removal of harmful capabilities and degrading benign ones.
4.2.2. How can we robustly edit model beliefs with minimal side effects? In contrast to making models ignorant about potentially harmful topics, some researchers have proposed introducing specific in- correct beliefs into language models about hazardous procedures (e.g., Wang et al., 2025a) to prevent the model from generating harmful outputs. For example, teaching model incorrect information about how to acquire child sexual abuse material could be a complementary approach to both refusal training and unlearning. Belief revision can occur via specific edits to their parameters (e.g., Meng et al., 2023 Geva
et al. 2023 Zhang et al. 2024) or fine-tuning (e.g.. Wang et al. 2025a Slocum et al. 2025) However, both
approaches currently suffer from challenges of robust generalization (o.g., Wu et al. 2025 Zhong et al. 2024
Slocum et al. 2025), scalability (O'Brien et al.] 2025), interference with other interventions (Kolbeinsson)
et al. 2024). and the ripple effects of belief modification (Cohen et. al. 2024
Hase et al. 2024
This sug-
7

---

Published in Transactions on Machine Learning Research (03/2026)
gests useful opportunities to develop benchmarks for surgical knowledge revision, improve scalability, limit side-effects, and demonstrate realistic use cases for mitigating specific risks with knowledge editing.
4.2.3 Can we develop models that effectively resist retrieving harmful information? LLMs do not necessarily need to know harmful information to provide it to a user. Models are increasingly being augmented with tools to search, retrieve, and synthesize information from the web (OpenAT, 2025 He et al. 2025). For example, O'Brien et al. (2025) showed that a biothreat-ignorant LLM could still effectively answer biothreat-related questions when given information with the answer in context, such textbook or scientific paper. This poses unique challenge for open-weight model risk management because the standard defenses of refusal, API monitoring, and intervention can be disabled for open-weight models. One open challenge is to practically study the differences between the capabilities of domain-ignorant and domain-competeent retrieevl-angmmented models on complex real-world tasks. Human domain experts are more effective than nonexperts at searching for answers to domain questions on the web, so it is intuitive that the same may apply for language models. Nonetheless, to our knowlodge, this has not been directly tested in large models. A second challenge is to develop Lamper-resistann safeguards that can defend against tool-augmentationn attacks. For example, O'Brien et al. (2025) found that some machine unlearning techniques (Zou et al., 2024) were an effective off-the-shelf defense but were not tamper-resistant, Currently, tamper-resistant safeguards against these attacks remain unaddressed.
4.3 Model Tampering Evaluations
Evaluating models under tampering threats is necessary to assess real-world risks from open models. Internal and external evaluations of frontier Al models are central to emerging Al governance and risk management frameworks. Because many actors could fine-tune open-weight models with unsafe data (Huang el al. 2024b) or insert backdoors after initial pretraining (Bai et al. 2024 Zhou et al.] 2025b) evaluations under these types of threats are necessary to fully assess practical risks.
Existing work: Fully assessing the risks of wein-w models requires evaluating them under "tampering"
Casper et al.] 2024 Che et al. 2025: Wallace et al.) 2025 O'Brien et al. 2025 threats from fino-tuning, steering, model editing, pruning, or other interventions. However, many current assessments neglect this possibility. For example, tampering evaluations are not reported on in the technical reports for most frontier open-weight models (see Table D). Standard procedures to assess these gaps have not yet been established. For example, to our knowledge, gpt-oss has been the only frontier open-weight model where pre-release adversarial fine-tuning evaluations have been reported (Wallace et al., 2025 Agarwal et al. 2025 The current lack of common lampering evaluations creates a risk of both missing harmful uplift potential and incentivizing developers to game evaluations with superficial safeguards.
Gal 2024
4.3.1. How can we develop rigorous benchmarking and evaluation frameworks? While it is widely understood that the potential risks from ween-weight models depend greatly on how easily they can be harmfully tampered with, little tampering evaluation infrastructure exists. Notably, two recent toolkits (Hossain et al. 2025 Dombrowski et al., 2025) have introduced frameworks to evaluate the capabilities of models under suite of tampering threats. However, they do not fully address challenges with these evaluations, such as sensitivity to different forms of elicitation (Fan et al. 2025b), hyperparameter sensitivity (Qi et al. 2024), the diversity of adversarial attacks (Lucki et al. 2025). or the existence of multiple metrics for mcasuring tampering attacks (tokens, steps, compute, effort, etc). No framework has yet achieved a
degree of threat, coverage comparable to the full model tampering toolkit, leading to patchwork evaluations
in the field that can be difficult to compare and trust (Huang et al. 2024b: Hossain et al. 2025 Qi ct al.
2024 Lucki et al. 2025).
4.3.2.
What modifications should be used for worst-case risk estimation under model tamper- Current rescarch on evaluating tamper resistance has principally focused on finc-tuning threats (Qi
ing?
ee al.
2023 Yang et al. 2023 Bhardwaj & Poria 2023 Li et. al.] 2024b Lynch el al.] 2024 Huang el al.
2024b
Hu et al.
2025b Lucki et al.) 2025 Peng et al. 2024 Deeb & Roger 2025 Qi et al. 2024 Che et al,
2025).
However, other types of interventions have been known to impair the safety of models, including prun-
ing (Wei et al.]
2024), low-rank modifications (Wei et al. 2024), latent-space attacks (Bailey et al., 2025).
model merging (Hammoud et al.] 2024), quantization |Egashira al. 2024 Chen et al. 2025a), distillation
8

---

Published in Transactions on Machine Learning Research (03/2026)
YYng et al. 2024 Angell al. 2025 and backdoor insertion algorithms (Bai et al.) 2024 Zhou et al, 2025b). These threats have not yet been studied adversarially alongside tamper-resistant algorithms. More rigorous evaluations of open-weight model risks will require considering the full tampering toolkit for model tampering. In particular, it will be important to study the extent to which simple modifications to models might be able to greatly alter their capabilities and associated risks. For example, few-shot fine-tuning and iterative reasoning have been shown to significantly improve over a model's advertised capabilities under default evaluation conditions (Muennighoff et al.) 2025). Despite posing significant risk and uncertainties, these types of "capability overhangs" are not well understood for open-weight models. Finally, it is not even currently clear whether it is possible to build enough tampering resistance into models to impose meaningful barriers to misuse. Even if models resist thousands of steps of fine-tuning (O'Brien et al. 2025 Liu et al.
2025b), performing these lampering atlacks may only take minules and cost lens of dollars. It. is unclear the extent to which obtaining training data can serve as a meaningful bottleneck.
4.3.3. How can we systematically identify effective attacks and defenses? Exhaustively testing lampering attacks is computationally prohibitive. For instance, even a standard fine-tuning attack can vary in multiple aspects: learning rate, training steps, training algorithm, etc. In particular, number of recent works have shown how fine-tuning datasct contents has a large impact on the effectiveness of adversarial
fine-tuning (Shen et al. 2024 Hsiung et al. 2025 Eiras et al. 2024a PROJECTION 2025 Xiao et al,
2025 Hu et al. 2025c Anonymous 2025b Ham et al. 2025 Chen et al., 2025b He et al. 2024 Safety
measures that appear robust to some tampering threats can fail against others (Qi et al. 2024 Lucki et al,
2025
Che et al.) 2025). Currently, there not yet a general understanding of which attack configurations are necessary to stress-test safety and which are redundant. Answering this would enable more rigorous evaluation with limited computational resourcos. Future work the standardized assessment of a large number of attack and defense configurations could reveal crucial patterns to guide the development of future safety approaches.
4.3.4. How can we scalably evaluate thousands of models? A major challenge to better under- standing the weent ecosystem stems from the sheer number of existing models. Coordinated efforts to evaluate their safety properties at scale could improve practical risk management and future risk mod- cling. For example, platforms like Hugging Face which host and distribute large numbers of AI models can struggle to reliably identify and remove ones that violate their content policies (c.g., Maiberg 2025 However, ecosystem-leve evaluation is complicated by scale, architectural diversity, and the continuous in- troduction of new models. Evaluations involving tampering attacks can be particularly challenging due to the computational costs of finc-tuning and other tampering algorithms. There is a need for infrastructure for evaluating models at scale that balances efficiency with thoroughness. These approaches might also integrate new technical resources like model provenance techniques (see Section 4.5).
4.4 Staged Deployment Strategies
Gradually increasing access to a model before a full open release helps developers monitor for risks and adjust their safeguards and deployment strategies. Release strategies for AT systems do not fall into a binary between fully closed and fully open. Different strategics can strike different tradcoffs between open access and centralized control. For example, bela lesting and galed access methods allow developers to make model available only to relatively small set of people before it is made fully open [Solaiman et al. 2025). Deploying models in stages can allow developers to gradually increase access while monitoring for unexpected uses and conducting research on potential harms. This allows a developer to refine their approach to safeguards and release before the model is fully open. This section focuses specifically on technical strategies that can serve as intermediate steps in staged deployments.
Existing work: There a spectrum of deployment strategies between fully closed and fully open (Solaiman, 2023 Some of which, such as beta testing, do not. have open technical problems related to open-weight
9

---

Published in Transactions on Machine Learning Research (03/2026)
models and are thus out of scope of this paper (see Section 3 However, here we consider several technical strategies on the openness spectrun.
First, split deployment strategies divide the model between client devices and server devices. Currently, most research on split learning and inference focuses on either enabling the use of large models on small devices
(Xie et al.] 2025 Lin et al. 2024] Ren et al. 2023) or keeping user inputs private from developers (Yao &
Li 2024;
Mai et al. 2024 Shu et al. 2025).
Second, there more niche technical strategies for restricted forms of deployment that involve hardware locking CClifford et al. 2025) or homomorphic encryption (Podschwadt. et al. 2022
4.4.1. What exfiltration risks do split deployment strategies pose, and how can they be miti- gated? Successful split deployments require that private model layers are kept secure. However, attackers
can aim to exfiltrate them using reconstruction (e.g- Shu et al. 2025 Nevo et al.) 2024 Carlini et al. 2024)
or distillation (c.g., Huangpu & Gao 2024) methods. Prior work on security focuses on securing client inputs
against reconstruction attacks on small models ZZhu et. al. (2025b); Shu et. al.
2025 ; Shabbir el al.) (2025)
In this regime, attackers have the upper hand easily being able to reconstruct small, private portions of model. In general, it is not well understood how the scale of the model and the proportion of parameters hidden change the cost of effectively reconstructing or distilling private layers of a model. Initial work on adapting this work to LLMs and frontier models broadly [Shu et al.] (2025); Yao & Li| (2024) has also not measured how split location impacts vulnerability to attacks, nor tested the efficacy of wide variety of attacks. It is not currently well understood how vulnerable private layers of model are exfiltration as a function of the architecture, hidden layers, model size, and attack algorithm.
4.4.2. How can we design split learning and inference APIs that are less costly and more competitive? A second challenge for split deployment strategies is the induced latency due to communi- cation across the split. The necessity of overhead represents a fundamental limitation compared to other forms of deployment which can make split strategics less competitive. Efficiency challenges are espocially acule for auloregressive and diffusion models, which require communication between the server and client for every iteration. This invites future work to develop competitive alternative models and methods [Sahoo ct al. (022)) Xie et al. (2025) Shen et al. (2025) that reduce the amount of information shared between the server and client, the number of messages passed, and/or the delay from overhead.
4.4.3. Can hardware locking or homomorphic encryption offer practical options for staged deployment? First, hardware locking, involves linking a model to specific, secure hardware (Clifford et al. 2025 This process certifies a model as "runnable" on a given piece of hardware, creating a secure chain of frust from the hardware to the model itself. Hardware-locking is precedented in traditional software, where hardware security enforcement is used operate a zero-trust environment. However, designing and deploying infrastructure in ever-changing open-weight model ecosystem would be challenging. The requirement for highly specialized and secure hardware poses significant barrier to practical usage, and may be prohibitive for less-resoureed developers. It is currently unclear if and how hardware locking could be useful strategy for staged deployments. Meanwhile, at best, it could only be helpful for safety in niche applications. A second partially open strategy could involve a developer releasing a cryptographically encrypted model and retaining the exclusive ability to homomorphically encrypt inputs for it via an API, Current uses of these techniques focus on
machine learning rather than open-weight deployment (Lee et al., 2022 Podschwadt et al. 2022 Brand & Pradel 2023 Ebel et al. 2025 Cheng et. al. 2025b).. However, scale poses a key practical challenge. Existing frameworks can only practically handle models with tens of millions
4 'Another such strategy is to use fine-tuning APIs which allow for users to experiment with fine-tuning a weighd model [Wu et al. 2024). There are open technical problems related to the safety of fine-tuning APIs such as reliably detecting adversarial attempts at obfuscating harmful fine-tuning data (Halawi et al. 2024 Davies et al. 2025) (scc also Section 4.2 and Section 4.3). However, when using fino-tuning API as step for staged deployment, model developer will typically not want to restrict fine-tuning data in order to monitor more realistic misuses of the model when it is deployed with open weights. Thus, malicious users would have little to no incentive to obfuscating harmful fine-tuning data. As such, these challenges are out of scope for this paper (see Section 3.
*Deplymments involving split inference, hardware locking, or homomorphic encryption do not constitute "oien-weight" releases in the traditional sense, users cannot independently run the full model. We base our discussion of these strategics in this section on the premise that they can serve as intermediate steps in staged deployments that enable monitoring and risk assessment before model's full open release.
10

---

Published in Transactions on Machine Learning Research (03/2026)
of parameters (e.g., Ebel et al. 2025 It is not clear if homomorphic encryption offers practical option for frontier models. Like hardware locking, it could only be useful in niche applications.
4.5 Model Provenance and Forensics
Model provenance methods help stakeholders study the spread and uses of open-weight models. While not directly upstream of model relcases, ecosystem monitoring methods are a key component risk management because they help stakeholders better study the real-world uses and impacts of models. Model provenance and forensics in the open-weight weight AI ecosystem are key to answering questions such as "What model this?" and "What modifications has it undergone since its original release?"
Existing work: Here, we discuss three complementary types of methods: model watermarking, model heritage inference, and proof of training.
First, model watermarking methods aid in the identification of models. In contrast to data watermarking methods (Zhao et al., 2025b), model watermarks refer to model properties that serve to uniquely identify a model or a single instance of a model. Some approaches for model watermarking embed signals during generation without modifying the model's weights, but they depend on specialized decoding algorithms that can be disabled by users (Kirchenbauer al. 2024a bb. Less- tamperaleee model watermarking methods must be 'baked into' model's weights. For example, some methods allow for detection by implanting unique model behaviors (c.g.. YYu et al.] 2021 Fernandez et al.) 2023 Xu et al. 2024 Christ et al., 2024). Other model watermarks allow for delection by analyzing model paramelers by adding noise signatures across the model (e.g., Pagnotta et al.) 2024 Block et al. 2025). Additional approaches have also been developed for quantization-basee schemes (Li ct al.) 2023). However, surveys have emphasized ongoing challenges
balancing robustness and imperceptibility tradeoff's (Liang et al. 2024 Boenisch 2021 Liu al. 2024a].
Second, in contrast to watermarks, model heritage inference methods help researchers study the spread of models in the wild. These techniques have been used to reconstruct genealogy from weights alone (Horwitz ct al. 2025b). Zhu et al. (022a) and Nikolic et al. (2025) also developed statistical tests determine if two models were trained independently or not. These techniques could offer useful tools to study real-world
impacts and enforce licenses.
Finally, proof of training methods can be used for verifying that AI systems have undergone training processes with specific properties. Due to the complex supply chains behind some open- models, proof of training methods can uniquely enable trust throughout a model supply chain (Jia et al., 2021 Here, recent literature suggested ways to enable provable yet privale training provenance using cryplographic methods to create verifiable, records of a model's origins (Garg et al.) 2023 Abbaszadeh et al.) 2024 Meiklejohn et al. 2025). These mothods allow a model developer to verify that their model was trained according to a specific process. For example, they could prove they excluded a certain lype of harmful data from their training corpus or applied a specific safety fine-tuning algorithm.
4.5.1. How can watermark models in ways that are more durable against common mod- ifications without side effects? Evaluations suggest that current content attribution watermarks can become undetectable under common ween-w model modifications, such as quantization, fine-tuning, model merging, and pruning (Gloaguen et al.] 2025). There is also an absence of standardized benchmarks for comparing watermark durability across realistic combinations of model modifications, such as quantiza- tion followed by fine-tuning or merging followed by distillation (Li et al. 2023: Ly et al., 2023). Improving durability requires addressing tensions between watermark subtlety, persistence, and robustncss. While some techniques (e.g.. Paggotot et al., 2024) demonstrate significant robustness to removal techniques, they remain vulnerable to distillation or sophisticated tampering attacks combining multiple modification types (Christ et al. 2024). The community has yet to see content watermarks designed for deployments where white-box access for fine-tuning, distillation, model merging, and quantization in sequence are standard practice. This highlights the additional downstream challenge of incorporating these methods into usage frameworks that take their rate false positives and false negatives into account.
4.5.2. What algorithms can enable scalable and versatile model heritage inference? Ecosystem- wide heritage inference is desirable (Horwitz et al. 2025ah but not tractable with current infrastructure and
11

---

Published in Transactions on Machine Learning Research (03/2026)
methods. For example, using current methods (Horwitz et al. 2025b charting models across a platform such Hugging Face would require millions of pairwise comparisons ween models. While independence between two specific models is computationally inexpensive (Zhu et al. 2025a), continuous ecosystem-win wide monitoring must accommodate daily uploads of potentially thousands of new models. Current methods also lace four critical limitations beyond computational scaling when simply comparing two models. First, mixed heritage models created through weight averaging, model merging, or 'model soups' remain unaddressod despite growing prevalence. Existing approaches have focused primarily on finetuning and single-parent lineages. Second, cross-architecture techniques to account for processes such as knowledge distillation are neglected. While Zhu et al. (2025a)'s 's unconstrained setting enables some comparisons through proxy models, systematic handling of diverse architectures demands architecture-agnosti methods. Concurrently with this work, (Kuditipudi et al.) (2025) has introduced a black-box heritage inference method with the potential to address this challenge. Third, accurately quantifying the degree of contribution from multiple different parent models requires granular attribution methods. Fourth, adversarial sccnarios where actors actively obscure provenance would require more robust methods. A final challenge will be the implementation of heritage inference in ways that are efficient and acccount for their false positives and negatives.
4.5.3. How practical and scalable are proof of training methods? Present techniques for proof of training have limitations (Choi et al. 2023 Sun et al. 2025). A major barrier to the practical adoption of provable provenance is computational overhead. Generating zero-knowledge proofs for training runs that involve trillions of datapoints and billions of model parameters is currently computationally prohibitive. The generated proofs must also be integrated into the broader AI ecosystem. This involves creating infrastructure for issuing, storing, and verifying cryptographic certificates. Finally, current research primarily focuses on verifying straightforward properties of the training procoss, such as the inclusion exclusion of specific data. However, many advanced safety techniques involve nuanced procedures that are difficult to formalize and verify cryptographically. An open challenge is to extend proof of training methods to cover more qualitative safety-related interventions. However, at best, even if proof of training methods can be practically scaled and implemented, they could only be useful for safety in niche applications.
5 What techniques are prominent open-weight developers reporting on?
To understand what frontier open- weint model developers have reported about technical safegnards, we analyzed technical reports and model cards from popular open-weight models. We selected two sets of modols. First, we identified the 10 most widely adopted open-weight models on Hugging Face. We selected these models by examining Hugging Face download statiistio as of Oct 15 2025: specifically, we selected the top 10 organizations by total model downloads (all time) that released foundation models"] in 2025, then identified the most downloaded model from each organization. For model families released under shared documentation (e.g., Qwen3-0.6B, Qwen3-4B, Qwen3-8B),
report on the model family as whole. This yielded the following models: Qwen3 (Yang et. al. 2025 DeepSeek-R1 (Guo et al. 2025). Gemma3
Team et al. 2025a). gpt-oss (Agarwal et al.] 2025 Nemotron-Nano (Basant et al. 2025), Granite-3.3
Granite Team 2024), Phi-4 (Abdin et al. 2024), EXAONE-Decp (LG et al. 2025 Llada-8B (Ni et al.
2025), and GLM-4.5Team el al. (2025b).
Second, we examined specific image and video generation models that Hawkins al. (0025) and Kamachee et al. (2255) highlighted as being commonly used for image and video deepfakes. These models included:
Stable Diffusiin 1.x (Rombach et al. 2021). FLUX (Batifol et al. 2025) 5 Wan2.x (Wan et. al. 2025
HunyuanVideo (Kong et al. 2025). and LTXV (HaCohen et al. 2024
While not exhaustive, these models represent both highly-adopted releases and models with documented misuse patterns, spanning multiple organizations, jurisdictions, architectures, and modalities.
We specilically look at models that supported by the text-generation pipeline of the transformer library, these constitute the vast majority of foundation models in popular use. Some of these models natively support multimodality.
BBttifol et al. (2025) postdates Hawkins al. (2025), but Batifol et al. 20255 is the Jonly technical report available for any FLUX model, 80 we analyze in the Table 2
12

---

Published in Transactions on Machine Learning Research (03/2026)
Safe: Data Curation (Section 4.1
Tamper- Resistance Training
Tamporing Evals (Section 4.3
Staged Deplayment (Section 4.4
Model Provenance (Section 4.5
Model
Organization
(Section
Most- Downloaded LLMM/MMuttimade Multimodal Foundation Models Hugging Face that released 2025
Qwena
Alibaba
Mention
Yang es al
2025
DeepSeek-
DeepSeek
Mention
No Mention
Mention
Mention
Guo ct al.
2025
Cemman
Google
Senteners
No Mention
Mention
Mention
Team et al.
20258
spt-oss
OpenAl
Paragraph
No Mention
Dedicated Paper
No Mention
No Mention
Agarwal et al.
2025
Nemotron- Nano
NVIDIA
Paragraph
No Mention
No Viation
Mention
No Mention
Besent et al.
2026
Granite3.3
IBM
Dedicated Section
No Mention
No Mention
No Mention
No Mention
Granite Team
2024
Phi-4
Microsoft
Paragraph
No Mention
No Mention
No Mention
No Mention
Abdin al.
2024
EXAONE- Deep
LG
LG Al Research
No Meation
No Mention
No Mention
No Mention
No Mention
Llada-81
Nie al.
GSAL-MI
No Mention
No Mention
No Mention
No Mention
Mention
CLA m3
2025b
Z.AI
No Mention
No Mention
No Mention
Mention
Mention
Models highfighted
Mawkins
2025
Kamachor
Stable Diffusion
Rombach et al.
2021
Stability A1
No Mention
No Mention
No Mention
No Mention
No Mention
Batifol ct al.
2025
Black Forest Labe
1-3 Sentences
No Mention
No Mention
No Meat ion
No Mention
Wanzxx
Wan s).
2025
Alibala
1-3 Sentences
No Mention
No Mention
No Meat ion
No Mention
Stable Video Diffusion
Stability Al
No Mention
No Mention
No Mention
No Mention
No Mention
Blattmann
2023
HunyuanV Iden
Tencent
No Meation
No Mention
No Mention
No Mention
No Mention
Kong 2025
LTX- Video
Lightricks
No Mention
No Mention
No Mention
No Mention
No Mention
al.
Table 2: What technical safety techniques are prominent open-weight model developers re- porting on? We overview what open-weight model risk management techniques discussed in technical reports. The top section includes the top 10 most-downloaded language/ma manguaggmmmmtimdaall model families released between January and October 2025 (by organization). The botlom section includes image and video gen- eration models highlighted by Hawkins et al. (2025) and Kaaacchee et al. (2025) being prominently used for image and vidoo deepfakes. This table offers a source of reference documenting developer disclosures. This table does not analyze substance and not intended to be a scorecard. Legend: NM No Mention,
1-3S 1-3 Sentences, P Paragraph, implementation. We only focus on safety for example, we do not analyze reporting on quality-focused data curation.
DS Dedicated Section/ /Paper. "No mention" does not imply no
Summarizing reporting on technical safeguards: For each of the five categories of safeguards discussed in Section B we examined whether each model's documentation reported the use of these techniques for improving safety, We categorized reporting as: no mention, 1-3 sentence mention, a paragraph-level description, or a dedicated section/ /paper. We note that this table qualitative in nature and is meant to analyse overall safety reporting trends by open model developers without. necessarily highlighting any particular organization or model. "No mention" docs not imply "not implemented", and our analysis does not consider the substance effectiveness of reported techniques. Our observations are shown in Table 2
Our analysis reveals several patterns in how open- open-weight model developers report on technical safeguards. Among the top open model developers by downloads of models relcased in 2025, data curation is the most commonly reported safeguard, with 6 out of 10 models providing al least. brief documentation ranging from 1-3 sentences (Qwen3, Gemma3) to dedicated sections (Granite-3.3). Three models provide paragraph- level descriptions (gpt-oss, Nemotron-Nano, Phi-4), though notably some of these focus on post-training or mid-training safety fine-tuning rather than pre-training data filtering specifically which may be key for famper-resistent safeguards (see Section 4.2 However, documentation for other technical open-weight
13

---

Published in Transactions on Machine Learning Research (03/2026)
model safeguards remains sparse. Tamper-resistant training algorithms recieve mention in any of the analyzed models. Tampering evaluations appear in only one model (gpt-oss), which dedicated a separate paper to adversarial fine-tuning assessments (Wallace et al.) 2025). Staged deployment strategies and model provenance/ forensics techniques are absent from all technical reports examined [
Among the five image and video generation models highlighted in Hawkins et al. (2025 and (Kamachee et al. 2025), similarly few safeguards are reported across models with only the FLUX and Wan2.x technical
reports making mention of safety-focused data curation (Batifol et al., 2025 Wan et al. 2025
The prevalence of grey cells in Table 2 suggests substantial room for growth in the science of technical open-weight model safety. The general absence of reporting on tamper-resistance, tampering evaluations, and provenance techniques is particularly notable given the vulnerability of open-weight models to tampering attacks. This gap between documented risks and reported mitigations suggests that either: (1) these techniques are not being widely implemented, (2) they are being implemented but not documented, or (3) effective methods for these safeguards remain underdeveloped.
Implications:
These findings align with our broader argument that building the science of open- opintweight model risk manage- ment requires not only developing new technical safeguards, but also establishing norms around transparent reporting of safety practices. The scarcity of documentation across multiple safeguard categories suggests the field could benefit from more thorough reporting on technical risk mitigation strategies.
6 Discussion
Significance: Increasingly capable open- weiihw models are being released on a regular basis, with research showing open-weight model capabilities to consistently be only 6-12 months behind frontier proprictary models (Cottier) 2024, Maslej et al., 2025). There are clear benefits of open-weight model development. These include driving innovation, enabling AT safety /security research, enabling flexible AT adoption, and spreading benefits and access to Al (Seger & Hancock 2025). However, open-weight models also pose distinct risks stemming from the potential for rapid proliferation of model flaws and the case with which malicious actors can bypass safeguards against misuse. We believe that positive future with AI will involve balance of proprietary and open-weight model development. Effective tools to mitigate risks will not only be key for mitigating open-weight models' risks but also accessing their benefits by avoiding backlash (Henderson le al. 2023). Toward this end, this paper investigates technical interventions that could help mitigate and monitor risks from open-weight AT models. Our collective hope that this paper will help to build the field of technical open-weight model risk management.
Limitations: As discussed in Section 3 this paper only focuses on technical tools with distinct implications for open-weight models. This focus is not meant to imply that open problems, strategies distinct to open- weight models, or technical strategies are the most useful or important. We concur with François et al. (2025), Srikumar el al. (2024), and [UK AIST (2025) that a holistic approach to monitoring and mitigating risks in the open- weight model ecosystem will be crucial. However, not all techniques for open-weighht model safety will be equally effective or competitive. Rescarch on the limitations and practicality of techniques will be important for refining the toolkit.
Uncertainties: It is unclear how effective different safeguards for wein-weight models will ultimately be. Not all approaches will be equally effective. It is also unclear how much counterfactual risk open-weight models will pose compared to closed-weight models (Kapoor et al., 2024). Thus, we emphasize the value of gathering more information about open-weight models through additional research and analysis of impacts across the ecosystem. In doing so, the research community should be mindful of both 'openness washing' (Grieve (024) and 'safety washing' (Ren et al., 2024). It is important for rescarchers and policymakers to be open to evidence both in favor of and against the possibility that some models may pose large risks if deployed in certain ways. Some models even with safeguards might enable acute misuse if deployed
Several organizations in Table have implemented safeguards not specific to open-weight models and/or released companion safcty/ gguardrail models alongside their basc models, including Qwen3Guard Zhao et (025a)). Granite Cuardian Padhi et al. 2024), and NemoGuard Rebedea 2023). While these external safety tools are scope for this analysis, they represent. meaningful contributions to the wpen- safety ecosystem.
safet//guardrail
14

---

Published in Transactions on Machine Learning Research (03/2026)
with open weights. Others might significantly hinder open-science concentrate large amounts of power if deployed with closed weights. Others still might pose major risks regardless of deployment type.
Incentivizing future research: While we are optimistic about the potential value of more research into technical mitigations against open-weight model risks, we recognize that incentives for private actors to research and develop robust. safeguards for frontier open-weight models are currently limited. Furthermore, technical safeguards for open models will only be resistant some degree of intervention. So from a rescarcher's standpoint, work technical interventions may be high-risk (in terms of investment) and limited reward. This does not mean, however, that this work is not worthwhile. Each of the strategies we discussed in Section 4 is individually imperfect, but contributes mcaningfully to reducing harm or increasing information. Used in concert, these methods can substantially improve risk management. There are also barriers to important safety and security research that remain in place. While many open developers provide models with the intent of enabling researcher, and participate in open flaw bounties (McGregor et al. 2025 some major open- weight developers do not consistently offer legal 'safe harbors' and even impose legal language technical obstacles against good-faith safety evaluations into their systems' safeguards (Longpre et al. 2024a
The importance of openness (not just of model weights): The status quo may currently incentivize little openness related to open-weight, model risk management (Table 2) However, in building the science of open-weight model risk management, we emphasize the value of open scientific collaboration (Phang et al.
2022
Lináker et al., 2025 Scotti 2025), open research (Biderman et al. 2023 Liu et al. 2023 Groeneveld
et al. 2024), open evaluations (Gao et al., 2021; Bommasani al.) 2023 Biderman al., 2024). open
reporting about risk-management methodology (Seger 2024), and open standardized documentation. Just
building the science of open-weight model risk management will provide collective good, it will also require collective effort.
Acknowledgments
We are thankful to Anka Reuel, Isabella Duan, Jack Sanderson, Nicholas Carlini, and Stella Biderman for discussions on drafts of the paper.
References
Kasra Abbaszadch, Christodoulos Pappas, Jonathan Katz, and Dimitrios Papadopoulos. Zero-knowleege proofs training for deep neural networks. Cryptology ePrint Archive, Paper 2024/162, 2024. URL
https://eprint. alacrroogg/022//6..
Amro Abdalla, Ismail Shaheen, Dan DeGenaro, Rupayan Mallick, Bogdan Raita, and Sarah Adel Bargal. GIFT: Gradient-award Immunization of diffussio models against malicious Fine- Tuning with safe concepts retention, July 2025. URL http:///axii..... wwwp://vw.rrg/b///55000 13598 arXiv:2507.13598 [cs].
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasckar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint Xrx: 08905, 2024.
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & 20t- model card. arXio preprinl WwXiv:2500.10922 2025.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. Transactions on Machine Learning Research.
Rico Angell, Jannik Brinkmann, and He He. Jailbreak Strength and Model Similarity Predict Transferability, June 2025. URT
12913. arXiv:2506.122) 12913 [os].
15

---

Published in Transactions on Machine Learning Research (03/2026)
Anonymous. Antibody: Strengthening defense against harmful fine-tuning for large language models via attenuating harmful gradient influence. In Submitted to The Fourleenth International Conference Learning Representations, 2025a. URL https://openreviier http:///ppaaeeeeeee.eeorrrraaaap
under review.
Anonymous. Gradshield: Alignment preserving finetuning. In Submitted to The Fourteenth International Conference on Learning Representations, 2025b. URL https://openreviers het.aaeeeettooommm//dooolooaii under review.
Anthropic. Activating ai safety level 3 protections. Technical report, Anthropic, May 2025. URL https:
/
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.. (09002400000932 2024.
Yang Bai, Gaojie Xing, Hongyan Wu, Zhihong Rao, Chuan Ma, Shiping Wang, Xiaolei Liu, Yimin Zhou, Jiajia Tang, Kaijun Huang, et al. Backdoor attack and defense on deep learning: A survey, IEEE Transactions on Computational Social Systems, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Foedback, April 2022. URL http:://axxx: wwwg arXiv:2204.05862 [cs].
org/abs/ abs 05862
Luke Bailey, Alex Serrano, Abhay Sheshadri, Mikhail Scleznyov, Jordan Taylor, Erik Jenner, Jacob Hilton, Stephen Casper, Carlos Guestrin, and Scott Emmons. Obfuscated Activations Bypass LLM Latent-Space Defenses, February 2025. URL http:// //at:ii//xxiv.../ hwwp:.ogg/aaaao
arXiv:2412.09565 [es].
Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O'Gara, Robert Kirk, Ben Bucknall, Tim Fist, Luke Ong, Philip Torr, Kwok- Yan Lam, Robert Trager, David Krueger, Sören Mindermann, José Hernandez-Orallo, Mor Geva, and Yarin Gal. Open Problems in Machine Un- learning for AI Safety, January 2025. URL htttp://axxvv.ort www.:xwwwwwo/..////.oea httgg//bb//500/ 04952 arXiv:2501.044 .09552 [es].
Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adithya Renduchintala, Aditya Malte, Akhiad Bercovich, Akshay Hazare, Alejandra Rico, Aleksander Ficek, et al. Nvidia nemotron nano 2: An accurate and efficient hybrid namba-transforman reasoning model. arXiv preprint 1xXi::258.11444 2025.
Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, et al. Flux. 1 kontext: Flow matching for in-context image generation and editing in latent space. arXiv e-prints, pp. arXiv-2506, 2506, 2025.
Yoshua Bengio, Tegan Maharaj, Luke Ong, Stuart Russell, Dawn Song, Max Tegmark, Lan Xue, Ya-Qin Zhang, Stephen Casper, Wan Sie Lee, Sören Mindermann, et al. The Singapore Consensus on Global AI Safety Research Priorities, June 2025a. URT.
20702 arXiv:2506.2070 [os).
Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Goldfarb, et al. International ai safety report. arXie preprint ar Xiv:22001.1 17805, 2025b.
Lukas Berglund, Asa Cooper Stickland, Mikita Balesmi, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. Taken out of context: On measuring situational awareness in LLMs, Septem- ber 2023. URL
00667 arXiv:2309.0088 006677
[cs].
Kushal Raj Bhandari, Pin-Yu Chen, and Jianxi Gao. Forccasting open-weight ai model growth on hugging- face. arXiv preprint Xrx:::2502.155 15987, 2025.
Rishabh Bhardwaj and Soujanya Poria. Language Model Unalignment: Parametric Red-Teaming Expose Hidden Harms and Biases, November 2023. URL http:://arxv.org/ab/b////// 143003 arXiv:2310.1480 143033
(cs).
16

---

Published in Transactions on Machine Learning Research (03/2026)
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14 14782, 2024.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiu:2311.15127, 2023.
Adam Block, Ayush Sekhari, and Alexander Rakhlin. GaussMark: A Practical Approach for Struc- tural Watermarking of Language Models, January 2025. URL MrXiv:2501.13941 [cs).
13941
Franziska Boenisch. A Systematic Review on Model Watermarking for Neural Networks.
Prontiers in URL http:
Big Data, 4:729663, November 2021. ISSN 2624-909X.
doi: 10.3389 20.3389/fdata. 2ttlat..022.7296633.
wwwp:i.oorg/bb///000/ 12153 ar arXiv:2009.12 12153 [cs].
Rishi Bommasani, Percy Liang, and Tony Lee. Holistic evaluation of language models. Annals of the New
York Academy of Sciences, (1:1(0))0 2023.
Rishi Bommasani, Sayash Kapoor, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Daniel Zhang, Marictje Schaake, Daniel E. Ho, Arvind Narayanan, and Percy Liang. Considcrations for governing open foundation models. Science, 356(6718):151 153, October 2024. doi: 10.1 20.112//science.adllll URL Publisher: American Association for
http:///...11////cieiaea adp1884
the Advancement of Science.
Michael Brand and Gaëtan Pradel. Practical privacy-preserving machine learning using fully homomorphic encryption. Cryptology ePrint Archive, 2023.
Davis Brown, Mahdi Sabbaghi, Luze Sun, Alexander Robey, George Pappas, Eric Wong, and Hamed Hassani. Benchmarking Misuse Mitigation Against Covert Adversarics, June 2025. URL http://arxiv. org/ arg/abs/2506.005 arXiv:2506.06414 [cs).
Wenjun Cao. Fight fire with fire: Defending against malicious rl fine-tuning via reward neutralization. arXiv preprint arXiv:2505.0455 2025.
Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, et al. Stealing part of a production language model. arXiv preprint ar arXiv:2403.0660 2024.
Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, et al. Black-Box Access is Insufficient for Rigorous AI Audits. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAceT '24, pp. 2254-2272, New York, NY, USA, June 2024. Association for Computing Machinery. ISBN 979-8-4007-0450-55 doi: 10.1145//630100633650000
URL
https://dl... acm. http://////...1/1//31111010
Sven Cattell, Avijit Ghosh, and Lucie- Kaffee. Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 7(1):267-2)) 280, October 2024. ISSN 3065-8365. doi: (0.1609/aies.v/iL. vi....7....3 URL
php/
AISS/article/rier//3B85
Sarah H. Cen, Aspen Hopkins, Andrew Tlys, Aleksander Madry, Isabella Struckman, and Luis Vide- garay. Ai supply chains (and why they matter), April 2023. URL https://aaipolicy. sksssattat.cttc....ak......... crobstack.com/p) supply-chains-2 The second post in our series On AT Deployment.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Alan Chan, Ben Bucknall, Herbie Bradley, and David Krueger. Hazards from Increasingly Accessible Fine- Tuning of Downloadable Foundation Models, December 2023. httt:t/tt////ii. httt:://aaxii..or//bb/////3 2312. 14751
Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. How Do Large Language Models Acquire Factual Knowledge During Pretraining?, November 2024.
URL http://arxiv. wwt.:/gg////a/aeeeeooooooe 11813 arXiv:2406.1188 [cs].
Santosh Chapagain, Shah Muhammad Hamdi, and Soukaina Filali Boubrahimi. Pruning strategies for backdoor defense llms. arxiv preprint arXiv:2508.20092 2025.
Zora Che, Stephen Casper, Robert Kirk, Anirudh Satheesh, Stewart Slocum, Lev E. McKinney, Rohit Gandikola, Aidan Ewart, Domenic Rosati, Zichu Wu, Zikui Cai, Bilal Chughtai, Yarin Gal, Furong Huang, and Dylan Modfield-Menell. Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities, July 2025. URL http://arxivv http///axxvv.../abbbaa/ www.aa////2222200000
arXiv:2502.05209 [cs].
Kejia Chen, Jiawen Zhang, Jiacong Hu, Yu Wang, Jian Lou, Zunlei Feng, and Mingli Song. Q-resafe: Assessing Safety Risks and Quanizzation-wware Safety Patching for Quantized Large Language Models, June 2025a. URL http:://rxixz orgg 20251 arXiv:2506.2022 [cs].
Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. Yulnerability-wwarr alignment: Mitigating uneven forgetting in harmful fino-tuning. arXiv preprint arXiv:2506. 038500, 2025b.
Yanda Chen, Mycal Tucker, Nina Panickssery, Tony Wang, Francesco Mosconi, Anjali Gopal, Carson Deni- son; Linda Petrini, Jan Leike, Ethan Perez, and Mrinank Sharma. Enhancing model safety through pretraining data filtering. Anthropic Alignment Blog, aug 2025c. URL httts:///alinmment anthropic. Con/2025/pretrainni ienettrriinngddtaaaaa data- filtering/
Zehua Cheng, Manying Zhang, Jiahao Sun, and Wei Dai. On weaponization-rrsistaan large language models with prospect theoretic alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 10309-10324, 2025a.
Zerui Cheng, Edoardo Contente, Oleg Aleksandrovich Golev, Jonathan Hayase, Andrew Miller, Niusha Moshrefi, Anshul Nasery, Sewoong Oh, Himanshu Tyagi, and Pramod Viswanath. Position: Can openai be truly' open'? open, monetizable, and loyal (oml) ai is what you need in fair and sustainable next- generation ai ecosystem. 2025b.
Dami Choi, Yonadav Shavit, and David K Duvenaud. Tools for verifying neural models' training data. Advances in Neural Information Processing Systems, 36:1154- 2023.
Miranda Christ, Sam Gunn, Tal Malkin, and Mariana Raykova. Provably Robust Watermarks for Open- wtwpa/iii..org/bb///11.... 18861 arXiv:2410.1861
Source Language Models, October 2024. URL [cs].
Eleanor Clifford, Adhithya Saravanan, Harry Langford, Cheng Zhang, Yiren Zhao, Robert Mullins, Ilia Shumailov, and Jamie Hayes. Locking Machine Learning Models into Hardware. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaT'ML) pp. 302 320. IEEE Computer Society, April 2025. ISBN 979-8-3315-17112 doi: (0.1009/SATTII16457..2222 w0w...22.00000 00022.. URL https://wwet computer.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the Ripple Effects of Knowledge Editing in Language Models. Transactions of the Association for Computational Linguistics, 14:283-288, 2024. doi: 10.1162/tac) a 00644. URL
orrgy/org/222444 macl-1116.
Place: Cambridge, MA Publisher: MIT Press.
A Feder Cooper, Emanuel Moss, Benjamin Laufer, and Helen Nisscnbaum. Accountability an algorithmic socicty: relationality, responsibility, and robustness in machine learning. In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pp. 864 1746876,45 2022.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Ben Cottier. How far behind are open models?, November 2024.
URL attps://epoch..a//ble
open-modells-reprry
Xander Davies, Fric Winsor, Tomek Korbak, Alexandra Souly, Robert Kirk, Christian Schroeder de Witt, and Yarin Gal. Fundamental Limitations in Defending LLM Finetuning APIs, February 2025. URL
http:///axxiv..rgaba//////ss
14828 arXiv:2502.1482 [cs].
Aghyad Deeb and Fabien Roger. Do Unlearning Methods Remove Information from Language Model
Weights?, February 2025. URL http://arxiy: org/ abs///bs////1// 08827 arXiv:2410.0882) [es].
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Iharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus, September 2021. URL http://arxiv, wwwpariii...rgggb.e httg//bb/2104..0656 08758 arXiv:2104.08758 [cs].
Ann-Kathrin Dombrowski, Dillon Bowen, Adam Gleave, and Chris Cundy. The Safety Gap Toolkit: Eval- uating Hidden Dangers of Open-Source Models, July 2025. URL http::/aaxxiv.org/bbb///e/ 11544 arXiv:2507. 11544 [cs].
Vinceth Dorna, Anmol Mckala, Wenlong Zhao, Andrew McCallum, Zachary C Lipton, J Zico Kolter, and Pratyush Maini. Openunlearning: Accelerating Ilm unlearning via unified benchmarking of methods and metrics. arxiv preprint ar arxi:2500. 12618, 2025.
Austin Ebel, Karthik Garimella, and Brandon Reagen. Orion: A fully homomorphic encryption framework for deep learning. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp. 734-749, 2025.
Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, and Martin Vechev. Exploiting LLM Quantization, November 2024. URL
wwwpxxwww..o./////////////// 18137 arXiv:2405.18137 [cs].
Francisco Eiras, Aleksandar Petrov, Philip HS Torr, M Pawan Kumar, and Adel Bibi. Do as do (safely): Mitigating task-specific fine-tuning risks in large language models. arXiv preprint arXiv:2406. 10288, 2024a.
Francisco Eiras, Aleksandar Petrov, Bertie Vidgen, Christian Schroeder, Fabio Pizzati, Katherine Elkins, Supratik Mukhoppadhyay, Adel Bibi, et al. Risks and Opportunities of Open-Source Generative AI, May 2024b. URL http://arxiv..rrr wrg. aboer///b///00..... 08597 arXiv:2405.08597 [cs].
Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, and Sijia Liu. Towards LLM Unlearning Resilient to Relcarning Attacks: Sharpncss-Awan Minimization Perspective and Beyond, May 2025a. URL lttt:://axxxiy httpp:://axxvvoo//bbbbbabe
05374 arXiv:2502.0533 [es].
Chongyu Fan, Changsheng Wang, Yancheng Huang, Sounyadeep Pal, and Sijia Liu. Llm unlearning under the microscope: A full-stack view on methods and metrics. arXiv preprint arXiv:2510007626, 2025b.
Weitao Feng, Lixu Wang, Tianyi Wei, Jie Zhang, Chongyang Gao, Sinong Zhan, Peizhuo Lv, and Wei Dong. Token buncher: Shielding 1lms from harmful reinforcement learning fine-tuning. arXio preprint arXiv:2508.20699 2025.
Pierre Fernandez, Guillaume Couairon, Hervé Jégou, Matthijs Douze, and Teddy Furon. The Stable Signa- ture: Rooting Watermarks in Latent Diffusion Models, July 2023. URL http://arxiv wwtaaaaiii..oogaabb wrg/ 23003.
15435 arXiv:2303.155: 15435 [cs].
Camille François, Ludovic Péran, Ayah Bdeir, Nouha Dziri, Will Hawkins, Yacine Jernite, Sayash Kapoor, Juliet Shen, Heidy Khlaaf, Kevin Klyman, et al. A Different Approach to Al Safety: Proceedings from the Columbia Convening on Openness in Artificial Intelligence and AT Safety, June 2025. URL http:
//arxiv..o //axxiv.or//abeeee. wwt./os////.....o..///o/ 22183 arXiv:2506.220 22183 [es].
Yarin Gal. Towards a science of ai evaluations. Blog post, 2024. URL https: wwwp///wwwcccoo cs ox ac. uk /eoppel yarin. heaihhagal/eebstte/llo...oo html
19

---

Published in Transactions on Machine Learning Research (03/2026)
Yarin Gal and Stephen Casper. Customizable ai systems that anyone adapt bring big opportunities and even bigger risks. Nature, :46 2206, 2025.
Hongcheng Gao, Tianyu Pang, Chao Du, Taihang Hu, Zhijie Deng, and Min Lin. Meta-wwlerrning on diffusion models: Preventing relearning unlearned concepts. arXiv preprint arXiu:2410.1277 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gb dataset of diverse text for language modeling. 2020.
arXiv preprint ar Xr::2
Leo Gao, Jonathan Tow, Stella Biderman, Shawn Black, Authony DiPofi, Charles Foster, Laurence Golding, Jasmine Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0.1. Sept, 10:8-9, 9, 2021.
Sanjam Garg, Aarushi Goel, Somesh Jha, Saced Mahloujifar, Mohammad Mahmoody, Guru- Vaamiiamsi Policharla, and Mingyuan Wang. Experinenting with zero-knowledge proofs of training. In Proceed- ings of the 2023 ACM SIGSAC Conference Computer and Communications Security, CCS '23, pp. 1880-1894, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700507. doi: 00.1145/355661553622222 URL
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting Recall of Factual Associ- alions in Auto-Regressive Language Models, October 2023. URL http://arxiv.. http:://axxiw.org/bbbs. 2304. 14767
arXiv:2304.14677 [cs].
Thiband Gloaguen, Nikola Jovanović, Robin Staab, and Martin Vechev. Towards Watermarking of Open- Source LLMs, February 2025. URL http://arxiv..rrg http://ww..rgaab//55........ 2a/////ass2222..222/ 10525. arXiv:2502.10525 [cs].
Goodfire. Under the hood of reasoning model. April 2025. URL
Google AI. Gemini 2.5 pro preview model card. Model card technical report, Google, May 2025.
IBM Granite Team. Granite 3.0 language models. URL: https://github. http:/iingggnnteerrenee
language-models, 2024.
Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov, and David Krueger. Stres-Teesting Capabil- ity Elicitation With Password-Locker Models, May 2024. URL http: http:::/aaxvvvorr///ee wwwp:iwwwwrggbb////0....oo 19550 arXiv:2405.19550 [cs].
Elisabeth Grieve. Openness hype and open washing: A critical analysis of openness discourses in generative ai. McMaster University Theses, 2024.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, lan Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402. 008088 2024.
Roger Grosse, Juhan Bac, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhosscin Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilé Lukošiūté, et al. Studying Large Language Model Generalization with Inffuence Functions, August 2023. httphttt://aaet wwtreivv.org/aan arg/abs//300/ 03296 arXiv:2308.0m 002060 [cs].
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang. Xingkai Yu, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcementt learning. Nature, 645(8081):633 638, September 2025. ISSN 1476-4687. doi: 10.
(028//511586-005--04222-- URL htttp::/swww.ttt Nature Publishing Group.
Publisher:
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint ar Xix: 2024.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Danny Halawi, Alexander Wei, Eric Wallace, Tony T. Wang, Nika Haghtalab, and Jacob Steinhardt. Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation, June 2024. URL 1/1//2206.00055 arXiv:2406.20053 [cs].
Seokil Ham, Yubin Choi, Yujin Yang, Seungju Cho, Younghun Kim, and Changick Kim. Safety-aligned weights are not enough: Refuusl--eaachen-nuilee finetuning enhances safety and downstream performance under harmful finetuning attacks. 2025. URL huttps://arxiis www.://gg//bbbbeeeeeeo 07356
Kimia Hamidieh, Lester Mackey, and David Melis.. Domain-aware scaling laws uncover data synergy. In NewrIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling, 2025.
Hasan Abed Al Kader Hammoud, Umberto Michieli, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem, and Mete Ozay. Model Merging and Safety Alignment: One Bad Model Spoils the Bunch, June 2024. URL http://arxiv. org/abs/ /206../446...0 14563. arXiv:2406.1455: 14563 [es]-
Muhammad Abdullah Hanif, Nandish Chattopadhyay, Bassem Ouni, and Muhammad Shafique. Survey on backdoor attacks on deep learning: Current trends, categorization, applications, research challenges, and
future prospects. IEEE Access, 2025.
Peler Hase, Thomas Hofweber, Xiang Zhou, Elias Stengel-Eskin, and Mohit Bansal. Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?, June 2024. URL http:
//arxiv. hwtp://orggbb///////// 1235//4466.11994 arXiv:2406.1998 [cs].
Will Hawkins, Chris Russell, and Brent Mittelstadt. Deepfakes on Demand: the rise of accessi-
ble non-consensual doepfake image generators, May 2025. URL http://aaxxvv.or///bb///55 03859.
urXiv:2505.03850 [cs].
Luxi He, Mengzhou Xia, and Peter Henderson. What is in your safe data? identifying benign data that breaks safety. Xr preprint arXiv:2404. 2440099000 2024.
Yichen He, Guanhua. Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, and Weinan E. PaSa: An LLM Agent for Comprehensive Academic Paper Search, May 2025. URL http://arxiv org/abs/250.. http://arra..orr/aa//55.11.
10120 arXiv:2501.100 10120 [cs].
Peter Henderson, Eric Mitchell, Christopher D. Manning, Dan Jurafsky, and Chelsea Finn. Self- Self-Destruetin Models: Increasing the Costs of Harmful Dual Uses of Foundation Models, August 2023. URL http:
wwwxiv.org/abss/// httg/abb/22211.0 14946 arXiv:2211.14948 [cs].
Felix Hofstätter, Toun van der Weij, Jayden Tooh, Rada Djoneva, Henning Bartsch, and Francis Rhys
Ward. The Elicitation Game: Evaluating Capability Elicitation Techniques, July 2025. URL http:
//arxiv. /larxiv.org/aae
02180 arXiv:2502.02150 [es].
Eliahu Horwitz, Nitzan Kurer, Jonathan Kahana, Liel Amar, and Yedid Hoshen. We should chart an atlas of all the world's models. arXiv preprint Xrv:220003. 10639, 2025a.
Eliahu Horwitz, Asaf Shul, and Yedid Hoshen. Unsupervised Model Tree Heritage Recovery, April 2025b.
URL http://arxiv org/ htr//bbs/24055. 18432 arXiv:2405.1843 [es].
Saad Hossain, Samanvay Vajpayee, and Sirisha Rambhatla. BaneTuneBed: A Toolkit for Benchmarking LLM Safety Aligmment in Fine-Tuning May 2025. http://ap////i....t/// awwp//6..//00..... 00676. arXiv:2506.00676
[cs].
Lei Hsiung, Tianyu Pang, Yung- Chen Tang, Linyue Song, Tsung-Ye Yi Ho, Pin-Yr Yu Chen, and Yaoqing Yang. Why 1lm safety guardrails collapse after fine-tuning: A similarity analysis between aligmment. and fine- tuning datascts. arXiv preprint Xrxiv:2506. 05346, 2025.
Nathan Hu, Benjamin Wright, Carson Denison, Sam Marks, Johannes Treutlein, Jonathan Uesato, and Evan Hubinger. Training on documents about reward hacking induces reward hacking. Anthropic Alignment Science Blog, 2025a. URL RoLLhttps:// https://alignments http:///ai.iot.tataooooo//a///// howp:oop/c..com222///////// 2025-10-07.
Accessed:
21

---

Published in Transactions on Machine Learning Research (03/2026)
Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. Unlearning or Obfuscatingg Jogging the Memory of Unlearned LLMs via Benign Relearning, March 2025b. URL http://arxii..or/abbs... 13356 arXiv:2406.1338 [es].
Zixuan Hu, Li Shen, Zhenyi Wang, Yongxian Wei, and Dacheng Tao. Adaptive defense against harmful fine- tuning for large language models via bayesian data scheduler. arXiv preprint arXiu:2510.27112,, 2025c.
Tiansheng Huang, Sihao Hu, Fatih IIhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine- tuning for large language models via attenuating harmful perturbation. arXiv preprint arXiv:2409. 01586, 2024a.
Tiansheng Huang, Sihao Hu, Fatih Maaan, Selim Furkan Tekin, and Ling Liu. Harmful Fine-tuning Atlckk and Defenses for Large Language Models: A Survey, December 2024b. URL http://arxiv.. wrw. /
2409.18166
Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. Advances in Neural Information Processing Systems, 37:74058 74088, 2024c.
Qionglin Huangpu and Huixiang Gao. Efficient model compression and knowledge distillation llama 2: Achieving high performance with reduced computational cost. 2024.
IWF.
What has changed in the ai csam landscape?
ai csam report update, july 2024. Technical report, Internet Watch Foundation, July 2024. URL https: //admin. org.
ppppauttpppppbb11cjj-2211111 pdf
Mona Jami Pour, Seyed Mohammadbagher Jafari, and Monireh Khani. How to Know Your Cus- tomers? Towards a Novel Framework for Online Customer Knowledge Absorptive Capacity. Journal of the Knowledge Economy, December 2024. ISSN 1868-7873.
doi: 10. 10.11077ss33122-0044455333 URL
Hengrui Jia, Mohammad Yaghini, Christopher A. Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasckaran, and Nicolas Papernot. Proof-oflcarning: Definitions and practice. In 2021 IEEE Sym- doi: 10.1109/5540000.2222..000 posium on Security and Privacy (SP), pp. 1039-1056, 2021.
Max Kamachee, Stephen Casper, Michelle L. Ding, Rui-Jie Yew, Anka Reuel, Stella Biderman, and Dylan Hadeield-Menell. Video deepfake abuse: How company choices predictably shape misuse patterns. Available at SSRN, November 2025. URL luttps://paperr: ssrn. www http//sll3//aaaa comm/ool/papprs....... papers. cfm?abstract httti/rracct___889300330 1d=5829303, SSRN ID: 5829303.
Nikhil Kandpal, Brian Lester, Colin Raffel, Sebastian Majstorovic, Stella Biderman, Baber Abbasi, Luca Soldaini, Enrico Shippole, Feder Cooper, Aviya Skowron, et al. The common pile vO. 1: An 8tb dataset of public domain and openly licensed text. arXie preprint arXin:2506. 255200000005000, 2025.
Sayash Kapoor, Rishi Bommasani, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Peter Cihon, Aspen Hopkins, Kevin Bankston, Stella Biderman, Miranda Bogen, et al. On the Societal Impact of Open Foundation Models, February 2024. URL http:://arxxiii wwwp://www..gg/b////0/00000 07918 MxXiv:2403.0991 [cs].
John Kirchenbauer, Jonas Gciping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A Watermark for Large Language Models, May 2024a. URL http://aaxxv..org/bb//211.0000 10226 arXiv:2301.1022
[cs].
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the Reliability of Watermarks for Large Language Models, May 2024b. URL http://arxiv.orgg/ wrgg/aa////..../////////
arXiv:2306.04334 [cs].
Arinbjorn Kolbeinsson, Kyle O'Brien, Tianjin Huang, Shanghua Gao, Shiwei Liu, Jonathan Richard Schwarz, Amurag Vaidya, Faisal Mahmood, Marinka Zitnik, Tianlong Chen, et al. Composable interventions for language models. arXiv preprint arXio:2407.06440 2024.
22

---

Published in Transactions on Machine Learning Research (03/2026)
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, et al. HunyuanVideo: A Systematic Framework For Large Video Generative Models, March 2025. URL http://arxiv. org/ /tgg//bs//1/../////////// 03603 arXiv:2412.03603 [cs].
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining language models with human preferences, 2023. URL
attps://arxiv.
Tomek Korbak, Joshua Clymer, Benjamin Hilton, Buck Shlegeris, and Geoffrey Irving. A sketch of an Al control safety case, January 2025. URL http::/arxiv... org/ http://oggggbbbb/b/aeeeoeeee htrg//bs/250011 17315
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan Van Esch, Nasanbayar Ulzii-Orshikh, Allah- sera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. Quality at glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Compulational Linguistics, 10: 50 72, 2022.
Rohith Kuditipudi, Jing Huang, Sally Zhu, Diyi Yang, Christopher Potts, and Percy Liang. Blackbox model provenance via palimpsestic membership inference. arXiv preprint arXiv:2510. 19796, 2025.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset." arxiv, 2023.
Bruce W. Lee, Addie Foote, Alex Infanger, Leni Shor, Harish Kamath, Jacob Goldman-Wetzler, Bryce Woodworth, Alex Cloud, and Alexander Matt Turner. Distillation Robustifies Unlearning, June 2025.
URL /tt.ttttp//axxvvooor////////// 2506. 062788 arXiv:2506.06778 [es].
Joon-Woo Lee, HyungChul Kang, Yongwoo Lee, Woosuk Choi, Jieun Fom, Maxim Deryabin, Funsang Lee, Junghyun Lee, Donghoon Yoo, Young-Sik Kim, et al. Privacy-preservinn machine learning with fully homomorphic encryption for deep neural network. iEEE Access, 10:30039-30054, 2022.
LG, Kyunghoon Bac, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Yemuk Choi, Scokhee Hong, Junwon Hwang, Hyojin Jeon, Kijeong Jeon, et al. Exaone deep: Reasoning enhanced language models. arXiv preprint Xiv:2500. 12524, 2025.
Boheng Li, Renjie Gu, Junjie Wang, Leyi Qi, Yiming Li, Run Wang, Zhan Qin, and Tianwei Zhang, Towards resilient
unlearning for diffusion models against downstream fine-tuning. arXiv preprint arXiv:2507. 16302, 2025a.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Tvgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37:14200-14282, 2024a.
Kenneth Li, Yida Chen, Fernanda Viégas, and Martin Wattenberg. When bad data leads to good models. arXiv preprint ar arXi:225550447 04741, 2025b.
Linyang Li, Botian Jiang, Pengyu Wang, Ke Ren, Hang Yan, and Xipeng Qiu. Watermarking LLMs with arXiv:2310.11237 Jes].
Weight Quantization, October 2023. URL http:////axxi..o//ab////////////
Shenghui Li, Edith C.-H. Ngai, Fanghua Ye, and Thiemo Voigt. CnrT--s--a-AAtaaee Jailbreaking Language Models during Federated Fiffcietr-F.ffcieent Fine-Tuning December 2024b. URL http://arxxvv.org/
abs/2411. 19335 arXiv:2411.19333 [cs].
Percy Liang, Rishi Bommasani, Kathleen A. Creel, and Rob Reich. The time now to develop commu- nity norms for the release of foundation models, 2022. URL https://crff. stanforrdeedu//222//000
commuity--nrrm. html
Yuqing Liang, Jiancheng Xiao, Wensheng Gan, and Philip S Yu. Watermarking techniques for large language models: A survey. arXiv preprint arX002449.00088, 2024.
23

---

Published in Transactions on Machine Learning Research (03/2026)
Zheng Lin, Xuanjie Hu, Yuxin Zhang, Zhe Chen, Zihan Fang, Xianhao Chen, Ang Li, Praneeth Vepakomma, and Yue Gao. Splitlora: A split parameter-effiient fine-tuning framework for large language models, 2024. URL
00952
Johan Linåker, Cailean Osborne, Jemnifer Ding, and Ben Burtenshaw. A cartography of open collaboration in open source ai: Mapping practices, motivations, and governance in 14 open large language model projects. arXiv preprint (xXm:2500225300 2025.
David Lindner and Mennatallah El-Assady. Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning, June 2022.
URL http://arxiv. org/ attp//bss////.////// 2206. 13316 arXiv:2206.138 [cs].
Aiwei Liu, Leyi Pan, Yijian Lu, Jingjing Li, Xuming Hu, Xi Zhang, Lijie Wen, Irwin King, Hui Xiong, and Philip S. Yu. A Survey of Text Watermarking in the Era of Large Language Models, August 2024a. URL
http://arxiv http://aaxxvvoo//aaaaa/ abs/
arXiv:2312.0791 [cs].
Guozhi Liu, Weiwei Lin, Qi Mu, Tiansheng Huang, Ruichao Mo, Yuren Tao, and Li Shen. Targeted vaccine: Safety alignment for large language models against harmful fine-tuning via layer-wise perturbation. IEEE Transactions on Information Forensics and Security, 2025a.
Guozhi Liu, Qi Mu, Tiansheng Huang, Xinhua Wang, Li Shen, Weiwei Lin, and Zhang Li. Pharmacist: Safety alignment data curation for large language models against harmful fine-tuning. arXie preprinl arXiv:2510. 10085, 2025b.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking Machine Unlearning for Large Language Models, December 2024b. URL http::ttaaxii...o/ has/2400.08788) arXiv:2402.08788 [cs).
Xiaoqun Liu, Jiacheng Liang, Muchao Ye, and Zhaohan Xi. Robustifying Safety-Aligned Large Lan- guage Models through Clean Data Curation, May 2024c. URL arXiv:2405.1998 [cs].
19358,
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, al. Llm360: Towards fully transparent open-source Ilms. arXiv preprinl ar Xrv:2231 06550, 2023.
Michelle Lo, Shay B Cohen, and Fazl Barez. Large language models relearn removed concepts. arXiv preprinl arXiv:2401.0181. 2024.
Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al. The data provenance initiative: A large scale audit of dataset licensing & attribution in ai. arXio preprint ar arxiv::310. 16787, 2023a.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity, November 2023b. URL
http://arxiv org/ wwaaab/30000 2305 13169 arXiv:2305.1 13169 [cs].
Shayne Longpre, Sayash Kapoor, Kevin Klyman, Ashwin Ramaswami, Rishi Bommasani, Borhane Blili- Hamelin, Yangsibo Huang, Aviya Skowron, Zheng-Xin Xin Yong, Suhas Kotha, et al. A safe harbor for ai evaluation and red teaming. arXiv preprint or Xii:2203304484. 2024a.
Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, William Bran- non, Robert Mahari, Naana Obeng-Marnu, Manan Dey, Mohammed Hamdy, et al. Bridging the data provenance gap across text, speech, and video. In The Thirteenth International Conference on Learning Representations, 2024b.
24

---

Published in Transactions on Machine Learning Research (03/2026)
Shayne Longpre, Stella Biderman, Alon Albalak, Hailey Schoelkopf, Daniel McDuff, Sayash Kapoor, Kevin Klyman, Kyle Lo, Gabriel Tharco,, Nay San, Maribeth Rauh, Aviya Skowron, Bertie Vidgen, Laura Wei- dinger, Arvind Narayanan, Victor Sanh, David Adelani, Percy Liang, Rishi Bommasani, Peter Henderson, Sasha Luccioni, Yacine Jernite, and Luca Soldaini. The Responsible Foundation Model Development Cheatsheet: A Review of Tools Resources, February 2025a. URL htat:://aaxziv.orr/bbbett 16746, 4rXiv:2406.16746 [cs].
Shayne Longpre, Kevin Klyman, Ruth E. Appel, Sayash Kapoor, Rishi Bommasani, Michelle Sahar, Sean McGregor, Avijit Ghosh, Borhanc Blili- Hamelin,,,, Nathan Butters, et al. In-House Evaluation Is Not Enough: Towards Robust. Third-Party Flaw Disclosure for Gereral-Purpose AI, March 2025b. URL
http::/arxxe org/abs/ 2503, 16861 arXiv:2503.168: [cs].
Peizhuo Lv, Pan Li, Shengzhi Zhang, Kai Chen, Ruigang Liang, Hualong Ma, Yue Zhao, and Yingjiu Li. A Robustness- Whitc-Box Watermark in Neural Networks. IEEE Transactions on Dependable and Secure Compuling, 20(6):5214 5229, November 2023. ISSN 1941-0018. doi: 101100/TDSS::22333322273
URL https://ieeexppoorr ieee. org /cc.//mmtmmm/t1088500000
Aengus Lynch, Phillip Guo, Aidan Fwart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlcarning in llms. arXiv preprint arXiv:2402. 16835, 2024.
Peihua Mai, Ran Yan, Zhe Huang, Youjia Yang, and Yan Pang. Split-and-denoise: Protect large language model inference with local differential privacy, 2024. URL https: http:///wxi1v.r////////111100000
Emanuel Maiberg.
Hugging
face
is
hosting 5,000
nonconsensual
ai
models of real people,
2025.
URL hstps://wwx http:///www00mmea...../ ww0.m000e.....mm co/
of peoppee
Pratyush Maini, Sachin Goyal, Dylan Sam, Alex Robey, Yash Savani, Yiding Jiang, Andy Zou, Matt Fredrik- son, Zacharcy C. Lipton, and J. Zico Kolter. Safety Pretraining: Toward the Next Generation of Safe A1,
September 2025. URL httt:://arxtt hwwp:/oorgggbaaaa.ee wwgg/bs//g/...o..////g//////// 200bb///9004.688000 16980 arXiv:2504. 16980 [cs].
Nestor Maslej, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Njenga Kariuki, Emily Capstick, Anka Reuel, Erik Brynjolfsson, John Etchemendy, et al. Artificial intelligence index report 2025. arXiv preprint ar Xrc:25004.0133 2025.
Scan McGregor, Allyson Ettinger, Nick Judd, Paul Albec, Liwci Jiang, Kavel Rao, William H Smith, Shayne Longpre, Avijit Ghosh, Christopher Fiorelli, et al. To err is ai: case study informing Ilm flaw reporting practices. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 28938-288 -2994,, 2025.
Sarah Meiklejohn, Hayden Blauzvern, Mihai Maruseac, Spencer Schrock, Laurent Simon, and Ilia Shu- mailov. Position: Machine learning models have a supply chain problem. In Porty-second International Conference on Machine Learning Position Paper Track, 2025. URL sttpp:://ppenreview.r net/forum?
id=2fohnbkMuo
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-Editin Memory in Transformer, August 2023. URL
272200002 arXiv:2210.07229 [cs].
Mcta AL. The Ilama 4 herd: The beginning of a new era of natively multimodal intelligence. Blog post, 2025. URL htttt::///at meta. http:///p/l.aaeaa/mmmtaaaaaanaea
Kyle Miller, Mia Hoffmann, and Rebecca Gelles. The use of open models in research. Research report, Center for Security and Emerging Technology, Georgetown University, October 2025. URL https:// cset botngettmmeeu/pppbcccatttooo/aea wtwp://nooh--uuu----oo-eeeeeee
Niklas Mucnnighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettle- moyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test- tistttmm scaling. arXiv preprint ar Xiv: 19393, 2025.
25

---

Published in Transactions on Machine Learning Research (03/2026)
Sella Nevo, Dan Lahav, Ajay Karpur, Yogev Bar-On, Henry Alexander Bradley, and Jeff Alstott. Securing ai model weights. Research reports, RAND, 2024.
Helen Ngo, Cooper Raterink, João G. M. Araújo, Ivan Zhang, Carol Chen, Adrien Morisot, and Nicholas Frosst. Mitigating harm in language models with conditionallikclhood filtration, November 2021. URL (xXiv2210.07799: [cs].
org/ http://arxixa wttaaaaiwiw..ogaaaaaaaa aa///b///220 /2008. 0000//700000070000
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Πla Sutskever, and Mark Chen. GLIDE: Towards Photorcalistic Image Generation and Editing with Text- Guided Diffusion Models, March 2022. URL
10741, arXiv:2112.10711 [cs].
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint ar arXiv:2502.000 2025.
Ivica Nikolic, Teodora Baluta, and Prateck Saxena. Model Provenance Testing for Large Language Models, February 2025. URL http://aaxii....rrr////555/2////
arXiv:2502.00706 [es].
Helen Nissenbaum. Accountability in computerized society. Science and engineering ethics 2(1):25 42, 1996.
Kyle O'Brien, Stephen Casper, Quentin Authony, Tomek Korbak, Robert Kirk, Xander Davies, Ishan Mishra, Geoffrey Irving, Yarin Gal, and Stella Biderman. Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs, August 2025. URRp: http::///rxt..org/aa///666 arg/abs/2508.
06601 arXiv:2508.06001 [cs].
Op Chatgpt agent system card. Technical report, OpenAI, July 2025. URL https://cee openai. com
ageet_sssttttmmtmmmmmmmmeam card launch. pdf
OpenAl. Deep research system card. Technical report, OpenAI, February 2025. URL https://cdn. openai. com seseepp-rrearrchssytteeeee card. cardd pdf Accessed: 2025-10-07.
OpenAl. Disrupting malicious uses of ai: June 2025. Technical report, OpenAl Global Affairs, 2025a. URL
maliciuus usess-seee---ee cores-of-a---jjeee 2025/
OpenAl. Sycophancy in gpt-4o: What happened and what we're doing about it. Web page, 2025b. URL http::///ppeaaa..ccconnnnee////s -0oo--/--///
Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchis, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Martín Santillán Cooper, Kieran Fraser, et al. Granite guardian. arXiv preprint arXiv:2412.07724, 2024.
Giulio Pagnotta, Dorjan Hitaj, Briland Hitaj, Fernando Perez-Cruz, and Luigi V. Mancini. TATTOOED: A Robust Deep Neural Network Watermarking Scheme based on Spread-Spectrrun Channel Coding, June arXiv:2202.06091 [cs].
2024. URL
Sounyadeep Pal, Changsheng Wang, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Llm unlearning reveals stronger-than-expeete coreset effect in current benchmarks. arXiv preprint arXiv:2504.100 10185, 2025.
Jiadong Pan, Hongcheng Gao, Zongyu Wu, Taihang Hu, Li Su, Qingming Huang, and Liang Li. Leveraging catastrophic forgetting to develop safe diffusion models against malicious finctuning. Advances in Neural Information Processing Systems, 37:115208 115232, 2024.
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. Data and its (dis)contents: survey of dataset development and use machine learning rescarch. Patterns, 2 (11), November 2021. ISSN 2666-3899. doi: 10.1016 10.1016/i.patter. http://.patter22221110000 2021.100336. URL httl:tttss//// cell. com/ /tttaaaa/absttaaet//266--9999999
Publisher: Elsevier.
26

---

Published in Transactions on Machine Learning Research (03/2026)
Guilherme Penedo, Hynek Kydličck, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Aduances in Neural Information Processing Systems, 37:30811 30849, 2024.
YuengYun Peng, Pin-Yo Yu Chen, Matthew Hull, and Duen Horng Chau. Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models, October 2024. URL http:://axxiii org/ abs/ 2405. 17374 arXiv:2405.1737 [cs].
Gabriel J Perin, Runjin Chen, Xuxi Chen, Nina ST Hirata, Zhangyang Wang, and Junyuan Hong. Lox: Low-rank extrapolation robustifics I1m safety against fino-tuning. arXiv preprint arXiv:2506. 15606, 2025.
Jason Phang, Herbie Bradley, Leo Gao, Louis Castricato, and Stella Biderman. Fleutherai: Going beyond"
open science' to' science in the open'. arviv preprint arXiv:2210.06411. 2022.
Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, Heidi Howard, al. Evaluating Frontier Mod- els for Dangerous Capabilities, April 2024. URL http:://arss www.ariw..../aaaaaeeee wtwp/ba////...o.././// 13793 arXiv:2403.13993
[cs].
Robert Podschwadt, Daniel Takabi, Peizhao Hu, Mohammad H Rafici, and Zhipeng Cai. A survey of deep learning architectures for privacy-preserving machine learning with fully homomorphic encryplion. IEEE Access, 10:117477-117500 2022.
ATTACK VIA SAFETY PROJECTION. Spard: Defending harmful fine-tuning attack via safety projection
with relevance -ivverrtt-diveiitty data selection. 2025.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine- tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend Tol, October 2023. URL
03693 arXiv:2310.03998 [cs].
Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang, Tinghao Xie, Luxi He, Matthew Jagielski, Milad Nasr, Prateek Mittal, and Peter Henderson. On Evaluating the Durability of Safeguards for Open-Weight LLMs, December 2024. URL
20100000022120000777 arXiv:2412.0709: [cs].
C Raffel, N Shazeer, A Roberts, K Lee, S Narang, M Matena, Y Zhou, W Li, and PJ Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arxiv preprint arxiv: 191010683. Published online, 2019.
Mohit Raghavendra, Vaskar Nath, and Sean Hendryx. Revisiting the Superficial Aligmment Hypothesis, September 2024. URL http:// http://aaxxi..orgga//////////// 2077/11770 arXiv:2410.03717 [cs].
Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe Ilm applications with programmable rails. arXie preprint arXiv:2310. 10501, 2023.
Richard Ren, Steven Basart, Adam Khoja, Alexander Pan, Alice Gatti, Long Phan, Xuwang Yin, Mantas Mazeika, Gabriel Mukobi, Ryan H. Kim, Stephen Fitz, and Dan Hendrycks. Safelywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? Advances in Newral Information Processing Systems, 37:68559-68599, December 2024. URL
www.rps..ccppp..rrreeeee wwwes/lllppppple patpp//pppr////////// and Bedd__Bhehaasss Track. html
Wei-Qing Ren, Yu-Ben Qu, Chao Dong, Yu-Qian Jing, Hao Sun, Qi-Hui Wu, and Song Guo. A survey on collaborative dnn inference for edge intelligence. Machine Intelligence Research, 20(3):370 395, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolttion image synthesis with latent diffusion models. arxiv 2022. arXiv preprint arXiv:2112.1011 2021.
Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, David Atanasov, Robie Gonzales, Sub- habrata. Majumdar, Carsten Maple, Hassan Sajjad, and Frank Rudzicz. Representation Noising: De- fence Mechanism Against Harnful Finetuning, October 2024. URL URL/tRtp:////// http:://axxi..org/bb///466 14577 arXiv:2405.14577 [cs].
27

---

Published in Transactions on Machine Learning Research (03/2026)
Domenic Rosati, Sebastian Dionicio, Xijie Zeng, Subhabrata Majumdar, Frank Rudzicz, and Hassan Sajjad. Locking open weight models with spectral deformation. In JCML Workshop on Technical AT Governance (TAIG), 2025.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 130136 180186-130184 Cur- ran Associates, Inc., 2024. URL https://prooeeedings www.nnew.reeeeeeee Saaper_filespppeeee 200012c667724000000e//rrr0022202
Debdeep Sanyal, Manodoep Ray, and Murari Mandal. Antidoto: Bi-level adversarial training for tamper- resistant llms. arxiv preprint orXiv::509. 08000, 2025.
Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Tbrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why Has Prodicting Downstream Capabilitiiss of Frontier AI Models with Scale Remained Elusive?, February 2025. URL httpp://arxiv.org//ae
04391
arXiv:2406.042: 043999 [cs].
Paul S Scotti. How structure open science collaborations online. 2025.
Elizabeth Soger. Open horizons:
Exploring nuanced technical and policy approaches openness ai. Toch- nical report, Demos in partnership with Mozilla, August 2024. URL http:/pppp/mmmms.....o////ppooop wo..://.cooo.///ppcteetetttt
uploads/2024/002 htt.//oooilaaaeerrr. _02_222222220 2024.pdf
Elizabeth Seger and Jamie Hancock. The open dividend: Building an openness strategy unlock the uk's ai potential. Technical report, Demos, June 2025. URL https://demos. wwtpiios....../wwooooooe contee///ootee////l//aa//////// 2025/66/The--ppe-DDiiidee BttehiidddddReeeett____ree 2025 a..2.cc-2.pd.
Elizabeth Seger, Noemi Dreksler, Richard Moulange, Emily Dardaman, Jonas Schuett, K. Wei, Christoph Winter, Mackenzie Arnold, Sean O hEigeartaigh, Anton Korinek, et al. Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives, September 2023. URL attt:///rxivv wwwlvv.orr//bbge httgg/ab//3311. 09227 arXiv:2311.09277 [cs].
Aqsa Shabbir, Halil fbrahim Kanpak, Alptekin Küpçü, and Sinem Sav. A laxonomy of attacks and defenses in split learning, 2025. URL https:// http:///xxxvv..oo.e 05820
Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, et al. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming, arXiv preprint arXiv:2501.18837, 2025.
Han Shen, Pin-Yu Chen, Payel Das, and Tianyi Chen. Seal: Safety-enhanced aligned 1lm fine-tuning via bilevel data selection. arXiv preprint arXiv:2410.07/77. 2024.
Hui Shen, Jingxuan Zhang, Boning Xiong, Rui Hu, Shoufa Chen, Zhongwei Wan, Xin Wang, Yu Zhang, Zixuan Gong, Guangyin Bao, et al. Efficient diffusion models: A survey, arXiv preprint Xxxx:2502.06800 2025.
Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Slephen Casper. Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs, July 2025. URL http:://axxiv.. org/abs/2407. 15549 arXiv:2407. 15549 [cs).
Yunmeng Shu, Shaofeng Li, Tian Dong, Yan Meng, and Haojin Zhu. Model inversion in split learning for personalized 1mss: New insights from information bottleneck thcory, 2025. URL https://arxiv.. org/
obb62500.05965
Vitaly Simonovich. Cato CTRLEM Threat Rescarch:
Uncovering Nythcon AI New
Platform of Uncensored LLMs, June 2025. URL
http://www..cttooooorr..s.s com/blog/
cato- eato-ctrl-nytheon--l aaal-aathenn-a-----ee--laaaeee of -ww-roomd------ceeeeed------nns
28

---

Published in Transactions on Machine Learning Research (03/2026)
Stewart Slocum, Julian Minder, Clément Dumas, Henry Sleight, Ryan Greenblatt, Samuel Marks, and Rowan Wang. Believe it or not: How deeply do 1lss believe implanted facts? arXiv preprint orXiv:2510 17941, 2025.
Trene Solaiman. The Gradient of Generative AT Reloase: Methods and Considerations, February 2023. URL http://arxiv. otg/aba/23000 http:///aaiii.ooaaaa////////////
04844 arXiv:2302.04844 [cs].
Irene Solaiman, Rishi Bommasani, Dan Hendrycks, Ariel Herbert- Yacine Jernite, Aviya Skowron, and Andrew Trask. Beyond Release: Access Considerations for Generative AI Systems, April 2025. URL http://arxiv.ort ha//abbs/250010 16701 arXiv:2502.16701 [cs].
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, al. Dolma: An open corpus of three trillion tokens for language model pretraining research. ar preprinl or urXie:2402.00 001:4402.0019,, 2024.
Filip Sondej and Yushi Yang. Collapse of irrelevant representations (cir) ensures robust and non-disruptive Ilm unlearning, arXiv preprint arXiv:2509.118 2025.
Madhulika Srikumar, Jiyoo Chang, Kasia Chmielinski, et al. Risk mitigation strategies
for the open foundation model value chain. Technical Report Rev. 3.1, Partnership
on AI, 2024. URL https://partuershipona http:///wccotteerrpppaaaaa/a/taa http:////pppppaaaaooooppa//////0 uploads/2024/07/ open food--ffudatiionnae wwwettiiionoodee-rriie rodiigiittttttitiitttt 1
Marco Antonio Stranisci and Christian Hardmeier. What Are They Filtering Out? A Survey of Filtering
Strategies for Harm Reduction in Pretraining Datasets, February 2025. URL http://arxiv.orr//ee
1503.05721 arXiv:2503.0572) [cs].
Zekun Sun, Zhihao Sui, Na Ruan, Conghui He, Dahua Lin, and Jie LI. Trustworthy dataset proof: Certifying the authentic use of dataset in training models for enhanced trust. 2025.
Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, Andy Zou, Dawn Song, Bo Li, Dan Hendrycks, and Mantas Mazeika. Resisant Safeguards for Open-Weight LLMs, February 2025. URL http://arxixt wwrexivvvorr/aae /b/// 2408. 00761
arXiv:2408.007: [cs].
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint xrXi::2503.199 19786, 2025a.
GLM-45 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xic, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models, August 2025b. URL httt:://axiit hwtp://rrg//ab////eeee 2t/1//2588006611111 arXiv:2508.0644 [cs].
National Telecommunictiios and Information Administration. Dual-use foundation models with widely available model weights. Technical report, NTIA / U.S. Department of Commerce, 2024. URL https:
//www.ntia.c www.nniiigg.....
Thorn and All Tech Is Human. Reducing the risk of synthetic content: Preventing generative AI from producing child sexual abuse material. Technical report, Thor and All Tech Human, February 2024. 055.0mmmaaaaaddaarro2oooooeee/e0 http:///wwhaaa/////////////////a
pdf Comments on Al Executive
Order Request for Information submitted to NIST.
Johannes
Treutlein, Dami Choi, Jan Betley, Sam Marks, Cem Anil, Roger Grosse, and Owain Evans. Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data. Advances in Newral Information Processing Systems, 37:140667- 140730, December 2024. URL https://proceedngss neurips. httpaa/apppppppp....ea/////////p wwtps///pp///////////a//p/ 12 Confeeeteoooeeeeeeeeeee html
29

---

Published in Transactions on Machine Learning Research (03/2026)
CK AISI Managing risks from increasingly capable open- ai systems. Blog post, 2025. URL https:
7/www.ais: aisi.
Emma Henderson Vaughan. NCMEC Releases New Data: 2024 in Numbers. URL httpss:/ncmec..
Risks of Open-Weight LLMs, August 2025. URL -aap----daaa-222--------eeeeee html [cs].
Eric Wallace, Olivia Watkins, Miles Wang. Kai Chen, and Chris Koch. Estimating Worst-Case Frontier
www.orggggab...../aa///// 03153 ar arXiv:250S.03153
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Whe Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, et al. Wan: Open and Advanced Large-Scale Video Generative Models, April 2025. URL http://axxiv, wttg 20314 arXiv:2503.20314 [cs].
Rowan Wang, Avery Griffin, Johannes Treutlein, Ethan Perez, Julian Michael, Fabien Roger, and Sam Marks. Modifying Ilm beliefs with synthetic document finetuning, Web post, 2025a. URL anthropic. com wtt.m///22mmmmffi/mm/mmm wtw.a//yyyyyneellle--a--a-------
Yuhui Wang, Rongyi Zhu, and Ting Wang.
Sclf-destructive language model. arXiv preprint
arXiv:2505.121 2025b.
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the Brittleness of Safety Alignment via Pruming and Low-Rank Modifications, October 2024. URL http://arxiv..eet http:://axxxv.orgaaago wwt.o//ggggggoooo.ooo/ [cs].
arXiv:2402.05162
Boyi Wei, Zora Che, Nathaniel Li, Udari Madhushani Sehwag, Jasper GÅktling, Samira Nedungadi, Julian Michael, Summer Yue, Dan Hendrycks, Peter Henderson, et al. Best practices for biorisk evaluations on open-woight bio-foundation models. arXiv preprint 1rXiu:2510.2762)) 2025.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint ar Xiv:222066. 07682, 2022.
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in Detoxifying Language Models, September 2021. URL
2445//0449990455555 arXiv:2109.07445 [cs].
Eric Wu, Kevin Wu, and James Zou. Finetunebench: How well do commercial fine-tuning apis infuse knowledge into lims? arXiv preprint 0rXiv:2411.05005 2024.
Suhang Wu, Ante Wang, Minlong Peng. Yujie Lin, Wenbo Li, Mingming Sun, and Jinsong Su. DocTER: Evaluating Document-based Knowledge Editing, July 2025. URL http://arxiv. org/ argg/bbs/3088.... 09954 arXiv:2308.09954 [cs).
Yuxin Xiao, Sana Tonekaboni, Walter Gerych, Vinith Suriyakumar, and Marzyeh Ghassemi. When style breaks safety: Defending language models against superficial style alignment. arXiv preprint arXiv:2506.07/33 2025.
Zuan Xie, Yang Xu, Hongli Xu, Yunming Liao, and Zhiwei Yao. A novel hat-shaped devico-cloud collabo- rative inference framework for large language models, 2025. URL
Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. Detoxifying Language Modols Risks Marginalizing Minority Voices, April 2021. URL http://arxx org/ 2204 06390 arXiv:2104.06390 [cs].
Xiaojun Xu, Yuanshun Yao, and Yang Liu. Learning to Watermark LLM-generated Text via Reinforcement Learning, March 2024. URL http: htttp://arxiva wwwpa/wwwwwr/bb/////..ooo 10553 1rX5v:2403.10553 [cs].
30

---

Published in Transactions on Machine Learning Research (03/2026)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, et al. Qwen3 Technical Report, May 2025. URL http://arxiv. arg/abs/25000 2508.0938880...0 MrXiv:2505.09385 [cs].
Mingke Yang. Yuqi Chen, Yi Liu, and Ling Shi. DistillSoq: A Framework for Safety Alignment Testing Large Language Models using Knowledge Distillation, July 2024. URL http://arxiv org/aba/247, http://araxv.orr/aaa/2007,
10106 arXiv:2407.10006 [cs] version: 1.
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin.
Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models, October 2023. URL http:
//axxiv. olarxiv.or//aat wwt.:/ggg/bbbbb wttp://////////.../////// 02949 arXiv:2310.02949 [cs].
Dixi Yao and Baochun Li. Is split learning privacy-preseree for fine-tuning large language models? IEEE Transactions on Big Dala, pp. 1 12, 2024. doi: 10.1 10.1109/TBDATA 201....11.22...211111.1.
Biao Yi, Tiansheng Huang, Baolei Zhang, Tong Li, Lihai Nie, Zheli Liu, and Li Shen. Ctrap: Embedding col- lapse trap to safeguard large language models from harmful fine-tunnng arXie preprint orXiv:2505.16559 2025.
Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, and Mario Fritz. Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14428-14437 14437, October 2021. doi: LLL19//ICCCV4992222222....2. URL ISSN: 2380-7504.
https://1seexplore.eeee ieee. org/ htgg///ocueen/9711166
Chen Yueh-Han, Nitish Joshi, Yulin Chen, Maksym Andriushchenko, Rico Angell, and He He. Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitors, June 2025. URL http://arxiv.
org/abs/2506. 10949. ar arXiv:2506.100 10949 [cs).
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, et al. A Com- prehensive Study of Knowlodge Editing for Large Language Models, November 2024. URL http: 4rXiv:2401.01266 [cs].
//arxiv.org/abs//40001
Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Tiu, Jingren Zhou, Junyang Lin, Baosong Yang, Chen Cheng, Jialong Tang, Jiandong Jiang, Jianwei Zhang, Jijie Xu, Ming Yan, Minmin Sun, Pei Zhang, Pengjun Xic, Qiaoyn Tang, Qin Zhu, Rong Zhang, Shibin Wu, Shuo Zhang, Tao He, Tianyi Tang, Tingyu Xia, Wei Liao, Weizhou Shen, Wenbiao Yin, Wenmeng Zhou, Wenyuan Yu, Xiaobin Wang, Xiaodong Deng, Xiaodong Xu, Xinyu Zhang, Yang Liu, Yeqiu Li, Yi Zhang, Yong Jiang, Yu Wan, and Yuxin Zhou. Qwen3guard technical report, 2025a. URL https://arxiv. www./osg////////oo//oooooooe 2211122211111166666
Xuandong Zhao, Sam Gunn, Miranda Christ, Jaiden Fairoze, Andres Fabrega, Nicholas Carlini, Sanjam Garg, Sanghyun Hong, Milad Nasr, Florian Tramer, Somesh Jha, Lei Li, Yu-Xiang Wang, and Dawn Song. Sok: Watermarking for ai-generated content, 2025b. URL httpps://axxx.
www.aaawww..i..rr..arrararaaaaae wwwp///g////......////// 18479
Amber Yijia Zheng and Raymond A Yeh. Imma: Immunizing text-to-image models against malicious adaptation. In European Conference on Computer Vision, pp. 458-475. Springer, 2024.
Amber Yijia Zhong, Codar Site Bai, Brian Bullins, and Raymond A Ych. Model immunizaiion from a condition number perspective. arXiv preprint ar XxXx:22555.2700 2025.
Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. MQUAKE:
Assessing Knowledge Editing in Language Models via Multi-Hop Questions, September 2024. URL http:
wwwrii.orr//bbs/22000 14795 arXiv:2305.14788 [cs].
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuczhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less More for Alignment, May 2023. URL http:///aww....r////////////////
11206] arXiv:2305.11200 [os].
31

---

Published in Transactions on Machine Learning Research (03/2026)
Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song, and Xin Eric Wang. The hidden risks of large reasoning models: A safety assessment of r1, 2025a.
URL https://arxiv.org/////// htt/a//iiiii.rra//////////// 2006b/2599.22999 12659
Shiji Zhou, Lianzhe Wang, Jiangnan Ye, Yongliang Wu, and Heng Chang. On the limitations and prospects of machine unlearning for generative ai. arXiv preprint arXiv:2408. 200022408000766 2024.
Yihe Zhou, Tao Ni, Wei-Bin Lee, and Qingchuan Zhao. survey on backdoor threats in large language models (Imms): Attacks, defenses, and evaluations. arXiv preprint orXim:2502.0524, 2025b.
Sally Zhu, Ahmed Ahmed, Rohith Kuditipudi, and Percy Liang, Independence Tests for Language Models, March 2025a. URL http://arxiv. org/ h/t///b///// 2atb../255221111 115502.12292 12292 arXiv:2502.1229) [cs].
Xiaochen Zhu, Xinjian Luo, Yuncheng Wu, Yangfan Jiang, Xiaokui Xiao, and Beng Chin Ooi. Passive inference allacks on split learning via adversarial regularization. In Network and Distributed System Security (NDSS) Symposium 2025, Feburary 2025b. doi: 20.14722/ndss..m 23030.
Yuxuan Zhu, Tengiun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Wciss, al. Establishing best practices for building rigorous agentic benchmarks. Xiviv preprinl ar 020:20007.222, 2025c.
Daniel Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adain Scherlis, Noa. Nabeshima, Benjamin Sainstein-Rau,, Daniel de Haas, Buck Shlegeris, and Nate Thomas. Ad- versarial training for high-stakes reliability. Advances in Neural Information Processing Systems, 35: 9274 9286, December 2022. URL https://prooeeedings meetoeeeeddinnnnnnnn........... $c4400dd11ad223344bbc0766666 Abstraatt www.ipp.cc.papprr.aaa..... cc/paper wwtpe/ppee.elleeppppepeeeeeeeeee
html
Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, 7ico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving Aligument, and Robustness with Circuit. Breakers, July 2024. URL http://arxiv. org. ab2//466. 04313 arXiv:2406.0433 04313 [es].
Jakub Lucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An Adver- sarial Perspective on Machine Unlcarning for AI Safety, May 2025. URL http://arxiv org/abs//44a 2409. 18025 arXiv:2409.1802 [cs].
32


Reviews:
[{"review_id": "IdvCGW7Csv", "reviewer": "Reviewer_YbcU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a structured survey of open technical problems in managing risks associated with open-weight AI models. It organizes sixteen open challenges across five stages of the model lifecycle, data curation, tamper-resistant training and unlearning, tampering-aware evaluation, staged deployment, and model provenance/forensics. The authors further provide a documentation-based audit of recent open-weight frontier models to illustrate gaps in transparency and reporting practices.\n\nStrengths:\n\n- The topic is timely and important as open-weight model capabilities grow and governance challenges intensify.\n\n- The taxonomy is clearly presented and helps readers navigate a rapidly evolving area.\n\n- The empirical documentation survey highlights a real and under-discussed disconnect between theoretical safety proposals and industry reporting.\n\nWeaknesses:\n\n- The survey largely summarizes existing concerns but provides limited interpretation; the open problems are listed clearly but without deeper conceptual insight or analysis of their interrelationships.\n\n- Discussion of feasibility, limitations, and trade-offs remains high level, which weakens the value of the taxonomy as a research roadmap.\n\n- Certain conceptual framings would benefit from greater definitional precision.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The central arguments of the paper, including the idea that open-weight models pose distinct risk-management challenges and that existing reporting practices show noticeable variation, are generally well motivated and supported through appropriate citations. At the same time, much of the discussion stays at a descriptive level rather than offering deeper analysis, and several conclusions rely on the implicit assumption that safeguards not mentioned in documentation were not implemented. The evidence provided is sufficient for highlighting broad issues in the field, although a more analytical treatment would help make the claims more convincing.", "audience_yn": "Yes", "audience_explanation": "The challenges associated with open-weight AI models are of clear interest to researchers working on model governance, robustness, alignment, and risk assessment. The taxonomy offered by the submission provides a structured lens through which to understand the emerging field of open-weight model risk management. In addition, the analysis of current reporting practices is a timely contribution. These aspects would be valuable to a meaningful subset of TMLR’s audience.", "requested_changes": "1. Provide deeper conceptual insight.\nThe survey would benefit from more analytical discussion explaining why the identified problems arise, how they relate to one another, and what principles structure the overall landscape.\n\n2. Discuss trade-offs and limitations more explicitly.\nAdding brief reflections on potential side effects or practical constraints of each safeguard category would make the taxonomy more informative.", "broader_impact_concerns": "No major ethical concerns arise from the technical content. The paper discusses defensive considerations rather than proposing new attack techniques. A short Broader Impact statement acknowledging the dual-use risk of discussing tampering pathways and clarifying the intended defensive purpose of the work would be helpful but is not strictly required.", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kp0Z0nyctK", "reviewer": "Reviewer_Wqgx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discuss open technical problems in open-weight AI model. This issues, in my understanding is the most fundamental problem in AI safety field, and in my understanding, is almost impossible to fully addressed in the current technical context.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This is a survey paper. Comprehensive reference to prior research has supported the claim of this paper.", "audience_yn": "Yes", "audience_explanation": "This survey is important to identify what's the most fundamental problem in safety field. Definitely of interest to the readers.", "requested_changes": "Several research directions on open-weight LLMs are not well discussed.\n\nSection 4.1, when talking about data curation, I suggest the authors to expand discussion on data curation based on different stages. Particularly, data curation for safety has been studied from three main aspects. \n\n* **How to fitler harmful data before the pre-training stage?**  \n\nAs far as I know two research falls in this angle. The authors should have enough knowledge on this aspect, but should also mention [2]. \n\n[1] Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs https://arxiv.org/abs/2508.06601\n\n[2] Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation Models https://arxiv.org/abs/2510.27629\n\n* **How to contruct better safety alignment data  at the safety alignment stage (after the pre-training) to make the model immune to tampering attack?**\n\n There are several research along this line, and should be discussed by the authors.  Example reference is include your task may vary[3] and Pharmacist[4]. \n\n[3] Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets\n\n[4] Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuning\n\n* **How to construct safety alignment data/filter harmful dta in the tampering attack phase?**\n\nThere are a line of research in this line as well and should be discussed. For example. [5][6][7]  explore safety data curation in the tampering attack stage. \n\n[5] Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models \n\n[6] When Style Breaks Safety: Defending LLMs Against Superficial Style Alignment\n\n[7] SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance–Diversity Data Selection\n\nOn the other hand,  [9][10][11][12] [13] explore  harmful data filteration in the tampering attak phase and should be disccussed.  \n\n[9] SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection (ICLR2025)\n\n[10] Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence https://openreview.net/forum?id=qur2ef8MqQ\n\n[11] Adaptive Defense against Harmful Fine-Tuning via Bayesian Data Scheduler (NeurIPS2025)\n\n[12] Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks https://openreview.net/forum?id=OK2GR1guwv\n\n[13] GradShield: Alignment Preserving Finetuning\n\n\nSection 4.2,  I think alignment stage defenses against tampering attack (or harmful fine-tuning) can be separated into twp directions. \nFirst line is represented by TAR, which the authors I guess is pretty familiar, but recently there is another line of research which is not discussed by this paper. \n\n* The first line of research is concerning **how to make safety alignment stronger** such that the model cannot learn harmful knowledge, represented by Vaccine, RepNoise, TAR. However, there are some missing references on this line of research, which should be added, i.e., [17-31] \n\n[14] Vaccine: Perturbation-aware alignment for large language model aginst harmful fine-tuning https://arxiv.org/abs/2402.01109\n\n[15] Representation noising effectively prevents harmful fine-tuning on LLMs https://arxiv.org/abs/2405.14577\n\n[16] Tamper-Resistant Safeguards for Open-Weight LLMs https://arxiv.org/abs/2408.00761\n\n\n**(Missing referece on defesense similar to TAR)**\n\n[17] Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation \n\n [18]  Leveraging Catastrophic Forgetting to Develop Safe Diffusion Models against Malicious Finetuning\n\n[19] Identifying and Tuning Safety Neurons in Large Language Models \n\n[20] Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation\n\n[21] Preserving Safety in Fine-Tuned Large Language Models: A Systematic Evaluation and Mitigation Strategy\n\n[22] On Weaponization-Resistant Large Language Models with Prospect Theoretic Alignment\n\n[23] Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralization \n\n[24] Model Immunization from a Condition Number Perspective\n\n[25]  Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning\n\n[26]  Locking Open Weight Models with Spectral Deformation\n\n[27] LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning\n\n[28] Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning\n\n[29] TOKEN BUNCHER: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuning \n\n[30] AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMs\n\n[31] Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence\n\n* The second line of research is more interesting. As demonstrated by Xiangyu attack paper, first line of defense, e.g., TAR and RepNoise are still not robust enough, as they are vulnerabe when the fine-tuning learning rate is large. To address this, the second line of  two subsequent research (CTRAP[32] and SEAM[33] ) explore another direction, which is to **embed a collapse trap during safety alignment to make the model collapse if harmful fine-tunig is detected**. I personnally think the collapse trap direction represented by CTRAP and SEAM is the most promising direction to eliminate the fine-tuning risk. Basically, they embed a \"shut-down button\" into the openweight LLMs and will be activated if the safety alignment is compromised (or the harmful knowledge is actiated) by harmful fine-tuning. I think the authors can expand discussion here. \n\n[32] CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuning https://arxiv.org/abs/2505.16559\n\n[33] Self-Destructive Language Model https://arxiv.org/abs/2505.12186", "broader_impact_concerns": "No concerns.", "additional_comments": "See requested change. I will participate in discussion with constant interation with the authors. I think this paper is important, but at the same time I think several lines of important research directions are missing.  I think I should help to better characterize this paper in these technical aspect.  Feel free to disagree my view in the rebuttal if your feel these lines of research are not relevant and I am open to discuss.", "extra_scores": {}, "extra_text": {}}, {"review_id": "JWPR9kQaK2", "reviewer": "Reviewer_bDvJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a structured overview of the open technical challenges in managing risks associated with open-weight AI models. The authors categorize sixteen open problems into five stages of the model lifecycle: data curation, tamper-resistant training, tampering evaluations, staged deployment, and model provenance/forensics—and analyze how current open-weight model reports address (or fail to address) these issues. The work aims to establish a clearer research agenda and highlight gaps in current practices, especially the lack of standardized evaluations for post-release model tampering.\n\nKey strengths:\n1. Provides a clear, systematic taxonomy of risk related technical challenges unique to open-weight settings.\n2. Offers practical guidance on where current open-weight reports fall short, supported by a cross-model qualitative analysis.\n3. The survey of technical reports from major 2025 releases (Table 2) provides timely and concrete evidence of the disconnect between theoretical safety techniques and actual industry practices.\n\nKey weaknesses:\n1. The model set analyzed in Section 5 is small and focused on high-profile systems, which may introduce sampling bias.\n2. The paper functions primarily as a position/survey piece: it organizes challenges and open problems but does not contribute new algorithms, proofs, or experimental methods.\n3. The Section 5 analysis depends solely on developer-provided reports rather than independent verification. “No Mention” does not imply the absence of a mitigation, reducing the strength of the survey’s conclusions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s main claims are supported by clear reasoning and well cited prior work. Its conceptual taxonomy is logically developed, and the identified technical gaps are grounded in existing literature. The Section 5 analysis of 2025 technical reports provides relevant evidence for the claim that current safety reporting is inadequate. However, this empirical component relies solely on developer-provided documentation rather than independent verification, which weakens the evidential strength. Overall, the claims are appropriately supported for a position paper, though some conclusions would benefit from stronger methodological rigor.", "audience_yn": "Yes", "audience_explanation": "The paper addresses topics, such as open-weight model safety, tamper-resistant training, and provenance mechanisms, that are directly relevant to researchers working on foundation models, security, and responsible AI. Its taxonomy of open technical problems provides a structured perspective that many in the TMLR community may find useful for guiding future research directions. Given the growing importance of open-weight models, these findings would likely interest a meaningful subset of TMLR’s audience.", "requested_changes": "Strengthening: \n1. Expand Model Survey: The survey focuses on 10 high-profile models. Expanding the dataset or at least discussing why other categories (e.g., smaller open-weight models, community finetunes, multilingual models) were not included would strengthen the representativeness of the ecosystem wide claims.\n\n2. Propose Concrete Metrics: For several open technical problems (e.g., tamper resistance, unlearning stability), the paper would be strengthened by proposing possible evaluation metrics or standardized protocols—such as “steps-to-harmfulness” benchmarks for adversarial fine-tuning—rather than only describing challenges qualitatively.", "broader_impact_concerns": "The work focuses on risk mitigation, presenting low direct ethical risk. However, a Broader Impact Statement should acknowledge a mild dual-use concern: the detailed discussion of tampering evaluations and attack vectors could theoretically assist adversarial actors in bypassing safeguards.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iS8HXgH8bS", "reviewer": "Reviewer_mc7U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper identifies several open technical problems related to the risks posed by open-weight frontier AI models, structured across the following lifecycle stages: training data curation, tamper-resistant training & unlearning algorithms, tampering-aware evaluations, staged deployment methods, as well as model provenance and forensics.\n\nThis submission’s motivation is that open-weight models have unique safety challenges relative to closed models: they can be modified freely, disseminated irreversibly, and used without oversight. The authors survey existing work, identify gaps, and articulate open research problems. The paper concludes with a small empirical analysis of the safety-related reporting practices of current open-weight model developers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This submission has clear and timely motivation. The introduction clearly establishes the escalating capabilities of open-weight models and the mounting difficulty of governing them due to unrestricted access and rapid proliferation. This framing is compelling and grounded in evidence from recent technical reports and open-weight releases. The paper has well-structured taxonomy. The five-part organization (training data → training → evaluations → deployment → monitoring) gives the reader a coherent conceptual map of what “open-weight model risk management” consists of. This structure mirrors and synthesizes multiple taxonomies in prior work. This submission also preent excellent survey of existing work. The authors assemble a broad range of relevant research and surface nuanced issues (e.g., the difficulty of filtering multilingual data; limits of post-training tamper resistance; exfiltration risks in split inference). The survey is one of the strongest components.", "audience_yn": "Yes", "audience_explanation": "This submission contributes a concrete set of open problems, many of which are indeed pressing and under-studied. These include making data filtering robust across harm categories, building tamper-resistant unlearning, benchmarking tampering attacks, watermarking or tracking modified models in the wild, evaluating thousands of ecosystem models efficiently. These iarevaluable to guide research priorities.\n\nThe submission also provide novel and interesting empirical analysis. The study of reporting practices across highly downloaded 2025 open-weight models (e.g., Qwen3, Gemma3, gpt-oss) is informative and highlights striking gaps — especially the lack of reporting on tamper resistance, tampering evaluations, staged deployment, and provenance.", "requested_changes": "This paper lack of prioritization for all the open problems. All problems are presented with roughly equal weight, but some are clearly more urgent or tractable than others. A prioritization framework—based on risk severity, tractability, or expected impact—would increase the paper’s utility for guiding the field.\n\nThere are also missing discussion of failure modes in “technical safeguards” themselves. Some safeguards (watermarking, unlearning, tamper resistance) may introduce new vulnerabilities like false negatives in forensics, over-confident safety claims leading to risk compensation, backdoors introduced during watermarking, and reduction in benign capabilities due to unlearning. These tradeoffs are only partially acknowledged. For each safeguard category, discuss risks introduced by the safeguard itself and highlight where these create second-order hazards.\n\nThe paper sometimes mixes “unique to open-weight” vs. “more challenging for open-weight”. The authors intend to focus on strategies with unique implications for open-weight models. However, tamper-resistant training is relevant for API-based fine-tuning in closed models. Heritage inference is useful for closed-model auditing. Split deployment and hardware locking are not “open-weight” strategies but restricted-weight strategies. This definitional ambiguity weakens the conceptual crispness.\n\nThe empirical developer-reporting study in this submission lacks clear methodology. The sampling procedure is described but could be more rigorous: How were “downloads” normalized across model families? How were ambiguous documentation statements interpreted? Were multiple reviewers involved to improve reliability?\n\nThere are limited discussion of non-technical constraints and incentives. Although the paper notes that incentives for private actors to implement safeguards are weak, the implications of this are under-explored. Many proposed safeguards (e.g., proof-of-training, hardware locking) are unlikely to be adopted without external pressure. The authors should expand the discussion about incentive structures, deployment economics and incentive-aligned governance mechanisms.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9qr7m6LMok", "reviewer": "Reviewer_hFix", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a valuable survey and taxonomy regarding the unique risk management challenges of open-weight AI models. Unlike API-based models, open-weight models allow users to bypass standard safeguards, necessitating different technical approaches. The paper outlines 16 open technical problems across five lifecycle stages, including training data curation, tamper-resistant training, and ecosystem monitoring. Additionally, the authors conduct a useful gap analysis of technical reports from prominent models (e.g., Qwen3, gpt-oss), demonstrating that safeguards like tamper-resistance and provenance are currently under-reported by developers", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides a well-supported characterization of the \"current state of the field\". The authors back their claims with a comprehensive literature review and an empirical audit of 15 recent frontier model technical reports in Section 5 . The distinction drawn between the risk surface of closed models versus open-weight models is logically sound and accurately reflects current technical realities.", "audience_yn": "Yes", "audience_explanation": "This work can be relevant to TMLR's audience, particularly researchers in AI safety, policy, and open-source development. The structured taxonomy provides a helpful roadmap for future research , and the consolidated analysis of current developer reporting practices offers a useful reference point for the community.", "requested_changes": "- Theoretical Limits of Tamper Resistance (Strengthen): In Section 4.2, please explicitly discuss the theoretical difficulty (or potential impossibility) of achieving robust \"tamper resistance\"  when adversaries have white-box access to weights and gradients. Acknowledging these fundamental limits would strengthen the technical grounding of the proposed solutions.\n\n- Feasibility of Hardware Locking: The discussion on hardware locking in Section 4.4.3 would benefit from a stronger critique regarding its practicality. As noted, specialized hardware requirements pose significant barriers , and expanding on the friction this creates for the open ecosystem would add necessary nuance.\n\n- Table 2 Clarification: Please clarify in Section 5 whether \"No Mention\" allows for the possibility that developers perform these evaluations (e.g., internal red teaming) but choose not to disclose them to avoid providing attack roadmaps. This can be an interesting avenue as well.", "broader_impact_concerns": "The paper focuses on risk mitigation and does not present direct ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)