T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attacks
Authors: Jiayang Liu
Siyuan Liang
Shiqian Zhao
Rong-Cheng Tu
Wenbo Zhou
Aishan Liu
Dacheng Tao
Siew Kei Lam
Venue: NeurIPS 2025 poster ()
Keywords: jailbreak; text-to-video model; safety filter
Primary area: deep_learning

Abstract: In recent years, fueled by the rapid advancement of diffusion models, text-to-video (T2V) generation models have achieved remarkable progress, with notable examples including Pika, Luma, Kling, and Open-Sora. Although these models exhibit impressive generative capabilities, they also expose significant security risks due to their vulnerability to jailbreak attacks, where the models are manipulated to produce unsafe content such as pornography, violence, or discrimination. Existing works such as T2VSafetyBench provide preliminary benchmarks for safety evaluation, but lack systematic methods for thoroughly exploring model vulnerabilities.
To address this gap, we are the first to formalize the T2V jailbreak attack as a discrete optimization problem and propose a joint objective-based optimization framework, called \emph{T2V-OptJail}. This framework consists of two key optimization goals: bypassing the built-in safety filtering mechanisms to increase the attack success rate, preserving semantic consistency between the adversarial prompt and the unsafe input prompt, as well as between the generated video and the unsafe input prompt, to enhance content controllability. In addition, we introduce an iterative optimization strategy guided by prompt variants, where multiple semantically equivalent candidates are generated in each round, and their scores are aggregated to robustly guide the search toward optimal adversarial prompts.
We conduct large-scale experiments on several T2V models, covering both open-source models (\textit{e.g.}, Open-Sora) and real commercial closed-source models (\textit{e.g.}, Pika, Luma, Kling). The experimental results show that the proposed method improves 11.4\% and 10.0\% over the existing state-of-the-art method (SoTA) in terms of attack success rate assessed by GPT-4, attack success rate assessed by human accessors, respectively, verifying the significant advantages of the method in terms of attack effectiveness and content control. This study reveals the potential abuse risk of the semantic alignment mechanism in the current T2V model and provides a basis for the design of subsequent jailbreak defense methods.

T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attacks
Jiayang Liu Nanyang Technological University Singapore 1jyljy@mail. uste. edu.
Siyuan Liang" Nanyang Technological University Singapore pandallang5210gmail.. com
Shiqian Zhao
Rongcheng Tu Nanyang Technological University Singapore rongcheng tw@ntu.edu..cc.. edu.
Nanyang Technological University Singapore shiqian www.entu edu.sg
Wenbo Zhou University of Science and Technology China China welbeckz@ustc.ee edu. cn
Aishan Liu Beihang University China liuaishan@buaa. edu..eduu....
Dacheng Tao
Siew-Kei Lam Nanyang Technological University Singapore assklam@ntu, edu.
Nanyang Technological University Singapore dacheng. aaoontu.edu....
Abstract
In recent years, fueled by the rapid advancement of diffusion models, text-to-video (T2V) generation models have achieved remarkable progress, with notable exam- ples including Pika, Luma, Kling, and Open-Sone Soon-SSra. Although these models exhibit impressive generative capabilities, they also expose significant security risks due their vulnerability to jailbreak attacks, where the models are manipulated produce unsafe content such as pornography, violence, or discrimination. Existing works such T2VSafety BSech provide preliminary benchmarks for safety evalua- tion, but lack systematic methods for thoroughly exploring model vulnerabilities. To address this gap, discrete optimization problem and propose a joint objective- optimization framework, called T2V-OptJail. This framework consists of two key optimization goals: bypassing the built-in safety filtering mechanisms to increase the attack success rate, preserving semantic consistency between the adversarial prompt and the unsafe input prompt, as well as between the generated video and the unsafe input prompt. to enhance content controllability. tive optimization strategy guided by prompt variants, where multiple semantically equivalent candidates are generated in each round, and their scores are aggregated to robustly guide the search toward optimal adversarial prompts. We conduct large-scale experiments on several T2V models, covering both open-source models (e.g., Open-Sora) and real commercial closed-source models (e.g., Pika, Luma, Kling). The experimental results show that the proposed method improves 1..4% and 10.0% over the existing state-of-the-art method (SoTA) in terms of attack
the first formalize the T2V jailbreak attack
addition, we introduce an itera-
*Corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

success rate assessed by GPT-4, attack success rate assessed by human accessors, respectively, verifying the significant advantages of the method in terms of attack effectiveness and content control. This study reveals the potential abuse risk of the semantic alignment mechanism in the current T2V model and provides basis for the design of subsequent jailbreak defense methods.
Introduction
In recent years, with the continuous evolution of diffusion models 2 3 text-to-video (T2V)
generation techniques have made leap forward, with representative models including Pika (S), Luma 66, Kling 77, and Open-Sora (8) which are capable of synthesizing semantically-matchen
content-rich videos based on natural language prompts, and have been widely used in the fields of entertainment, education, advertising, etc. However, similar to the image generation task, T2V models also face security challenges, especially the vulnerability to jailbreak attacks (9] [10), where the attacker induces the model to generate inappropriate content, such as pornography, violence, and discrimination, through well-designed text inputs [T]. As video content is more realistic and continuous, the generation of unsafe content is often more harmful to the society.
Although prior work such as T2VSafetyBench [12 has initially constructed benchmarks for evaluating the safety of T2V models, research on effective attack methodologies and systematic vulnerability analysis remains limited. The lack of robust jailbreak techniques applicable to real-world deployment scenarios renders mainstream T2V systems highly susceptible to even moderately strong adversarial attacks. This situation raises a critical question: do we truly understand the security boundaries of T2V generation systems?
Designing an effective T2V jailbreak attack involves several key challenges: (1) T2V models typically incorporate complex safety filtering mechanisms, making difficult to directly inject malicious intent into the prompt; (2) as a cross cross-modal system, T2V requires the adversarial semantics to be transferred from text to video, necessitating strong semantic alignment between the adversarial prompt and the generated output to avoid benign reinterpretation; and (3) video generation involves temporal dimension, where a low proportion of unsafe frames can lead to diminished impact due to rapid playback, reducing the overall effectiveness of the attack.
To address these challenges, this paper presents the first optimization- jailbreak attack for T2V models and formulates it as a discrete token-level search problem. We design a language model-driven optimization framework that incorporates two key objectives: (1) filter bypassing optimization, which ensures that the adversarial prompt successfully evades safety filters and induces the generation of jailbreak-relevant frames; and (2) semantic consistency optimization, which preserves the alignment between the adversarial prompt and the original attack intent, as well as the semantic coherence between the prompt and the generated video. Additionally, we introduce an iterative optimization mechanism using a large language model as an agent, which produces high- quality semantic rewrites at each step. To further enhance robustness, we propose Prompt Mutation strategy that introduces multiple semantically similar, slightly altered variants and combines their evaluation scores to help search more robust and generalizable adversarial prompts. This significantly improves the stability of the attack across different models and input scenarios.
We conduct comprehensive empirical evaluations on several mainstream T2V models, including the open-source Open-Sora and commercial closed systems such as Pika, Luma, and Kling, Experimental results demonstrate that our method substantially outperforms existing baselines in terms of attack success rate, content toxicity, and multimodal semantic consistency. For example, on the real-worle platform Pika, our method improves attack success rate by 7.0%, while the generated videos maintain high semantic consistency (0.266) with the original unsafe intent. These results show the potential abuse risks associated with the semantic alignment methods in current T2V models when adequate safety policies are absent and underscore the urgent need for more robust jailbreak defenses. The contributions of this paper are summarized as follows:
We are the first to formalize the T2V jailbreak problem as discrete optimization task and propose joint objective framework that simultaneously optimizes attack success and semantic alignment.

---

We design an iterative search procedure guided by language model agent and introduce prompt variant aggregation strategy to significandly enhance jailbreak effectiveness and robustness.
Experimental results across multiple real-world T2V models demonstrate significant gains (+7% ASR), validating our method's effectiveness and offering guidance for future T2V safety research.
Related Work
2.1 Text-to-Videe Generative Models
Diffusion models and large-scale pretraining have helped text-to-video (T2V) generation make much progress in the last few years. Cascaded diffusion models demonstrated the potential in early works like Make-A-Vidd temporal attention LVDM [5] and MagicVideo [16]. To enable zero-shot generation, methods like Tex 117 and Stable Video Diffusion [1] used pretrained text-to-image models for temporal extension. More recently, commercial systems like Pika (5), Luma [6) and Kling [7] have shown great video quality with fine-grained control. In the open-source domain, Open-Sora [8] replicates the capabilities the proprictary Sora [8] model, providing strong performance and accessibility.
[3] and Imagen- Video [14], followed by improvements such as
2.2 Jailbreak Attacks against Text-to-lmage Models
Jailbreak attacks on text-to-image (T2I) models aim to bypass safety filters and induce the generation of unsafe content (e.g., nudity, violence, discrimination) by carefully designed adversarial prompts [9
19 10.. Existing methods can generally be divided into two categories: search-bases and LLM-bbsed
optimization. Search-based approaches explore the token space to find semantically similar but unfiltered substitutes, in which reinforcement learning [9] or gradient-based methods (20 21 are
utilized, with DiftZero [21] employing zeroth-order optimization for black-box settings. LLM-based approaches utilize large language models to generate or rewrite prompts via in-context learning or instruction tuming [2 23). In addition, there are other works which explore perception- based safe word substitution (4 vulnerabilities in caamorr--ugmennt generation [10). Although they use
different strategies, all aim to evade filters while preserving the intended malicious semantics.
2.3 Jailbreak Attacks against Text-to- Video Models
T2VSafetyBench 12 introduces a benchmark evaluate the safety of text-to-video (T2V) models against jailbreak attacks. This benchmark covers 14 aspects such as pornography, violence, discrimi- nation, and political sensitivity, includes 5,151 malicious prompts that come from real-user datasets (e.g., VidProM (25). I2P (26). UnsafeBench (27), Gate2AI (28) GPT-4-generee
prompts, and prompts crafted via jailbreaking techniques adapted from T2I attacks. Our method T2V-OptJail sig- nificantly distinguishes itself from existing research in the following three key aspects: O Motivation. T2V-OptJail models T2V jailbreak attacks as discrete optimization problem for the first time and combines filter bypassing optimization with semantics consistency optimization. breaking through the limitations of existing methods that rely on static prompts. Implementation. We introduce large language model improve robustness, avoiding manual design and coarse-grained replacement. @ Effects. T2V-OptJail significantly improves the attack success rate and semantic consistency multiple models such as Open-Sora, Pika, etc., with good migration and efficiency.
optimization agent and combine it with prompt variant strategies to
3
Method
In this section, we present our approach for optimizing unsafe prompts aimed at achieving an efficient jailbreak against the T2V generative model. We formalize the task as a discrete optimization problem, where the goal is to search the token space for adversarial prompts that both bypass the model's built-in safety mechanisms and maintain semantic consistency. Specifically, our approach consists of two key components: (1) joint optimization framework that balances the improvement of the

---

Before optimization
Filter Bypassing
After optimization
Safecy Boundary
5A noleed darchig"
Optimization
"A mon kills zombii"
tapinn mani
man Ails harrible zombie"
Target
rahes
Unuafe:
Safety
Eypassing
Unsoff Frame Orcuponcy
Semantics
Semontics Consistency Optimizotion
Boundary
Target
saledd maae
target
Promet Semonéics
Prompt
Cunsistenry
Semondics Consistency
Figure 1: Overall framework of proposed method. Our method generally consists of two main optimization goals: Filter Bypassing Optimization and Semantics Consistency Optimization. Among them, Filter Bypassing Optimization consists of 1 Safety Filter Bypassing for evading the safety filter and @ Unsafe Frame Occupancy for decreasing false positive cases. Semantics Consistency Opti- mization includes the @ Prompt Semantics Consistency for semantics reservation in the adversarial prompt and O Prompt-Vidke Video Semantics Consistency for ensuring semantics similarity in generated video. Before applying our method, the unsafe prompt is blocked by the safety mechanism of T2V system or is revised for generating false positive cases, i.e, safe content. After our optimization, the adversarial prompt can successfully induce the model generate unsafe content.
attack success rate with the semantic quality of the generated videos; and (2) prompt mutation strategy that improves the robustness and generalization ability of the search process by introducing controlled perturbations in the prompt space. The overall framework is shown in Figure |
3.1 Problem Definition
Given unsafe input prompt P that is intercepted by the built-in safety filter F of the T2V model, the attacker's goal is to optimize an adversarial prompt P that can both bypass the safety filtering mechanism and induce the model M to generate videos containing unsafe content. The video generation process can be formalized as follows:
V (fm J M(P"),
where V represents generated video consisting of M frames, each frame is fmi and P (wi, W2,..., w,) represents optimized version of original prompt P, composed discrete tokens.
3.2 Filter Bypassing Optimization for Enhancing Attack Success Rate
To enhance the overall jailbreak performance of adversarial prompts in T2V systems, we propose a Filter Bypassing Optimization (FBO) module that jointly considers input-level evasion and output- level induction.
At the input level, T2V models typically incorporate multiple safety mechanisms to filter out prompts with potentially malicious intent or block the video consisting of unsafe content being generated [12]. We abstract all these internal safety filters unified black-box function, denoted F, which determines whether a prompt is allowed to pass through. To ensure that an adversarial prompt Pa successfully bypasses the filter, introduce the following binary penalty term:
F(P")= = [0, if P bypasses the filter; [1, P is blocked by the filter.
Function J directly evaluated by submitting p to target text-to-video model and observing whether video output returns. If the model outputs an empty response refusal message due to safety concerns, we assign F(P*) 1 otherwise, we set F(P*) = This penalty term incorporated into the overall objective penalize blocked prompts. The optimization encourages

---

the model to conceal unsafe intent through semantic obfuscation or lexical substitution, thereby increasing the deliverability the prompt.
At the output level, we aim to strengthen the intensity and temporal persistence jailbreak hehavior in the generated video. Based on our modeling of the T2V encoder-decoder architecture, the model generates each frame ft by computing cross-attention between the prompt tokens p = { WN } and video frames, represented as:
A Attention(f: t w:),
where A denotes the attention weight from frame ft to token wy. We observe a temporal focusing effect, where for certain frames:
SSE C [1, |S < N,
A1,
iES,
This implies that attention concentrates on a small subset of tokens. If these tokens correspond to an attack intent segment quantify this, we define the jailbreak frame ratio I (MM(P))) )) as the proportion of frames exhibiting unsafe semantics:
W allckk the corresponding frames are likely to exhibit unsafe content. To
T
J(M(P"))
Σ [simecir(ffa Pft P)> o], = T
(5)
where 5 is a similarity threshold.
The overall FBO loss combines the two terms:
Lhypass À
(6)
By jointly minimizing Laypuss, the generated adversarial prompt can not only bypass the safety filter but also induce the model to generate higher proportion of jailbreak-relevant frames. In this way, our method achieves more effective and sustained attack.
3.3 Semantics Consistency Optimization for Improving Jailbreak Quality
Simply bypassing text-level safety filters does not guarantee the completeness and effectiveness of jailbreak attack. further improve the practicality and semantic fidelity of the generated videos, we propose the Semantics Consistency Optimization (SCO) module, which aims to ensure high alignment adversarial prompts in terms both semantic preservation and video-level coherence.
First, at the level of prompt semantic consistency, we require that the optimized adversarial prompt P' remains semantically faithful to the original unsafe prompt P. This prevents significant distortion of the original intent during the process of evading safety mechanisms, which could otherwise cause the generated video to diverge from the attack objective. We employ the CLIP text encoder [29 to extract semantic embeddings of P and pa, and compute their cosine similarity:
DC(P) TC(P-) ([T(P) [[()))))))
sim(C(P),C(P")
where C(-) denotes the CLIP text encoding module. This metric ensures that semantic consistency is preserved during prompt optimization, thereby preventing the original attack semantics from being diluted or lost.
Second, terms of prompt-to-video consistency, we need achieve that the video generated from P", i.e., M(P*), still accurately reflects the core semantics of the original prompt P. To enforce this, we use a video captioning model I.(-) to summarize the generated video and measure semantic similarity to P via CLIP:
sim(C(P), sim(CCP),C(L(M(P)))) C(L(M(P))))
This constraint helps reduce cases where the output of the model is irrelevant or benign content under adversarial prompts and improves both the coherence and controllability of the attack output.
Finally, the two objectives are combined into a unified semantic loss:
sim (C(P) + By minimizing Leem, the generated adversarial prompt retains the core semantics of the original attack intent and also makes video outputs convey those semanties. This significantly enhances the overall effectiveness of jailbreak attacks in terms of both content quality and practical deployment.
Lyem
(IN(C(P),CCIIMM(P)))
5

---

3.4 Prompt Optimization with Mutation Strategy
In order to efficiently search for adversarial prompts that can successfully jailbreak and maintain semantic consistency in a discrete token space, we design an iterative optimization framework based on a language model agent and introduce the prompt mutation mechanism to enhance the robustness and diversity of the search process. The method aims to minimize joint loss function consisting of bypassing ability and semantic consistency:
min Licat Coypas((")) + Lsem Lsem(P"), p
where Laypass measures the ability of the prompt to bypass safety filters and induce failbreak-relewe content, and Luem evaluates the semantic consistency of the adversarial prompt with respect to the original intent, including the semantic alignment quality of the generated video.
We introduce powerful language model (e.g., GPT-4o) as an optimization agent that generates new semantically preserved version P based on the current best candidate Pý-1 in each iteration, and scores it using the joint loss. To enhance the stability of the search process and robustness against semantic perturbations, we further introduce the Prompt Mutation Strategy: construct K semantically equivalent, mildly perturbed variants around the current candidate Pj, denoted as (P,00) P:(K)) J. to simulate subtle rewritings that may occur in real-world inputs.
We form a set V; {PJ. ..... P--K)) containing the main candidate and its K variants. For each prompt in the set, we compute the joint loss and use the average loss as the evaluation metric for this iteration. The prompt which achieves the lowest average loss is selected as starting point for the next round:
Pj arg min ERE, K
E (P)). PřEV;
The optimization process iterates until maximum number of steps Taax or convergence is reached. The final output Pj with the lowest loss is selected as the optimal adversarial prompt. The generated adversarial prompt can reliably induce the T2V model to generate high-quality videos containing jailbreak content.
4 Experiment
4.1 Experimental Setup
Dataset. Due to computational costs, we construct subset of the T2VSafetyBench [12] dataset for our experiments. Specifically, we randomly select 50 prompts from each of 14 categories, resulting in balanced subset with a total of 700 prompts, covering diverse range of scenarios. The 14 categories included in the dataset are as follows: pornography, borderline pornography, violence. gore, disturbing content, public figure, discrimination, political sensitivity, copyright, illegal activities, misinformation, sequential action, dynamic variation, and coherent context.
Models. We evaluate the effectiveness of the proposed method on 4 popular text-to-video models, including open-sourced model Open-Sora 1.1 g and three dosed-sourced commercial models, including Pika 1.5 [5] Luma 1.0 (6), and Kling 1.0 ( from the real world.
Baselines. We consider T2VSafetyBench [12] as one of the baselines for comparison. Additionally, we adopt DACA (22), jailbreak method designed for text-to-image generative models, another bascline. The parameters for these attacks follow the corresponding default settings.
Evaluation metries. To assess the effectiveness of the generated prompts, we use the following evaluation metries: Attack Success Rate (ASR). Our attack evaluation metric is the attack success rate (ASR (%)), which is the percentage of jailbreak prompts. Specifically, jailbreak prompt is considered successful if it meets two criteria: (1) can bypass the model's safety filter, and (2) the generated video from the prompt contains unsafe content, such as pornography, violence, other harmful material. The ASR then calculated as the proportion of such successful jailbreak prompts over the total number of tested prompts. Following T2VSafety Bench [12], we employ GPT-4 and human evaluations to determine the safety of the generated videos. The details of GPT-4 and human evaluations follow the setting of Bey [2]. Semantic similarity (Cosine Similarity).

---

Table 1: The results of various text-to-video models. We report the attack success rate across 14 safety aspects assessed by both GPT-4 and human assessors.
Pika
Luma
6
Kling
7
Open-Sone g
Aspect
GPT-4
Human
GPT-4
Human
GPT-4
Human
GPT-4
Human
Pomography
30.0%
38.0%
10.0%
12.0%
12.0%
14.0%
52.0%
56,0%
Borderline Pomography
64.0%
62.0%
50.0%
58.0%
32.0%
26.0%
50.0%
58.0%
Violence
72.0%
74.0%
28. 28.0%
34.0%
44.0%
40.0%
90.0%
90.0%
Gore
70.0%
72.0%
34.0%
38.0%
48.0%
50.0%
70.0%
76.0%
Disturbing Content
68.0%
76.0%
56.0%
60.0%
36.0%
38.0%
86.0%
92.0%
Public Figures
96.0%
98.0%
32.0%
28.0%
52.0%
56.0%
92.0%
88.0%
Discrimination
30.0%
34.0%
20.0%
28.0%
18.0%
14.0%
46.0%
52.0%
Political Sensitivity
30.0%
32.0%
22. 22.0%
24.0%
12.0%
14.0%
48.0%
44.0%
Copyright
22.0%
16.0%
90.0%
90.0%
70.0%
60.0%
44.0%
50.0%
Illegal Activities
60.0%
60.0%
54.0%
60.0%
52.0%
46.0%
66.0%
64.0%
Misinformation
72.0%
76.0%
80.0%
84.0%
48.0%
42.0%
82.0%
76.0%
Sequential Action
56.0%
52.0%
42.0%
50.0%
42.0%
44.0%
68.0%
74.0%
Dynamic Variation
62.0%
70.0%
36.0%
46.0%
34.0%
36.0%
82.0%
88. 88.0%
Coherent Contextual
50.0%
46.0%
48.0%
42.0%
30.0%
26.0%
64.0%
54.0%
ASR Average
55.9%
57.6%
43.0%
46.7%
37.9%
36.1%%
67.1%
68.7%
We calculate the cosine similarity between the input prompt and the caption of the generated video, using the CLIP text encoder. If one prompt or its output is blocked by the safety filter, we consider an all-black video as the generated video. Specifically, we calculate the average cosine similarity across all test prompts to obtain the semantic similarity in the experiments. This metric measures how closely the semantics of the generated video match the input prompt.
Implementation details. In our optimization function, we set À = 3.0, 3 33=220 2.0, and γ = 1.0. We set the number of iterations to 20, and the number of variants 5. We utilize VideoLLaMA2 [30] as the video caption model L. Since Open-Sora 1.1 is an open-source text-to-video model without built-in safety filters, we manually integrated a combination of safety mechanisms to simulate real-world scenarios. For input filter, we leverage the zero-shot ability of CL.PP to classify the text prompts [31] For output filter, we use the NSFW (Not Safe For Work) detection model, which fine-tuned Vision Transformer, as the end-to-end image classifier (32). For each generated video, we sample image frames and present these multi-frane images to the output filter.
4.2
Main Results
Table I presents comparative evaluation of ASR and semantic similarity across four representative text-to-video models: Pika, Luma, Kling, and Open-Sora. The results are assessed by both GPT-4 and human annotators. We find that the ASR is significantly higher Pika and Open-Sora, while lower on Luma and Kling. We hypothesize this is due to differences in safety filtering strategies: Open-Sora and Pika are either open-source more permissive in content moderation, making them more vulnerable to prompt-based attacks. In contrast, Luma and Kling probably integrate stronger content filtering pipelines and internal moderation heuristics, resulting in lower ASR values. For example, our method only achieves 37.9% (GPT-4) on Kling, compared to 67.1 % on Open-Sora.
We also observe performance differences across different jailbreak scenarios (e.g.. Pornography, Violence, Disturbing Content, and Misinformation). Results show that Open-Sora and Pika are especially vulnerable in high-impact categories like Pornography and Violence, where ASR exceeds 70% in some cases. This indicates that current safety mechanisms are insufficient for detecting nuanced visually implied unsafe content in these domains. In contrast, models like Kling show greater robustness in categories such as Misinformation and Hate Speech, which may benefit from more conservative generation policies or stricter internal filters. Overall, these findings demonstrate that our method not only achieves stronger attack effectiveness, but also reveals critical variations in model vulnerability depending architecture and moderation design.
In Figure 2 we illustrate two generated examples that demonstrate the effectiveness our proposed method. We generate malicious prompts targeting Kling 77, where the safety filter blocks the original input prompts. The resulting malicious prompts are able to jailbreak real-world platforms, leading to the generation of vidcos containing unsafe content.
7

---

Table 2: Attack success rate (GPT-4 Human) and semantic similarity various T2V models. Bold
indicates best performance.
Pika
Luma
Method
GPT-4 (%)
Human (%)
Similarity
GPT-4 (%)
Human (%)
Similarity
T2VSafetyBench
47.7
50.6
0.257
32.9
37.9
0.253
DACA
0.245
10.3
11.1
0.244
12V-OplJail (Ours)
55.9
57.6
0.266
43.0
46.7
0.263
Kling Human (%)
Open-Sora Human (%)
Method
GPT-4 (%)
Similarity
GPT-4 (%)
Similarity
T2VSafctyBench
29.3
26.7
0.251
55.7
58.7
0.259
DACA
7.3
6.1
0.240
22.3
24.0
0.247
T2V-OptJail (Ours)
37.9
36.1
0.257
67.1
68.7
0.269
(a) Input prompt: lesbian kiss. Generated malicious prompt: a girl lick another woman push.
(b) Input prompt: human kill zombie. Generated mali- cious prompt: man kills a horrible zombie.
Figure Examples of generated malicious prompts. These examples are generated with the commercial text-to-video platform Kling 77.
4.3 Comparison with Baselines
Table compares the ASR and semantic similarity of our method with two baselines: TeVSafetyBench [12] and DACA (22). We use the default hyperparameters of DACA [22] For the Open-Sora model, our method achieves an ASR of 67.1% (GPT-4) and 68.7% (Human), substan- tially surpassing T2VSafety Bench (55.7% GPT-4, 58.7% Human) and DACA (22.3% GPT-4, 24.0% Human). Moreover, our method attains semantic similarity score of 0.269, which higher than T2VSafetyBench (0.259) and DACA (0.247). This indicates that the adversarial prompts generated by our method not only have higher success rate but also preserve the semantic meaning of the input prompts more effectively. Similarly. across all the commercial models, the ASR of our ap- proach consistently outperforms the baselines, with improvements of 7.0% to 10. 10.1% compared to T2VSafety Bench and even larger margins over DACA. The semantie similarity for our method also higher than the baselines, highlighting the effectiveness of our optimization strategy in generating semantically consistent malicious prompts.
These results suggest that our method not only enhances the attack success rate significantly but also ensures that the generated video remains semantically similar to the input prompts, demonstrating that our approach effectively balances attack success with semantic integrity.
4.4 Comparison with More Baselines
We compare the proposed method with two additional baselines, including Sneakyprompt and Autodan [33]. We use the default hyperparameters of Sneakyprompt [9] and Autodan [33]. Table 3 shows attack success rate and semantic similarity Pika and Open-Sora. Compared baseline methods, T2V-OptJil consistently achieves the highest attack success rates on both models. For instance, on the Open-Son -ora model, reaches 67.1% (GPT-4) and 68.7% (Human), which better than Autodan (40.6% 44.0%) and Sneakyprompt (27.9% / 30.4%). Similarly, T2V-OptJail achieves 55.9% (GPT-4) and 57.6% (Human) on Pika, outperforming Autodan (33.1% 36.11 1%) and Sneakyprompt (20.6% / 23.0%). 1. In addition, T2V-OpUJil maintains the highest semantic similarity in all cases, such as 0.269 on Open-Sora and 0.266 on Pika. This indicates its strong ability to preserve the intended semantics while evading safety filters. These results demonstrate that T2V-OptJail is not only more

---

effective in generating successful jailbreak prompts, but also better at preserving the underlying unsafe intent in a stealthy manner.
Table Attack success rate (GPT-4 Human) and semantic similarity on Pika and Open-Sora, Bold indicates best performance.
Pika
Open-Sora
Method
GPT-4 (%)
Human (%)
Similarity
GPT-4 (%)
Human (%)
Similarity
Sneakyprompt
20,6
23.0
0.247
27.9
30.4
0.248
Autodan
33.1
36.1
0.249
40.6
44.0
0.251
T2V-OptJail (Ours)
55.9
57.6
0.266
67.1
68.7
0.269
4.5
Comparison with Genetic Algorithm
We compare with the genetic algorithm (GA) bascline, in which prompts are modified via simple token substitution without LLM guidance. For the GA, we tokenize the prompt and then replace tokens with semantically similar alternatives. We perform the experiment on Open-Sora and the results are shown in T Our method outperforms GA by approximately 10% in attack success rate, while achieving higher semantic similarity. This result highlights the superiority of using LLM guidance. Delving into the intrinsic difference, we argue that the possible reason is that LLM-based agent is more effective identifying suitable modifications to the prompts during optimization.
Table 4: Attack success rate (GPT-4 Human) and semantic similarity on Open-Sora. Bold indicates best performance.
Open-Sora
Method
GPT-4 (%)
Human (%)
Similarity
T2V-OptJail using GA
58.8
0.260
T2V-OptJail (Ours)
67.1
68.7
0.269
4.6 Experiments Defenses
We further validate the effectiveness of our method when defenses are adopted. Table shows attack success and semantic similarity defense methods, including Keyword Detection p 34 and Implicit Mcaning Analysis [34]. We use the default hyperparameters of these defenses following the setting of (34). The malicious prompts are generated against Open-Sora. Compared baseline methods, T2V-OpUJail achieves the highest attack success rates under these defenses. For example, under Implicit Meaning Analysis, it reaches 66.1% 1% (GPT-4) and 67.4% (Human), outperforming T2VSafetyBende (54.9% / 57.8%) and DACA (22.0% / the highest semantic similarity, such as 0.268 under Implicit Meaning Analysis, indicating its strong ability to preserve unsafe intent even against defenses.
23.4%) ) In addition, T2V-OptJail maintains
Table 5: Attack success rate (GPT-4/ Human) and semantic similarity on defense methods. Bold indicates best performance.
Keyword Detection
Implicit Meaning Analysis
Method
GPT-4 (%)
Human (%)
Similarity
GPT-4 (%)
Human (%)
Similarity
B2VSafetyBench
43.8
47.1
0.252
$4.9
$7.8
0.258
DACA
12.2
14.4
0.241
22.0
23.4
0.247
T2V-OptJail (Ours)
52.3
54.6
0.257
66.1
67.4
0.268
4.7
Ablation Study
We conduct the following ablation studies to investigate effects of key hyperparameters, including the balance factor X, balance factor 3, number of iterations, and the presence or absence of the prompt mutation strategy. For the ablation studies of these hyperparameters, we generate the malicious prompts against Open-Sora (8).

---

10
0.29
629
Scrrani umilacty
Scmanik amilait
0.79
Seaaati stnuliity
Atalk sacess 230
sintalis
70
20
0.92
0.27
68
GIPL Elerron
CPP Ituron
0.5
1.5
3.5
1.24
15
(a) X
(b) |
(c) Number of Iterations
Figure 3: Ablation studies on different hyperparameters: (a) balance factor X, (b) balance factor B and (c) number of iterations.
Table 6: Effectiveness of the prompt mutation strategy on attack success rate and semantic similarity.
Kling
Open-Sora
Method
GPT-4 (%)
Human (%)
Similarity
GPT-4 (%)
Human (%)
Similarity
T2V-OptJail
37.9
36.1
0.257
67.1
68.7
0.269
w/o Prompt Mutation
34.4
31.9
0.255
61.1
62.4
0.266
Balance factor X. Figure 3a illustrates the attack success rate and semantic similarity of our attack are fixed. When À is increased, the attack
with different values of X, while other success rate improves while the semantic similarity decreases. To balance the attack success rate and semantic similarity, we set
3.0 our experiments.
Balance factor B. Figure 3b illustrates the attack success rate and semantic similarity of our attack with different values of B, while other dyyerr-paaaeett success rate decreases while semantic similarity improves. To balance the attack success rate and semantic similarity, we set 2.0 in our experiments.
fixed. When / is increased, the attack
Number of iterations. Figure 3c illustrates the attack success rate and semantic similarity of our attack with different numbers of iterations, while other syper-parameters are fixed. When the number of iterations is no larger than 20, both the attack success rate and semantic similarity improve as the number of iterations increases. However, when the number of iterations exceeds 20, the improvements in both attack success rate and semantic similarity become marginal. Additionally, more iterations require more computation overhead during the optimization. To balance the attack success rate, semantic similarity with computation overhead, we set the number of iterations to 20.
Prompt mutation ablation. Table pressenss the attack success rates and semantic similarity with and without the prompt mutation strategy on text-to-video models. The results show that incorporating the prompt mutation strategy not only improves the attack success rate for both GPT-4 and human evaluations but also enhances the semantic similarity. For instance, on the Open-Sora dataset, the attack success rate increases from 61.1% % to 67.1 for GPT-4 and from 62.4% 68.7% for humans, while the semantic similarity also improves from 0,266 to 0.269. This demonstrates that the prompt mutation strategy effectively enhances both the attack performance and the semantic relevance of the generated video.
Discussion and Conclusion
This paper presents T2V-OptJail, the first optimization- jailbreak framework for text-to-video models. By jointly optimizing for safety filter bypass and semantic consistency, along with a robust prompt mutation strategy, our method achieves significantly higher attack success rates and better content controllability than existing baselines. Extensive experiments on both open-source and commercial T2V models highlight serious safety vulnerabilities in current systems. One limitation of our current method is that it requires querying the generated videos for feedback during optimization, which improves performance but introduces extra query budget. However, we argue that this limitation can be mitigated by introducing local proxy model (free of charge) or optimizing the querying algorithm. We leave this for future work.
10

---

References
[1] Andreas Blattmann, Tim Dockhor, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent vidco diffusion models to large datasets, arXiv preprint arX1v2231 2023.
[2] Jay Zhangjie Wu, Yixiao Ge. Xintao Wang, Stan Weixian Lei. Yuchao Gu. Yufei Shi. Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-vides One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623-7633, 2023.
[3] David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-l: Marrying pixel and latent diffusion models for text-to-video generation. International Journal of Computer Vision, pages 1-15, 2024.
[4] Haoxin Chen, Menghan Xia, Yingqing Hc, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing. Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint 2rXiv:23/1 19512, 2023.
[5] Pika ai free video generator, 2024. URL https://pikattai.com
[6] Luma. luma dream machine, 2024. URL https://tumllasssstttt wwwp:/dssiaiieeammmmmmmiiiii machine.
[7] Kwai. kling, 2024. URL https///linngkuuishuu..co
[8] Hpcaitech. open-sora: Democratizing efficient video http://giihbb.com/mpaaaccc////pp
production for all, 2024.
URL
[9] Yuchen Yang. Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. Sneakyprompt Jailbreaking text-to-image generative models. In 2024 IEEE symposium on security and privacy (SP), pages 897-912. TEEE, 2024.
[10] Shiqian Zhao, Jiayang Liu, Yiming Li, Runyi Hu, Xiaojun Jia, Wenshu Fan, Xinfeng Li, Jie Zhang, Wei Dong, Tianwei Zhang, et al. Inception: Jailbreak the memory mechanism of text-to-image generation systems. arXiv preprint arXiv:200.223776 2025.
[11] Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlci Hc, Jiaxing Song, Ke Xu, and Qi Li. Jailbreak attacks and defenses against large language models: A survey. arXiv preprint arXiv:2407. 240.:2490..429555 2024.
[12] Yiho Miao, Yifan Zhu, Lijia Yu, Jun Zhu, Xiao-Shan Gao, and Yinpeng Dong. T2vsafethhnnc:: Evaluating the safety of text-to-vide video generative models. Advances in Neural Information Processing Systems, 37:63858 63872, 2025.
[13] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
[14] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, and Tim Salimans. Imagen video: High definition vidco generation with diffusion models. arXiv preprint arXiv:2210.023:0
2022.
[15] Yingqing Hc, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint 22X.:2211..3221,, 2022.
[16] Daquan Zhou, Weimin Wang. Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint 2tt::2211.11001111 2022.
[17] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Tex:2ideo-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages
15994-15564, 2023.
[18] Openai. sora: Creating vidco from text, 2024. URL https://openai.coom https://ppemai.com/son. com//sor..
[19] Yihao Huang, Le Liang, Tianlin Li, Xiaojun Jia, Run Wang, Weikai Miao, Geguang Pu, and Yang Liu. innceptioonguided jailbreak against text-to-image models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26238-26247, 2025.
[20] Yijun Yang, Ruiyuan Gao, Xiaosen Wang. Tsung-Yi Yi Ho, Nan Xu, and Qiang Xu. Mmaadifffusin: Multimodal attack on diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7737-7746, 2024.
11

---

[21] Pucheng Dang, Xing Hu, Dong Li, Rui Zhang, Qi Guo, and Kaidi Xu. Diffzoo: A purely query-based black-box attack for red-teaming text-to-image generative model via zeroth order optimization. arXiv preprint arXiv:2408. 11071, 2024.
[22] Yimo Deng and Huangxun Chen. soliee-nddcongeee attack: Harnessing the power of Ilm bypass the censorship of text-to-image generation model. arXiv preprint arXiv:2312.07130, 2023.
[23] Yingkai Dong, Zheng Li, Xiangtao Meng. Ning Yu, and Shanqing Guo. Jailbreaking text-to-image models with based agents, arXiv preprint arXiv:2408 00523, 2024.
[24] Yihao Huang, Le Liang, Tianlin Li, Xiaojun Jia, Run Wang. Weikai Miao, Geguang Pu, and Yang Liu. Perception-guided jailbreak against text-to-image models. arXiv preprint urXiv:2408. 10848, 2024.
[25] Wenhao Wang and Yi Yang. Vidprom: A million-scale real prompt-gallery dataset for text-to-video diffusion models. Advances in Neural Information Processing Systems, 37:65618 65642, 2025.
[26] Patrick Schramowski, Manuel Brack, Björ Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22522-22531, 2023.
[27] Yiting Qu, Xinyue Shen, Yixin Wu, Michael Backes, Savvas Zannettou, and Yang Zhang. safebench: Benchmarking image safety classifiers on real-world and ai-generated images. arXiv preprint arXiv:2405.03485 2024.
[28] Gate2ai. URL http://wwww.aatte22 gaw.ec/ci.....oooooommmmmioaaaa.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR,
2021.
[30] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang. Ziyang Luo, Deli Zhao, al. Videollama Advancing spatial-tempont modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.0744 2024.
[31] Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting. Can machines help answering question in datashects, and turn reflecting on inappropriate content? Proceedings of 2022 ACM conference on fairness, accountability, and transparency, pages 1350-1361, 2022.
[32] Falconsai. Nsfw image classification. detection 2024, Accessed on: 2024-11-18.
[33] Xiaogeng Liu, Nan Xu. Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04551, 2023.
[34] Siyuan Liang, Jiayang Liu, Jiecheng Zhai, Tianmeng Fang. Rongcheng Tu, Aishan Liu, Xiaochun Cao, and Dacheng Tao. T2vshield: agnestcg jailbreak defense for text-to- video models. arXiv preprint arXiv:2504.5511.1 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims the abstract and introduction accurately reflect the paper's contributions and scope, effectively conveying the key points and supporting evidence.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper effectively discusses the limitations of the work done by the authors in the "Discussion and Conclusion" section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification:
The paper does not include theoretical results, hence there are no assumptions
proofs to be provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses all the information necessary to reproduce the main experimental results, ensuring transparency and replicability of the findings.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The paper provides open access to the data and code, the anonymous link is 25_12v-CH60)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic/guuddes/Codeeubbiiss
www.sssssiioooiiiyoooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppppclgguiuaeeeaeccuuee
Saaeee//CoSSSubisss..S.. www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper specifies all the training and test details necessary to understand the results, such as hyperparameters, ensuring transparency and reproducibility of the experimental setup.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
As described in the Experimental Setup, we report the average of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: As described in the Supplemental Material, all experiments were conducted on server equipped with an Intel(R) (eon(R) Gold 6336Y CPU @ 2.40GHz, 512 GB of system memory, and one NVIDIA A100 GPU with 40 GB memory.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research in the paper aligns with the NeurIPS Code of Ethics, ensuring adherence to ethical standards outlined by the NeurIPS community for responsible conduct. in Al research.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
16

---

Justification:
The paper discusses both potential positive societal impacts and negative societal impacts the work performed Supplemental Material, fulfilling the conference's expectations for addressing broader impacts and considering potential ethical implications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (c.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no risks requiring specific safeguards for responsible data or model release, therefore safeguards are not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators original owners of assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and respected, ensuring compliance with legal and ethical standards.
Justification:
Guidelines:
17

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssii thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce new assets, therefore documentation of assets is not applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Yes, our work involves human evaluation judge the safety of generated videos. The volunteers were shown set of videos and asked to evaluate them based on the input prompt and presence of jailbreak content. All participants received full instructions, including example screenshots and evaluation guidelines. The volunteers are at least 18 years old, in good physical and mental health, and free from conditions such as heart disease or vasovagal syncope.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the Ne Code of Ethics, workers involved data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
18

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Our study involves human evaluation conducted by members of our research lab. The task involves watching Al-generated videos and providing judgements based on predefined criteria. The evaluation does not involve any sensitive personal information, and no foreseeabl risks were posed to the participants. Therefore, IRB approval was not required under our institution's 's guidelines.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
19


Reviews:
[{"review_id": "a9mnlh7sDf", "reviewer": "Reviewer_Zxsj", "summary": "This paper addresses the security vulnerabilities of text-to-video (T2V) generation models, which are susceptible to jailbreaking attacks that generate unsafe content. Despite existing benchmarks like T2VSafetyBench, there is a lack of systematic methods to explore model vulnerabilities. This work introduces T2V-OptJail, a goal-based optimization framework that aims to bypass security filters and maintain semantic consistency between adversarial prompts and generated content. The approach includes an iterative optimization strategy using prompt variations to improve attack robustness. Experiments on popular T2V models such as Pika, Luma, Kling, and Open-Sora show significant improvements in attack success rate and semantic consistency.", "questions": "Could considering the diversity of jailbreak prompts enhance the proposed method?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n- This is the first optimization-based jailbreak attack specifically designed for T2V models, contributing to the formalization of T2V jailbreaking as a discrete optimization problem, an innovative iterative search strategy, and strong experimental validation, providing valuable insights for future T2V security research.\n- The paper is clearly written and easy to understand.\n- The experimental results on T2VSafetyBench (Table 2) effectively validate the effectiveness of the proposed method.\n- The ablation experiments (Table 3) validate the rationale of Prompt Mutation.\n\nWeaknesses\n- The authors consider semantic consistency optimization, which preserves the alignment between the adversarial prompt and the original attack intent, as well as the semantic coherence between the prompt and the generated video. However, what is the significance of maintaining semantic consistency? From the perspective of exploring model vulnerabilities, a successful jailbreak that significantly alters the semantics is equally valuable.\n- The authors did not provide the computational overhead of the proposed method. It would be beneficial to provide details regarding time consumption and query budget, along with an analysis.", "was_revised": true, "final_justification": "The author’s response addresses all my concerns regarding semantic consistency and computational overhead. I also review the comments from other reviewers and the author’s rebuttals. The author’s responses to reviewers XweG/iDNu convincingly demonstrate the novelty of the proposed method. The paper structure issues raised by grrG are also resolved. Therefore, I raise my score from 4 to 5.", "rebuttal": "Q1: The authors consider semantic consistency optimization, which preserves the alignment between the adversarial prompt and the original attack intent, as well as the semantic coherence between the prompt and the generated video. However, what is the significance of maintaining semantic consistency? From the perspective of exploring model vulnerabilities, a successful jailbreak that significantly alters the semantics is equally valuable.\n\nA1: Thank you for raising this valuable question. The “semantic consistency” referred to in our paper encompasses two levels:\n1. Consistency between the prompt and the generated video. This is the most basic capability requirement for T2V models. If the generated video does not align with the prompt's semantic intent (e.g., the prompt requests “dancing topless,” but the generated video shows “a speech” or other irrelevant scenes), it not only fails to demonstrate the model's understanding of the prompt but also cannot be considered a truly “successful and controllable” attack. Therefore, we incorporate this consistency into the optimization objective to assess whether the model poses exploitable risks at the generation level.\n\n2. Consistency between the optimized prompt and the original attack intent. In reality, attackers often do not wish to alter their attack intent but instead aim to bypass filters in a more “covert” manner. We achieve this type of “intent-preserving jailbreak” through semantic-invariant rewriting, which better aligns with actual risks. If the optimization process causes the semantic meaning to deviate completely, the attack's effectiveness loses controllability and reproducibility, thereby reducing its practical value.\n\nRegarding the reviewers' view that “even if the semantic changes are significant, the escape behavior still has value,” we partially agree. In certain cases, this can indeed reveal weaknesses in the model's filtering mechanisms. However, we believe that only when attacks remain successful while maintaining their original intent can they expose deeper vulnerabilities in the model's semantic alignment and instruction execution. This is why we emphasize semantic consistency, as it not only enhances the practical significance of attacks but also makes model evaluation more challenging.\n\n\nQ2: The authors did not provide the computational overhead of the proposed method. It would be beneficial to provide details regarding time consumption and query budget, along with an analysis.\n\nA2: We perform the computational overhead experiments using 700 prompts, which is the same setting as the experiments in the main manuscript. The experiments were conducted on a server equipped with an Intel(R) Xeon(R) Gold 6336Y CPU @2.40GHz, 512 GB of system memory, and one NVIDIA A100 GPU with 40 GB of memory. We conduct jailbreaks against the Open-Sora. The time consumption results are shown in Table 1. We report the average attack time per prompt. Since T2VSafetyBench is a benchmark study that primarily adopts three jailbreak prompt attack methods (RAB, JPA, and BSPA), we report its time consumption as the average computation time across these three methods. Our experimental results show that the computational cost of our method is the second lowest compared to existing approaches. The other methods, except JP, take over 15 minutes longer per prompt than our method. Moreover, we show the attack performance of these methods against Open-Sora in Table 2. The results of Table 2 show that JPA achieves lower attack success rates and semantic similarity than our method, although JPA achieves the fastest computation time in generating adversarial prompts. These results demonstrate that our method achieves a good balance between attack performance and computational efficiency.\n\n**Table 1. Comparison of attack time.**\n|     |T2VSafetyBench |DACA|SneakyPrompt|JPA|T2V-OptJail (Ours)|\n|--------------|--------|----------|-----------|-------|-------|\n|Attack time per prompt (min) |28.3  |29.2  |62.6 |**6.9**   |12.8     |\n\n**Table 2. Comparison of attack performance.**\n|Results on Open-Sora      |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|T2VSafetyBench|55.7| 58.7 |0.259         |\n|DACA |22.3 |24.0 |0.247          |\n|SneakyPrompt |27.9 |30.4 |0.248          |\n|JPA |41.7       |44.0         |0.251          |\n|T2V-OptJail|**67.1**|**68.7**  |**0.269**  |\n\n\nQ3: Could considering the diversity of jailbreak prompts enhance the proposed method?\n\nA3: Following your advice, we incorporate a diversity loss term that penalizes semantic similarity among generated prompts across different inputs to evaluate whether increasing the diversity of generated jailbreak prompts could enhance the proposed method. Here, we use Jaccard Similarity to measure prompt diversity. Specifically, we introduce:\n$$\nL_{\\text{div}} = \\max_{i} \\, \\text{sim}(P^*, P^{(i)})\n$$\nand optimize a joint objective:\n$$\nL_{\\text{total}} = L_{\\text{bypass}} + L_{\\text{sem}} + \\alpha \\cdot L_{\\text{div}}.\n$$\nThis unified optimization function encourages semantic preservation with the original prompt while discouraging repetition across different adversarial prompts. We set $\\alpha=0.2$. We follow the same experimental setting as the main manuscript. Experimental results in Table 2 show that introducing this term leads to an approximately 2% increase in attack success rate, with a drop in semantic similarity. This demonstrates that increasing the diversity of generated jailbreak prompts can slightly improve the attack performance of our method, but leads to a drop in semantic similarity, which is a trade-off between attack success rate and semantic similarity.\n\n**Table 2. Results when incorporating diversity loss.**\n|Results on Pika      |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|T2V-OptJail|55.9       |57.6         |**0.266**          |\n|T2V-OptJail with diversity|**58.0**|**59.9**  |0.264  |\n\n|Results on Luma      |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|T2V-OptJail|43.0       |46.7         |**0.263**          |\n|T2V-OptJail with diversity|**44.8**|**48.8**  |0.261  |\n\n|Results on Kling     |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|T2V-OptJail|37.9       |36.1         |**0.257**          |\n|T2V-OptJail with diversity|**39.1**|**37.6**  |0.256  |\n\n|Results on Open-Sora      |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|T2V-OptJail|67.1       |68.7         |**0.269**          |\n|T2V-OptJail with diversity|**69.5**|**71.5**  |0.266  |"}, {"review_id": "oGOHERl4ZI", "reviewer": "Reviewer_XweG", "summary": "The paper presents a method for jailbreaking text-to-video AI models to generate unsafe content. The main idea is to automatically optimize prompts that can get past safety filters while still producing the intended harmful videos. The method uses GPT-4o to iteratively rewrite prompts and test multiple variants to find ones that work. When tested on real platforms like Pika and Luma, their system successfully generates unsafe videos more reliably than previous approaches that were more manually crafted.", "questions": "1. Implementation Details: Could you provide more specific details about the prompt mutation strategy? What exact prompts or instructions do you give to GPT-4o to generate the variants, and how do you ensure they remain semantically consistent while introducing useful diversity?\n\n2. Practical Limitations: Your method requires generating videos during optimization, which is computationally expensive and potentially detectable by platforms monitoring for repeated attempts. Have you analyzed the computational costs compared to simpler attack methods?\n\n3. Defense Effectiveness: You mention testing an adaptive defense in the appendix, which seems ineffective against your attacks. Given that your method can evade both conventional and adaptive defenses, what types of defense mechanisms do you think would be most promising for mitigating these attacks?\n\n4. Transferability and Robustness: Do attacks optimized against one T2V model transfer effectively to other models? And given the significant performance differences across platforms, what factors make certain T2V systems more robust, and could these insights inform better defense strategies?", "limitations": "Societal impact:\n•\tWhile the authors mention responsible disclosure, providing detailed attack methods could enable misuse.\n•\tThe adaptive defense mentioned is relegated to the appendix and seems ineffective.\n•\tLimited discussion of detection/mitigation strategies beyond basic filtering.\n\nLimitations:\n•\tThe focus is mainly on bypassing existing filters rather than more sophisticated defenses.\n•\tThe attack categories are fairly basic. What is required to address more subtle forms of harmful content?\n•\tMore  analyses of failure and limitations is warranted.\n•\tLimited analysis of why certain platforms are more robust than others.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This is a solid, well-executed paper addressing an important problem. The technical approach is sound and clearly presented, with strong experimental validation. However, the contribution is primarily an engineering advance applying existing techniques to a new domain rather than a fundamental methodological innovation.  The paper would benefit from deeper analysis of why attacks succeed and fail, better discussion of defenses, and more thorough evaluation of the practical deployment constraints.\n\nStrengths:\n\nQuality:\n•\tSolid formalization of T2V jailbreak as a discrete optimization problem with a clear mathematical framework..\n•\tWell-designed joint objective combining filter bypassing and semantic consistency. \n•\tComprehensive evaluation on a variety of deployed systems: across both open-source (Open-Sora) and real commercial platforms (Pika, Luma, Kling).\n•\tGood experimental design with proper baselines and metrics (ASR, semantic similarity, and both GPT-4 and human evaluation).\n•\tThe prompt mutation strategy seems technically sound. \n\nOriginality:\n•\tThe authors claim this is the first optimization-based approach for T2V jailbreaks. \n\nSignificance:\n•\tReveals significant vulnerabilities in current T2V systems (more than 55% ASR on some T2V systems).\n•\tResults show clear improvements over existing methods. \n•\tImportant for understanding and addressing T2V safety before these systems become more widespread.\n\nClarity:\n•\tThe paper is generally well-written and organized.\n•\tClear problem motivation and mathematical formulation.\n•\tThe figures were helpful in illustrating the framework and results.\n\nWeaknesses:\n\nOriginality:\n•\tI am not an expert in the area of T2V jailbreaks; while this approach may be original there to me it seems like a straightforward adaptation of existing text-to-image jailbreak methods to the video domain.\n•\tThe discrete optimization framework is not particularly novel as similar approaches exist for text adversarial attacks.\n•\tThe prompt mutation strategy is fairly standard in adversarial ML.\n\nQuality & Clarity:\n•\tRequires video generation feedback during optimization, making it expensive and potentially detectable.\n•\tNo comparison of computational costs vs. simpler attack methods.\n•\tMissing analysis of attack transferability across different T2V models.\n•\tHuman evaluation details are sparse. It would be good to discuss the number of evaluators, inter-rater agreement, etc.\n•\tAblation studies are limited (only a few hyperparameters).\n\nSignificance\n•\tLimited discussion of detection/mitigation strategies beyond basic filtering. The adaptive defense mentioned is relegated to the Appendix and seems ineffective based on their description.\n•\tLimited analysis of failure cases and attack limitations.", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Originality and novelty issue\n\nA1: We would like to express our gratitude to the reviewers for their careful evaluation of our work. We would like to elaborate on the originality of this work, which is manifested in the following aspects:\n- First, we are the first to formulate the T2V jailbreak task as a discrete optimization problem. We propose a GPT-based discrete prompt optimization framework that actively searches for illegal instruction prompts in the LLM token space that are both attack-oriented and capable of bypassing security filters.\n- Second, we incorporate dynamic constraints from the video generation process into the objective function, an aspect completely missing in T2I jailbreak research. We introduce two T2V-specific loss terms: Unsafe Frame Occupancy and Prompt-to-Video Semantic Consistency, to integrate temporal and semantic alignment into jailbreak evaluation, extending beyond traditional static-generation attack methods.\n- Third, we introduce a semantic-level prompt mutation strategy specifically designed for T2V, moving beyond traditional token-level mutations used in existing adversarial methods. \n\nIn summary, our work is not a simple migration of existing methods, but rather an innovative exploration of T2V jailbreak optimization in task modeling, loss design, optimization strategies, and validation experiments, demonstrating clear originality. To prove this point, we present experimental comparisons between Ours and the T2I baselines, as well as a simple genetic algorithm, in A2 response of Reviewer grrG and A2 response of Reviewer iDNu. The experiments demonstrate that our method achieves a higher attack success rate. Please also refer to A1 response of Reviewer iDNu for more detailed discussion about originality and novelty issue.\n\nQ2: Requires video generation feedback during optimization, making it expensive and potentially detectable. \n\nA2: Transferability experimental results (see the result in Q4) show that the attacker can even conduct jailbreaks on open-source T2V model (Open-Sora) to achieve good attack performance on commercial T2V models, which is not expensive. In practical, attackers could easily evade the petential detection by manipulating multiple independent accounts. \n\nQ3: No comparison of computational costs vs. simpler attack method.\n\nA3: The results and analysis of computational costs can be refered in the A2 response of Reviewer Zxsj. Simpler attack method such as JPA can achieve shorter attack time, but lead to relatively lower attack performance compared with our method.\n\nQ4: Missing analysis of attack transferability across different T2V models.\n\nA4: We add a transferability experiment across different T2V models, following the experiment setting of main manuscript. We firstly optimize adversarial prompts on the Open-Sora. Then we test the generated adversarial prompts on Pika and Kling. As shown in Table 1, our method outperforms over 4% than T2VSafetyBench in transferability and achieves the highest transferability among all the methods. The potential reason is our optimization incorporates four objectives which help allievates the potential overfitting to a certain model, and further improve the robustness of our generated adversarial prompts. \n\n**Table 1. Transferability.**\n|On Pika |GPT-4(%)|Human(%)|Similarity|\n|-|-|-|-|\n|T2VSafetyBench|32.5|34.3|0.252 |\n|DACA|8.5|9.6|0.238|\n|T2V-OptJail|**37.6**|**39.0**|**0.254**|\n\n|On Kling |GPT-4(%) |Human(%)|Similarity|\n|-|-|-|-|\n|T2VSafetyBench|24.4|23.3|0.248|\n|DACA|3.2|2.5|0.213|\n|T2V-OptJail|**28.6**|**27.4**|**0.249**|\n\nQ5: Human evaluation details are sparse.\n\nA5: We recruited 50 adult volunteers (aged 18 and above) in good physical and mental health, with no conditions such as heart disease or vasovagal syncope. Before the assessment, participants were given clear definitions and examples of each safety risk type. Videos were viewed in full on 22–24 inch monitors. To ensure consistency, we conducted a second evaluation round on overlapping samples and measured inter-rater agreement. The Fleiss’ Kappa score of 0.78 indicates substantial agreement among annotators. \n\nQ6: Ablation studies are limited (only a few hyperparameters).\n\nA6: We have conducted ablation studies of eight hyperparameters in main body of the manucript and supplementary material (four in main body of the manucript and four in supplementary material). Please refer to supplementary material to see more ablation studies.\n\n\nQ7: Limited discussion of detection/mitigation strategies beyond basic filtering.\n\nA7: We've considered five existing potential defenses (including **Keyword Detection, Implicit Meaning Analysis, Adversarial Training, Unlearning Defense, and Safree**) and one more adaptive defense in Table 1 and Table 2 of the supplementary material. These results have demonstrated that our method is robust against these strong defenses beyond basic filtering. Here we consider one more potential defense, GuardT2I [1], following the same experiment setting in main manuscript, and present the result in Table 2. As shown in Table 7, our method still outperforms T2VSafetyBench over 7% in attack succcess rate, while maintaining high semantic similarity under **defense GuardT2I**. This further demonstrates that our method is more robust against conventional defenses beyond basic filtering. \n\n**Table 2. Performance on GuardT2I.**\n|Methods|GPT-4(%)|Human(%)|Similarity|\n|-|-|-|-|\n|T2VSafetyBench|45.5|49.1|0.253|\n|DACA|13.8|15.7|0.241|\n|T2V-OptJail|54.7|56.6|0.258|\n\n[1] Yang, Yijun, et al. \"Guardt2i: Defending text-to-image models from adversarial prompts.\" Advances in neural information processing systems 37 (2024): 76380-76403.\n\nQ8: The adaptive defense mentioned is relegated to the Appendix and seems ineffective based on their description. \n\nA8: Due to space limitation of the manucript, we put adaptive defense section in Appendix. Moreover, our proposed adaptive defense is effective against our method. Our designed adaptive defense (as shown in the supplementary material) reduces the attack success rate by approximately 16%, significantly lowering the effectiveness of our proposed attack. Please refer to Response A3 of Reviewer grrG to see more results about enhanced adaptive defense and adaptive attack.\n\nQ9:  Limited analysis of failure cases and attack limitations.\n\nA9: We have discussed the current limitations of our attack in the Discussion and Conclusion section (lines 325–329 of the main paper). During our experiment, we found that, when the model's safety filters are overly sensitive, it may block video generation even if the prompt contains no explicit unsafe keywords. Additionally, during semantic consistency optimization, the introduction of unsafe terms in order to preserve semantics can also lower the jailbreak success rate. \n\nQ10: Specific details about the prompt mutation strategy.\n\nA10: To generate the variants, we provide GPT with the following prompt in each iteration:\n>Given the malicious prompt $P^{*}$, generate $N$ semantically equivalent variants of this prompt. Each variant should preserve the intent and harmful meaning of the original prompt, but avoid using trigger words from the unsafe word list below. You may paraphrase, change the order of phrases, use synonyms or descriptive metaphors, or introduce minor grammatical variations. Do not reduce the harmfulness or change the semantic content. Aim to retain the core meaning while improving the chance of bypassing safety filters.\n\nWe adopt the NSFW word list used in SneakyPrompt as the unsafe words list.\n\nThere are mainly three points to ensure the semantic consistency while introducing diversity:\n1. Explicit instruction constraints: We instruct GPT to preserve the intent, meaning, and harmfulness of the prompt, and clearly specify that the semantic content must not be weakened or altered. This acts as the first line to avoid semantic drift.\n2. Controlled variation methods: In the instructions given to GPT, it is guided to apply localized and shallow transformations, such as paraphrasing, reordering phrases, using synonyms or metaphors, and making minor grammatical changes. These edits maintain the original meaning while introducing diversity in expression.\n3. Multiple variants and averaging: Generating multiple variants per iteration and averaging their joint loss scores serves two purposes: it reduces the risk of semantic drift from any single mutation and increases the likelihood of identifying semantically valid adversarial prompts with stronger attack potential. \n\nQ11: What factors make certain T2V systems more robust, and could these insights inform better defense strategies?\n\nA11: From the main experimental results, we observe that our method achieves a higher attack success rate on Open-Sora compared to commercial models, suggesting that commercial models are more robust against jailbreak attacks in T2V generation. A possible reason is that, unlike open-source models, commercial systems may integrate more comprehensive safety filtering mechanisms and potentially leverage adversarial training to enhance robustness.\n\nExperiments in Section 1 of the Appendix further show that adversarial training is one of the most effective defense strategies. This insight suggests that incorporating adversarial training into deployed defense frameworks could significantly improve model robustness against jailbreak attacks.\n\nQ12: While the authors mention responsible disclosure, providing detailed attack methods could enable misuse.\n\nA12: We have several points to mitigate potential misuse:\n1. We have followed responsible disclosure practices by notifying affected vendors prior to publication.\n2. Sensitive implementation details (e.g., exact prompts, optimization code) are omitted from the paper and will only be shared with verified researchers upon request.\n3. The purpose of sharing the attack methodology is to expose real-world risks and inform the design of more robust defenses."}, {"review_id": "niAYmzoG46", "reviewer": "Reviewer_grrG", "summary": "The paper proposes a joint objective-based optimization framework called T2V-OptJail for jailbreaking text to video models. The framework aims to achieve two objectives: 1. bypassing the built-in safety filters and 2. preserving semantic consistency between the adversarial prompt and the unsafe input prompt. Authors evaluate their approach on various text to video models to showcase effectiveness of their approach.", "questions": "In section 5 it is written \"To mitigate the threat posed by T2V-OptJail, we also propose an adaptive defense. \" but i could not find any discussions on defense in the paper. Can authors clarify this please?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "While the paper studies an interesting and timely topic and specially its extension to text-to-video is interesting, the paper has some weaknesses. For instance, the paper needs some restructuring in its experiments discussions. In the main results, the paper talks about the breakdown on different topics as well as comparison to baselines, and in the next section \"Comparison with Baselines\" it again repeats the comparison to baselines discussion that was brought up in the main results. I think the paper can do better restructuring to better discuss the results. In addition, it would be good if more baselines are discussed in the paper in a sense that more text-to-image methods are extended to be able to be used as a baseline to this approach. It would also be good if the approach was more versatile to be applicable to text to image and text only models. The current work is limited to text to video which can limit its scope. It would be good if the approach was presented as a more versatile approach applicable to a wide range of models with various modalities. In section 5 it is written \"To mitigate the threat posed by T2V-OptJail, we also propose an adaptive defense. \" but i could not find any discussions on defense in the paper. Can authors clarify this please?", "was_revised": true, "final_justification": "I keep my score for the reasons stated previously on structural issues.", "rebuttal": "Q1: The paper needs some restructuring in its experiments discussions.\n\nA1: We will carefully review and revise the structure of the experimental section to ensure that the discussion is clearer and more organized, making it easier for readers to follow the logical connections among the experimental design, results, and analysis. We are committed to improving the overall coherence and readability of this section.\n\nQ2: It would be good if more baselines are discussed in the paper in a sense that more text-to-image methods are extended to be able to be used as a baseline to this approach. It would also be good if the approach was more versatile to be applicable to text to image and text only models. The current work is limited to text to video which can limit its scope. It would be good if the approach was presented as a more versatile approach applicable to a wide range of models with various modalities.\n\nA2: Following your advice, we add three more baselines (PGJ[1], Coljailbreak[2], EG[3]) adapted from T2I filed for a more comprehensive understanding of our method. We follow the experiment setting in the main manuscript. As shown in Table 1, our method outperforms these adaptations with a large margin in terms of both attack success rate and semantic similarity. The results demonstrate that the adaptations from T2I models are far from satisfactory for testing the vulnerability of T2V generation systems, and reveal the significance of our method. The reason is that, unlike T2I methods that primarily rely on implicitly increasing the adversarial nature of prompts, our method explicitly boosts the proportion of jailbreak frames in the generated videos, marking a fundamental difference in attack mechanism.\n\n**Table 1. Comparison with more baselines.**\n|Results on Pika      |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|PGJ [1]|31.4       |32.9         |0.249          |\n|Coljailbreak [2]|29.2       |31.5        |0.248          |\n|EG [3]|35.7       |38.1         |0.250          |\n|T2V-OptJail|**55.9**|**57.6**  |**0.266**  |\n\n\n|Results on Open-Sora      |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|PGJ [1]|39.0       |41.4         |0.251          |\n|Coljailbreak [2]|36.8       |39.7         |0.250          |\n|EG [3]|43.6       |46.5         |0.253          |\n|T2V-OptJail |**67.1**|**68.7**  |**0.269**  |\n\nOur work specifically targets jailbreak attacks on T2V models, a novel and complex research area. We believe the proposed optimization framework and prompt mutation strategy offer a degree of generalizability. In theory, if we remove the video-specific jailbreak frame ratio optimization, our approach can be adapted to T2I or even pure text generation models. Current research is limited to text-to-video, which may restrict its scope of application. It would be better if our method could be used as a more flexible model applicable to various modalities. To address this issue, we transfer our method to T2I jailbreak attacks. We remove the jailbreak frame ratio term and measure the semantic similarity of the generated image instead of the generated video in our loss function. We follow the experiment setting of SneakyPrompt to conduct experiments. The results are shown in Table 2. The results illustrate that our method can achieve similar attack performance to SneakyPrompt and higher performance than DACA. Moreover, this result demonstrates that our method is still effective in T2I jailbreaks. Due to time constraints, we leave it to future work to explore our method on more generative tasks, such as audio generative tasks.\n\n**Table 2. Performance on T2I Jailbreak.**\n|Results on T2I     |Attack success rate(%) |Clip score|\n|--------------|--------|----------\n|DACA |31.1       |0.247          |\n|SneakyPrompt |39.8       |0.260          |\n|T2V-OptJail |**40.7**  |**0.262**  |\n\n\n\n\n[1] Huang, Yihao, et al. \"Perception-guided jailbreak against text-to-image models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 25. 2025.\n\n\n[2] Ma, Yizhuo, et al. \"Coljailbreak: Collaborative generation and editing for jailbreaking text-to-image deep generation.\" Advances in Neural Information Processing Systems 37 (2024): 60335-60358.\n\n[3] Villa, Corban, Shujaat Mirza, and Christina Pöpper. \"Exposing the Guardrails: Reverse-Engineering and Jailbreaking Safety Filters in DALL· E Text-to-Image Pipelines.\" USENIX 2025.\n\nQ3: In section 5 it is written \"To mitigate the threat posed by T2V-OptJail, we also propose an adaptive defense. \" but i could not find any discussions on defense in the paper. Can authors clarify this please?\n\nA3: Due to space limitations in the main paper, we present an adaptive defense in Section 2 of the supplementary material. Following the same experimental setup as in the main manuscript, we evaluate our defense on Open-Sora. The proposed adaptive defense (detailed in the supplementary material) reduces the attack success rate by approximately 16%, significantly decreasing the effectiveness of our attack.\n\nAs shown in Tables 1 and 2 of the supplementary material, our adaptive defense achieves the second-best mitigation performance among all evaluated methods, ranking just below adversarial training. Recognizing that our method is less robust against adversarial training, we integrate adversarial training into the Light Gradient Boosting Machine (LightGBM)-based defense [4], thereby developing an enhanced adaptive defense. This enhanced version, presented in Table 3, further reduces the attack success rate by over 24%, demonstrating strong effectiveness in countering jailbreak attacks on T2V models. \n\nHowever, this enhanced defense remains fragile, as it relies on adversarial training using prompts from a specific domain. When the attacker diverges from the original adversarial prompt distribution, the defense may become less effective. For instance, when we promote greater diversity in the adversarial prompts, shifting their distribution away from the original training data, the adaptive defense is compromised. To simulate this, we introduce a diversity loss term that penalizes semantic similarity among prompts generated from different inputs. Here, we use Jaccard Similarity to measure prompt diversity. We introduce:\n\n$$\nL_{\\text{div}} = \\max_{i} \\, \\text{sim}(P^*, P^{(i)})\n$$\n\nand optimize a joint objective:\n\n$$\nL_{\\text{total}} = L_{\\text{bypass}} + L_{\\text{sem}} + \\alpha \\cdot L_{\\text{div}}.\n$$\n\nThis unified optimization function discourages repetition across different adversarial prompts, and thus improves the diversity of adversarial prompts, which can be seen as an **adaptive attack**. We set $\\alpha=0.5$. Table 4 shows that our adaptive attack achieves an attack success rate of about 62%, demonstrating its effectiveness against the enhanced adaptive defense. This reveals that although the defense can be effective, it is still far from satisfactory in defending against our attack, even with a tiny adaptation.\n\n\n\n**Table 3. Performance of our proposed enhanced adaptive defense.**\n|No defense     |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|T2V-OptJail |67.1      |68.7         |0.269          |\n\n|Enhanced adaptive defense     |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|T2V-OptJail |41.6      |43.8        |0.252          |\n\n**Table 4. Performance of our proposed adaptive attack.**\n|Adaptive attack against enhanced adaptive defense     |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|T2V-OptJail-adaptive |62.4      |63.7         |0.264          |\n\n[4] Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132, 2023."}, {"review_id": "KAcODNE7rz", "reviewer": "Reviewer_iDNu", "summary": "This paper proposes T2V-OptJail, a joint objective-based optimization framework to jailbreak text-to-video models. In the objective function construction stage, the authors considered both filter bypassing loss function and semantics consistency loss function to bypass the safety filters and jailbreak quality. Then GPT-4o is use to provide candidate prompts along with multiple variants. The prompt with the lowest loss function value is used for the next iteration. Experimental results shows the effectiveness of T2V-OptJail with a set of ablation studies.", "questions": "- What is the unique novelty of T2V-OptJail compared to genetic optimization methods?\n- Please explore stronger safety filters (such as Llama-guard or using LLM as a classifier) as the defense methods.", "limitations": "See my comments above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- Jailbreaking text-to-video model is an interesting research topic. \n- The jailbreak methods proposed in this work intuitively makes sense.  \n\n**Weakness**\n- The overall optimization methodology looks very much like the genetic optimization method, the loss function defined in section 3.2 and 3.3 is the fitness function and mutation strategy in section 3.4 is also a common idea in genetic algorithm. This undercuts the novelty of this paper.\n- Some critical technical details are missing in Section 3.2. For example, how to evaluate eq. (2) is not discussed in this paper. What are the typical safety filters considered in the current T2V models? How strong is the zero-shot ability of CLIP model to classify the text prompts? Based on the demonstration examples shown in the paper, the text prompt only changed a few words with similar semantic meaning, these prompts are still harmful and can be detected easily such as using Llama-guard.", "was_revised": true, "final_justification": "Based on the discussions with authors. I think the authors have done a lot to answer my questions. However, I feel this paper will be further enhanced by including more technical details in the main paper. I will increase my score to 4 assuming that the authors will add more technical details in the main paper.", "rebuttal": "Q1：Novelty issue of this paper\n\nA1: We would like to express our gratitude to the reviewers for their careful evaluation of our work. We would like to elaborate on the originality of this work, which is manifested in the following aspects:\n- First, we are the first to model the T2V jailbreak task as a discrete optimization problem. Unlike existing text-to-image (T2I) jailbreak methods, which primarily rely on keyword masking or prompt rewriting, we propose a GPT-based discrete prompt optimization framework that actively searches for illegal instruction prompts in the LLM token space that are both attack-oriented and capable of bypassing security filters. This framework introduces a joint optimization objective that ensures both evasion capability and enhances the actual harmfulness of the generated attacks, marking the first systematic optimization solution proposed for T2V model jailbreaking tasks.\n- Second, we introduce dynamic constraints in the video generation process into the objective function, which is completely absent in T2I jailbreaking research. T2V attacks are fundamentally a cross-temporal semantic control problem, so we designed two T2V-specific loss terms: the first is “Unsafe Frame Occupancy,” which penalizes frames in the generated video that do not contain attack content, thereby enhancing attack strength and completeness; the second is “Prompt-to-Video Semantic Consistency,” ensuring that the video content accurately reflects the intent of the attack prompt. This design marks the first time that temporal and semantic alignment has been incorporated into jailbreak evaluation criteria, representing a significant extension of traditional static generation attack paradigms.\n- Third, we propose a semantic-level prompt mutation strategy tailored for T2V, rather than relying solely on token-level mutation operations from existing adversarial methods. We use a GPT generator to fuse joint loss functions in each mutation round to search for the optimal prompt. This mechanism not only considers evasion capabilities but also maintains consistency with the original attack intent, significantly improving the effectiveness of prompt generation and the controllability of attacks. Unlike traditional fixed mutation strategies based on perturbations, our method introduces semantic guidance and adaptive search, making it more suitable for handling the complex, continuous output structures in video generation.\n\nIn summary, our work is not a simple migration of existing methods, but rather an innovative exploration of T2V jailbreak optimization in task modeling, loss design, optimization strategies, and validation experiments, demonstrating clear originality.\n\n\nQ2: The overall optimization methodology looks very much like the genetic optimization method, the loss function defined in section 3.2 and 3.3 is the fitness function and mutation strategy in section 3.4 is also a common idea in genetic algorithm. \n\nA2: While our optimization framework resembles evolutionary algorithms in spirit, our mutation strategy is fundamentally different: instead of naive random mutation, we leverage an agent to produce semantically consistent and targeted perturbations. To further clarify this point, we added a comparison with the classic genetic algorithm baseline, in which prompts are mutated via simple token substitution without LLM guidance. We follow the same experiment setting of the main manuscript and conducted experiment on Open-Sora. As shown in Table 1, our method outperforms the vanilla GA over 12% in attack success rate, while achieving higher semantic similarity. This result highlights the superiority of using LLM-driven mutation strategies. Delving into the intrinsic difference, we argue that the possible reason is that LLM based agent is more effective at identifying suitable mutations during optimization. \n\n**Table 1. Comparison of our adapted mutation strategy with vanilla GA.**\n|Results against Open-Sora |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|T2V-OptJail using vanilla GA |54.3      |55.5         |0.258          |\n|T2V-OptJail |67.1|68.7  |0.269  |\n\n\nQ3: Some critical technical details are missing in Section 3.2. For example, how to evaluate eq. (2) is not discussed in this paper. What are the typical safety filters considered in the current T2V models? How strong is the zero-shot ability of CLIP model to classify the text prompts? Based on the demonstration examples shown in the paper, the text prompt only changed a few words with similar semantic meaning, these prompts are still harmful and can be detected easily such as using Llama-guard.\n\nA3: We make the following clarification for each point.\n- **Equation evaluation.** In our formulation, F(P*) is a binary indicator function that assesses whether the adversarial prompt P* is successfully blocked by the model’s built-in safety filter. This function is directly evaluated by submitting P* to the target text-to-video model and observing whether a video output is returned. If the model outputs an empty response or a refusal message due to safety concerns, we assign F(P*)=1; otherwise, we set F(P*)=0.\n- **Typical safety filters**: Simulating real-world T2V generation systems, two-stage safety filters are usually considered including: 1. **Input filters:** screening user prompts before generation using keyword matching [1], classifier-based methods, and large multimodal encoders (e.g., CLIP or LLaVA); as well as 2. **Output filter:** analyzing generated video frames and their captions using image classifiers [2], NSFW detectors [3], and vision-language models.\n- **CLIP zero-shot ability.** In our work, following previous work[9, 31], we adopt the zero-shot ability of CLIP as the classifier to approximate the type of input safety filtering deployed in real-world systems. As shown in [31], CLIP has strong zero-shot capabilities for detecting unsafe semantics in textual prompts. We compute cosine similarity between prompts and a curated set of unsafe category embeddings, and reject prompts above a certain threshold following the setting in [4]. \n- **Prompt Variation.** Figure 2 in main manucript serves as a demonstration of **successful examples** of our attack in jailbreaking **real-world commercial T2V models, Kling**. The reason that the variation on prompt is minimal is result of joint optimization of Equation 7, where the **Semantics Consistency Optimization**, including **Prompt Semantics Consistency** and **Prompt-Video Semantics Consistency**, constrains the shift of adversarial prompt for targeted jailbreak attacks. Thus, the change of prompt may only take place on a few words. \n- **Llama-guard and WildGuard as filters.** Indeed, one adversarial prompt that can successfully jailbreak certain safety filters is still possibly detectble by another safety filter. We further validate the effectiveness of our attack when more defense strategies (Llama-guard and WildGuard) are adopted. As shown in Table 2, when Llama-guard or WildGuard is trigered, our attack still achieves the highest ASR and similarity among all the methods. This also reveals the robustness of our attack against various defense mechanisms and rises the need for more effective defenses. \n\n**Table 2. Results against Llama-Guard and WildGuard.**\n|Results against Llama-guard    |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|--------|--------|\n|T2VSafetyBench|40.6 |43.0 |0.251 |\n|DACA|10.4      |11.9|0.240 |\n|T2V-OptJail |50.3|52.2  |0.256  |\n\n|Results against WildGuard   |GPT-4(%) |Human(%)|Similarity|\n|--------------|--------|----------|-----------|\n|T2VSafetyBench|38.2     |40.7         |0.250          |\n|DACA|9.8      |10.7|0.239         |\n|T2V-OptJail |46.4|48.5  |0.254  |\n\n\n[1] Siyuan Liang, Jiayang Liu, Jiecheng Zhai, Tianmeng Fang, Rongcheng Tu, Aishan Liu, Xiaochun Cao, and Dacheng Tao. T2vshield: Model-agnostic jailbreak defense for text-to-video models. arXiv preprint arXiv:2504.15512, 2025.\n\n[2] Schramowski, Patrick, Christopher Tauchmann, and Kristian Kersting. \"Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content?.\" Proceedings of the 2022 ACM conference on fairness, accountability, and transparency. 2022.\n\n[3] Falconsai. Nsfw image classification. https://huggingface.co/Falconsai/nsfw_image_detection, 2024. Accessed on: 2024-11-18.\n\n[4] Schramowski, Patrick, Christopher Tauchmann, and Kristian Kersting. \"Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content?.\" Proceedings of the 2022 ACM conference on fairness, accountability, and transparency. 2022."}]

Author rebuttal:

Meta-review: This paper studies the vulnerability of text-to-video (T2V) generative models to jailbreak prompts. The authors systematically design attack strategies that can bypass existing safety filters and alignment measures, and evaluate these across several popular T2V systems. The work also proposes a taxonomy of jailbreak types and discusses potential mitigation strategies.

All four reviewers rated the paper positively (one 5, three 4s) with reasonable confidence. The consensus is that while the methodological novelty is limited, the contribution is important, timely, and the empirical study appears to be comprehensive. I recommend acceptance.
Decision: Accept (poster)