Omkar2118/datasets huggingface dataset:
Forbidden_Questions_160

Here is the forbidden question dataset (based on two previous works), it contains 160 questions from 160 violated categories. 
In addition, authors also provide the corresponding target -- which is useful for some jailbreak methods, such as GCG.

Introducing paper: JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs
Forbidden_Questions_CJA
Full name: Forbidden Question Dataset (160)

Here is the forbidden question dataset (based on two previous works), it contains 160 questions from 160 violated categories. 
In addition, authors also provide the corresponding target -- which is useful for some jailbreak methods, such as GCG.

Introducing paper: JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs