Package: prompt-injection-bench
Prompt Injection BenchmarkingThe mother of all prompt injection benchmarking repositories, this is the one you have been waiting for (or soon will be)Analysing OpenAI and Azure OpenAI GPT-4, Gemini Pro and Azure OpenAI Jailbreak Detection (CyberPunk mode)This repository contains Python code to analyze the Hugging Face Jailbreak dataset against OpenAI's ChatGPT-4 an Gemini Pro models.
The code sends prompts from the dataset, processes and tabulates results.What is a Prompt Jailbreak attack?The term 'Jailbreak' is used to describe a situation where a language model is manipulated to generate text that is harmful, inappropriate, or otherwise violates community guidelines. This can include generating hate speech, misinformation, or other harmful content.Let me tell you a story: Once I tried to Jailbreak my  iPhone to side load apps and it worked until a reboot, then I had a paper weigh in the format of an iPhoneIn the world of LLMs the termjailbreakis used veryloosely. Someattacksare just ridiculous and silly, others are serious.Anxious to check the results?The table below shows the total injection attack prompts according to the data set and the number of detected attacks by ChatGPT-4 and Gemini Pro.Latest RunPromptsGPT-4GeminiAzure OpenAIAzure OpenAI w/ Jailbreak Detection139Detected13353136Not Attack643Missed Attack0820Previous runsPromptsGPT-4GeminiAzure OpenAIAzure OpenAI w/ Jailbreak Detection139Detected13149134Not AttackTBDTBDMissed AttackTBDTBDmany more but...no spaceImportant DetailsPrompts can be blocked by built-in or explicit safety filters. Therefore the code becomes nuanced and to gather proper results you need to catch certain exceptions (see code)Next steps (soon)Add more models to the analysis.Enable Safety filters excplicitly and see variation in results assuming a safety block is consider anattackRequirementsTo run this code, you need the following:Python 3OpenAI Python libraryHugging Face datasets libraryA valid OpenAI API keyA valid Google API keyA valid Azure Open AI endpoint and all the config that comes with itIf you do not have all these keys the code will skip that LLM, no need to worry.SetupClone this repository:gitclonehttps://github.com/BenderScript/PromptInjectionBench.gitcdPromptInjectionBenchCreate a.envCreate a .env file in the project root directory that contains your OpenAI API, Azure and Google Keys.
If you do not have all these keys the code will skip that LLM, no need to worry.OPENAI_API_KEY=your key>
#
GOOGLE_API_KEY=<your key>
# Azure
AZURE_OPENAI_API_KEY=your key>
AZURE_MODEL_NAME=gpt-4
AZURE_OPENAI_ENDPOINT=your endpoint>
AZURE_OPENAI_API_VERSION=<your api verion, normally 2023-12-01-preview>
AZURE_OPENAI_DEPLOYMENT=<your deployment name. This is the name you gave when deploying the model>Install the required libraries using pip3 and the provided requirements.txt file:pip3install-rrequirements.txtRunning the AnalysisTo analyze the Jailbreak dataset with ChatGPT-4 and Gemini Pro, execute the following Python script:uvicornprompt_injection_bench.server:prompt_bench_app--reload--port9002If Everything goes well, you should see the following page athttp://127.0.0.1:9001This script loads the dataset, iterates through prompts, sends them to ChatGPT-4, and detects potential injection attacks in the generated responses.TestingSee the demo below where the App checks a prompt with a malicious URL and injection.Code Structureserver.py: The main script that loads the dataset, drives the test, and performs the analysis.gemini*: The gemini pro codeopenai*: The openai codeazure*: The azure codeSkipping "Benign" PromptsIn the interest of time, the code skips prompts labeled as "benign." This helps focus the analysis on potentially harmful prompts where injection attacks might occur.Tabulating Attacks vs. Detected AttacksThe script tabulates the total number of generated texts and the number of texts where potential injection attacks are detected. It counts these attacks based on the presence of specific keywords such as "I'm sorry," "apologize," or "developed by OpenAI" in the generated text.ResultsThe script will print information about each prompt, including its type, the generated text from ChatGPT-4, and whether an injection attack was detected based on the presence of specific keywords.LicenseThis code is provided under theApache License 2.0. If you liked this code, cite it and let me know.For more information about OpenAI's GPT-4 model and the Hugging Face Jailbreak dataset, please refer to the official documentation and sources:OpenAI GPT-4Hugging Face Jailbreak DatasetAzure Jailbreak Risk DetectionGemini API