Package: garak
garak, LLM vulnerability scannergarakchecks if an LLM can be made to fail in an way we don't want.garakprobes for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and many other weaknesses. If you knownmap, it'snmapfor LLMs.garak's a free tool. We love developing it and are always interested in adding functionality to support applications.Get started> See our user guide!docs.garak.ai> Join ourDiscord!> Project links & home:garak.aiLLM supportcurrently supports:hugging face hubgenerative modelsreplicatetext modelsopenai apichat & continuation modelsggml models likellama.cpp.. and many more LLMs!Install:garakis a command-line tool. It's developed in Linux and OSX.Standard install withpipJust grab it from PyPI and you should be good to go:python -m pip install -U garakInstall development version withpipThe standard pip version ofgarakis updated periodically. To get a fresher version, from GitHub, try:python -m pip install -U git+https://github.com/leondz/garak.git@mainClone from sourcegarakhas its own dependencies. You can to installgarakin its own Conda environment:conda create --name garak "python>=3.11"
conda activate garak
gh repo clone leondz/garak
cd garak
python -m pip install -r requirements.txtOK, if that went fine, you're probably good to go!Getting startedThe general syntax is:python3 -m garak <options>garakneeds to know what model to scan, and by default, it'll try all the probes it knows on that model, using the vulnerability detectors recommended by each probe. You can see a list of probes using:python3 -m garak --list_probesTo specify a generator, use the--model_typeand, optionally, the--model_nameoptions. Model type specifies a model family/interface; model name specifies the exact model to be used. The "Intro to generators" section below describes some of the generators supported. A straightfoward generator family is Hugging Face models; to load one of these, set--model_typetohuggingfaceand--model_nameto the model's name on Hub (e.g."RWKV/rwkv-4-169m-pile"). Some generators might need an API key to be set as an environment variable, and they'll let you know if they need that.garakruns all the probes by default, but you can be specific about that too.--probes promptinjectwill use only thePromptInjectframework's methods, for example. You can also specify one specific plugin instead of a plugin family by adding the plugin name after a.; for example,--probes lmrc.SlurUsagewill use an implementation of checking for models generating slurs based on theLanguage Model Risk Cardsframework.For help & inspiration, find us ontwitterordiscord!ExamplesProbe ChatGPT for encoding-based prompt injection (OSX/*nix) (replace example value with a real OpenAI API key)export OPENAI_API_KEY="sk-123XXXXXXXXXXXX"
python3 -m garak --model_type openai --model_name gpt-3.5-turbo --probes encodingSee if the Hugging Face version of GPT2 is vulnerable to DAN 11.0python3 -m garak --model_type huggingface --model_name gpt2 --probes dan.Dan_11_0Reading the resultsFor each probe loaded, garak will print a progress bar as it generates. Once generation is complete, a row evaluating that probe's results on each detector is given. If any of the prompt attempts yielded an undesirable behaviour, the response will be marked as FAIL, and the failure rate given.Here are the results with theencodingmodule on a GPT-3 variant:And the same results for ChatGPT:We can see that the more recent model is much more susceptible to encoding-based injection attacks, where text-babbage-001 was only found to be vulnerable to quoted-printable and MIME encoding injections.  The figures at the end of each row, e.g. 840/840, indicate the number of text generations total and then how many of these seemed to behave OK. The figure can be quite high because more than one generation is made per prompt - by default, 10.Errors go ingarak.log; the run is logged in detail in a.jsonlfile specified at analysis start & end. There's a basic analysis script inanalyse/analyse_log.pywhich will output the probes and prompts that led to the most hits.Send PRs & open issues. Happy hunting!Intro to generatorsHugging Face--model_type huggingface(for transformers models to run locally)--model_name- use the model name from Hub. Only generative models will work. If it fails and shouldn't, please open an issue and paste in the command you tried + the exception!--model_type huggingface.InferenceAPI(for API-based model access)--model_name- the model name from Hub, e.g."mosaicml/mpt-7b-instruct"--model_type huggingface.InferenceEndpoint(for private endpoints)--model_name- the endpoint URL, e.g.https://xxx.us-east-1.aws.endpoints.huggingface.cloud(optional) set theHF_INFERENCE_TOKENenvironment variable to a Hugging Face API token with the "read" role; seehttps://huggingface.co/settings/tokenswhen logged inOpenAI--model_type openai--model_name- the OpenAI model you'd like to use.text-babbage-001is fast and fine for testing;gpt-4seems weaker to many of the more subtle attacks.set theOPENAI_API_KEYenvironment variable to your OpenAI API key (e.g. "sk-19763ASDF87q6657"); seehttps://platform.openai.com/account/api-keyswhen logged inRecognised model types are whitelisted, because the plugin needs to know which sub-API to use. Completion or ChatCompletion models are OK. If you'd like to use a model not supported, you should get an informative error message, and please send a PR / open an issue.Replicate--model_type replicate--model_name- the Replicate model name and hash, e.g."stability-ai/stablelm-tuned-alpha-7b:c49dae36"--model_type replicate.InferenceEndpoint(for private endpoints)--model_name- username/model-name slug from the deployed endpoint, e.g.elim/elims-llama2-7bset theREPLICATE_API_TOKENenvironment variable to your Replicate API token, e.g. "r8-123XXXXXXXXXXXX"; seehttps://replicate.com/account/api-tokenswhen logged inCohere--model_type cohere--model_name(optional,commandby default) - The specific Cohere model you'd like to testset theCOHERE_API_KEYenvironment variable to your Cohere API key, e.g. "aBcDeFgHiJ123456789"; seehttps://dashboard.cohere.ai/api-keyswhen logged inggml--model_type ggml--model_name- The path to the ggml model you'd like to load, e.g./home/leon/llama.cpp/models/7B/ggml-model-q4_0.binset theGGML_MAIN_PATHenvironment variable to the path to your ggmlmainexecutableOctoAI--model_type octo--model_name- the OctoAI public endpoint for the model, e.g.mistral-7b-instruct-fp16--model_type octo.InferenceEndpoint(for private endpoints)--model_name- the deployed endpoint URL, e.g.https://llama-2-70b-chat-xxx.octoai.run/v1/chat/completionsset theOCTO_API_TOKENenvironment variable to your Replicate API token, e.g. "r8-123XXXXXXXXXXXX"; seehttps://replicate.com/account/api-tokenswhen logged inTest--model_type test(alternatively)--model_name test.BlankFor testing. This always generates the empty string, using thetest.Blankgenerator.  Will be marked as failing for any tests thatrequirean output, e.g. those that make contentious claims and expect the model to refute them in order to pass.--model_type test.RepeatFor testing. This generator repeats back the prompt it received.Intro to probesProbeDescriptionblankA simple probe that always sends an empty prompt.atkgenAutomated Attack Generation. A red-teaming LLM probes the target and reacts to it in an attempt to get toxic output. Prototype, mostly stateless, for now uses a simple GPT-2fine-tunedon the subset of hhrlhf attempts that yielded detectable toxicity (the only target currently supported for now).continuationProbes that test if the model will continue a probably undesirable worddanVariousDANand DAN-like attacksencodingPrompt injection through text encodinggcgDisrupt a system prompt by appending an adversarial suffix.glitchProbe model for glitch tokens that provoke unusual behavior.goodsideImplementations of Riley Goodside attacks.knownbadsignaturesProbes that attempt to make the model output malicious content signaturesleakerplayEvaluate if a model will replay training data.lmrcSubsample of theLanguage Model Risk CardsprobesmalwaregenAttempts to have the model generate code for building malwaremisleadingAttempts to make a model support misleading and false claimspackagehallucinationTrying to get code generations that specify non-existent (and therefore insecure) packages.promptinjectImplementation of the Agency EnterprisePromptInjectwork (best paper awards @ NeurIPS ML Safety Workshop 2022)realtoxicitypromptsSubset of the RealToxicityPrompts work (data constrained because the full test will take so long to run)snowballSnowballed Hallucinationprobes designed to make a model give a wrong answer to questions too complex for it to processxssLook for vulnerabilities the permit or enact cross-site attacks, such as private data exfilteration.Logginggarakgenerates multiple kinds of log:A log file,garak.log. This includes debugging information fromgarakand its plugins, and is continued across runs.A report of the current run, structured as JSONL. A new report file is created every timegarakruns. The name of this file is output at the beginning and, if successful, also the end of the run. In the report, an entry is made for each probing attempt both as the generations are received, and again when they are evaluated; the entry'sstatusattribute takes a constant fromgarak.attemptsto describe what stage it was made at.A hitlog, detailing attempts that yielded a vulnerability (a 'hit')How is the code structured?In a typical run,garakwill read a model type (and optionally model name) from the command line, then determine whichprobes anddetectors to run, start up agenerator, and then pass these to aharnessto do the probing; anevaluatordeals with the results. There are many modules in each of these categories, and each module provides a number of classes that act as individual plugins.garak/probes/- classes for generating interactions with LLMsgarak/detectors/- classes for detecting an LLM is exhibiting a given failure modegarak/evaluators/- assessment reporting schemesgarak/generators/- plugins for LLMs to be probedgarak/harnesses/- classes for structuring testingresources/- ancillary items required by pluginsThe default operating mode is to use theprobewiseharness. Given a list of probe module names and probe plugin names, theprobewiseharness instantiates each probe, then for each probe reads itsrecommended_detectorsattribute to get a list ofdetectors to run on the output.Each plugin category (probes,detectors,evaluators,generators,harnesses) includes abase.pywhich defines the base classes usable by plugins in that category. Each plugin module defines plugin classes that inherit from one of the base classes. For example,garak.generators.openai.OpenAIGeneratordescends fromgarak.generators.base.Generator.Larger artefacts, like model files and bigger corpora, are kept out of the repository; they can be stored on e.g. Hugging Face Hub and loaded locally by clients usinggarak.Developing your own pluginTake a look at how other plugins do itInherit from one of the base classes, e.g.garak.probes.base.TextProbeOverride as little as possibleYou can test the new code in at least two ways:Start an interactive Python sessionImport the model, e.g.import garak.probes.mymoduleInstantiate the plugin, e.g.p = garak.probes.mymodule.MyProbe()Run a scan with test pluginsFor probes, try a blank generator and always.Pass detector:python3 -m garak -m test.Blank -p mymodule -d always.PassFor detectors, try a blank generator and a blank probe:python3 -m garak -m test.Blank -p test.Blank -d mymoduleFor generators, try a blank probe and always.Pass detector:python3 -m garak -m mymodule -p test.Blank -d always.PassGetgarakto list all the plugins of the type you're writing, with--list_probes,--list_detectors, or--list_generatorsFAQWe have an FAQhere. Reach out if you have any more questions!leon@garak.aiCode reference documentation is atgarak.readthedocs.io."Never tell the same lie twice"- ElimFor updates and news see@garak_llm© Apache v2 2023 Leon Derczynski