Repository: RedHatProductSecurity/rapidast
Pull request #315: support advanced Garak configs
Author: login: jeremychoi
Created: 2025-03-03T04:02:54Z | Merged: 2025-03-05T09:09:17Z

(updated) Support Garak config format as is under the 'garak_config' key, according to the discussion

The changed config example looks like the following now.
```
scanners:
  garak:
    # garak_executable_path: 
    garak_config:
      plugins:
        model_type: rest
        generators:
          ...
      run:
        generations: 1

```


Diff:
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 19fd0c2f..e9ffaa24 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -43,6 +43,7 @@ repos:
   # R0801: Similar lines in 2 files. Disabled because it flags any file even those which are unrelated
   # R1705: Unnecessary "elif" after "return", remove the leading "el" from "elif" (no-else-return)
   # R1710: Either all return statements in a function should return an expression, or none of them should. (inconsistent-return-statements)
+  # W0201: Attribute 'state' defined outside __init__ (attribute-defined-outside-init)
 
   - repo: https://github.com/PyCQA/pylint
     #rev: v3.0.3
@@ -54,5 +55,5 @@ repos:
           - --max-line-length=120
           - --min-public-methods=0
           - --good-names=o,w,q,f,fp,i,e
-          - --disable=E0401,W1201,W1203,C0114,C0115,C0116,C0411,W0107,W0511,W0702,R0801,R1705,R1710
+          - --disable=E0401,W1201,W1203,C0114,C0115,C0116,C0411,W0107,W0511,W0702,R0801,R1705,R1710,W0201
         language_version: python3
diff --git a/README.md b/README.md
index a739206a..5d5dc919 100644
--- a/README.md
+++ b/README.md
@@ -480,8 +480,10 @@ The following is an example to launch a scan:
 ```yaml
 scanners:
   garak:
-    model_type: huggingface
-    model_name: gpt2
+    parameters:
+      plugins:
+        model_type: huggingface
+        model_name: gpt2
 ```
 
 #### Generic scanner
diff --git a/config/config-template-garak.yaml b/config/config-template-garak.yaml
index 1b855f45..d6b80884 100644
--- a/config/config-template-garak.yaml
+++ b/config/config-template-garak.yaml
@@ -14,16 +14,21 @@ application:
 # `scanners' is a section that configures scanning options
 scanners:
   garak:
-    model_type: huggingface                         # required, e.g. hugginngface, openai, rest
-    model_name: gpt2                                # optional, but a specific model type requires a model name or path
-    #generators:                                    # optional, providing more options for the selected model type, e.g. RestGenerator
-    #  rest:
-    #    RestGenerator:
-    #      uri:
-    #      method:
-    #      headers:
-    #      response_json_field:
-    #      req_template_json_object:
-    #      request_timeout: 60
-    #probe_spec: all                               # default: all, or a list of probes like "probe1,probe2"
-    #garak_executable_path: /usr/local/bin/garak    # default: /usr/local/bin/garak
+    parameters:
+      plugins:
+        model_type: huggingface                         # required, e.g. hugginngface, openai, rest
+        model_name: gpt2                                # optional, but a specific model type requires a model name or path
+        #probe_spec: all                                # default: all, or a list of probes like "probe1,probe2"
+        #generators:                                    # optional, providing more options for the selected model type, e.g. RestGenerator
+        #  rest:
+        #    RestGenerator:
+        #      uri:
+        #      method:
+        #      headers:
+        #      response_json_field:
+        #      req_template_json_object:
+        #      request_timeout: 60
+      #run:
+      #  generations: 5                                 # optional, the number of times each prompt is sent for inference, affecting performance
+
+    #executable_path: /usr/local/bin/garak    # default: /usr/local/bin/garak
diff --git a/config/schemas/6/rapidast_schema.json b/config/schemas/6/rapidast_schema.json
index 2aace157..14f766ca 100644
--- a/config/schemas/6/rapidast_schema.json
+++ b/config/schemas/6/rapidast_schema.json
@@ -639,25 +639,16 @@
                         "type": "object",
                         "description": "Garak Scanner",
                         "properties": {
-                            "model_name": {
-                                "type": "string"
-                            },
-                            "model_type": {
-                                "type": "string"
-                            },
-                            "generators": {
+                            "parameters": {
                                 "type": "object",
                                 "properties": {}
                             },
-                            "probe_spec": {
-                                "type": "string"
-                            },
-                            "garak_executable_path": {
+                            "executable_path": {
                                 "type": "string"
                             }
                         },
                         "required": [
-                            "model_type"
+                            "parameters"
                         ]
                     }
                 ]
diff --git a/scanners/garak/garak-config-template.yaml b/scanners/garak/garak-config-template.yaml
deleted file mode 100644
index b3b62d22..00000000
--- a/scanners/garak/garak-config-template.yaml
+++ /dev/null
@@ -1,39 +0,0 @@
----
-system:
-    verbose: 0
-    narrow_output: false
-    parallel_requests: false
-    parallel_attempts: false
-    lite: true
-    show_z: false
-
-run:
-    seed:
-    deprefix: true
-    eval_threshold: 0.5
-    generations: 5
-    probe_tags:
-
-plugins:
-    model_type:
-    model_name:
-    probe_spec: all
-    detector_spec: auto
-    extended_detectors: false
-    buff_spec:
-    buffs_include_original_prompt: false
-    buff_max:
-    detectors: {}
-    generators: {}
-    buffs: {}
-    harnesses: {}
-    probes:
-        encoding:
-            payloads:
-                - default
-
-reporting:
-    report_prefix:
-    taxonomy:
-    report_dir: garak_runs
-    show_100_pass_modules: true
diff --git a/scanners/garak/garak_none.py b/scanners/garak/garak_none.py
index 2ce524b0..b01c8530 100644
--- a/scanners/garak/garak_none.py
+++ b/scanners/garak/garak_none.py
@@ -7,6 +7,7 @@
 from dataclasses import field
 from typing import Any
 from typing import Dict
+from typing import Optional
 
 import dacite
 import yaml
@@ -18,12 +19,12 @@
 
 
 @dataclass
+# pylint: disable=too-many-instance-attributes
 class GarakConfig:
-    model_type: str
-    model_name: str = field(default="test_model")
-    probe_spec: str = field(default="all")  # all or a list of probes like "probe1,probe2"
-    garak_executable_path: str = field(default="/usr/local/bin/garak")
-    generators: Dict[str, Any] = field(default_factory=dict)
+    parameters: Optional[Dict[str, Any]] = field(default_factory=dict)
+
+    # The path to the Garak executable
+    executable_path: str = field(default="/usr/local/bin/garak")
 
 
 CLASSNAME = "Garak"
@@ -34,16 +35,13 @@ class GarakConfig:
 class Garak(RapidastScanner):
     """Scanner implementation for Garak LLM security testing tool."""
 
-    GARAK_CONFIG_TEMPLATE = "garak-config-template.yaml"
     GARAK_RUN_CONFIG_FILE = "garak-run-config.yaml"
     TMP_REPORTS_DIRNAME = "garak_runs"
     MIN_GARAK_VERSION = "0.10.2"
 
     def _check_garak_version(self):
         try:
-            result = subprocess.run(
-                [self.cfg.garak_executable_path, "--version"], capture_output=True, text=True, check=True
-            )
+            result = subprocess.run([self.cfg.executable_path, "--version"], capture_output=True, text=True, check=True)
             version_match = re.search(r"v(\d+\.\d+\.\d+)", result.stdout.strip())
             if not version_match:
                 raise ValueError(f"Could not find version number in output: {result.stdout}")
@@ -56,7 +54,7 @@ def _check_garak_version(self):
                     f"Garak version {current_version} is not supported. Version {min_version} or higher is required."
                 )
         except FileNotFoundError as exc:
-            raise RuntimeError(f"Garak is not found at {self.cfg.garak_executable_path}") from exc
+            raise RuntimeError(f"Garak is not found at {self.cfg.executable_path}") from exc
         except (subprocess.SubprocessError, IndexError, ValueError) as e:
             raise RuntimeError(f"Failed to check Garak version: {str(e)}") from e
 
@@ -81,36 +79,37 @@ def setup(self):
         if self.state != State.UNCONFIGURED:
             raise RuntimeError(f"Garak scanning setup encountered an unexpected state: {self.state}")
 
+        def _search_model_type(config):
+            if isinstance(config, dict):
+                for key, value in config.items():
+                    if key == "model_type":
+                        return True
+                    if isinstance(value, dict):
+                        if _search_model_type(value):  # Recursively search in sub-dictionaries
+                            return True
+            return False
+
         # Check Garak version
         self._check_garak_version()
 
-        try:
-            template_path = os.path.join(MODULE_DIR, self.GARAK_CONFIG_TEMPLATE)
-            with open(template_path, "r", encoding="utf-8") as stream:
-                self.automation_config = yaml.safe_load(stream)
-
-            # Update values from the config template with user configured values
-            self.automation_config.update(
-                {
-                    "plugins": {
-                        "model_name": self.cfg.model_name,
-                        "model_type": self.cfg.model_type,
-                        "probe_spec": self.cfg.probe_spec,
-                        "generators": self.cfg.generators,
-                    },
-                    "reporting": {"report_dir": self.workdir_reports_dir},
-                }
-            )
+        self.automation_config = self.cfg.parameters
+
+        # Update reporting with RapiDAST workdir directory
+        self.automation_config["reporting"] = {"report_dir": self.workdir_reports_dir}
 
-            # Write updated config
+        # XXX check at least if model_type is defined to prevent a Garak error in advance
+        if not _search_model_type(self.automation_config):
+            raise ValueError("model_type is not defined in the Garak configuration")
+
+        try:
             garak_run_conf_path = os.path.join(self.workdir, self.GARAK_RUN_CONFIG_FILE)
             with open(garak_run_conf_path, "w", encoding="utf-8") as f:
                 yaml.dump(self.automation_config, f)
 
         except yaml.YAMLError as exc:
-            raise RuntimeError(f"Failed to parse config '{template_path}': {exc}") from exc
+            raise RuntimeError(f"Failed to write a Garak config: {exc}") from exc
 
-        self.garak_cli = [self.cfg.garak_executable_path, "--config", garak_run_conf_path]
+        self.garak_cli = [self.cfg.executable_path, "--config", garak_run_conf_path]
 
         if self.state != State.ERROR:
             self.state = State.READY
@@ -133,7 +132,7 @@ def run(self):
 
         except FileNotFoundError:
             logging.error(
-                f"Garak is not found at {self.cfg.garak_executable_path}. Please ensure Garak is installed in your PATH"
+                f"Garak is not found at {self.cfg.executable_path}. Please ensure Garak is installed in your PATH"
             )
             self.state = State.ERROR
         except subprocess.SubprocessError as e:
@@ -146,13 +145,17 @@ def postprocess(self):
 
         super().postprocess()
 
+        # pylint: disable=broad-exception-caught
         try:
             shutil.copytree(self.workdir_reports_dir, self.results_dir, dirs_exist_ok=True)
-        # pylint: disable=broad-exception-caught
+        except FileNotFoundError as exc:
+            logging.error(
+                f"There is no result, possibly because the configuration is not fully set up to run a scan: {exc}"
+            )
+            self.state = State.ERROR
         except Exception as excp:
             logging.error(f"Unable to save results: {excp}")
-            # pylint: disable=attribute-defined-outside-init
-            # it's a false positive: it's defined in the RapidastScanner class
+
             self.state = State.ERROR
 
         if not self.state == State.ERROR:
diff --git a/tests/scanners/garak/test_garak.py b/tests/scanners/garak/test_garak.py
index 2540aa8d..e393131c 100644
--- a/tests/scanners/garak/test_garak.py
+++ b/tests/scanners/garak/test_garak.py
@@ -32,10 +32,14 @@ def test_setup_garak(mock_check_garak_version, test_config):
     }
 
     test_garak_config_in_rapidast = {
-        "model_name": test_model_name,
-        "model_type": test_model_type,
-        "probe_spec": test_probe_spec,
-        "generators": test_generators,
+        "parameters": {
+            "plugins": {
+                "model_name": test_model_name,
+                "model_type": test_model_type,
+                "probe_spec": test_probe_spec,
+                "generators": test_generators,
+            }
+        }
     }
 
     test_config.set("scanners.garak", test_garak_config_in_rapidast)
@@ -50,7 +54,7 @@ def test_setup_garak(mock_check_garak_version, test_config):
 
     assert test_garak.garak_cli
     assert test_garak.garak_cli[:3] == [
-        test_garak.cfg.garak_executable_path,
+        test_garak.cfg.executable_path,
         "--config",
         os.path.join(test_garak.workdir, test_garak.GARAK_RUN_CONFIG_FILE),
     ]


Reviews:
user: login: jeremychoi
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T04:04:47Z
user: login: ccronca
body: I'm not completely sure about the configuration management approach here. The extra layer on top of the Garak config doesn’t seem to add much value and might actually limit advanced users. For example, if they want to tweak the 'reporting' section, they might not be able to. It could be simpler if we just let advanced users bring in their own config file
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-03T08:27:05Z
user: login: ccronca
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-05T07:35:52Z
user: login: ccronca
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T08:48:06Z
user: login: ccronca
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T08:51:08Z
user: login: ccronca
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-05T08:51:17Z
user: login: jeremychoi
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T09:10:29Z
user: login: jeremychoi
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T09:29:12Z

Review comments:
diff_hunk: @@ -54,5 +55,5 @@ repos:
           - --max-line-length=120
           - --min-public-methods=0
           - --good-names=o,w,q,f,fp,i,e
-          - --disable=E0401,W1201,W1203,C0114,C0115,C0116,C0411,W0107,W0511,W0702,R0801,R1705,R1710
+          - --disable=E0401,W1201,W1203,C0114,C0115,C0116,C0411,W0107,W0511,W0702,R0801,R1705,R1710,W0201
path: .pre-commit-config.yaml
user: login: jeremychoi
body: adding this for now, as the issue occurs again. This will need to be more investigated separately. 
created_at: 2025-03-03T04:04:47Z
author_association: MEMBER
diff_hunk: @@ -639,25 +639,16 @@
                         "type": "object",
                         "description": "Garak Scanner",
                         "properties": {
-                            "model_name": {
-                                "type": "string"
-                            },
-                            "model_type": {
-                                "type": "string"
-                            },
-                            "generators": {
+                            "parameters": {
path: config/schemas/6/rapidast_schema.json
user: login: ccronca
body: Since `model_type` is a required field, wouldn't it make more sense to define it here rather than checking for its existence in the code? After all, that's what the schema file is for
```
"parameters": {
    "type": "object",
    "properties": {
        "plugins": {
            "type": "object",
            "properties": {
                "type": "object",
                "properties": {
                    "model_type": {
                        "type": "string"
                    }
                }
            },
            "required": ["model_type"]
        }
    },
    "required": ["plugins"]
},
```
created_at: 2025-03-05T08:48:06Z
author_association: COLLABORATOR
diff_hunk: @@ -18,12 +19,12 @@
 
 
 @dataclass
+# pylint: disable=too-many-instance-attributes
 class GarakConfig:
-    model_type: str
-    model_name: str = field(default="test_model")
-    probe_spec: str = field(default="all")  # all or a list of probes like "probe1,probe2"
-    garak_executable_path: str = field(default="/usr/local/bin/garak")
-    generators: Dict[str, Any] = field(default_factory=dict)
+    parameters: Optional[Dict[str, Any]] = field(default_factory=dict)
path: scanners/garak/garak_none.py
user: login: ccronca
body: The `parameters` property is not `Optional`, right? It should always be initialized
created_at: 2025-03-05T08:51:07Z
author_association: COLLABORATOR
diff_hunk: @@ -18,12 +19,12 @@
 
 
 @dataclass
+# pylint: disable=too-many-instance-attributes
 class GarakConfig:
-    model_type: str
-    model_name: str = field(default="test_model")
-    probe_spec: str = field(default="all")  # all or a list of probes like "probe1,probe2"
-    garak_executable_path: str = field(default="/usr/local/bin/garak")
-    generators: Dict[str, Any] = field(default_factory=dict)
+    parameters: Optional[Dict[str, Any]] = field(default_factory=dict)
path: scanners/garak/garak_none.py
user: login: jeremychoi
body: Sorry I just noticed your comment after the merge. You're correct, so I'll create a new PR for it.
created_at: 2025-03-05T09:10:29Z
author_association: MEMBER
diff_hunk: @@ -639,25 +639,16 @@
                         "type": "object",
                         "description": "Garak Scanner",
                         "properties": {
-                            "model_name": {
-                                "type": "string"
-                            },
-                            "model_type": {
-                                "type": "string"
-                            },
-                            "generators": {
+                            "parameters": {
path: config/schemas/6/rapidast_schema.json
user: login: jeremychoi
body: will address this too in a separate PR.
created_at: 2025-03-05T09:29:12Z
author_association: MEMBER

PR comments:
user: login: jperezdealgaba
created_at: 2025-03-03T10:08:42Z
body: @ccronca Adding my two cents here. It would be good if we only have one configuration file so if advanced users want to modify something, maintaining different configuration files can be a mess. If they could do everything in the rapidast config file, that would be awesome.
author_association: CONTRIBUTOR
user: login: ccronca
created_at: 2025-03-03T10:36:29Z
body: > @ccronca Adding my two cents here. It would be good if we only have one configuration file so if advanced users want to modify something, maintaining different configuration files can be a mess. If they could do everything in the rapidast config file, that would be awesome.

Good point! Having a single file makes total sense. My main concern is that it looks like Rapidast is duplicating the Garak config (at least the top-level stuff) in its own config. So, every time the Garak config changes, Rapidast has to be updated too. Since Rapidast isn’t really doing anything with that config, it might be better to just merge the default values from Rapidast with whatever the user provides and pass that to Garak. Garak can then check if the config is correct or not
author_association: COLLABORATOR
user: login: jperezdealgaba
created_at: 2025-03-03T11:48:55Z
body: @ccronca Thank you for the response. That sounds like a valid approach!
author_association: CONTRIBUTOR
user: login: jeremychoi
created_at: 2025-03-04T04:03:18Z
body: @ccronca Passing the garak config schema as is was actually one of options that I also considered so I'm okay with that approach too. The reason I ended up implementing it this way for now is that we may want to simplify the format as much as possible and add some schema checks for the top level at least, rather than just passing it to Garak ( later when we want to improve the config schema, the simplicity will matter more). 

How do the following three examples look to you? 

Example 1:  (current)
```
scanners:
  garak:
    model_type: rest
    generators: {}
    run:
      generations: 1
    system:
```

VS Example 2 (just using Garak config as is)

```
scanners:
  garak:
     plugin:
       model_type: xxx
       generators: {} 
     run:
       generations: 1
     system:
```

I have also thought of supporting both by adding a key like 'garak_config'

Example 3    (supporting both simplified format and the original garak config format. in this case, we need to decide which value of the same config option will be used if both exists )
```
scanners:
  garak:
     model_type: xxx
     generators: {}
     garak_config:
       plugin:
         model_type: yyy
         generators: {} 
       run:
          generations: 1
       system:
```

Which one would you suggest?

@jperezdealgaba what is your preference too from an end user perspective?

author_association: MEMBER
user: login: jperezdealgaba
created_at: 2025-03-04T07:29:27Z
body: IMHO, I would use number two so users can copy/paste it later and use it with Garak as a stand-alone
author_association: CONTRIBUTOR
user: login: ccronca
created_at: 2025-03-04T08:11:11Z
body: I’d avoid having multiple ways to define the same config items, so option 3 is out. Between 1 and 2, I prefer the second one since it follows Garak’s existing structure, meaning we don’t need to map any config item, Rapidast can just pass the config as is.

We could wrap all Garak-related configs under a parent key like "garak_config", so it's clear what belongs to Garak vs. any future Rapidast-specific settings.

So, I’d go with option 2, but with everything under a key.
author_association: COLLABORATOR
user: login: jperezdealgaba
created_at: 2025-03-05T07:32:58Z
body: It looks in my opinion with the new changes!
author_association: CONTRIBUTOR