Repository: promptfoo/promptfoo
Pull request #3262: feat: base64 loader for images
Author: login: typpo
Created: 2025-03-02T04:02:47Z | Merged: 2025-03-03T04:34:42Z



Diff:
diff --git a/examples/amazon-bedrock/cat.jpg b/examples/amazon-bedrock/cat.jpg
new file mode 100644
index 000000000000..36e10b1e3777
Binary files /dev/null and b/examples/amazon-bedrock/cat.jpg differ
diff --git a/examples/amazon-bedrock/nova_multimodal_prompt.json b/examples/amazon-bedrock/nova_multimodal_prompt.json
new file mode 100644
index 000000000000..6cf0c80cd77d
--- /dev/null
+++ b/examples/amazon-bedrock/nova_multimodal_prompt.json
@@ -0,0 +1,16 @@
+[
+  {
+    "role": "user",
+    "content": [
+      {
+        "image": {
+          "format": "jpg",
+          "source": { "bytes": "{{image}}" }
+        }
+      },
+      {
+        "text": "What is this a picture of?"
+      }
+    ]
+  }
+]
diff --git a/examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml b/examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml
new file mode 100644
index 000000000000..e38304c2352d
--- /dev/null
+++ b/examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml
@@ -0,0 +1,17 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Nova Eval with Images'
+
+prompts:
+  - file://nova_multimodal_prompt.json
+
+providers:
+  - id: bedrock:amazon.nova-pro-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+
+tests:
+  - vars:
+      image: file://cat.jpg
diff --git a/site/docs/configuration/expected-outputs/index.md b/site/docs/configuration/expected-outputs/index.md
index b897d2babea2..92b2d30b0416 100644
--- a/site/docs/configuration/expected-outputs/index.md
+++ b/site/docs/configuration/expected-outputs/index.md
@@ -348,7 +348,7 @@ print(json.dumps({
 
 ## Load assertions from CSV
 
-The [Tests file](/docs/configuration/parameters#tests-file) is an optional format that lets you specify test cases outside of the main config file.
+The [Tests file](/docs/configuration/parameters#tests-and-vars) is an optional format that lets you specify test cases outside of the main config file.
 
 To add an assertion to a test case in a vars file, use the special `__expected` column.
 
diff --git a/site/docs/configuration/guide.md b/site/docs/configuration/guide.md
index ab559295876d..5623ae3e5ff9 100644
--- a/site/docs/configuration/guide.md
+++ b/site/docs/configuration/guide.md
@@ -787,7 +787,7 @@ promptfoo eval -c config1.yaml -c config2.yaml -c config3.yaml
 
 ## Loading tests from CSV
 
-YAML is nice, but some organizations maintain their LLM tests in spreadsheets for ease of collaboration. promptfoo supports a special [CSV file format](/docs/configuration/parameters#tests-file).
+YAML is nice, but some organizations maintain their LLM tests in spreadsheets for ease of collaboration. promptfoo supports a special [CSV file format](/docs/configuration/parameters#tests-and-vars).
 
 ```yaml
 prompts:
diff --git a/site/docs/configuration/parameters.md b/site/docs/configuration/parameters.md
index eeaf0c26bd08..b20af5347942 100644
--- a/site/docs/configuration/parameters.md
+++ b/site/docs/configuration/parameters.md
@@ -374,7 +374,7 @@ In this example, the `body` variable is passed through the `allcaps` filter befo
 
 If `prompts` is not defined in the config, then by default Promptfoo will use a "passthrough" prompt: `{{prompt}}`. This prompt simply passes through content of the `prompt` variable.
 
-## Tests File
+## Tests and Vars
 
 If you have a lot of tests, you can optionally keep them separate from the main config file.
 
@@ -548,6 +548,88 @@ module.exports = (output, { vars }) => {
 };
 ```
 
+### Loading images and other file types
+
+Promptfoo supports loading various file types as variables in your prompts:
+
+```yaml title="promptfooconfig.yaml"
+tests:
+  - vars:
+      text_file: file://path/to/text.txt
+      yaml_file: file://path/to/data.yaml
+      pdf_document: file://path/to/document.pdf
+      image: file://path/to/image.png
+      video: file://path/to/video.mp4
+```
+
+When loading files as variables:
+
+- **Text files** are loaded as plain text
+- **YAML/YML files** are loaded as JSON strings
+- **PDF files** are parsed to extract text (requires `pdf-parse` package)
+- **Image files** (jpg, jpeg, png, gif, bmp, webp, svg) are loaded as base64 strings
+- **Video files** (mp4, webm, ogg, mov, avi, wmv, mkv, m4v) are loaded as base64 strings
+
+This is particularly useful for multimodal models that can process text, images, and videos. For example, to include an image in a prompt for a vision model:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - |-
+    [
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": "Describe what you see in this image:"
+          },
+          {
+            "type": "image_url",
+            "image_url": {
+              "url": "data:image/jpeg;base64,{{image}}"
+            }
+          }
+        ]
+      }
+    ]
+
+tests:
+  - vars:
+      image: file://path/to/image.jpg
+```
+
+Similarly, for video content:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - |-
+    [
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": "Describe what happens in this video:"
+          },
+          {
+            "type": "video_url",
+            "video_url": {
+              "url": "data:video/mp4;base64,{{video}}"
+            }
+          }
+        ]
+      }
+    ]
+
+tests:
+  - vars:
+      video: file://path/to/video.mp4
+```
+
+The media files will be automatically loaded and converted to base64 strings that can be used directly in your prompts for multimodal models.
+
+To disable this behavior, set the `PROMPTFOO_DISABLE_MULTIMEDIA_AS_BASE64` environment variable to `false`.
+
 ## Output File
 
 The results of the evaluation are written to this file. For example:
diff --git a/src/envars.ts b/src/envars.ts
index 455aa8f16579..674a4428338b 100644
--- a/src/envars.ts
+++ b/src/envars.ts
@@ -21,6 +21,8 @@ export type EnvVars = {
   PROMPTFOO_DISABLE_CONVERSATION_VAR?: boolean;
   PROMPTFOO_DISABLE_ERROR_LOG?: boolean;
   PROMPTFOO_DISABLE_JSON_AUTOESCAPE?: boolean;
+  PROMPTFOO_DISABLE_MULTIMEDIA_AS_BASE64?: boolean;
+  PROMPTFOO_DISABLE_PDF_AS_TEXT?: boolean;
   PROMPTFOO_DISABLE_REDTEAM_MODERATION?: boolean;
   PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION?: boolean;
   PROMPTFOO_DISABLE_REF_PARSER?: boolean;
diff --git a/src/evaluatorHelpers.ts b/src/evaluatorHelpers.ts
index 2b8d5352f787..6d26a2b1cc9d 100644
--- a/src/evaluatorHelpers.ts
+++ b/src/evaluatorHelpers.ts
@@ -13,7 +13,7 @@ import { runPython } from './python/pythonUtils';
 import telemetry from './telemetry';
 import type { ApiProvider, NunjucksFilterMap, Prompt } from './types';
 import { renderVarsInObject } from './util';
-import { isJavascriptFile } from './util/file';
+import { isJavascriptFile, isImageFile, isVideoFile } from './util/file';
 import invariant from './util/invariant';
 import { getNunjucksEngine } from './util/templates';
 import { transform } from './util/transform';
@@ -122,8 +122,27 @@ export async function renderPrompt(
         vars[varName] = JSON.stringify(
           yaml.load(fs.readFileSync(filePath, 'utf8')) as string | object,
         );
-      } else if (fileExtension === 'pdf') {
+      } else if (fileExtension === 'pdf' && !getEnvBool('PROMPTFOO_DISABLE_PDF_AS_TEXT')) {
+        telemetry.recordOnce('feature_used', {
+          feature: 'extract_text_from_pdf',
+        });
         vars[varName] = await extractTextFromPDF(filePath);
+      } else if (
+        (isImageFile(filePath) || isVideoFile(filePath)) &&
+        !getEnvBool('PROMPTFOO_DISABLE_MULTIMEDIA_AS_BASE64')
+      ) {
+        telemetry.recordOnce('feature_used', {
+          feature: 'load_image_as_base64',
+        });
+        logger.debug(`Loading image as base64: ${filePath}`);
+        try {
+          const imageBuffer = fs.readFileSync(filePath);
+          vars[varName] = imageBuffer.toString('base64');
+        } catch (error) {
+          throw new Error(
+            `Failed to load image ${filePath}: ${error instanceof Error ? error.message : String(error)}`,
+          );
+        }
       } else {
         vars[varName] = fs.readFileSync(filePath, 'utf8').trim();
       }
diff --git a/src/util/file.ts b/src/util/file.ts
index c1cd207875ba..519f0c082928 100644
--- a/src/util/file.ts
+++ b/src/util/file.ts
@@ -12,3 +12,27 @@ export const JAVASCRIPT_EXTENSIONS = ['js', 'cjs', 'mjs', 'ts', 'cts', 'mts'];
 export function isJavascriptFile(filePath: string): boolean {
   return new RegExp(`\\.(${JAVASCRIPT_EXTENSIONS.join('|')})$`).test(filePath);
 }
+
+/**
+ * Checks if a file is an image file based on its extension. Non-exhaustive list.
+ *
+ * @param filePath - The path of the file to check.
+ * @returns True if the file has an image extension, false otherwise.
+ */
+export function isImageFile(filePath: string): boolean {
+  const imageExtensions = ['jpg', 'jpeg', 'png', 'gif', 'bmp', 'webp', 'svg'];
+  const fileExtension = filePath.split('.').pop()?.toLowerCase() || '';
+  return imageExtensions.includes(fileExtension);
+}
+
+/**
+ * Checks if a file is a video file based on its extension. Non-exhaustive list.
+ *
+ * @param filePath - The path of the file to check.
+ * @returns True if the file has a video extension, false otherwise.
+ */
+export function isVideoFile(filePath: string): boolean {
+  const videoExtensions = ['mp4', 'webm', 'ogg', 'mov', 'avi', 'wmv', 'mkv', 'm4v'];
+  const fileExtension = filePath.split('.').pop()?.toLowerCase() || '';
+  return videoExtensions.includes(fileExtension);
+}
diff --git a/src/util/testCaseReader.ts b/src/util/testCaseReader.ts
index 534a2697dccf..62475ffaa9c8 100644
--- a/src/util/testCaseReader.ts
+++ b/src/util/testCaseReader.ts
@@ -24,7 +24,7 @@ import type {
 } from '../types';
 import { isJavascriptFile } from './file';
 
-export async function readVarsFiles(
+export async function readTestFiles(
   pathOrGlobs: string | string[],
   basePath: string = '',
 ): Promise<Record<string, string | string[] | object>> {
@@ -191,7 +191,7 @@ async function loadTestWithVars(
 ): Promise<TestCase> {
   const ret: TestCase = { ...testCase, vars: undefined };
   if (typeof testCase.vars === 'string' || Array.isArray(testCase.vars)) {
-    ret.vars = await readVarsFiles(testCase.vars, testBasePath);
+    ret.vars = await readTestFiles(testCase.vars, testBasePath);
   } else {
     ret.vars = testCase.vars;
   }
diff --git a/test/util/testCaseReader.test.ts b/test/util/testCaseReader.test.ts
index b5de040db20e..4bdafa75807d 100644
--- a/test/util/testCaseReader.test.ts
+++ b/test/util/testCaseReader.test.ts
@@ -13,7 +13,7 @@ import {
   readStandaloneTestsFile,
   readTest,
   readTests,
-  readVarsFiles,
+  readTestFiles,
 } from '../../src/util/testCaseReader';
 
 jest.mock('proxy-agent', () => ({
@@ -938,7 +938,7 @@ describe('readVarsFiles', () => {
     jest.mocked(fs.readFileSync).mockReturnValue(yamlContent);
     jest.mocked(globSync).mockReturnValue(['vars.yaml']);
 
-    const result = await readVarsFiles('vars.yaml');
+    const result = await readTestFiles('vars.yaml');
 
     expect(result).toEqual({ var1: 'value1', var2: 'value2' });
   });
@@ -952,7 +952,7 @@ describe('readVarsFiles', () => {
       .mockReturnValueOnce(yamlContent2);
     jest.mocked(globSync).mockReturnValue(['vars1.yaml', 'vars2.yaml']);
 
-    const result = await readVarsFiles(['vars1.yaml', 'vars2.yaml']);
+    const result = await readTestFiles(['vars1.yaml', 'vars2.yaml']);
 
     expect(result).toEqual({ var1: 'value1', var2: 'value2' });
   });


Reviews:
user: login: mldangelo
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T03:28:11Z
user: login: mldangelo
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T03:29:03Z
user: login: mldangelo
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T03:29:48Z
user: login: mldangelo
body: Thanks @typpo 
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-03T03:31:04Z
user: login: mldangelo
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T04:35:47Z
user: login: typpo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T04:47:01Z

Review comments:
diff_hunk: @@ -0,0 +1,17 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Bedrock Nova Eval with Images'
+
+prompts:
+  - file://nova_multimodal_prompt.json
+
+providers:
+  - id: bedrock:amazon.nova-pro-v1:0
+    config:
+      region: 'us-east-1'
+      inferenceConfig:
+        temperature: 0.7
+        max_new_tokens: 256
+
+tests:
+  - vars:
+      image: file://cat.jpg
path: examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml
user: login: mldangelo
body: don't leave us hanging! Add a cat image! 
created_at: 2025-03-03T03:28:10Z
author_association: MEMBER
diff_hunk: @@ -548,6 +548,86 @@ module.exports = (output, { vars }) => {
 };
 ```
 
+### Loading images and other file types
+
+Promptfoo supports loading various file types as variables in your prompts:
+
+```yaml title="promptfooconfig.yaml"
+tests:
+  - vars:
+      text_file: file://path/to/text.txt
+      yaml_file: file://path/to/data.yaml
+      pdf_document: file://path/to/document.pdf
+      image: file://path/to/image.png
+      video: file://path/to/video.mp4
+```
+
+When loading files as variables:
+
+- **Text files** are loaded as plain text
+- **YAML/YML files** are loaded as JSON strings
+- **PDF files** are parsed to extract text (requires `pdf-parse` package)
+- **Image files** (jpg, jpeg, png, gif, bmp, webp, svg) are loaded as base64 strings
+- **Video files** (mp4, webm, ogg, mov, avi, wmv, mkv, m4v) are loaded as base64 strings
+
+This is particularly useful for multimodal models that can process text, images, and videos. For example, to include an image in a prompt for a vision model:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - |-
+    [
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": "Describe what you see in this image:"
+          },
+          {
+            "type": "image_url",
+            "image_url": {
+              "url": "data:image/jpeg;base64,{{image}}"
+            }
+          }
+        ]
+      }
+    ]
+
+tests:
+  - vars:
+      image: file://path/to/image.jpg
+```
+
+Similarly, for video content:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - |-
+    [
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": "Describe what happens in this video:"
+          },
+          {
+            "type": "video_url",
+            "video_url": {
+              "url": "data:video/mp4;base64,{{video}}"
+            }
+          }
+        ]
+      }
+    ]
+
+tests:
+  - vars:
+      video: file://path/to/video.mp4
+```
+
+The media files will be automatically loaded and converted to base64 strings that can be used directly in your prompts for multimodal models.
path: site/docs/configuration/parameters.md
user: login: mldangelo
body: nit, consider adding a note about how to disable 
created_at: 2025-03-03T03:29:03Z
author_association: MEMBER
diff_hunk: @@ -18,6 +18,18 @@ import invariant from './util/invariant';
 import { getNunjucksEngine } from './util/templates';
 import { transform } from './util/transform';
 
+function isImageFile(filePath: string): boolean {
path: src/evaluatorHelpers.ts
user: login: mldangelo
body: nit, move into src/util/file.ts
created_at: 2025-03-03T03:29:47Z
author_association: MEMBER
diff_hunk: @@ -548,6 +548,88 @@ module.exports = (output, { vars }) => {
 };
 ```
 
+### Loading images and other file types
+
+Promptfoo supports loading various file types as variables in your prompts:
+
+```yaml title="promptfooconfig.yaml"
+tests:
+  - vars:
+      text_file: file://path/to/text.txt
+      yaml_file: file://path/to/data.yaml
+      pdf_document: file://path/to/document.pdf
+      image: file://path/to/image.png
+      video: file://path/to/video.mp4
+```
+
+When loading files as variables:
+
+- **Text files** are loaded as plain text
+- **YAML/YML files** are loaded as JSON strings
+- **PDF files** are parsed to extract text (requires `pdf-parse` package)
+- **Image files** (jpg, jpeg, png, gif, bmp, webp, svg) are loaded as base64 strings
+- **Video files** (mp4, webm, ogg, mov, avi, wmv, mkv, m4v) are loaded as base64 strings
+
+This is particularly useful for multimodal models that can process text, images, and videos. For example, to include an image in a prompt for a vision model:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - |-
+    [
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": "Describe what you see in this image:"
+          },
+          {
+            "type": "image_url",
+            "image_url": {
+              "url": "data:image/jpeg;base64,{{image}}"
+            }
+          }
+        ]
+      }
+    ]
+
+tests:
+  - vars:
+      image: file://path/to/image.jpg
+```
+
+Similarly, for video content:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - |-
+    [
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": "Describe what happens in this video:"
+          },
+          {
+            "type": "video_url",
+            "video_url": {
+              "url": "data:video/mp4;base64,{{video}}"
+            }
+          }
+        ]
+      }
+    ]
+
+tests:
+  - vars:
+      video: file://path/to/video.mp4
+```
+
+The media files will be automatically loaded and converted to base64 strings that can be used directly in your prompts for multimodal models.
+
+To disable this behavior, set the `PROMPTFOO_DISABLE_MULTIMEDIA_AS_BASE64` environment variable to `false`.
path: site/docs/configuration/parameters.md
user: login: mldangelo
body: TRUE!!!
created_at: 2025-03-03T04:35:47Z
author_association: MEMBER
diff_hunk: @@ -548,6 +548,88 @@ module.exports = (output, { vars }) => {
 };
 ```
 
+### Loading images and other file types
+
+Promptfoo supports loading various file types as variables in your prompts:
+
+```yaml title="promptfooconfig.yaml"
+tests:
+  - vars:
+      text_file: file://path/to/text.txt
+      yaml_file: file://path/to/data.yaml
+      pdf_document: file://path/to/document.pdf
+      image: file://path/to/image.png
+      video: file://path/to/video.mp4
+```
+
+When loading files as variables:
+
+- **Text files** are loaded as plain text
+- **YAML/YML files** are loaded as JSON strings
+- **PDF files** are parsed to extract text (requires `pdf-parse` package)
+- **Image files** (jpg, jpeg, png, gif, bmp, webp, svg) are loaded as base64 strings
+- **Video files** (mp4, webm, ogg, mov, avi, wmv, mkv, m4v) are loaded as base64 strings
+
+This is particularly useful for multimodal models that can process text, images, and videos. For example, to include an image in a prompt for a vision model:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - |-
+    [
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": "Describe what you see in this image:"
+          },
+          {
+            "type": "image_url",
+            "image_url": {
+              "url": "data:image/jpeg;base64,{{image}}"
+            }
+          }
+        ]
+      }
+    ]
+
+tests:
+  - vars:
+      image: file://path/to/image.jpg
+```
+
+Similarly, for video content:
+
+```yaml title="promptfooconfig.yaml"
+prompts:
+  - |-
+    [
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "text",
+            "text": "Describe what happens in this video:"
+          },
+          {
+            "type": "video_url",
+            "video_url": {
+              "url": "data:video/mp4;base64,{{video}}"
+            }
+          }
+        ]
+      }
+    ]
+
+tests:
+  - vars:
+      video: file://path/to/video.mp4
+```
+
+The media files will be automatically loaded and converted to base64 strings that can be used directly in your prompts for multimodal models.
+
+To disable this behavior, set the `PROMPTFOO_DISABLE_MULTIMEDIA_AS_BASE64` environment variable to `false`.
path: site/docs/configuration/parameters.md
user: login: typpo
body: oops
created_at: 2025-03-03T04:47:01Z
author_association: CONTRIBUTOR

PR comments:
user: login: gru-agent[bot]
created_at: 2025-03-02T04:03:08Z
body: ## TestGru Assignment

### Summary

| Link | CommitId | Status | Reason |
| ---- | -------- | ------ | ------ |
| [Detail](https://gru.ai/:test/promptfoo@github/promptfoo/a6b2016e-f630-4ac4-a0ff-ba908ac5ce00) | 01af6623a5f2a1bca0f445c31b34d43935318a2d | 🚫 Skipped | No files need to be tested {"site/docs/configuration/parameters.md":"target file(site/docs/configuration/parameters.md) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","src/envars.ts":"target file(src/envars.ts) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","src/evaluatorHelpers.ts":"target file(src/evaluatorHelpers.ts) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","src/util/testCaseReader.ts":"target file(src/util/testCaseReader.ts) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","test/util/testCaseReader.test.ts":"ta… |

> [!TIP]
> You can `@gru-agent` and leave your feedback. TestGru will make adjustments based on your input

author_association: CONTRIBUTOR
user: login: gru-agent[bot]
created_at: 2025-03-02T04:03:11Z
body: ## TestGru Assignment

### Summary

| Link | CommitId | Status | Reason |
| ---- | -------- | ------ | ------ |
| [Detail](https://gru.ai/:test/promptfoo@github/promptfoo/1636bcd5-5c14-4a87-9d86-59e0f2d31de6) | 01af6623a5f2a1bca0f445c31b34d43935318a2d | 🚫 Skipped | No files need to be tested {"site/docs/configuration/parameters.md":"target file(site/docs/configuration/parameters.md) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","src/envars.ts":"target file(src/envars.ts) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","src/evaluatorHelpers.ts":"target file(src/evaluatorHelpers.ts) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","src/util/testCaseReader.ts":"target file(src/util/testCaseReader.ts) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","test/util/testCaseReader.test.ts"… |

> [!TIP]
> You can `@gru-agent` and leave your feedback. TestGru will make adjustments based on your input

author_association: CONTRIBUTOR
user: login: gru-agent[bot]
created_at: 2025-03-02T04:03:14Z
body: ## TestGru Assignment

### Summary

| Link | CommitId | Status | Reason |
| ---- | -------- | ------ | ------ |
| [Detail](https://gru.ai/:test/promptfoo@github/promptfoo/7397cfa6-d660-4a3d-9de7-876d3b2216a0) | 01af6623a5f2a1bca0f445c31b34d43935318a2d | ✅ Finished | Cancelled by auto rebase |

### Files

| File | Pull Request |
| ---- | ------------ |
| src/envars.ts | 🛑 Cancelled (Cancelled by auto rebase) |
| src/evaluatorHelpers.ts | 🛑 Cancelled (Cancelled by auto rebase) |
| src/util/testCaseReader.ts | 🛑 Cancelled (Cancelled by auto rebase) |

> [!TIP]
> You can `@gru-agent` and leave your feedback. TestGru will make adjustments based on your input

author_association: CONTRIBUTOR
user: login: gru-agent[bot]
created_at: 2025-03-02T04:06:29Z
body: ## TestGru Assignment

### Summary

| Link | CommitId | Status | Reason |
| ---- | -------- | ------ | ------ |
| [Detail](https://gru.ai/:test/promptfoo@github/promptfoo/d9917d7b-1834-4dfa-aa3e-ffd160880b49) | fd30a46e9c2ea18dc0a04a61556f2159bae6fb81 | 🚫 Skipped | No files need to be tested {"examples/amazon-bedrock/nova_multimodal_prompt.json":"target file(examples/amazon-bedrock/nova_multimodal_prompt.json) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml":"target file(examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","site/docs/configuration/parameters.md":"target file(site/docs/configuration/parameters.md) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","src/envars.ts":"target file(src/envars.ts) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.… |

> [!TIP]
> You can `@gru-agent` and leave your feedback. TestGru will make adjustments based on your input

author_association: CONTRIBUTOR
user: login: gru-agent[bot]
created_at: 2025-03-02T04:06:32Z
body: ## TestGru Assignment

### Summary

| Link | CommitId | Status | Reason |
| ---- | -------- | ------ | ------ |
| [Detail](https://gru.ai/:test/promptfoo@github/promptfoo/b371320b-8496-49d0-b770-ff9439f580f4) | fd30a46e9c2ea18dc0a04a61556f2159bae6fb81 | 🚫 Skipped | No files need to be tested {"examples/amazon-bedrock/nova_multimodal_prompt.json":"target file(examples/amazon-bedrock/nova_multimodal_prompt.json) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml":"target file(examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","site/docs/configuration/parameters.md":"target file(site/docs/configuration/parameters.md) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","src/envars.ts":"target file(src/envars.ts) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,*… |

> [!TIP]
> You can `@gru-agent` and leave your feedback. TestGru will make adjustments based on your input

author_association: CONTRIBUTOR
user: login: gru-agent[bot]
created_at: 2025-03-02T04:06:35Z
body: ## TestGru Assignment

### Summary

| Link | CommitId | Status | Reason |
| ---- | -------- | ------ | ------ |
| [Detail](https://gru.ai/:test/promptfoo@github/promptfoo/8b920bdc-8aee-47b7-92e6-bfb56b303fdc) | fd30a46e9c2ea18dc0a04a61556f2159bae6fb81 | ⏳ Queued |  |

### Files

| File | Pull Request |
| ---- | ------------ |
| src/envars.ts | [⏳ Queued](https://gru.ai/:test/promptfoo@github/promptfoo/8b920bdc-8aee-47b7-92e6-bfb56b303fdc?filePath=src/envars.ts) |
| src/evaluatorHelpers.ts | [⏳ Queued](https://gru.ai/:test/promptfoo@github/promptfoo/8b920bdc-8aee-47b7-92e6-bfb56b303fdc?filePath=src/evaluatorHelpers.ts) |
| src/util/testCaseReader.ts | [⏳ Queued](https://gru.ai/:test/promptfoo@github/promptfoo/8b920bdc-8aee-47b7-92e6-bfb56b303fdc?filePath=src/util/testCaseReader.ts) |

> [!TIP]
> You can `@gru-agent` and leave your feedback. TestGru will make adjustments based on your input

author_association: CONTRIBUTOR
user: login: gru-agent[bot]
created_at: 2025-03-02T04:06:41Z
body: ## TestGru Assignment

### Summary

| Link | CommitId | Status | Reason |
| ---- | -------- | ------ | ------ |
| [Detail](https://gru.ai/:test/promptfoo@github/promptfoo/8b920bdc-8aee-47b7-92e6-bfb56b303fdc) | fd30a46e9c2ea18dc0a04a61556f2159bae6fb81 | ✅ Finished | Cancelled by auto rebase |

### Files

| File | Pull Request |
| ---- | ------------ |
| src/envars.ts | 🔴 Closed https://github.com/promptfoo/promptfoo/pull/3263 |
| src/evaluatorHelpers.ts | 🔴 Closed https://github.com/promptfoo/promptfoo/pull/3265 |
| src/util/testCaseReader.ts | 🚫 Skipped (There's no need to update the test code) |

> [!TIP]
> You can `@gru-agent` and leave your feedback. TestGru will make adjustments based on your input

author_association: CONTRIBUTOR
user: login: gru-agent[bot]
created_at: 2025-03-02T04:06:52Z
body: ## TestGru Assignment

### Summary

| Link | CommitId | Status | Reason |
| ---- | -------- | ------ | ------ |
| [Detail](https://gru.ai/:test/promptfoo@github/promptfoo/0a5a5a46-88ba-4f24-8741-0a0ecd5624f7) | fae5e1b23068887402448ae1d204da328da051c0 | ✅ Finished |  |

### Files

| File | Pull Request |
| ---- | ------------ |
| src/envars.ts | 🔴 Closed https://github.com/promptfoo/promptfoo/pull/3264 |
| src/evaluatorHelpers.ts | 🔴 Closed https://github.com/promptfoo/promptfoo/pull/3266 |
| src/util/testCaseReader.ts | 🚫 Skipped (There's no need to update the test code) |

> [!TIP]
> You can `@gru-agent` and leave your feedback. TestGru will make adjustments based on your input

author_association: CONTRIBUTOR
user: login: gru-agent[bot]
created_at: 2025-03-02T04:06:55Z
body: ## TestGru Assignment

### Summary

| Link | CommitId | Status | Reason |
| ---- | -------- | ------ | ------ |
| [Detail](https://gru.ai/:test/promptfoo@github/promptfoo/1d7de7ed-9320-453d-8e94-92cb7f514594) | fae5e1b23068887402448ae1d204da328da051c0 | 🚫 Skipped | No files need to be tested {"examples/amazon-bedrock/nova_multimodal_prompt.json":"target file(examples/amazon-bedrock/nova_multimodal_prompt.json) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml":"target file(examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","site/docs/configuration/parameters.md":"target file(site/docs/configuration/parameters.md) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","src/envars.ts":"target file(src/envars.ts) not in work scope \n include: src/app/src/**/*.tsx \n exclude: node_modules,*… |

> [!TIP]
> You can `@gru-agent` and leave your feedback. TestGru will make adjustments based on your input

author_association: CONTRIBUTOR
user: login: gru-agent[bot]
created_at: 2025-03-02T04:06:58Z
body: ## TestGru Assignment

### Summary

| Link | CommitId | Status | Reason |
| ---- | -------- | ------ | ------ |
| [Detail](https://gru.ai/:test/promptfoo@github/promptfoo/ef414c5f-4874-432a-b002-3253a0dd2046) | fae5e1b23068887402448ae1d204da328da051c0 | 🚫 Skipped | No files need to be tested {"examples/amazon-bedrock/nova_multimodal_prompt.json":"target file(examples/amazon-bedrock/nova_multimodal_prompt.json) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml":"target file(examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","site/docs/configuration/parameters.md":"target file(site/docs/configuration/parameters.md) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.test.ts,**/*.test.tsx,**/*.spec.ts,**/*.spec.tsx,**/*.d.ts,**/*.test.js,**/*.spec.js","src/envars.ts":"target file(src/envars.ts) not in work scope \n include: src/app/src/**/*.ts \n exclude: node_modules,**/*.… |

> [!TIP]
> You can `@gru-agent` and leave your feedback. TestGru will make adjustments based on your input

author_association: CONTRIBUTOR