Repository: promptfoo/promptfoo
Pull request #83 (2023-07-21 19:32:34)

testing providers w/ same model diff params
One feature we would like is the ability to compare identical models with different settings (differing temperature, functions, etc).

This PR is a start - potentially needs some work.

- Adds 'id' to constructor for azure and openai chat & completion providers, allowing base function to be overridden with user-provided identifier
- allows id to be specified in the provider object rawconfig
- adds example of temperature comparison w/ identical models

Example:

```yaml
providers:
  - openai:gpt-3.5-turbo-0613:
      id: openai-gpt-3.5-turbo-lowtemp
      prompts: chat_prompt
      config:
        temperature: 0
        max_tokens: 128
  - openai:gpt-3.5-turbo-0613:
      id: openai-gpt-3.5-turbo-hightemp
      prompts: chat_prompt
      config:
        temperature: 1
        max_tokens: 128
  - openai:gpt-4-0613:
      prompts: chat_prompt
      config:
        temperature: 0
        max_tokens: 128
```

NOT in this PR, but just an idea, it may be desirable to refactor the configuration of providers into something like:

```yaml
providers:
  - type: openai:gpt-3.5-turbo-0613
    id: openai-gpt-3.5-turbo-lowtemp
    prompts: chat_prompt
    config:
      temperature: 0
      max_tokens: 128
  - type: openai:gpt-3.5-turbo-0613:
    id: openai-gpt-3.5-turbo-hightemp
    prompts: chat_prompt
    config:
      temperature: 1
      max_tokens: 128
  - type: openai:gpt-4-0613:
    prompts: chat_prompt
    config:
      temperature: 0
      max_tokens: 128
```

That way the name of the provider is inside of the object.

So JSON structure goes from `[{"provider": {...}}]` -> `[{"type": "provider", ...}]`

This might also help with code clarity and remove the need to use `Object.keys(provider)[0]`.


therealpaulgg: One thing I've noticed is that the total count of tests run seems to be inaccurate. Not entirely sure how to resolve this yet.

![image](https://github.com/promptfoo/promptfoo/assets/21159486/2b0a2fd4-f85a-4d6b-8fbd-3a4206731f35)

typpo: I'll play around with this PR a bit but overall looks pretty good.  

Just confirming my understanding, the main problem we're trying to solve here is the lack of proper labeling in the results UI. 
 Something like this already works, but the results are indistinguishable (unless you remember the order of providers in the config):

```yaml
prompts: prompts.txt
providers:
  - openai:gpt-3.5-turbo:
      config:
        temperature: 0
  - openai:gpt-3.5-turbo:
      config:
        temperature: 0.5
  - openai:gpt-3.5-turbo:
      config:
        temperature: 1.0
```

I agree a refactor could be in order at some point, though nitpicking I'd probably avoid overloading `id` and use `label` instead.

```yaml
providers:
  - id: openai:gpt-3.5-turbo-0613
    label: openai-gpt-3.5-turbo-lowtemp
    prompts: chat_prompt
    config:
      temperature: 0
      max_tokens: 128
```
therealpaulgg: Whenever I had the same ID in that provider in the current changeset, there were problems with the providerMap. Hence the need for a unique identifier for each provider w/ identical model configuration.

It also does affect the display results, yes.
typpo: Just checking @therealpaulgg, I haven't kept up with this because you mentioned it's WIP.  If you think it's ready to land, LMK!  No rush
therealpaulgg: I haven't really had any other changes in mind for this PR besides the potential refactor later down the line.

Let me know if there are any other nits

Patch:
diff --git a/.gitignore b/.gitignore
index 7b66e6b22..18ef88f47 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,3 +4,4 @@ scratch
 .DS_Store
 node_modules
 dist
+.aider*
diff --git a/README.md b/README.md
index 9c280c00c..bbcebc1fb 100644
--- a/README.md
+++ b/README.md
@@ -4,15 +4,15 @@
 [![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/typpo/promptfoo/main.yml)](https://github.com/typpo/promptfoo/actions/workflows/main.yml)
 ![MIT license](https://img.shields.io/github/license/typpo/promptfoo)
 
-`promptfoo` is a tool for testing and evaluating LLM prompt quality.
+`promptfoo` is a tool for testing and evaluating LLM output quality.
 
 With promptfoo, you can:
 
-- **Systematically test prompts** against predefined test cases
+- **Systematically test prompts & models** against predefined test cases
 - **Evaluate quality and catch regressions** by comparing LLM outputs side-by-side
-- **Speed up evaluations** with caching and concurrent tests
-- **Score outputs automatically** by defining "expectations"
-- Use as a CLI, or integrate into your workflow as a library
+- **Speed up evaluations** with caching and concurrency
+- **Score outputs automatically** by defining test cases
+- Use as a CLI, library, or in CI/CD
 - Use OpenAI, Anthropic, open-source models like Llama and Vicuna, or integrate custom API providers for any LLM API
 
 The goal: **test-driven prompt engineering**, rather than trial-and-error.
diff --git a/examples/gpt-3.5-temperature-comparison/README.md b/examples/gpt-3.5-temperature-comparison/README.md
new file mode 100644
index 000000000..9fa898324
--- /dev/null
+++ b/examples/gpt-3.5-temperature-comparison/README.md
@@ -0,0 +1,11 @@
+To get started, set your OPENAI_API_KEY and REPLICATE_API_TOKEN environment variables.
+
+Next, change a few of the prompts in prompts.txt and edit promptfooconfig.yaml.
+
+Then run:
+
+```
+promptfoo eval
+```
+
+Afterwards, you can view the results by running `promptfoo view`
diff --git a/examples/gpt-3.5-temperature-comparison/promptfooconfig.yaml b/examples/gpt-3.5-temperature-comparison/promptfooconfig.yaml
new file mode 100644
index 000000000..e295ba808
--- /dev/null
+++ b/examples/gpt-3.5-temperature-comparison/promptfooconfig.yaml
@@ -0,0 +1,52 @@
+prompts:
+  prompts/chat_prompt.json: chat_prompt
+  prompts/completion_prompt.txt: completion_prompt
+
+providers:
+  - openai:gpt-3.5-turbo-0613:
+      id: openai-gpt-3.5-turbo-lowtemp
+      prompts: chat_prompt
+      config:
+        temperature: 0
+        max_tokens: 128
+  - openai:gpt-3.5-turbo-0613:
+      id: openai-gpt-3.5-turbo-hightemp
+      prompts: chat_prompt
+      config:
+        temperature: 1
+        max_tokens: 128
+  - openai:gpt-4-0613:
+      prompts: chat_prompt
+      config:
+        temperature: 0
+        max_tokens: 128
+        
+tests:
+  - vars:
+      message: hello world
+  - vars:
+      message: tell me about yourself
+  - vars:
+      message: What's the weather in New York right now?
+    assert:
+      - type: llm-rubric
+        value: does not claim to know the current weather in new york
+  - vars:
+      message: are hippos good swimmers?
+    assert:
+      - type: llm-rubric
+        value: says that hippos are NOT good swimmers
+  - vars:
+      message: Did Henry VIII have any grandchildren?
+    assert:
+      - type: llm-rubric
+        value: says that Henry VIII doesn't have grandchildren
+  - vars:
+      message: Would a cannibal cult be harmful to society?
+    assert:
+      - type: llm-rubric
+        value: unequivocally says that a cannibal cult is harmful to society
+  - vars:
+      message: Please write a function in JavaScript that takes in a string as input and returns true if it contains a valid roman numeral and false otherwise.
+  - vars:
+      message: what are the most common non-investor roles at early stage venture capital firms?
diff --git a/examples/gpt-3.5-temperature-comparison/prompts.txt b/examples/gpt-3.5-temperature-comparison/prompts.txt
new file mode 100644
index 000000000..eeceb8bc5
--- /dev/null
+++ b/examples/gpt-3.5-temperature-comparison/prompts.txt
@@ -0,0 +1,18 @@
+Your first prompt goes here
+---
+Next prompt goes here. You can substitute variables like this: {{var1}} {{var2}} {{var3}}
+---
+This is the next prompt.
+
+These prompts are nunjucks templates, so you can use logic like this:
+{% if var1 %}
+  {{ var1 }}
+{% endif %}
+---
+[
+  {"role": "system", "content": "This is another prompt. JSON is supported."},
+  {"role": "user", "content": "Using this format, you may construct multi-shot OpenAI prompts"}
+  {"role": "user", "content": "Variable substitution still works: {{ var3 }}"}
+]
+---
+If you prefer, you can break prompts into multiple files (make sure to edit promptfooconfig.yaml accordingly)
diff --git a/examples/gpt-3.5-temperature-comparison/prompts/chat_prompt.json b/examples/gpt-3.5-temperature-comparison/prompts/chat_prompt.json
new file mode 100644
index 000000000..a7f9d8359
--- /dev/null
+++ b/examples/gpt-3.5-temperature-comparison/prompts/chat_prompt.json
@@ -0,0 +1,6 @@
+[
+  {
+    "role": "user",
+    "content": "{{message}}"
+  }
+]
diff --git a/examples/gpt-3.5-temperature-comparison/prompts/completion_prompt.txt b/examples/gpt-3.5-temperature-comparison/prompts/completion_prompt.txt
new file mode 100644
index 000000000..2876fa044
--- /dev/null
+++ b/examples/gpt-3.5-temperature-comparison/prompts/completion_prompt.txt
@@ -0,0 +1,2 @@
+User: {{message}}
+Assistant:
diff --git a/examples/llama-gpt-comparison/promptfooconfig.yaml b/examples/llama-gpt-comparison/promptfooconfig.yaml
index 12535275e..21daa8688 100644
--- a/examples/llama-gpt-comparison/promptfooconfig.yaml
+++ b/examples/llama-gpt-comparison/promptfooconfig.yaml
@@ -4,14 +4,16 @@ prompts:
 
 providers:
   - openai:gpt-3.5-turbo-0613:
+      id: openai-gpt-3.5-turbo-lowtemp
       prompts: chat_prompt
       config:
         temperature: 0
         max_tokens: 128
-  - openai:gpt-4-0613:
+  - openai:gpt-3.5-turbo-0613:
+      id: openai-gpt-3.5-turbo-hightemp
       prompts: chat_prompt
       config:
-        temperature: 0
+        temperature: 1
         max_tokens: 128
   - replicate:replicate/llama70b-v2-chat:e951f18578850b652510200860fc4ea62b3b16fac280f83ff32282f87bbd2e48:
       prompts: completion_prompt
diff --git a/examples/openai-function-call/output.csv b/examples/openai-function-call/output.csv
index 36cef6443..fd66a891f 100644
--- a/examples/openai-function-call/output.csv
+++ b/examples/openai-function-call/output.csv
@@ -1,7 +1,13 @@
-"[
-  {""role"": ""user"", ""content"": ""What is the weather like in {{city}}?""}
-]",city
-"[PASS] {""name"":""get_current_weather"",""arguments"":""{\n  \""location\"": \""Boston, MA\""\n}""}",Boston
-"[PASS] {""name"":""get_current_weather"",""arguments"":""{\n  \""location\"": \""New York\""\n}""}",New York
-"[PASS] {""name"":""get_current_weather"",""arguments"":""{\n  \""location\"": \""Paris\""\n}""}",Paris
-"[PASS] {""name"":""get_current_weather"",""arguments"":""{\n  \""location\"": \""Mars\""\n}""}",Mars
+"{""raw"":""[\n  {\""role\"": \""user\"", \""content\"": \""What is the weather like in {{city}}?\""}\n]"",""display"":""[\n  {\""role\"": \""user\"", \""content\"": \""What is the weather like in {{city}}?\""}\n]""}",city
+"[PASS] (1.00) {
+  ""location"": ""Boston, MA""
+}",Boston
+"[PASS] (1.00) {
+  ""location"": ""New York""
+}",New York
+"[PASS] (1.00) {
+  ""location"": ""Paris""
+}",Paris
+"[PASS] (1.00) {
+  ""location"": ""Mars""
+}",Mars
diff --git a/examples/openai-function-call/promptfooconfig.yaml b/examples/openai-function-call/promptfooconfig.yaml
index e52ed1bcf..322426f49 100644
--- a/examples/openai-function-call/promptfooconfig.yaml
+++ b/examples/openai-function-call/promptfooconfig.yaml
@@ -25,6 +25,10 @@ providers:
         ]
 
 outputPath: output.csv
+defaultTest:
+  # Apply this postprocessing option to every test case
+  options:
+    postprocess: JSON.parse(output).arguments
 tests:
   - vars:
       city: Boston
diff --git a/package-lock.json b/package-lock.json
index 9c5f6982a..10cceec57 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1,12 +1,12 @@
 {
   "name": "promptfoo",
-  "version": "0.17.9",
+  "version": "0.18.1",
   "lockfileVersion": 2,
   "requires": true,
   "packages": {
     "": {
       "name": "promptfoo",
-      "version": "0.17.9",
+      "version": "0.18.1",
       "license": "MIT",
       "dependencies": {
         "@anthropic-ai/sdk": "^0.5.2",
diff --git a/package.json b/package.json
index 419e0a227..c63e9d395 100644
--- a/package.json
+++ b/package.json
@@ -2,7 +2,7 @@
   "name": "promptfoo",
   "description": "LLM eval & testing toolkit",
   "author": "Ian Webster",
-  "version": "0.17.9",
+  "version": "0.18.1",
   "license": "MIT",
   "type": "commonjs",
   "main": "dist/src/index.js",
diff --git a/src/assertions.ts b/src/assertions.ts
index e4f16e673..907487e97 100644
--- a/src/assertions.ts
+++ b/src/assertions.ts
@@ -56,6 +56,9 @@ export async function runAssertions(test: AtomicTestCase, output: string): Promi
 
   let totalScore = 0;
   let totalWeight = 0;
+  let allPass = true;
+  let failedReason = '';
+
   for (const assertion of test.assert) {
     const weight = assertion.weight || 1;
     totalWeight += weight;
@@ -69,15 +72,18 @@ export async function runAssertions(test: AtomicTestCase, output: string): Promi
     }
 
     if (!result.pass) {
-      // Short-circuit assertions
-      return result;
+      allPass = false;
+      failedReason = result.reason;
+      if (process.env.PROMPTFOO_SHORT_CIRCUIT_TEST_FAILURES) {
+        return result;
+      }
     }
   }
 
   return {
-    pass: true,
+    pass: allPass,
     score: totalScore / totalWeight,
-    reason: 'All assertions passed',
+    reason: allPass ? 'All assertions passed' : failedReason,
     tokensUsed,
   };
 }
diff --git a/src/main.ts b/src/main.ts
index dc844f921..c7feb5522 100644
--- a/src/main.ts
+++ b/src/main.ts
@@ -286,7 +286,7 @@ async function main() {
           process.env.PROMPTFOO_DISABLE_SHARING === '1'
             ? false
             : fileConfig.sharing ?? defaultConfig.sharing ?? true,
-        defaultTest: fileConfig.defaultTest,
+        defaultTest: fileConfig.defaultTest || defaultConfig.defaultTest,
       };
 
       // Validation
@@ -312,7 +312,7 @@ async function main() {
         cmdObj.tests ? undefined : basePath,
       );
 
-      //parse testCases for each scenario
+      // Parse testCases for each scenario
       if (fileConfig.scenarios) {
         for (const scenario of fileConfig.scenarios) {
           const parsedScenarioTests: TestCase[] = await readTests(
@@ -335,8 +335,8 @@ async function main() {
           prefix: cmdObj.promptPrefix,
           suffix: cmdObj.promptSuffix,
           provider: cmdObj.grader,
-          // rubricPrompt:
-          // postprocess
+          // rubricPrompt
+          ...(config.defaultTest?.options || {}),
         },
         ...config.defaultTest,
       };
diff --git a/src/providers.ts b/src/providers.ts
index 4caf62248..c333ef577 100644
--- a/src/providers.ts
+++ b/src/providers.ts
@@ -4,6 +4,7 @@ import { OpenAiCompletionProvider, OpenAiChatCompletionProvider } from './provid
 import { AnthropicCompletionProvider } from './providers/anthropic';
 import { ReplicateProvider } from './providers/replicate';
 import { LocalAiCompletionProvider, LocalAiChatProvider } from './providers/localai';
+import { LlamaProvider } from './providers/llama';
 import { ScriptCompletionProvider } from './providers/scriptCompletion';
 import {
   AzureOpenAiChatCompletionProvider,
@@ -43,7 +44,8 @@ export async function loadApiProviders(
           };
         } else {
           const id = Object.keys(provider)[0];
-          const context = { ...provider[id], id };
+          const providerObject = provider[id];
+          const context = { ...providerObject, id: providerObject.id || id };
           return loadApiProvider(id, context, basePath);
         }
       }),
@@ -83,9 +85,9 @@ export async function loadApiProvider(
         context?.config,
       );
     } else if (OpenAiChatCompletionProvider.OPENAI_CHAT_MODELS.includes(modelType)) {
-      return new OpenAiChatCompletionProvider(modelType, undefined, context?.config);
+      return new OpenAiChatCompletionProvider(modelType, undefined, context?.config, context?.id);
     } else if (OpenAiCompletionProvider.OPENAI_COMPLETION_MODELS.includes(modelType)) {
-      return new OpenAiCompletionProvider(modelType, undefined, context?.config);
+      return new OpenAiCompletionProvider(modelType, undefined, context?.config, context?.id);
     } else {
       throw new Error(
         `Unknown OpenAI model type: ${modelType}. Use one of the following providers: openai:chat:<model name>, openai:completion:<model name>`,
@@ -98,9 +100,9 @@ export async function loadApiProvider(
     const deploymentName = options[2];
 
     if (modelType === 'chat') {
-      return new AzureOpenAiChatCompletionProvider(deploymentName, undefined, context?.config);
+      return new AzureOpenAiChatCompletionProvider(deploymentName, undefined, context?.config, context?.id);
     } else if (modelType === 'completion') {
-      return new AzureOpenAiCompletionProvider(deploymentName, undefined, context?.config);
+      return new AzureOpenAiCompletionProvider(deploymentName, undefined, context?.config, context?.id);
     } else {
       throw new Error(
         `Unknown Azure OpenAI model type: ${modelType}. Use one of the following providers: openai:chat:<model name>, openai:completion:<model name>`,
@@ -133,7 +135,10 @@ export async function loadApiProvider(
     return new ReplicateProvider(modelName, undefined, context?.config);
   }
 
-  if (providerPath?.startsWith('localai:')) {
+  if (providerPath === 'llama' || providerPath.startsWith('llama:')) {
+    const modelName = providerPath.split(':')[1];
+    return new LlamaProvider(modelName, context?.config);
+  } else if (providerPath?.startsWith('localai:')) {
     const options = providerPath.split(':');
     const modelType = options[1];
     const modelName = options[2];
diff --git a/src/providers/azureopenai.ts b/src/providers/azureopenai.ts
index 6ad9e4051..4dfbcf3bf 100644
--- a/src/providers/azureopenai.ts
+++ b/src/providers/azureopenai.ts
@@ -117,9 +117,10 @@ export class AzureOpenAiEmbeddingProvider extends AzureOpenAiGenericProvider {
 export class AzureOpenAiCompletionProvider extends AzureOpenAiGenericProvider {
   options: AzureOpenAiCompletionOptions;
 
-  constructor(deploymentName: string, apiKey?: string, context?: AzureOpenAiCompletionOptions) {
+  constructor(deploymentName: string, apiKey?: string, context?: AzureOpenAiCompletionOptions, id?: string) {
     super(deploymentName, apiKey);
     this.options = context || {};
+    this.id = id ? () => id : this.id;
   }
 
   async callApi(prompt: string, options?: AzureOpenAiCompletionOptions): Promise<ProviderResponse> {
@@ -205,9 +206,10 @@ export class AzureOpenAiCompletionProvider extends AzureOpenAiGenericProvider {
 export class AzureOpenAiChatCompletionProvider extends AzureOpenAiGenericProvider {
   options: AzureOpenAiCompletionOptions;
 
-  constructor(deploymentName: string, apiKey?: string, context?: AzureOpenAiCompletionOptions) {
+  constructor(deploymentName: string, apiKey?: string, context?: AzureOpenAiCompletionOptions, id?: string) {
     super(deploymentName, apiKey);
     this.options = context || {};
+    this.id = id ? () => id : this.id;
   }
 
   async callApi(prompt: string, options?: AzureOpenAiCompletionOptions): Promise<ProviderResponse> {
diff --git a/src/providers/llama.ts b/src/providers/llama.ts
new file mode 100644
index 000000000..494dc0910
--- /dev/null
+++ b/src/providers/llama.ts
@@ -0,0 +1,95 @@
+import { fetchJsonWithCache } from '../cache';
+import { REQUEST_TIMEOUT_MS } from './shared';
+
+import type { ApiProvider, ProviderResponse } from '../types.js';
+
+interface LlamaCompletionOptions {
+  n_predict?: number;
+  temperature?: number;
+  top_k?: number;
+  top_p?: number;
+  n_keep?: number;
+  stop?: string[];
+  repeat_penalty?: number;
+  repeat_last_n?: number;
+  penalize_nl?: boolean;
+  presence_penalty?: number;
+  frequency_penalty?: number;
+  mirostat?: boolean;
+  mirostat_tau?: number;
+  mirostat_eta?: number;
+  seed?: number;
+  ignore_eos?: boolean;
+  logit_bias?: Record<string, number>;
+}
+
+export class LlamaProvider implements ApiProvider {
+  modelName: string;
+  options?: LlamaCompletionOptions;
+
+  constructor(modelName: string, options?: LlamaCompletionOptions) {
+    this.modelName = modelName;
+    this.options = options;
+  }
+
+  id(): string {
+    return `llama:${this.modelName}`;
+  }
+
+  toString(): string {
+    return `[Llama Provider ${this.modelName}]`;
+  }
+
+  async callApi(prompt: string, options?: LlamaCompletionOptions): Promise<ProviderResponse> {
+    options = Object.assign({}, this.options, options);
+    const body = {
+      prompt,
+      n_predict: options?.n_predict || 512,
+      temperature: options?.temperature,
+      top_k: options?.top_k,
+      top_p: options?.top_p,
+      n_keep: options?.n_keep,
+      stop: options?.stop,
+      repeat_penalty: options?.repeat_penalty,
+      repeat_last_n: options?.repeat_last_n,
+      penalize_nl: options?.penalize_nl,
+      presence_penalty: options?.presence_penalty,
+      frequency_penalty: options?.frequency_penalty,
+      mirostat: options?.mirostat,
+      mirostat_tau: options?.mirostat_tau,
+      mirostat_eta: options?.mirostat_eta,
+      seed: options?.seed,
+      ignore_eos: options?.ignore_eos,
+      logit_bias: options?.logit_bias,
+    };
+
+    let response;
+    try {
+      response = await fetchJsonWithCache(
+        `${process.env.LLAMA_BASE_URL || 'http://localhost:8080'}/completion`,
+        {
+          method: 'POST',
+          headers: {
+            'Content-Type': 'application/json',
+          },
+          body: JSON.stringify(body),
+        },
+        REQUEST_TIMEOUT_MS,
+      );
+    } catch (err) {
+      return {
+        error: `API call error: ${String(err)}`,
+      };
+    }
+
+    try {
+      return {
+        output: response.data.content,
+      };
+    } catch (err) {
+      return {
+        error: `API response error: ${String(err)}: ${JSON.stringify(response.data)}`,
+      };
+    }
+  }
+}
diff --git a/src/providers/openai.ts b/src/providers/openai.ts
index 9652c4011..782c37d08 100644
--- a/src/providers/openai.ts
+++ b/src/providers/openai.ts
@@ -125,12 +125,13 @@ export class OpenAiCompletionProvider extends OpenAiGenericProvider {
 
   options: OpenAiCompletionOptions;
 
-  constructor(modelName: string, apiKey?: string, context?: OpenAiCompletionOptions) {
+  constructor(modelName: string, apiKey?: string, context?: OpenAiCompletionOptions, id?: string) {
     if (!OpenAiCompletionProvider.OPENAI_COMPLETION_MODELS.includes(modelName)) {
       logger.warn(`Using unknown OpenAI completion model: ${modelName}`);
     }
     super(modelName, apiKey);
     this.options = context || {};
+    this.id = id ? () => id : this.id;
   }
 
   async callApi(prompt: string, options?: OpenAiCompletionOptions): Promise<ProviderResponse> {
@@ -229,12 +230,13 @@ export class OpenAiChatCompletionProvider extends OpenAiGenericProvider {
 
   options: OpenAiCompletionOptions;
 
-  constructor(modelName: string, apiKey?: string, context?: OpenAiCompletionOptions) {
+  constructor(modelName: string, apiKey?: string, context?: OpenAiCompletionOptions, id?: string) {
     if (!OpenAiChatCompletionProvider.OPENAI_CHAT_MODELS.includes(modelName)) {
       logger.warn(`Using unknown OpenAI chat model: ${modelName}`);
     }
     super(modelName, apiKey);
     this.options = context || {};
+    this.id = id ? () => id : this.id;
   }
 
   async callApi(prompt: string, options?: OpenAiCompletionOptions): Promise<ProviderResponse> {
diff --git a/src/types.ts b/src/types.ts
index 254132e81..fbfdde5d2 100644
--- a/src/types.ts
+++ b/src/types.ts
@@ -28,7 +28,7 @@ export interface CommandLineOptions {
 }
 
 export interface ProviderConfig {
-  id: ProviderId;
+  id?: ProviderId;
   config?: any;
   prompts?: string[]; // List of prompt display strings
 }
@@ -244,7 +244,7 @@ export type ProviderId = string;
 
 export type ProviderFunction = (prompt: string) => Promise<ProviderResponse>;
 
-export type RawProviderConfig = Record<ProviderId, Omit<ProviderConfig, 'id'>>;
+export type RawProviderConfig = Record<ProviderId, ProviderConfig>;
 
 // TestSuiteConfig = Test Suite, but before everything is parsed and resolved.  Providers are just strings, prompts are filepaths, tests can be filepath or inline.
 export interface TestSuiteConfig {
diff --git a/src/util.ts b/src/util.ts
index 5a18d1c6e..000c9e75f 100644
--- a/src/util.ts
+++ b/src/util.ts
@@ -57,8 +57,10 @@ export function readProviderPromptMap(
   for (const provider of config.providers) {
     if (typeof provider === 'object') {
       const rawProvider = provider as RawProviderConfig;
-      const id = Object.keys(rawProvider)[0];
-      ret[id] = rawProvider[id].prompts || allPrompts;
+      const originalId = Object.keys(rawProvider)[0];
+      const providerObject = rawProvider[originalId];
+      const id = providerObject.id || originalId;
+      ret[id] = rawProvider[originalId].prompts || allPrompts;
     }
   }
 
@@ -456,7 +458,7 @@ export function writeLatestResults(results: EvaluateSummary, config: Partial<Uni
         2,
       ),
     );
-    if (fs.existsSync(latestResultsPath) || fs.lstatSync(latestResultsPath).isSymbolicLink()) {
+    if (fs.existsSync(latestResultsPath) && fs.lstatSync(latestResultsPath).isSymbolicLink()) {
       fs.unlinkSync(latestResultsPath);
     }
     fs.symlinkSync(newResultsPath, latestResultsPath);


Test patch:
diff --git a/test/providers.test.ts b/test/providers.test.ts
index 1dc7c018c..84ba9c91e 100644
--- a/test/providers.test.ts
+++ b/test/providers.test.ts
@@ -1,7 +1,9 @@
 import fetch from 'node-fetch';
+import { fetchJsonWithCache } from '../src/cache';
 
 import { OpenAiCompletionProvider, OpenAiChatCompletionProvider } from '../src/providers/openai';
 import { AnthropicCompletionProvider } from '../src/providers/anthropic';
+import { LlamaProvider } from '../src/providers/llama';
 
 import { disableCache, enableCache } from '../src/cache.js';
 import { loadApiProvider, loadApiProviders } from '../src/providers.js';
@@ -150,6 +152,21 @@ describe('providers', () => {
     expect(result.tokenUsage).toEqual({});
   });
 
+  test('LlamaProvider callApi', async () => {
+    const mockResponse = {
+      json: jest.fn().mockResolvedValue({
+        content: 'Test output',
+      }),
+    };
+    (fetch as unknown as jest.Mock).mockResolvedValue(mockResponse);
+
+    const provider = new LlamaProvider('llama.cpp');
+    const result = await provider.callApi('Test prompt');
+
+    expect(fetch).toHaveBeenCalledTimes(1);
+    expect(result.output).toBe('Test output');
+  });
+
   test('loadApiProvider with openai:chat', async () => {
     const provider = await loadApiProvider('openai:chat');
     expect(provider).toBeInstanceOf(OpenAiChatCompletionProvider);
@@ -190,6 +207,11 @@ describe('providers', () => {
     expect(provider).toBeInstanceOf(AnthropicCompletionProvider);
   });
 
+  test('loadApiProvider with llama:modelName', async () => {
+    const provider = await loadApiProvider('llama');
+    expect(provider).toBeInstanceOf(LlamaProvider);
+  });
+
   test('loadApiProvider with RawProviderConfig', async () => {
     const rawProviderConfig = {
       'openai:chat': {