Repository: lastmile-ai/aiconfig (language: python)

Issue:
[AIC-py][eval] refactor enough to mock aiconfig and add more testing
[AIC-py][eval] refactor enough to mock aiconfig and add more testing

---
Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/lastmile-ai/aiconfig/pull/430).
* #432
* __->__ #430
* #428
* #426


Discussion / hints:

Fix patch:
diff --git a/python/requirements.txt b/python/requirements.txt
index 84e49e072..80b58143d 100644
--- a/python/requirements.txt
+++ b/python/requirements.txt
@@ -14,5 +14,5 @@ nest_asyncio
 prompt_toolkit
 mock
 pytest-asyncio
-lastmile-utils==0.0.7
+lastmile-utils==0.0.8
 hypothesis==6.91.0
\ No newline at end of file
diff --git a/python/src/aiconfig/eval/README.md b/python/src/aiconfig/eval/README.md
index e1e30ec76..af68ddfdc 100644
--- a/python/src/aiconfig/eval/README.md
+++ b/python/src/aiconfig/eval/README.md
@@ -15,7 +15,7 @@ Inputs:
 # The first element of each pair is the test input.
 # Each one will get put into "the_query" in the input prompt:
 # "Tell me 3 fun attractions related to {{the_query}} to do in NYC."
-# See the aiconfig (python/src/aiconfig/eval/custom_eval/examples/travel/travel_parametrized.aiconfig.json).
+# See the aiconfig (https://github.com/lastmile-ai/aiconfig/blob/9669b9e5614d5b1b0603ddf3ec61556cd79bfc14/python/src/aiconfig/eval/examples/travel/travel_parametrized.aiconfig.json#L4).
 test_inputs_with_substrings = [
     ("different kinds of cuisines", "Magnolia Bakery"),
     ("Iconic midtown skyscrapers", "Empire State Building"),
@@ -41,7 +41,7 @@ Output:
 
 Make sure AIConfig library is installed (pip3 install python-aiconfig).
 
-Usage: Please see [this notebook](https://github.com/lastmile-ai/aiconfig/blob/d79cf7dadf934b0ce09bc671cfde37aaecf05c1e/python/src/aiconfig/eval/custom_eval/examples/travel/travel_eval.ipynb#L4) for an example of running evaluation on an existing AIConfig. Run the notebook cells in order. This should be sufficient to get started, but the sections below provide more detail.
+Usage: Please see [this notebook](https://github.com/lastmile-ai/aiconfig/blob/9669b9e5614d5b1b0603ddf3ec61556cd79bfc14/python/src/aiconfig/eval/examples/travel/travel_eval.ipynb#L4) for an example of running evaluation on an existing AIConfig. Run the notebook cells in order. This should be sufficient to get started, but the sections below provide more detail.
 
 ## Custom Eval Usage Steps
 
@@ -50,7 +50,7 @@ Let's take a closer look at what we're testing and how. The travel example linke
 `the_query -> get_activities.input -> get_activities.output -> gen_itinerary.input -> gen_itinerary.output`.
 
 The leaf "the_query" is the placeholder for test case inputs, like a function argument. The root "gen_itinerary.output" is the value that will be tested, like the function's return value. You must specify exactly one leaf input and it must be named "the_query". You must also specify the name of the prompt you want to test (see test suite settings, `ts_settings = TestSuiteWithInputsSettings...
-` in the [notebook](https://github.com/lastmile-ai/aiconfig/blob/d79cf7dadf934b0ce09bc671cfde37aaecf05c1e/python/src/aiconfig/eval/custom_eval/examples/travel/travel_eval.ipynb#L4)).
+` in the [notebook](https://github.com/lastmile-ai/aiconfig/blob/9669b9e5614d5b1b0603ddf3ec61556cd79bfc14/python/src/aiconfig/eval/examples/travel/travel_eval.ipynb#L4)).
 
 Zooming in on the edges, each of which is either prompt string formatting or model execution:
 
@@ -97,13 +97,13 @@ You should get one passing test and one failing test:
 
 This is very similar to the custom eval case, but leverages Promptfoo. The idea is to define an AIConfig as a custom provider that you will specify in your promptfoo file. The AIConfig library contains a run script that appears as a black box to Promptfoo and therefore can easily integrate with Promptfoo as long as you provide a small amount of extra configuration. Essentially, you make a Promptfoo test yaml file with a custom AIConfig provider that points to your test suite settings. Details below.
 
-1. Write your test cases in a Promptfoo config file. ([Example](https://github.com/lastmile-ai/aiconfig/blob/d79cf7dadf934b0ce09bc671cfde37aaecf05c1e/python/src/aiconfig/eval/promptfoo/examples/travel/travel_promptfooconfig.yaml)).
+1. Write your test cases in a Promptfoo config file. ([Example](https://github.com/lastmile-ai/aiconfig/blob/9669b9e5614d5b1b0603ddf3ec61556cd79bfc14/python/src/aiconfig/eval/examples/travel/travel_promptfoo_config.yaml#L4)).
 
 - Note that the `prompts` list must contain one literal prompt (in this case, "question") which is then assigned test values in your Promptfoo test cases (see `vars`). This prompt variable is just a placeholder for Promptfoo's use and is completely distinct from the prompts in your AIConfig. Think of it roughly like a function parameter you need to call the Promptfoo helper function.
 
-2. Define an AIConfig test suite settings file. It must have the name of the AIConfig prompt you want to test and the file path to your aiconfig. ([Example](https://github.com/lastmile-ai/aiconfig/blob/d79cf7dadf934b0ce09bc671cfde37aaecf05c1e/python/src/aiconfig/eval/promptfoo/examples/travel/travel_aiconfig_test_suite_settings.json)).
+2. Define an AIConfig test suite settings file. It must have the name of the AIConfig prompt you want to test and the file path to your aiconfig. ([Example](https://github.com/lastmile-ai/aiconfig/blob/9669b9e5614d5b1b0603ddf3ec61556cd79bfc14/python/src/aiconfig/eval/examples/travel/travel_aiconfig_test_suite_settings.json#L3)).
 
-3. Put one item in `providers` that runs the AIConfig run script with the path to your test suite settings file ([Example](https://github.com/lastmile-ai/aiconfig/blob/d79cf7dadf934b0ce09bc671cfde37aaecf05c1e/python/src/aiconfig/eval/promptfoo/examples/travel/travel_promptfooconfig.yaml#L3)).
+3. Put one item in `providers` that runs the AIConfig run script with the path to your test suite settings file ([Example](https://github.com/lastmile-ai/aiconfig/blob/9669b9e5614d5b1b0603ddf3ec61556cd79bfc14/python/src/aiconfig/eval/examples/travel/travel_promptfoo_config.yaml#L4)).
 
 4. Make sure your API key(s) are available as environment variables, including to subshell processes. This can be done using `export`. See [here](https://github.com/lastmile-ai/aiconfig/#set-your-openai-api-key) for more detail.
 
diff --git a/python/src/aiconfig/eval/common.py b/python/src/aiconfig/eval/common.py
index b2aa7c855..7d6f270e3 100644
--- a/python/src/aiconfig/eval/common.py
+++ b/python/src/aiconfig/eval/common.py
@@ -52,6 +52,9 @@ def check_value_range(cls, values: dict[str, Any]) -> dict[str, Any]:
         if worst_value < best_value and not worst_value <= value <= best_value:
             raise ValueError(
                 f"""
+                    [{values["interpretation"].name}]
+                    {values["interpretation"].description}
+
                     Value {value} is not in range [{worst_value}, {best_value}]. 
                     You defined worst_value = {worst_value} and best_value = {best_value},
                     but got value outside that range.
@@ -60,6 +63,9 @@ def check_value_range(cls, values: dict[str, Any]) -> dict[str, Any]:
         if worst_value > best_value and not worst_value >= value >= best_value:
             raise ValueError(
                 f"""
+                    [{values["interpretation"].name}]
+                    {values["interpretation"].description}
+
                     Value {value} is not in range [{worst_value}, {best_value}]. 
                     You defined worst_value = {worst_value} and best_value = {best_value},
                     but got value outside that range.
diff --git a/python/src/aiconfig/eval/examples/travel/travel_eval.ipynb b/python/src/aiconfig/eval/examples/travel/travel_eval.ipynb
index 9991ed4e0..b83491a83 100644
--- a/python/src/aiconfig/eval/examples/travel/travel_eval.ipynb
+++ b/python/src/aiconfig/eval/examples/travel/travel_eval.ipynb
@@ -162,22 +162,22 @@
       "Test input:\n",
       " different kinds of cuisines \n",
       "Function:\n",
-      " <function brevity at 0x16ab927a0>\n",
+      " <function brevity at 0x15d4195a0>\n",
       "\n",
       "Test input:\n",
       " different kinds of cuisines \n",
       "Function:\n",
-      " functools.partial(<function contains_substring at 0x16ab92710>, substring='Magnolia Bakery', case_sensitive=False)\n",
+      " functools.partial(<function contains_substring at 0x15d419480>, substring='Magnolia Bakery', case_sensitive=False)\n",
       "\n",
       "Test input:\n",
       " iconic midtown skyscrapers \n",
       "Function:\n",
-      " <function brevity at 0x16ab927a0>\n",
+      " <function brevity at 0x15d4195a0>\n",
       "\n",
       "Test input:\n",
       " iconic midtown skyscrapers \n",
       "Function:\n",
-      " functools.partial(<function contains_substring at 0x16ab92710>, substring='Empire State Building', case_sensitive=False)\n"
+      " functools.partial(<function contains_substring at 0x15d419480>, substring='Empire State Building', case_sensitive=False)\n"
      ]
     }
    ],
@@ -235,8 +235,10 @@
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td>different kinds of cuisines</td>\n",
-       "      <td>Visit Chelsea Market for global cuisine exploration, stop by Smorgasburg for diverse street food, end at Eataly for immersive Italian experience.</td>\n",
-       "      <td>145.0</td>\n",
+       "      <td>1. Start at Top of the Rock for skyline views.\n",
+       "2. Visit Empire State Building, admire city panoramas. \n",
+       "3. Explore Grand Central Terminal, appreciate architecture, dine, shop.</td>\n",
+       "      <td>174.0</td>\n",
        "      <td>brevity</td>\n",
        "      <td>Absolute text length</td>\n",
        "      <td>1.0</td>\n",
@@ -245,7 +247,9 @@
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td>different kinds of cuisines</td>\n",
-       "      <td>Visit Chelsea Market for global cuisine exploration, stop by Smorgasburg for diverse street food, end at Eataly for immersive Italian experience.</td>\n",
+       "      <td>1. Start at Top of the Rock for skyline views.\n",
+       "2. Visit Empire State Building, admire city panoramas. \n",
+       "3. Explore Grand Central Terminal, appreciate architecture, dine, shop.</td>\n",
        "      <td>0.0</td>\n",
        "      <td>contains_substring</td>\n",
        "      <td>1.0 (pass) if contains given substring</td>\n",
@@ -255,8 +259,10 @@
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td>iconic midtown skyscrapers</td>\n",
-       "      <td>Day 1: Visit Empire State Building; revel in architecture, history, views. Day 2: Explore Rockefeller Center; admire city panoramas, enjoy other attractions. Day 3: Tour Chrysler Building, appreciate art, stop at Chase Manhattan Plaza.</td>\n",
-       "      <td>235.0</td>\n",
+       "      <td>1. Start at Top of the Rock for skyline views.\n",
+       "2. Visit Empire State Building, admire city panoramas. \n",
+       "3. Explore Grand Central Terminal, appreciate architecture, dine, shop.</td>\n",
+       "      <td>174.0</td>\n",
        "      <td>brevity</td>\n",
        "      <td>Absolute text length</td>\n",
        "      <td>1.0</td>\n",
@@ -265,7 +271,9 @@
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td>iconic midtown skyscrapers</td>\n",
-       "      <td>Day 1: Visit Empire State Building; revel in architecture, history, views. Day 2: Explore Rockefeller Center; admire city panoramas, enjoy other attractions. Day 3: Tour Chrysler Building, appreciate art, stop at Chase Manhattan Plaza.</td>\n",
+       "      <td>1. Start at Top of the Rock for skyline views.\n",
+       "2. Visit Empire State Building, admire city panoramas. \n",
+       "3. Explore Grand Central Terminal, appreciate architecture, dine, shop.</td>\n",
        "      <td>1.0</td>\n",
        "      <td>contains_substring</td>\n",
        "      <td>1.0 (pass) if contains given substring</td>\n",
@@ -283,16 +291,24 @@
        "2   iconic midtown skyscrapers   \n",
        "3   iconic midtown skyscrapers   \n",
        "\n",
-       "                                                                                                                                                                                                                               aiconfig_output  \\\n",
-       "0                                                                                            Visit Chelsea Market for global cuisine exploration, stop by Smorgasburg for diverse street food, end at Eataly for immersive Italian experience.   \n",
-       "1                                                                                            Visit Chelsea Market for global cuisine exploration, stop by Smorgasburg for diverse street food, end at Eataly for immersive Italian experience.   \n",
-       "2  Day 1: Visit Empire State Building; revel in architecture, history, views. Day 2: Explore Rockefeller Center; admire city panoramas, enjoy other attractions. Day 3: Tour Chrysler Building, appreciate art, stop at Chase Manhattan Plaza.   \n",
-       "3  Day 1: Visit Empire State Building; revel in architecture, history, views. Day 2: Explore Rockefeller Center; admire city panoramas, enjoy other attractions. Day 3: Tour Chrysler Building, appreciate art, stop at Chase Manhattan Plaza.   \n",
+       "                                                                                                                                                                  aiconfig_output  \\\n",
+       "0  1. Start at Top of the Rock for skyline views.\n",
+       "2. Visit Empire State Building, admire city panoramas. \n",
+       "3. Explore Grand Central Terminal, appreciate architecture, dine, shop.   \n",
+       "1  1. Start at Top of the Rock for skyline views.\n",
+       "2. Visit Empire State Building, admire city panoramas. \n",
+       "3. Explore Grand Central Terminal, appreciate architecture, dine, shop.   \n",
+       "2  1. Start at Top of the Rock for skyline views.\n",
+       "2. Visit Empire State Building, admire city panoramas. \n",
+       "3. Explore Grand Central Terminal, appreciate architecture, dine, shop.   \n",
+       "3  1. Start at Top of the Rock for skyline views.\n",
+       "2. Visit Empire State Building, admire city panoramas. \n",
+       "3. Explore Grand Central Terminal, appreciate architecture, dine, shop.   \n",
        "\n",
        "   value         metric_name                      metric_description  \\\n",
-       "0  145.0             brevity                    Absolute text length   \n",
+       "0  174.0             brevity                    Absolute text length   \n",
        "1    0.0  contains_substring  1.0 (pass) if contains given substring   \n",
-       "2  235.0             brevity                    Absolute text length   \n",
+       "2  174.0             brevity                    Absolute text length   \n",
        "3    1.0  contains_substring  1.0 (pass) if contains given substring   \n",
        "\n",
        "   best_possible_value  worst_possible_value  \n",
@@ -366,14 +382,14 @@
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>different kinds of cuisines</th>\n",
-       "      <th>Visit Chelsea Market for global cuisine exploration, stop by Smorgasburg for diverse street food, end at Eataly for immersive Italian experience.</th>\n",
-       "      <td>145.0</td>\n",
+       "      <th>1. Start at Top of the Rock for skyline views.\\n2. Visit Empire State Building, admire city panoramas. \\n3. Explore Grand Central Terminal, appreciate architecture, dine, shop.</th>\n",
+       "      <td>174.0</td>\n",
        "      <td>0.0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>iconic midtown skyscrapers</th>\n",
-       "      <th>Day 1: Visit Empire State Building; revel in architecture, history, views. Day 2: Explore Rockefeller Center; admire city panoramas, enjoy other attractions. Day 3: Tour Chrysler Building, appreciate art, stop at Chase Manhattan Plaza.</th>\n",
-       "      <td>235.0</td>\n",
+       "      <th>1. Start at Top of the Rock for skyline views.\\n2. Visit Empire State Building, admire city panoramas. \\n3. Explore Grand Central Terminal, appreciate architecture, dine, shop.</th>\n",
+       "      <td>174.0</td>\n",
        "      <td>1.0</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
@@ -381,15 +397,15 @@
        "</div>"
       ],
       "text/plain": [
-       "metric_name                                                                                                                                                                                                                                                              brevity  \\\n",
-       "input                       aiconfig_output                                                                                                                                                                                                                                        \n",
-       "different kinds of cuisines Visit Chelsea Market for global cuisine exploration, stop by Smorgasburg for diverse street food, end at Eataly for immersive Italian experience.                                                                                              145.0   \n",
-       "iconic midtown skyscrapers  Day 1: Visit Empire State Building; revel in architecture, history, views. Day 2: Explore Rockefeller Center; admire city panoramas, enjoy other attractions. Day 3: Tour Chrysler Building, appreciate art, stop at Chase Manhattan Plaza.    235.0   \n",
+       "metric_name                                                                                                                                                                                                   brevity  \\\n",
+       "input                       aiconfig_output                                                                                                                                                                             \n",
+       "different kinds of cuisines 1. Start at Top of the Rock for skyline views.\\n2. Visit Empire State Building, admire city panoramas. \\n3. Explore Grand Central Terminal, appreciate architecture, dine, shop.    174.0   \n",
+       "iconic midtown skyscrapers  1. Start at Top of the Rock for skyline views.\\n2. Visit Empire State Building, admire city panoramas. \\n3. Explore Grand Central Terminal, appreciate architecture, dine, shop.    174.0   \n",
        "\n",
-       "metric_name                                                                                                                                                                                                                                                              contains_substring  \n",
-       "input                       aiconfig_output                                                                                                                                                                                                                                                  \n",
-       "different kinds of cuisines Visit Chelsea Market for global cuisine exploration, stop by Smorgasburg for diverse street food, end at Eataly for immersive Italian experience.                                                                                                           0.0  \n",
-       "iconic midtown skyscrapers  Day 1: Visit Empire State Building; revel in architecture, history, views. Day 2: Explore Rockefeller Center; admire city panoramas, enjoy other attractions. Day 3: Tour Chrysler Building, appreciate art, stop at Chase Manhattan Plaza.                 1.0  "
+       "metric_name                                                                                                                                                                                                   contains_substring  \n",
+       "input                       aiconfig_output                                                                                                                                                                                       \n",
+       "different kinds of cuisines 1. Start at Top of the Rock for skyline views.\\n2. Visit Empire State Building, admire city panoramas. \\n3. Explore Grand Central Terminal, appreciate architecture, dine, shop.                 0.0  \n",
+       "iconic midtown skyscrapers  1. Start at Top of the Rock for skyline views.\\n2. Visit Empire State Building, admire city panoramas. \\n3. Explore Grand Central Terminal, appreciate architecture, dine, shop.                 1.0  "
       ]
      },
      "execution_count": 5,
@@ -476,22 +492,22 @@
       "Test output:\n",
       " Begin at Chelsea Market for diverse food options. Continue to Queens for immersive food tours. Conclude at Smorgasburg for unique outdoor food market experience \n",
       "Function:\n",
-      " <function brevity at 0x16ab927a0>\n",
+      " <function brevity at 0x15d4195a0>\n",
       "\n",
       "Test output:\n",
       " Begin at Chelsea Market for diverse food options. Continue to Queens for immersive food tours. Conclude at Smorgasburg for unique outdoor food market experience \n",
       "Function:\n",
-      " functools.partial(<function contains_substring at 0x16ab92710>, substring='Magnolia Bakery', case_sensitive=False)\n",
+      " functools.partial(<function contains_substring at 0x15d419480>, substring='Magnolia Bakery', case_sensitive=False)\n",
       "\n",
       "Test output:\n",
       " 1. Empire State Building: Observation deck visit, explore exhibits and historical displays. 2. Rockefeller Center: Visit \"Top of the Rock\", ice-skating, NBC Studio tour, shopping and dining. 3. Chrysler Building: Admire exterior and iconic spire, photo opportunities. \n",
       "Function:\n",
-      " <function brevity at 0x16ab927a0>\n",
+      " <function brevity at 0x15d4195a0>\n",
       "\n",
       "Test output:\n",
       " 1. Empire State Building: Observation deck visit, explore exhibits and historical displays. 2. Rockefeller Center: Visit \"Top of the Rock\", ice-skating, NBC Studio tour, shopping and dining. 3. Chrysler Building: Admire exterior and iconic spire, photo opportunities. \n",
       "Function:\n",
-      " functools.partial(<function contains_substring at 0x16ab92710>, substring='Empire State Building', case_sensitive=False)\n"
+      " functools.partial(<function contains_substring at 0x15d419480>, substring='Empire State Building', case_sensitive=False)\n"
      ]
     }
    ],
diff --git a/python/src/aiconfig/eval/lib.py b/python/src/aiconfig/eval/lib.py
index c7e3c4839..4e089162c 100644
--- a/python/src/aiconfig/eval/lib.py
+++ b/python/src/aiconfig/eval/lib.py
@@ -1,4 +1,3 @@
-import asyncio
 import logging
 from dataclasses import dataclass
 from typing import Generic, NewType, Sequence, Tuple, TypeVar
@@ -35,8 +34,13 @@ async def run_test_suite_with_inputs(
     test_suite: UserTestSuiteWithInputs,
     settings: TestSuiteWithInputsSettings,
 ) -> pd.DataFrame:
+    aiconfig = AIConfigRuntime.load(settings.aiconfig_path)  # type: ignore[fixme, no-untyped-call]
     res = await run_test_suite_helper(
-        TestSuiteWithInputsSpec(test_suite=test_suite, settings=settings)
+        TestSuiteWithInputsSpec(
+            test_suite=test_suite,
+            prompt_name=settings.prompt_name,
+            aiconfig=aiconfig,
+        )
     )
     return res.unwrap_or_raise(ValueError)
 
@@ -133,21 +137,9 @@ def eval_res_to_df(
     return df
 
 
-async def run_aiconfig(
-    settings: TestSuiteWithInputsSettings, input_datum: TextInput
-) -> TextOutput:
-    """Helper to run the AIConfig which makes the data to be evaluated."""
-    prompt_name = settings.prompt_name
-    aiconfig_path = settings.aiconfig_path
-    aiconfig = AIConfigRuntime.load(aiconfig_path)  # type: ignore[fixme, no-untyped-call]
-
-    output = await run_aiconfig_helper(aiconfig, prompt_name, input_datum)
-    return output.map(TextOutput).unwrap()
-
-
 async def user_test_suite_with_inputs_to_eval_params_list(
-    test_suite: UserTestSuiteWithInputs, settings: TestSuiteWithInputsSettings
-) -> Sequence[SampleEvaluationParams[TextInput, TextOutput]]:
+    test_suite: UserTestSuiteWithInputs, prompt_name: str, aiconfig: AIConfigRuntime
+) -> Result[Sequence[SampleEvaluationParams[TextInput, TextOutput]], str]:
     out: list[SampleEvaluationParams[TextInput, TextOutput]] = []
     grouped: dict[str, list[SampleEvaluationFunction[str]]] = {}
     for input_datum, eval_fn in test_suite:
@@ -156,28 +148,35 @@ async def user_test_suite_with_inputs_to_eval_params_list(
         grouped[input_datum].append(eval_fn)
 
     all_inputs = list(grouped.keys())
-    outputs = await asyncio.gather(
-        *(run_aiconfig(settings, TextInput(input_datum)) for input_datum in all_inputs)
-    )
 
-    # This zip is safe because we have defined an order for the keys in `all_inputs`
-    # them apped run_aiconfig over that list.
-    # Docs: https://docs.python.org/3/library/asyncio-task.html#running-tasks-concurrently
-    #  > If all awaitables are completed successfully, the result is an aggregate list
-    # of returned values. The order of result values corresponds to the order
-    # of awaitables in aws.
-    outputs_by_input = dict(zip(all_inputs, outputs))
-
-    for input_datum, eval_fns in grouped.items():
-        for eval_fn in eval_fns:
-            out.append(
-                SampleEvaluationParams(
-                    input_sample=TextInput(input_datum),
-                    output_sample=outputs_by_input[input_datum],
-                    evaluation_fn=eval_fn,
+    async def _run(input_datum: str) -> Result[TextOutput, str]:
+        return (await run_aiconfig_helper(aiconfig, prompt_name, input_datum)).map(
+            TextOutput
+        )
+
+    res_outputs = await cu.result_reduce_list_all_ok_async(list(map(_run, all_inputs)))
+
+    def _zip_inputs_outputs(outputs: list[TextOutput]):
+        # This zip is safe because we have defined an order for the keys in `all_inputs`
+        # them apped run_aiconfig over that list.
+        # Docs: https://docs.python.org/3/library/asyncio-task.html#running-tasks-concurrently
+        #  > If all awaitables are completed successfully, the result is an aggregate list
+        # of returned values. The order of result values corresponds to the order
+        # of awaitables in aws.
+        outputs_by_input = dict(zip(all_inputs, outputs))
+
+        for input_datum, eval_fns in grouped.items():
+            for eval_fn in eval_fns:
+                out.append(
+                    SampleEvaluationParams(
+                        input_sample=TextInput(input_datum),
+                        output_sample=outputs_by_input[input_datum],
+                        evaluation_fn=eval_fn,
+                    )
                 )
-            )
-    return out
+        return out
+
+    return res_outputs.map(_zip_inputs_outputs)
 
 
 def user_test_suite_outputs_only_to_eval_params_list(
@@ -209,7 +208,8 @@ async def run_aiconfig_helper(
 @dataclass(frozen=True)
 class TestSuiteWithInputsSpec:
     test_suite: UserTestSuiteWithInputs
-    settings: TestSuiteWithInputsSettings
+    prompt_name: str
+    aiconfig: AIConfigRuntime
 
 
 @dataclass(frozen=True)
@@ -225,15 +225,16 @@ async def run_test_suite_helper(
 ) -> Result[pd.DataFrame, str]:
     async def _get_eval_params_list(
         test_suite_spec: TestSuiteSpec,
-    ) -> Sequence[SampleEvaluationParams[TextInput, TextOutput]]:
+    ) -> Result[Sequence[SampleEvaluationParams[TextInput, TextOutput]], str]:
         match test_suite_spec:
-            case TestSuiteWithInputsSpec(test_suite=test_suite, settings=settings):
+            case TestSuiteWithInputsSpec(
+                test_suite=test_suite, prompt_name=prompt_name, aiconfig=aiconfig
+            ):
                 return await user_test_suite_with_inputs_to_eval_params_list(
-                    test_suite, settings
+                    test_suite, prompt_name, aiconfig
                 )
             case TestSuiteOutputsOnlySpec(test_suite=test_suite):
-                return user_test_suite_outputs_only_to_eval_params_list(test_suite)
+                return Ok(user_test_suite_outputs_only_to_eval_params_list(test_suite))
 
     eval_params_list = await _get_eval_params_list(test_suite_spec)
-    eval_res = evaluate(eval_params_list)
-    return eval_res.map(eval_res_to_df)
+    return eval_params_list.and_then(evaluate).map(eval_res_to_df)


Test patch:
diff --git a/python/tests/test_eval.py b/python/tests/test_eval.py
index f523e935b..f8f486903 100644
--- a/python/tests/test_eval.py
+++ b/python/tests/test_eval.py
@@ -1,17 +1,47 @@
+import os
+from typing import Any
 import hypothesis.strategies as st
 import lastmile_utils.lib.core.api as cu
 import pandas as pd
 import pytest
-from result import Ok
+from result import Err, Ok
+from aiconfig.Config import AIConfigRuntime
 from aiconfig.eval.api import (
     TestSuiteWithInputsSettings,
     brevity,
-    run_test_suite_with_inputs,
     substring_match,
+    run_test_suite_with_inputs,
+    run_test_suite_outputs_only,
 )
-from aiconfig.eval.lib import run_test_suite_outputs_only
+from aiconfig.eval.lib import TestSuiteWithInputsSpec, run_test_suite_helper
+
 from hypothesis import given
 
+from aiconfig.model_parser import InferenceOptions
+
+
+def current_dir():
+    return os.path.dirname(os.path.realpath(__file__))
+
+
+class MockAIConfigRuntime(AIConfigRuntime):
+    def __init__(self):
+        pass
+
+    async def run_and_get_output_text(
+        self,
+        prompt_name: str,
+        params: dict[Any, Any] | None = None,
+        options: InferenceOptions | None = None,
+        **kwargs,
+    ) -> str:
+        params_ = params or {}
+        assert params_.keys() == {
+            "the_query"
+        }, 'For eval, AIConfig params must have just the key "the_query".'
+        the_query = params_["the_query"]
+        return f"output_for_{prompt_name}_the_query_{the_query}"
+
 
 def test_metrics():
     assert brevity("hello").value == 5.0
@@ -23,13 +53,17 @@ def test_metrics():
 
 @pytest.mark.asyncio
 async def test_run_with_inputs_sanity_check():
-    """No easy way to mock LLM calls,
-    so just test the imports and sanity check output."""
+    """No easy way to mock LLM calls from outside run_test_suite_with_inputs.
+
+    Instead, give empty list and just test the imports and sanity check output."""
+
+    path = os.path.join(
+        current_dir(),
+        "../src/aiconfig/eval/examples/travel/travel_parametrized.aiconfig.json",
+    )
     out = await run_test_suite_with_inputs(
         [],
-        TestSuiteWithInputsSettings(
-            prompt_name="test", aiconfig_path="some/path/we/wont/read"
-        ),
+        TestSuiteWithInputsSettings(prompt_name="test", aiconfig_path=path),
     )
     assert isinstance(out, pd.DataFrame)
     assert out.shape == (0, 0)
@@ -53,18 +87,18 @@ async def test_run_with_outputs_only_basic():
 @given(st.data())
 @pytest.mark.asyncio
 async def test_run_test_suite_outputs_only(data: st.DataObject):
-    Metrics = [brevity, substring_match("hello")]
-    TestPairs = st.tuples(st.text(min_size=1), st.sampled_from(Metrics))
+    metrics = [brevity, substring_match("hello")]
+    test_pairs = st.tuples(st.text(min_size=1), st.sampled_from(metrics))
     user_test_suite_outputs_only = data.draw(
         st.lists(
-            TestPairs,
+            test_pairs,
             min_size=1,
         )
     )
 
-    out = await run_test_suite_outputs_only(user_test_suite_outputs_only)
-    assert out.shape[0] == (len(user_test_suite_outputs_only))
-    assert out.columns.tolist() == [
+    df = await run_test_suite_outputs_only(user_test_suite_outputs_only)
+    assert df.shape[0] == (len(user_test_suite_outputs_only))
+    assert df.columns.tolist() == [
         "input",
         "aiconfig_output",
         "value",
@@ -73,11 +107,68 @@ async def test_run_test_suite_outputs_only(data: st.DataObject):
         "best_possible_value",
         "worst_possible_value",
     ]
-    inputs = out["input"].astype(str).tolist()  # type: ignore[no-untyped-call]
+    inputs = df["input"].astype(str).tolist()  # type: ignore[no-untyped-call]
     assert cu.only(inputs) == Ok("Missing")
 
-    out_brevity = out[out["metric_name"] == "brevity"]
+    df_brevity = df[df["metric_name"] == "brevity"]
     assert (
-        out_brevity["aiconfig_output"].apply(len)  # type: ignore[no-untyped-call]
-        == out_brevity["value"]  # type: ignore[no-untyped-call]
+        df_brevity["aiconfig_output"].apply(len)  # type: ignore[no-untyped-call]
+        == df_brevity["value"]  # type: ignore[no-untyped-call]
     ).all()
+
+
+@given(st.data())
+@pytest.mark.asyncio
+async def test_run_test_suite_with_inputs(data: st.DataObject):
+    """In test_run_test_suite_outputs_only, we test the user-facing function (e2e)
+    In this case that's harder because run_test_suite_with_inputs takes
+    an aiconfig path, not object.
+    In order to test with a mock AIConfig object, in this test we go one level down
+    and test run_test_suite_helper().
+
+    Also see test_run_with_inputs_sanity_check.
+    """
+    metrics = [brevity, substring_match("hello")]
+    test_pairs = st.tuples(st.text(min_size=1), st.sampled_from(metrics))
+    user_test_suite_with_inputs = data.draw(
+        st.lists(
+            test_pairs,
+            min_size=1,
+        )
+    )
+
+    input_data, _ = cu.unzip(user_test_suite_with_inputs)
+
+    mock_aiconfig = MockAIConfigRuntime()
+
+    out = await run_test_suite_helper(
+        TestSuiteWithInputsSpec(
+            test_suite=user_test_suite_with_inputs,
+            prompt_name="prompt0",
+            aiconfig=mock_aiconfig,
+        )
+    )
+
+    match out:
+        case Ok(df):
+            assert isinstance(df, pd.DataFrame)
+            assert df.shape[0] == (len(user_test_suite_with_inputs))
+            assert df.columns.tolist() == [
+                "input",
+                "aiconfig_output",
+                "value",
+                "metric_name",
+                "metric_description",
+                "best_possible_value",
+                "worst_possible_value",
+            ]
+            inputs = df["input"].astype(str).tolist()  # type: ignore[no-untyped-call]
+            assert set(df["input"]) == set(input_data)  # type: ignore[no-untyped-call]
+
+            df_brevity = df[df["metric_name"] == "brevity"]
+            assert (
+                df_brevity["aiconfig_output"].apply(len)  # type: ignore[no-untyped-call]
+                == df_brevity["value"]  # type: ignore[no-untyped-call]
+            ).all()
+        case Err(e):
+            assert False, f"expected Ok, got Err({e})"