Repository: lastmile-ai/aiconfig (language: python)

Issue:
option to add a list of outputs to config
add_output takes in a single output, it would be nice to add a list of outputs at once


Discussion / hints:
@tanya-rai @Ankush-lastmile!
I would like to pick this up. Can I have some more context for the issue?
Amazing! Yeah you should be able to. @Ankush-lastmile will provide some context today so you can get started. 
@Ankush-lastmile Any context on this?
Hey @sp6370, thanks for following up on this! I apologize for the delay. I've been tied up with a few other tasks. I'll make sure to provide you with the necessary context by the eod so you can start working on the issue. Thanks for your patience!
@Ankush-lastmile No worries! Please take your time.
Hi @sp6370! Apologies for the delay in getting back to you.

Starting with the python sdk, you can find the current method definition of `add_output` method [in schema.py](https://github.com/lastmile-ai/aiconfig/blob/5cce4ebd8eec70222f3fb7560e4c3343b5a9ee95/python/src/aiconfig/schema.py#L511)  

As you can see, this method currently accepts a single output. The goal is to extend its functionality to handle multiple outputs. A possible approach is to introduce a new method, `add_outputs`, which would be similar in design to add_output. However, instead of a single output, it would accept a list of outputs. This method would then iterate over the list, utilizing the existing `add_output` method for each individual item.

I also notice a typo in the docstring, feel free to fix that up as well!

Looking forward to your thoughts and contributions on this! Please feel free to reach out if you have any questions or need further clarification.


@Ankush-lastmile Could you elaborate what does not output do here?
https://github.com/lastmile-ai/aiconfig/blob/main/python/src/aiconfig/schema.py#L533

As per my understanding `not output` => User hasn't given anything to add to the prompt output. In that case, the function should just exit. 

If it's empty then, why are we adding it to `prompt.outputs`?

Is there something that I am missing here? 
Great observation, @sp6370! Your understanding seems correct – if 'output' is non-existent, it ideally shouldn't proceed to add anything to 'prompt.outputs'. Exiting the function or raising an error in this scenario would be a more appropriate action. 

This does look like a potential oversight in our current implementation. Your eye for this detail is much appreciated! We should consider addressing this as part of the enhancement. 

I Would recommend that we throw an exception here if Output is None, considering the fact that the Output object being passed in is _not_ optional.
Gotchya! Could you go ahead and create a issue for this and assign to me?
For the this issue, I will implement error handling as discussed above and raise the PR later today.
>Could you go ahead and create a issue for this and assign to me?

Great! Created #529 

I'm unable to assign your user to the issue for some reason at the moment, I'll try again later today. Looking forward to see your incoming pr's

@Ankush-lastmile I have sent you a message over Discord. Could you have a look at my request?
I can't seem to get the `add_output` working. 

```python
import os
from aiconfig import AIConfigRuntime
import aiconfig_extension_gemini
from aiconfig.schema import ExecuteResult

os.environ['GOOGLE_API_KEY'] = ""

config = AIConfigRuntime(**{
    "name": "Exploring Gemini",
    "description": "",
    "schema_version": "latest",
    "metadata": {
        "models": {
            "gemini-pro": {
                "stream": True
            }
        },
        "default_model": "gemini-pro",
        "model_parsers": {}
    },
    "prompts": [
        {
            "name": "multilinguality",
            "input": "Can you teach me how to say duck in Mexican?",
            "metadata": {
                "model": "gemini-pro",
                "remember_chat_context": False
            }
        },
        {
            "name": "pronounciation",
            "input": "Can you explain how to pronounce the {{language}} one?",
            "metadata": {
                "model": "gemini-pro",
                "remember_chat_context": False
            }
        }
    ]
})

config.save("gemini_demo.aiconfig.json")

from aiconfig import CallbackManager, InferenceOptions
aiconfig = AIConfigRuntime.load("gemini_demo.aiconfig.json")

test_result = ExecuteResult(
    output_type="execute_result",
    execution_count=0.0,
    data={"role": "assistant", "content": "test output"},
    metadata={"finish_reason": "stop"},
)

aiconfig.add_output("multilinguality", test_result)

#aiconfig.delete_output("multilinguality")
await aiconfig.run("multilinguality")
output_text = aiconfig.get_output_text("multilinguality")
print(f'Gemini output: {output_text} \n')
```

Output:
<img width="916" alt="image" src="https://github.com/lastmile-ai/aiconfig/assets/26037101/51c961ba-9f21-46c7-b92d-6de62b94a544">

Expected Output:
`Gemini output: Pato test output`

Can you let me know what I am missing here?
@Ankush-lastmile Any thoughts on How should I go about resolving this?
> Any thoughts on How should I go about resolving this?


I noticed your approach with the run() method in the code.  `run()` clears any previous outputs each time it's executed.

One thing you could try is adjusting the sequence of your operations. I would reccomend trying to either add the output after executing run or see if you can achieve your goal without using run at all initially.

Here's some pseudocode to illustrate:
```python
output = ExecuteResult(...)
aiconfig.add_output("multilinguality", output)

print(aiconfig.get_prompt("multilinguality")) # print prompt. Should have new output added
```

This might help in better understanding how your changes are affecting the output. Give it a shot and let me know your findings!

Fix patch:
diff --git a/python/src/aiconfig/schema.py b/python/src/aiconfig/schema.py
index df23faa43..bc5d30e5b 100644
--- a/python/src/aiconfig/schema.py
+++ b/python/src/aiconfig/schema.py
@@ -696,6 +696,29 @@ def add_output(self, prompt_name: str, output: Output, overwrite: bool = False):
         else:
             prompt.outputs.append(output)
 
+    def add_outputs(self, prompt_name: str, outputs: List[Output], overwrite: bool = False):
+        """
+        Add multiple outputs to the prompt with the given name in the AIConfig
+
+        Args:
+            prompt_name (str): The name of the prompt to add the outputs to.
+            outputs (List[Output]): List of outputs to add.
+            overwrite (bool, optional): Overwrites the existing output if True. Otherwise appends the outputs to the prompt's output list. Defaults to False.
+        """
+        prompt = self.get_prompt(prompt_name)
+        if not prompt:
+            raise IndexError(
+                f"Cannot add outputs. Prompt '{prompt_name}' not found in config."
+            )
+        if not outputs:
+            raise ValueError(
+                f"Cannot add outputs. No outputs provided for prompt '{prompt_name}'."
+            )
+        if overwrite:
+            prompt.outputs = outputs
+        else:
+            prompt.outputs.extend(outputs)
+
     def delete_output(self, prompt_name: str):
         """
         Deletes the outputs for the prompt with the given prompt_name.


Test patch:
diff --git a/python/tests/test_programmatically_create_an_AIConfig.py b/python/tests/test_programmatically_create_an_AIConfig.py
index 43bb8624b..a220ebf2f 100644
--- a/python/tests/test_programmatically_create_an_AIConfig.py
+++ b/python/tests/test_programmatically_create_an_AIConfig.py
@@ -587,6 +587,109 @@ def test_add_output_existing_prompt_no_overwrite(ai_config_runtime: AIConfigRunt
 
     assert ai_config_runtime.get_latest_output("GreetingPrompt") == None
 
+def test_add_outputs_existing_prompt_no_overwrite(ai_config_runtime: AIConfigRuntime):
+    """Test adding outputs to an existing prompt without overwriting."""
+    original_result = ExecuteResult(
+        output_type="execute_result",
+        execution_count=0,
+        data="original result",
+        metadata={
+            "raw_response": {"role": "assistant", "content": "original result"}
+        },
+    )
+    prompt = Prompt(
+        name="GreetingPrompt",
+        input="Hello, how are you?",
+        metadata=PromptMetadata(model="fakemodel"),
+        outputs=[original_result],
+    )
+    ai_config_runtime.add_prompt(prompt.name, prompt)
+    
+    assert ai_config_runtime.get_latest_output("GreetingPrompt") == original_result
+
+    test_result1 = ExecuteResult(
+        output_type="execute_result",
+        execution_count=0,
+        data="test output 1",
+        metadata={
+            "raw_response": {"role": "assistant", "content": "test output 1"}
+        },
+    )
+    test_result2 = ExecuteResult(
+        output_type="execute_result",
+        execution_count=0,
+        data="test output 2",
+        metadata={
+            "raw_response": {"role": "assistant", "content": "test output 2"}
+        },
+    )
+    ai_config_runtime.add_outputs("GreetingPrompt", [test_result1, test_result2])
+
+    assert ai_config_runtime.get_latest_output("GreetingPrompt") == test_result2
+    assert prompt.outputs == [original_result, test_result1, test_result2]
+
+def test_add_outputs_existing_prompt_with_overwrite(ai_config_runtime: AIConfigRuntime):
+    """Test adding outputs to an existing prompt with overwriting."""
+    original_result = ExecuteResult(
+        output_type="execute_result",
+        execution_count=0,
+        data="original result",
+        metadata={
+            "raw_response": {"role": "assistant", "content": "original result"}
+        },
+    )
+    prompt = Prompt(
+        name="GreetingPrompt",
+        input="Hello, how are you?",
+        metadata=PromptMetadata(model="fakemodel"),
+        outputs=[original_result],
+    )
+    ai_config_runtime.add_prompt(prompt.name, prompt)
+    
+    assert ai_config_runtime.get_latest_output("GreetingPrompt") == original_result
+
+    test_result1 = ExecuteResult(
+        output_type="execute_result",
+        execution_count=0,
+        data="test output 1",
+        metadata={
+            "raw_response": {"role": "assistant", "content": "test output 1"}
+        },
+    )
+    test_result2 = ExecuteResult(
+        output_type="execute_result",
+        execution_count=0,
+        data="test output 2",
+        metadata={
+            "raw_response": {"role": "assistant", "content": "test output 2"}
+        },
+    )
+    ai_config_runtime.add_outputs("GreetingPrompt", [test_result1, test_result2], True)
+
+    assert ai_config_runtime.get_latest_output("GreetingPrompt") == test_result2
+    assert prompt.outputs == [test_result1, test_result2]
+
+def test_add_undefined_outputs_to_prompt(ai_config_runtime: AIConfigRuntime):
+    """Test for adding undefined outputs to an existing prompt with/without overwriting. Should result in an error."""
+    prompt = Prompt(
+        name="GreetingPrompt",
+        input="Hello, how are you?",
+        metadata=PromptMetadata(model="fakemodel"),
+    )
+    ai_config_runtime.add_prompt(prompt.name, prompt)
+    assert ai_config_runtime.get_latest_output("GreetingPrompt") == None
+    # Case 1: No outputs, overwrite param not defined
+    with pytest.raises(
+        ValueError,
+        match=r"Cannot add outputs. No outputs provided for prompt 'GreetingPrompt'.",
+    ):
+        ai_config_runtime.add_outputs("GreetingPrompt", [])
+    # Case 2: No outputs, overwrite param set to True
+    with pytest.raises(
+        ValueError,
+        match=r"Cannot add outputs. No outputs provided for prompt 'GreetingPrompt'.",
+    ):
+        ai_config_runtime.add_outputs("GreetingPrompt", [], True)
 
 def test_extract_override_settings(ai_config_runtime: AIConfigRuntime):
     initial_settings = {"topP": 0.9}