Repository: stanfordnlp/dspy

Issue:
Error when running Evaluate:  'str' object has no attribute '_store'
I am trying out DSPy 2.3.6 with a simple transcript summarization example:

```python
class Summarizer(dspy.Module):
    def __init__(self):
        super().__init__()
        self.summarizer = dspy.Predict('transcript -> summary')
        
    def forward(self, transcript):
        return dspy.Prediction(summary=self.summarizer(transcript=transcript))
```

I created a number of test examples, with:

```python
testset=[x.with_inputs('transcript') for x in [
    dspy.Example(transcript='...', summary='...'),
    ...
]]
```

I also created a metric function, which I have verified is working as expected.

However, when I run this code:

```
evaluate = dspy.evaluate.Evaluate(devset=testset, metric=metric, num_threads=1, display_progress=True, display_table=True)
evaluate(Summarizer())
```

I get this error, repeating once for each of test case:

```
Error for example in dev set: 		 'str' object has no attribute '_store'
```

I have verified that the test cases in `testset` each have a `_store` attribute, so its not that. So I'm a bit stumped, and hoping someone has the answer before I go step through all the code line by line.


 


Discussion:
Okay, so bizzarely I have narrowed it down to the metric. This is despite the fact that I can manually call my metric function and it works, but for some reason when called from the optimizer it doesn't work. Code is based pretty much on the DSPy documentation [here](https://dspy-docs.vercel.app/docs/building-blocks/metrics#intermediate-using-ai-feedback-for-your-metric). Here's a fairly minimal version:

```python
class Assess(dspy.Signature):
    """Assess the quality of a summary of a transcript along the specified dimension."""
    summary = dspy.InputField()
    assessment_question = dspy.InputField()
    assessment_answer = dspy.OutputField(desc="No or Yes")
    

def metric(example: dspy.Example, prediction, trace=None):
        
    transcript, answer, summary = example.transcript, example.summary, prediction.summary
    
    with dspy.context(lm=ollama):
        # This next line is the one that results in the error when called from the optimizer.
        content_eval = dspy.Predict(Assess)(summary=summary, assessment_question=\
                            f"Is the assessed text a good summary of this transcript, capturing all the important details?\n\n{transcript}?")
    return content_eval.to_lower().endswith('yes')
```
I have narrowed the problem down to this line: https://github.com/stanfordnlp/dspy/blob/1d300c834b6db20847196adb276c6c988f3674b6/dsp/templates/template_v2.py#L81

This code seems to be assuming that the dictionary values are strings, but it turns out they can be instances of `Prediction`, which then have an overloaded `__eq__` operator that tries to compare `_store` contents - but of course `""` does not have a `_store` property.

```python
            has_value = [
                field.input_variable in example
                and example[field.input_variable] is not None
                and example[field.input_variable]) != ""
                for field in self.fields
            ]
```

I tried a naive fix of casting the LHS to a `str` but that just causes a blowup later. I don't know enough about the innards of this library to make further progress.

@gramster : did you try CoT instead of Predict ? I remember testing that as well but forgot the outcome.. (just to un-block you until the bug is fixed)
Good idea, but it still results in the same error.
I think I may have found the issue. In my Summarizer.forward method, I am returning a `Prediction`. I based my code off the RAG sample in the tutorial in the docs here: https://dspy-docs.vercel.app/docs/tutorials/rag

This seems to be the problem. If I just return the result from `dspy.ChainOfThought` (or presumably also `dspy.Predict`), I get through the evaluation stage. 

Perhaps the turtorial is out of date.

Also I had to remove the `display_table=True` argument to `Evaluate` as that threw an exception in pandas.
Also ran in to the same problem. @gramster  Did you end up not using the module set up of init and forward set up but directly use chainofthought? Just trying to understand the solution clearly
i has same problem...how to solve it?

please help me:



same :(
The solution for me was to not wrap the result in a `Prediction`. So I changed my code to:

```
class Summarizer(dspy.Module):
    def __init__(self):
        super().__init__()
        self.summarizer = dspy.Predict('transcript -> summary')
        
    def forward(self, transcript):
        return self.summarizer(transcript=transcript)
```
That's strange because `self.summarizer(transcript=transcript)` itself returns a Prediction object. 

Calling the forward function vs Wrapping it again in Prediction should not make a difference... especially in its _store attribute...

This seems to be a bug?


in fact one of two times I saw this error being returned was during the evaluation of an assertion / suggestion..
Ok, I have found a workaround for this. Basically I needed to overwrite the `__eq__` method by providing my own Prediction class and wrap the result using the new class. 

```python
from dspy import Prediction


class MyPrediction(Prediction):
    def __init__(self, prediction: Prediction):
        self.__dict__.update(vars(prediction))
        
    def __eq__(self, other):
        if isinstance(other, str):
            return False # because _store is intended to be a dict object
        return self._store == other._store

# and in my forward method

def forward(self, *args, **kwargs):
    ....
    return MyPrediction(result_from_chain_of_thought_or_predict)
    ...
```



Patch:
diff --git a/dspy/primitives/example.py b/dspy/primitives/example.py
index 55b06ad8d1..5165a442f9 100644
--- a/dspy/primitives/example.py
+++ b/dspy/primitives/example.py
@@ -54,7 +54,7 @@ def __str__(self):
         return self.__repr__()
     
     def __eq__(self, other):
-        return self._store == other._store
+        return isinstance(other, Example) and self._store == other._store
     
     def __hash__(self):
         return hash(tuple(self._store.items()))


Test patch:
diff --git a/tests/primitives/test_example.py b/tests/primitives/test_example.py
index 2f27996a24..8900f564a3 100644
--- a/tests/primitives/test_example.py
+++ b/tests/primitives/test_example.py
@@ -59,6 +59,7 @@ def test_example_eq():
     example1 = Example(a=1, b=2)
     example2 = Example(a=1, b=2)
     assert example1 == example2
+    assert example1 != ""
 
 
 def test_example_hash():