Repository: griptape-ai/griptape

Issue:
Evaluation engine criteria causes errors when intermediate evaluation_steps generation returns poorly formatted JSON/dict
**Description:**

When "criteria" is used in the evaluation engine, during the automagic generation of evaluation_steps, the intermediate JSON/dict is sometimes improperly formatted, leading to runtime errors.

**Steps to Reproduce:**
Running this as-is will work (sending something to the evaluation steps directly).  Commenting "evaluation_steps" out, and "criteria" in, shows the problem:

```
from griptape.structures import Pipeline
from griptape.engines import EvalEngine
from griptape.tasks import PromptTask
from griptape.rules import Rule

from dotenv import load_dotenv
load_dotenv() # Load the environment variables

rules    =  [
                "Answer with a json object, with no additional markup.",
                "Talk like a pirate.",
            ]

pipeline = Pipeline(
    tasks = [
        PromptTask(
            "Respond to this user: '{{ args[0] }}'"
            "{% if args[1] %}Use this feedback when answering.{{ args[1] }}{% endif %}"
        ),
    ],
    rules=[ Rule( rule ) for rule in rules ],
)

engine = EvalEngine(
    #criteria=[
    evaluation_steps=[
        f"Determine whether the following rules have been met: {rules}",
    ]
)

pipeline.run( "Hi there" )
score, reason = engine.evaluate(
    input         = pipeline.tasks[0].input.value,
    actual_output = pipeline.output.value,
)
```

Expected Behavior: The intermediate JSON/dict for evaluation_steps should always include a steps key, an be properly formatted for use.

Environment:

Griptape version: 1.1.1
Python version: 3.10-3.12
OS: OSX

**Debug text pulled to look at the json directly**
```
('{"type": "object", "properties": {"steps": {"type": "array", "items": '
 '{"type": "string"}}}, "required": ["steps"], "additionalProperties": false, '
 '"$id": "Output Format", "$schema": '
 '"http://json-schema.org/draft-07/schema#"}')
```

**How the error reads:**
```
Traceback (most recent call last):
  File ".../GitHub/griptape-TS-1_1/evalEngine.py", line 33, in <module>
    score, reason = engine.evaluate(
  File ".../GitHub/griptape-TS-1_1/.venv/lib/python3.10/site-packages/griptape/engines/eval/eval_engine.py", line 86, in evaluate
    self.evaluation_steps = self._generate_steps(evaluation_params)
  File ".../GitHub/griptape-TS-1_1/.venv/lib/python3.10/site-packages/griptape/engines/eval/eval_engine.py", line 115, in _generate_steps
    return parsed_result["steps"]
KeyError: 'steps'
```

Hints from issue comments:
@collindutter: Fixed via #1519 

Solution patch:
diff --git a/.github/ISSUE_TEMPLATE/preapproved.md b/.github/ISSUE_TEMPLATE/preapproved.md
index bf5525ab..59331f55 100644
--- a/.github/ISSUE_TEMPLATE/preapproved.md
+++ b/.github/ISSUE_TEMPLATE/preapproved.md
@@ -1,6 +1,6 @@
 ---
-name: Pre-Discussed and Approved Topics 
-about: Only for topics already discussed and approved in the GitHub Discussions section. 
+name: Pre-Discussed and Approved Topics
+about: Only for topics already discussed and approved in the GitHub Discussions section.
 ---
 
 Is this a reproducible bug? If not, please open a new [Github Discussion](https://github.com/orgs/griptape-ai/discussions/new/choose).
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 33a90d82..03488923 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -6,29 +6,15 @@ repos:
       args: [--py39-plus, --keep-runtime-typing]
 - repo: local
   hooks:
-  - id: ruff-lint
-    name: Ruff
-    entry: make lint
-    language: system
-    types: [python]
-- repo: local
-  hooks:
-  - id: ruff-format
-    name: Ruff
+  - id: format
+    name: Format
     entry: make format
     language: system
     types: [python]
 - repo: local
   hooks:
-  - id: pyright
-    name: Pyright
-    entry: make check/types
-    language: system
-    types: [python]
-- repo: local
-  hooks:
-  - id: typos
-    name: Typos
-    entry: make check/spell
+  - id: types
+    name: Check
+    entry: make check
     language: system
     types: [python]
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5cab7c77..acdde9dd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -22,6 +22,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Deprecated
 
 - `FuturesExecutorMixin.futures_executor`. Use `FuturesExecutorMixin.create_futures_executor` instead.
+### Changed
+
+- `EvalEngine` to use structured output when generating evaluation steps.
 
 ## [1.1.1] - 2025-01-03
 
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 2db9d822..8a85628d 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -16,8 +16,8 @@ friendly to new contributors are tagged with "good first issue".
 **I have a bug!**
 
 1. Search the issue tracker and discussions for similar issues.
-2. If you don't have steps to reproduce, open a discussion.
-3. If you have steps to reproduce, open an issue.
+1. If you don't have steps to reproduce, open a discussion.
+1. If you have steps to reproduce, open an issue.
 
 **I have an idea for a feature!**
 
@@ -26,7 +26,7 @@ friendly to new contributors are tagged with "good first issue".
 **I've implemented a feature!**
 
 1. If there is an issue for the feature, open a pull request.
-2. If there is no issue, open a discussion and link to your branch.
+1. If there is no issue, open a discussion and link to your branch.
 
 **I have a question!**
 
@@ -58,7 +58,6 @@ Pull requests should be associated with a previously accepted issue.
 **If you open a pull request for something that wasn't previously discussed,**
 it may be closed or remain stale for an indefinite period of time.
 
-
 > [!NOTE]
 >
 > **Pull requests are NOT a place to discuss feature design.** Please do
@@ -75,16 +74,19 @@ The [Griptape Extension Template](https://github.com/griptape-ai/griptape-extens
 ## Dev Environment
 
 Install all dependencies via Make:
+
 ```shell
 make install
 ```
 
 Run tests:
+
 ```shell
 make test/unit
 ```
 
 Run checks:
+
 ```shell
 make check
 ```
diff --git a/griptape/engines/eval/eval_engine.py b/griptape/engines/eval/eval_engine.py
index b3bd258b..045c0d64 100644
--- a/griptape/engines/eval/eval_engine.py
+++ b/griptape/engines/eval/eval_engine.py
@@ -11,7 +11,6 @@ from griptape.common import Message, PromptStack
 from griptape.configs import Defaults
 from griptape.engines import BaseEvalEngine
 from griptape.mixins.serializable_mixin import SerializableMixin
-from griptape.rules import JsonSchemaRule
 from griptape.utils import J2
 
 if TYPE_CHECKING:
@@ -89,7 +88,6 @@ class EvalEngine(BaseEvalEngine, SerializableMixin):
         system_prompt = self.generate_steps_system_template.render(
             evaluation_params=", ".join(param for param in evaluation_params),
             criteria=self.criteria,
-            json_schema_rule=JsonSchemaRule(STEPS_SCHEMA.json_schema("Output Format")),
         )
         user_prompt = self.generate_steps_user_template.render()
 
@@ -99,6 +97,7 @@ class EvalEngine(BaseEvalEngine, SerializableMixin):
                     Message(system_prompt, role=Message.SYSTEM_ROLE),
                     Message(user_prompt, role=Message.USER_ROLE),
                 ],
+                output_schema=STEPS_SCHEMA,
             ),
         ).to_artifact()
 
@@ -111,7 +110,6 @@ class EvalEngine(BaseEvalEngine, SerializableMixin):
             evaluation_params=", ".join(param for param in evaluation_params),
             evaluation_steps=self.evaluation_steps,
             evaluation_text="\n\n".join(f"{key}: {value}" for key, value in evaluation_params.items()),
-            json_schema_rule=JsonSchemaRule(RESULTS_SCHEMA.json_schema("Output Format")),
         )
         user_prompt = self.generate_results_user_template.render()
 
@@ -121,6 +119,7 @@ class EvalEngine(BaseEvalEngine, SerializableMixin):
                     Message(system_prompt, role=Message.SYSTEM_ROLE),
                     Message(user_prompt, role=Message.USER_ROLE),
                 ],
+                output_schema=RESULTS_SCHEMA,
             ),
         ).to_text()
 
diff --git a/griptape/templates/engines/eval/results/system.j2 b/griptape/templates/engines/eval/results/system.j2
index 0fb8583a..3cd9c078 100644
--- a/griptape/templates/engines/eval/results/system.j2
+++ b/griptape/templates/engines/eval/results/system.j2
@@ -6,5 +6,3 @@ Evaluation Steps:
 {{ evaluation_steps }}
 
 {{ evaluation_text }}
-
-{{ json_schema_rule }}
diff --git a/griptape/templates/engines/eval/steps/system.j2 b/griptape/templates/engines/eval/steps/system.j2
index e6792ba8..5c3f70a7 100644
--- a/griptape/templates/engines/eval/steps/system.j2
+++ b/griptape/templates/engines/eval/steps/system.j2
@@ -3,5 +3,3 @@ You MUST make it clear how to evaluate {{ evaluation_params }} in relation to on
 
 Evaluation Criteria:
 {{ criteria }}
-
-{{ json_schema_rule }}


Test patch:
diff --git a/tests/unit/engines/eval/test_eval_engine.py b/tests/unit/engines/eval/test_eval_engine.py
index 5a7dcb80..7625a80d 100644
--- a/tests/unit/engines/eval/test_eval_engine.py
+++ b/tests/unit/engines/eval/test_eval_engine.py
@@ -13,13 +13,11 @@ class TestEvalEngine:
         return EvalEngine(
             criteria="foo",
             prompt_driver=MockPromptDriver(
-                mock_output=json.dumps(
-                    {
-                        "steps": ["mock output"],
-                        "score": 0.0,
-                        "reason": "mock output",
-                    }
-                ),
+                mock_structured_output={
+                    "steps": ["mock output"],
+                    "score": 0.0,
+                    "reason": "mock output",
+                }
             ),
         )
 
@@ -74,12 +72,10 @@ class TestEvalEngine:
         engine = EvalEngine(
             evaluation_steps=["foo"],
             prompt_driver=MockPromptDriver(
-                mock_output=json.dumps(
-                    {
-                        "score": 0.0,
-                        "reason": "mock output",
-                    }
-                ),
+                mock_structured_output={
+                    "score": 0.0,
+                    "reason": "mock output",
+                }
             ),
         )
         score, reason = engine.evaluate(