Repository: Shopify/roast (Ruby)

Issue:
CodingAgent JSON parsing fails when response includes surrounding text
# CodingAgent JSON parsing fails when response includes surrounding text

## Problem

When using `json: true` with CodingAgent steps, the JSON parsing logic only looks for markdown code block indicators (```` ```json` and ```` ```) on the first and last lines of the response. This fails when the coding agent provides explanatory text around the JSON output, which is common in real-world usage, despite giving claude explicit instructions to the contrary.

## Current Behavior

The existing parsing logic in `lib/roast/workflow/agent_step.rb` (lines 40-52) expects:
- First line to be exactly ```` ```json` or ```` ```
- Last line to be exactly ```` ```
- JSON content between these lines

## Failing Case

When CodingAgent returns:
```
Here's my analysis of the code:

I found several issues that need attention:

```json
{
  "issues": ["performance", "security"],
  "severity": "high"
}
```

Please review these findings.
```

The parser fails because the code blocks are not on the first/last lines.

## Expected Behaviour

The parser should extract JSON from markdown code blocks anywhere in the response, ignoring surrounding explanatory text.

## Impact

- CodingAgent steps with `json: true` fail unpredictably
- Users must avoid providing context around JSON responses
- Reduces the usefulness of structured JSON output from agents


Discussion:

Patch:
diff --git a/lib/roast/workflow/agent_step.rb b/lib/roast/workflow/agent_step.rb
index 2623cea5..c6f4a2e2 100644
--- a/lib/roast/workflow/agent_step.rb
+++ b/lib/roast/workflow/agent_step.rb
@@ -37,19 +37,8 @@ def call
             raise result
           end
 
-          # Strip Markdown code block indicators surrounding JSON data in response if present
-          lines = result.strip.split("\n")
-
-          # Check if first line is ```json or ``` (with optional trailing whitespace)
-          # and last line is ``` (with optional trailing whitespace)
-          cleaned_result = if lines.length >= 2 &&
-              (lines.first.strip == "```json" || lines.first.strip == "```") &&
-              lines.last.strip == "```"
-            # Remove first and last lines
-            lines[1..-2].join("\n")
-          else
-            result
-          end
+          # Extract JSON from markdown code blocks anywhere in the response
+          cleaned_result = extract_json_from_markdown(result)
 
           begin
             result = JSON.parse(cleaned_result)
@@ -64,6 +53,23 @@ def call
         # Apply coercion if configured
         apply_coercion(result)
       end
+
+      private
+
+      def extract_json_from_markdown(text)
+        # Look for JSON code blocks anywhere in the text
+        # Matches ```json or ``` followed by content, then closing ```
+        json_block_pattern = /```(?:json)?\s*\n(.*?)\n```/m
+
+        match = text.match(json_block_pattern)
+        if match
+          # Return the content inside the code block
+          match[1].strip
+        else
+          # No code block found, return original text
+          text.strip
+        end
+      end
     end
   end
 end


Test patch:
diff --git a/test/roast/workflow/agent_step_test.rb b/test/roast/workflow/agent_step_test.rb
index 93bfd01a..ce17895b 100644
--- a/test/roast/workflow/agent_step_test.rb
+++ b/test/roast/workflow/agent_step_test.rb
@@ -453,4 +453,50 @@ def teardown
     assert_instance_of String, result
     assert_equal json_string, result
   end
+
+  test "agent step parses JSON code blocks anywhere in response" do
+    # Create a mock workflow
+    workflow = mock
+    workflow.stubs(:resource).returns(nil)
+    workflow.stubs(:output).returns({})
+    workflow.stubs(:transcript).returns([])
+    workflow.stubs(:append_to_final_output)
+    workflow.stubs(:file).returns(nil)
+    workflow.stubs(:config).returns({})
+
+    # Create agent step with json: true
+    agent_step = Roast::Workflow::AgentStep.new(workflow, name: "test_agent")
+    agent_step.json = true
+
+    # Mock the prompt loader
+    Roast::Helpers::PromptLoader.stubs(:load_prompt).returns("Generate JSON output")
+
+    # Mock CodingAgent to return response with text around JSON code block
+    response_with_text = <<~RESPONSE
+      Here's my analysis of the code:
+
+      I found several issues that need attention. Let me provide the detailed results:
+
+      ```json
+      {
+        "issues": ["performance", "security", "maintainability"],
+        "severity": "high",
+        "recommendations": ["refactor loops", "sanitize input", "add tests"]
+      }
+      ```
+
+      Please review these findings and let me know if you need clarification.
+    RESPONSE
+
+    Roast::Tools::CodingAgent.expects(:call).returns(response_with_text)
+
+    # Execute the step
+    result = agent_step.call
+
+    # Verify the result is parsed JSON with surrounding text stripped
+    assert_instance_of Hash, result
+    assert_equal ["performance", "security", "maintainability"], result["issues"]
+    assert_equal "high", result["severity"]
+    assert_equal ["refactor loops", "sanitize input", "add tests"], result["recommendations"]
+  end
 end