Repository: github/gh-aw (2026-03-28T12:28:10Z)

Issue:
[cli-tools-test] audit: tool usage parser extracts common English words as tool call names from agent logs
During daily CLI tools exploratory testing (run [§23632386989](https://github.com/github/gh-aw/actions/runs/23632386989)), a reproducible parsing bug was found in the `audit` tool's tool usage extraction logic.

### Problem Description

When the Copilot CLI agent's STDIO log contains natural language text that mentions tool-like words (e.g., "the agent **calls** a **command**"), the audit tool incorrectly extracts those words as tool call names. This contaminates the `tool_usage` report with English stop-words like `calls`, `call`, `to`, `for`, `or`, `on`, `with`, `since`, `launches`, `misuse`, `unified`, etc.

### Affected Run

- **Run ID**: [§23578198071](https://github.com/github/gh-aw/actions/runs/23578198071) — `GPL Dependency Cleaner (gpclean)`, `completed/failure`
- **Reproduced via**: `audit` MCP tool on this run ID

### Evidence

The `run_summary.json` → `metrics.ToolCalls` for this run contains:

````json
[
  {"Name": "calls",   "CallCount": 20, "MaxInputSize": 0, "MaxOutputSize": 0},
  {"Name": "call",    "CallCount": 5,  "MaxInputSize": 0, "MaxOutputSize": 0},
  {"Name": "to",      "CallCount": 3,  "MaxInputSize": 0, "MaxOutputSize": 0},
  {"Name": "or",      "CallCount": 1,  "MaxInputSize": 0, "MaxOutputSize": 0},
  {"Name": "for",     "CallCount": 1,  "MaxInputSize": 0, "MaxOutputSize": 0},
  {"Name": "on",      "CallCount": 1,  "MaxInputSize": 0, "MaxOutputSize": 0},
  {"Name": "since",   "CallCount": 2,  "MaxInputSize": 0, "MaxOutputSize": 0},
  {"Name": "with",    "CallCount": 1,  "MaxInputSize": 0, "MaxOutputSize": 0},
  {"Name": "misuse",  "CallCount": 1,  "MaxInputSize": 0, "MaxOutputSize": 0},
  {"Name": "unified", "CallCount": 1,  "MaxInputSize": 0, "MaxOutputSize": 0},
  {"Name": "launches","CallCount": 1,  "MaxInputSize": 0, "MaxOutputSize": 0},
  // REAL tools mixed in:
  {"Name": "startHttpServer", "CallCount": 1, ...},
  {"Name": "noop",            "CallCount": 1, ...},
  {"Name": "label_agent",     "CallCount": 1, ...}
]
```

Note: All false-positive entries have `MaxInputSize: 0` and `MaxOutputSize: 0`, unlike real tool calls which typically carry payloads.

The `ToolSequences` confirm the pattern — sequences like `["or", "calls", "call", "call", "call"]` and a 21-entry sequence of repeated `calls`/`since` clearly come from parsed natural language prose, not actual tool invocations.

### Downstream Impact

This bug causes cascading inaccuracies across the entire audit report:

| Field | Incorrect Value | Correct Value |
|-------|----------------|---------------|
| `behavior_fingerprint.tool_breadth` | `"broad"` (15 "tool types") | `"narrow"` (3 real tools) |
| `performance_metrics.most_used_tool` | `"calls (20 calls)"` | `"startHttpServer"` or `"noop"` |
| `observability_insights` | "The agent used 8 turns across **15 tool types**" | "...across **3 tool types**" |
| `comparison.delta` (other runs) | Baseline tool_breadth comparisons are wrong | — |

### Likely Root Cause

The parser extracting tool names from `agent-stdio.log` appears to use a regex or tokenizer that's too broad — it likely matches words near patterns like `● toolname` or `` ` toolname ` `` but also catches words in nearby context lines that are free text. The Copilot CLI log format uses `● toolname` for actual tool invocations, so the regex may need to be anchored more strictly to that exact prefix.

### Steps to Reproduce

```
Use the audit MCP tool with run_id_or_url: "23578198071"
Check tool_usage[] in the response — entries like "calls", "to", "for" are present
Compare with agent-stdio.log at /tmp/gh-aw/aw-mcp/logs/run-23578198071/agent-stdio.log
````

### Expected Behavior

`tool_usage` should only contain entries for actual tool invocations (e.g., `bash`, `edit`, `startHttpServer`, MCP tool names). English words from log prose should never appear as tool names.

### Actual Behavior

Common English words (`calls`, `call`, `to`, `for`, `on`, `or`, `with`, `since`, `launches`, `misuse`, `unified`) appear as tool names with call counts, polluting all downstream metrics.

### Environment

- **Repository**: github/gh-aw
- **Testing run**: [§23632386989](https://github.com/github/gh-aw/actions/runs/23632386989)
- **Date**: 2026-03-27
- **Affected runs confirmed**: 23578198071 (gpclean)

### Impact

- **Severity**: High
- **Frequency**: Affects any run where the Copilot CLI agent's log contains natural language that includes common verb-like words near the tool-call parsing window
- **Workaround**: None — the `tool_usage`, `behavior_fingerprint.tool_breadth`, and `most_used_tool` fields cannot be trusted for affected runs

**References:**
- [§23578198071](https://github.com/github/gh-aw/actions/runs/23578198071) — Affected run (gpclean failure)
- [§23632386989](https://github.com/github/gh-aw/actions/runs/23632386989) — Testing run that discovered this issue




> Generated by [Daily CLI Tools Exploratory Tester](https://github.com/github/gh-aw/actions/runs/23632386989) · [◷](https://github.com/search?q=repo%3Agithub%2Fgh-aw+is%3Aissue+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Fdaily-cli-tools-tester%22&type=issues)
> - [x] expires <!-- gh-aw-expires: 2026-04-03T05:28:19.057Z --> on Apr 3, 2026, 5:28 AM UTC

<!-- gh-aw-agentic-workflow: Daily CLI Tools Exploratory Tester, engine: copilot, model: auto, id: 23632386989, workflow_id: daily-cli-tools-tester, run: https://github.com/github/gh-aw/actions/runs/23632386989 -->

<!-- gh-aw-workflow-id: daily-cli-tools-tester -->
<!-- gh-aw-workflow-call-id: github/gh-aw/daily-cli-tools-tester -->


Discussion:

Patch:
diff --git a/pkg/cli/copilot_agent_logs.go b/pkg/cli/copilot_agent_logs.go
index 0bd39af8b9a..2442a2b8fb0 100644
--- a/pkg/cli/copilot_agent_logs.go
+++ b/pkg/cli/copilot_agent_logs.go
@@ -8,13 +8,28 @@ import (
 )
 
 var (
-	agentTurnPattern     = regexp.MustCompile(`(?i)task.*iteration|agent.*turn|step.*\d+`)
-	agentToolCallPattern = regexp.MustCompile(`(?i)tool.*call|executing.*tool|calling.*(\w+)`)
+	agentTurnPattern = regexp.MustCompile(`(?i)task.*iteration|agent.*turn|step.*\d+`)
+	// agentToolCallPattern matches log lines that indicate an actual tool invocation.
+	// It requires either a bullet point prefix (● toolname) or a colon separator after
+	// the keyword (e.g. "Tool call:", "Calling:", "Executing tool:"). This prevents
+	// false positives from natural language prose that contains words like "tool calls",
+	// "calling a function", etc.
+	agentToolCallPattern = regexp.MustCompile(`(?i)\btool\s+call\s*:|\btool\s*:|executing\s+tool\s*:|executing\s*:|calling\s*:|using\s+tool\s*:|^\s*●\s*\S`)
 	toolNamePatterns     = []*regexp.Regexp{
-		regexp.MustCompile(`(?i)tool[:\s]+([a-zA-Z0-9_-]+)`),
-		regexp.MustCompile(`(?i)calling[:\s]+([a-zA-Z0-9_-]+)`),
-		regexp.MustCompile(`(?i)executing[:\s]+([a-zA-Z0-9_-]+)`),
-		regexp.MustCompile(`(?i)using[:\s]+tool[:\s]+([a-zA-Z0-9_-]+)`),
+		// "● toolname" - bullet point format used by Copilot coding agent STDIO logs
+		regexp.MustCompile(`^\s*●\s*([a-zA-Z0-9_][a-zA-Z0-9_-]*)`),
+		// "Tool call: toolname" - colon separator required
+		regexp.MustCompile(`(?i)\btool\s+call\s*:\s*([a-zA-Z0-9_][a-zA-Z0-9_-]*)`),
+		// "Tool: toolname" - colon immediately after "tool" required
+		regexp.MustCompile(`(?i)\btool\s*:\s*([a-zA-Z0-9_][a-zA-Z0-9_-]*)`),
+		// "Calling: toolname" - colon separator required
+		regexp.MustCompile(`(?i)\bcalling\s*:\s*([a-zA-Z0-9_][a-zA-Z0-9_-]*)`),
+		// "Executing tool: toolname" - colon separator required
+		regexp.MustCompile(`(?i)\bexecuting\s+tool\s*:\s*([a-zA-Z0-9_][a-zA-Z0-9_-]*)`),
+		// "Executing: toolname" - colon separator required
+		regexp.MustCompile(`(?i)\bexecuting\s*:\s*([a-zA-Z0-9_][a-zA-Z0-9_-]*)`),
+		// "Using tool: toolname" - colon separator required
+		regexp.MustCompile(`(?i)\busing\s+tool\s*:\s*([a-zA-Z0-9_][a-zA-Z0-9_-]*)`),
 	}
 )
 
@@ -48,7 +63,7 @@ func ParseCopilotCodingAgentLogMetrics(logContent string, verbose bool) workflow
 		}
 
 		// Extract tool calls from agent logs
-		if matches := agentToolCallPattern.FindStringSubmatch(line); len(matches) > 1 {
+		if agentToolCallPattern.MatchString(line) {
 			toolName := extractToolName(line)
 			if toolName != "" {
 				// Track tool call


Test patch:
diff --git a/pkg/cli/copilot_agent_test.go b/pkg/cli/copilot_agent_test.go
index 2829182d6e8..456672496cc 100644
--- a/pkg/cli/copilot_agent_test.go
+++ b/pkg/cli/copilot_agent_test.go
@@ -131,6 +131,7 @@ func TestParseCopilotCodingAgentLogMetrics(t *testing.T) {
 		expectedTurns  int
 		expectedErrors int
 		expectedTools  int
+		expectNoTools  bool // true when the test asserts that zero tool calls are extracted
 	}{
 		{
 			name: "parses agent turns",
@@ -164,6 +165,30 @@ Executing tool: read
 `,
 			expectedTools: 4,
 		},
+		{
+			name: "extracts tool from bullet point format",
+			logContent: `
+● startHttpServer
+● noop
+● label_agent
+`,
+			expectedTools: 3,
+		},
+		{
+			name: "does not extract common English words as tool names",
+			logContent: `
+The agent tool calls a command which launches the unified process
+Since the tool or alternative is needed, we use it
+The tool calls for analysis
+tool call to the API
+The agent is making tool calls with certain parameters
+agent calls a bash command
+executing something important
+calling the helper function
+`,
+			expectedTools: 0,
+			expectNoTools: true,
+		},
 		{
 			name: "extracts token usage from JSON",
 			logContent: `
@@ -182,6 +207,7 @@ Task step 1 complete
 			expectedTurns:  0,
 			expectedErrors: 0,
 			expectedTools:  0,
+			expectNoTools:  true,
 		},
 	}
 
@@ -196,6 +222,14 @@ Task step 1 complete
 			if tt.expectedTools > 0 && len(metrics.ToolCalls) < 1 {
 				t.Errorf("Expected tool calls to be detected, got none")
 			}
+
+			if tt.expectNoTools && len(metrics.ToolCalls) > 0 {
+				var names []string
+				for _, tc := range metrics.ToolCalls {
+					names = append(names, tc.Name)
+				}
+				t.Errorf("Expected no tool calls, but got %d: %v", len(metrics.ToolCalls), names)
+			}
 		})
 	}
 }
@@ -231,6 +265,48 @@ func TestExtractToolName(t *testing.T) {
 			line:     "Just a regular log line",
 			expected: "",
 		},
+		// False positive prevention: common English words near "tool" without colon separator
+		{
+			name:     "does not extract 'calls' from 'tool calls' in prose",
+			line:     "The agent tool calls a command",
+			expected: "",
+		},
+		{
+			name:     "does not extract 'call' from 'tool call' without colon",
+			line:     "Making a tool call to the API",
+			expected: "",
+		},
+		{
+			name:     "does not extract 'or' from 'tool or' in prose",
+			line:     "The tool or alternative approach",
+			expected: "",
+		},
+		{
+			name:     "does not extract 'for' from 'tool for' in prose",
+			line:     "Use a tool for building the project",
+			expected: "",
+		},
+		{
+			name:     "does not extract word from 'calling' without colon",
+			line:     "The agent is calling a function here",
+			expected: "",
+		},
+		// Bullet point format
+		{
+			name:     "extracts tool name from bullet point format",
+			line:     "● startHttpServer",
+			expected: "startHttpServer",
+		},
+		{
+			name:     "extracts snake_case tool name from bullet point",
+			line:     "● label_agent",
+			expected: "label_agent",
+		},
+		{
+			name:     "extracts tool from bullet with leading whitespace",
+			line:     "  ● noop",
+			expected: "noop",
+		},
 	}
 
 	for _, tt := range tests {