# Beyond Manual Media Coding: Evaluating Large Language Models and Agents for News Content Analysis
## 4 Results
### Per-Question Performance and Insights  
Figure 2 presents the final evaluation metrics, i.e., the weighted F1-scores for all questions, computed by comparing each model's output to the ground truths provided by the expert human annotator. Table A1 of Appendix A.2 also presents the evaluation metrics per question alongside the question types.  
Figure 2: Heatmap presenting the average F1-scores per question for all evaluated models.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Average Scores** & **Llama 4 Maverick Direct-LLM Prompt** & **Llama 4 Maverick Multi-Agent Team** \\ \hline Global & 0.757 & **0.805** \\ Binary & 0.841 & **0.910** \\ Categorical & 0.704 & **0.739** \\ Multi-label & 0.681 & **0.711** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Meta’s Llama 4 Maverick and Meta’s Llama 4 Maverick multi-agent team evaluation.  
Binary questions, which mostly involve direct presence or absence detections (e.g., mentions, references), yield consistently high scores across models, typically exceeding 0.80, with only minor variations between low- and high-capacity models. The Llama-Maverick-based agentic team maintains this strong performance, virtually matching [PERSON]'s performance of 0.911 by increasing direct M1's 0.841 to 0.910.  
Categorical questions, which demand nuanced assessments of prominence, sentiment, or political framing, display notable performance variability across systems. These tasks often involve subtle judgments requiring discourse-level comprehension, such as evaluating how prominently a policy is discussed or inferring sentiment toward economic and political consequences. While Claude-3-7-Sonnet outperforms others on tasks like evaluating economic impacts (Q23), the agentic approach still provides competitive results on several questions (e.g., 0.791 on Q26 vs. direct M1's 0.594), though inherently subjective judgments remain more difficult to standardize.  
Multi-label questions, requiring identification of multiple applicable entities (such as industries or stakeholders), present greater challenges. Direct models frequently struggle with these tasks, exhibiting generally lower F1-scores, especially in smaller-scale systems. For example, in question Q21, Meta-Llama-3-1 scores as low as 0.210. Direct Meta-Llama-4-Maverick achieves only 0.536 on Q14, whereas Claude-3-7-Sonnet reaches around 0.593. Here, the agentic system demonstrates clear advantages by distributing cognitive load among specialized roles and consolidating findings, resulting in more balanced multi-label extraction (e.g., achieving 0.643 on Q14).  
These patterns indicate that while simpler binary tasks are reliably addressed by both large direct models and the agentic system, structured multi-agent workflows are particularly effective for distributed extraction tasks. Notably, even binary questions such as Q25 can prove to be difficult if they require deep political understanding and indirect contextual connections, while the agentic system improves Meta-Llama-4-Maverick's scores in nuanced tasks such as Q11 and Q26 (sentiment and ideological framing), underscoring the benefit of collaborative reasoning for subjective classifications, fine-grained categorical judgments continue to represent a critical limitation for automated media content analysis irrespective of the model or methodology used.