# Beyond Manual Media Coding: Evaluating Large Language Models and Agents for News Content Analysis
## 5 Discussion  
In this section, we discuss the findings of our comprehensive evaluation of LLMs' capacity for structured media codebook analysis. We address two fundamental research questions regarding LLM-based media coding efficacy and the potential benefits of agentic workflows, through a systematic experimentation on our curated dataset of 200 annotated news articles spanning 26 codebook-derived questions. _RQ1: To What Extent Can LLMs Accurately Respond to Media Analysis Questions Derived from Structured Codebooks?_  
Our results confirm that modern LLMs are capable of responding to structured media analysis questions with reasonable efficacy, comparable to human accuracy in challenging \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Models** & **Input** & **Output** & **Total** & **Input** & **Output** & **Inference** \\  & **Tokens (M)** & **Tokens (K)** & **Tokens (M)** & **Cost/M (S) *** & **Cost/M (S) *** & **Time (s)** \\ \hline
**Meta-Llama-4-Maverick** & 3.15 & 356 & 3.50 & 0.15 & 0.60 & 4508 \\
**Qwen3** & 3.39 & 369 & 3.76 & 0.13 & 0.60 & 5300 \\
**Meta-Llama-3-3** & 4.10 & 440 & 4.54 & 0.05 & 0.18 & 8288 \\
**DeepSeek-R1** & 2.90 & 267 & 3.16 & 0.50 & 2.15 & 13,517 \\
**DeepSeek-R1-Distill-Qwen** & 3.47 & 497 & 3.97 & 0.075 & 0.15 & 11,513 \\
**Claude-3-7-Sonnet** & 2.99 & 482 & 3.47 & 3.0 & 15.0 & 8067 \\
**Meta-Llama-3-1** & 3.28 & 373 & 3.66 & 0.016 & 0.022 & 5409 \\
**Agentic System (Llama-4)** & 3.92 & 882 & 4.80 & 0.15 & 0.60 & 8967 \\ \hline \hline \end{tabular}  
* Pricing data for various LLMs was sourced from [81]. Open-weight LLMs can be deployed locally, but the providers’ Cost/M was used as resource utilization indicator.  
\end{table} 
Table 6: Token usage and indicative costs across evaluated models for processing the full dataset of 200 articles. Figure 3: Model performance comparison showing cost effectiveness versus average F1-score over the 200-article dataset. Each bubble represents a different language model, with bubble size being proportional to total token usage (input + output tokens). The x-axis shows the total inference cost in USD, while the y-axis represents the average F1-score across evaluation tasks. Pricing data for various LLMs was sourced from [81]. content analysis tasks [82, 83]. Our methodology for direct LLM prompting followed best practices, including system-role definition, schema-constrained response formats, and explicit reasoning instructions, achieving global F1-scores ranging from 0.636 to 0.822 across seven models of small-, medium-, and high-tier capacities. Anthropic's Claude 3.7 Sonnet achieved the highest overall performance, followed by DeepSeek R1 and Meta's Llama 3.3. The reported variability in performance across model tiers underscores that raw LLM capabilities play a critical role, especially in handling nuanced, multi-label questions. Nevertheless, our findings suggest that with carefully designed and structured prompts, even direct-call LLMs can handle codebook-based content analysis requiring advanced reasoning and contextual understanding without prior training. Despite no known prior exposure to the codebook's thematic definitions, models generally respected its structural logic, suggesting that prompt engineering alone can elicit schema-consistent behavior, especially in mid- and high-tier models. _RQ2: Can Agentitic Workflows Enhance the Performance of LLMs in Codebook-Based News Content Analysis?_  
To evaluate the impact of agentic orchestration, we developed a multi-agent coordination layer atop Meta's Llama 4 Maverick. This agentic architecture integrated multi-agent coordination and planning, tool utilization, and sub-task delegation, while preserving schema consistency. The system yielded a consistent improvement in performance, with the global average F1-score increasing from 0.757 (Direct-LLM prompt) to 0.805 under agentic execution, with consistent gains across binary, categorical, and multi-label classification tasks.